how the web crawlers works and what is it?
154221
Let's say you write a program (in either Java or .Net) to retrieve the contents of a single web page.  Then, in your program, you search the page for links to other pages.  Next, recursively, you retrieve those pages and search for links on them.  Etc.  Etc.\n\nYou save all the links in a database with (perhaps) the text content of the page.  You rerun this process nightly and build you "portfolio" of linked pages.\n\nYou might "prime" you database with links from known pages with links to lots of other pages.  Over time, you will have lots of links...
776650
A search engine or search service is a document retrieval system designed to help find information stored on a computer system, such as on the World Wide Web, inside a corporate or proprietary network, or in a personal computer. The search engine allows one to ask for content meeting specific criteria (typically those containing a given word or phrase) and retrieves a list of items that match those criteria. This list is often sorted with respect to some measure of relevance of the results. Search engines use regularly updated indexes to operate quickly and efficiently. Without further qualification, search engine usually refers to a Web search engine, which searches for information on the public Web. Other kinds of search engine are enterprise search engines, which search on intranets, personal search engines, which search individual personal computers, and mobile search engines. Different selection and relevance criteria may apply in different environments, or for different uses.\n\nSome search engines also mine data available in newsgroups, large databases, or open directories like DMOZ.org. Unlike Web directories, which are maintained by human editors, search engines operate algorithmically. Most websites which call themselves search engines are actually front ends to search engines owned by other companies
495838
The information collected in HTML can be processed by scripting.\n\nFor example do a calculation, check some conditions and performing actions.
637890
Inktomi Slurp is yahoo's little spider. If you let slurp crawl your website, then you let yahoo index your website. Your website will then show up in the Yahoo Search results. \nWhat are the advantages of it? It depends on if you want traffic to your website. If you want traffic it is advantageous. If you don't like people on your website, then it is a disadvantage\n\ni do see \nMozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/... \nMozilla/5.0 (compatible; SlurpC; http://help.yahoo.com/help/... \nMozilla/5.0 (compatible; SlurpS; http://help.yahoo.com/help/... \n\ni also see \nFAST MetaWeb Crawler (helpdesk at fastsearch dot com) \nthought fast had stopped crawling a long time ago .... \n(must pay more attention to log files / stats etc) \n\nSreeVenkatesh.I
110722
It will take sometime for the search engines to pick up your site. You  can submit your site to search engines. However you need to build your site crawler freindly. Crawler are the robotic application who are on intrnet in search of the websites and add the site to the search engine's database.\n\nHere is guideline on how to develope a crawler friendly webstie.\n\n\nhttp://www.google.com/webmasters/guidelines.html\n\nHere is another link on how to add your site to yahoo search engine.\n\nhttp://help.yahoo.com/help/us/dir/suggest/suggest-01.html\n\nfollow the steps and add your site to yahoo.
785742
The most common search engine type is the Crawler-Based Search Engines.\n\nCrawler-based search engines, such as Google, create their listings automatically. They "crawl" or "spider" the web, then people search through what they have found.\n\nIf you change your web pages, crawler-based search engines eventually find these changes, and that can affect how you are listed. Page titles, body copy and other elements all play a role.\n\nThere are differences in the ways various search engines work, but they all perform three basic tasks:\n\n    * They search the Internet -- or select pieces of the Internet -- based on important words.\n    * They keep an index of the words they find, and where they find them.\n    * They allow users to look for words or combinations of words found in that index. \n\nEarly search engines held an index of a few hundred thousand pages and documents, and received maybe one or two thousand inquiries each day. Today, a top search engine will index hundreds of millions of pages, and respond to tens of millions of queries per day.
249187
They have interactive websites that have robots. When you sign up, (when you press register), your e-mail is inserted into their database. The same page that inserts your e-mail in to the database, sends a greeting e-mail to you. After that, with those e-mails, they create newsletters and send them to you, and the other users. You must have a web space and install various applications on it like mailing lists, etc. for more details, let me know.
1076004
It used to work a lot better than it does now.  Major websites now have people who's specific job it is to find the accounts on bugmenot for their sites and delete those accounts.\n\nThese companies make money by the amount of information they collect.  It's kinda scary, actually.
1091124
No, the search engines don't - The webmasters do.\nSearch Engine algorithms will bury web pages that are attempting to manipulate search engine bots, crawlers, spiders (whatever you want to call them). It's a never ending game.
57680
It probably just creates a web page with html on the page to redirect you to the large link.\n\nRight-click to download their example, open in Notepad, and look at the html...  http://tinyurl.com/6
966019
First You Need a Website and Then post on it and when google will crawl that site, It will also index the Picture
6343
A web crawler always analyses the links, as well the the location of image files put up in the source code of a webpage, and crawls forward by following the links. The text present on the page, serves as the keyword for searching the image. The ranking of the images with the keywords depends on various factors, including \n\n(1) Name of the image file \n(2) Distance of the text from the location of the image on the webpage\n\nIn that manner, MSN gives much better image results as compared to Google and Yahoo. Hope this answers your query.
472979
Designing a good search engine involves a lot of planning. The different components of a search engine are:\n1. A Bot / Crawler program that scans different websites and extracts the content.\n2. A parser program  that eliminates the HTML tags and other words that you dont wish to store.\n3. An Index - which is a list of keywords and link to the URL in which this was found.\n4. A Ranking algorithm that can assign some kind of weightage to the entries in the index to identify which entry in the index is more relevant for a keyword. This could get very complex so that the context of the content, relevancy, freshness, reliability of the content and many other factors are considered before a ranking is done.\n5. Storage architecture - The index could grow to phenomenal sizes (more so if not planned well) and so you need to decide what kind of storage you wish and how data can be retrieved quickly from many physical stores.\n6. A Web interface to query the index and display results appropriately.\n\nI'm no Guru in search engine tech - but briefly this is my understanding. Good luck!
180418
The previous answers are absolute nonsense and do not even answer the question.\n\nYour feeling that it is the url is correct and also the product pages are more than 2 or 3 url hops away from the home page.\n\nYou can solve this problem easily by doing three things.\n\n1. Use Apache's mod_rewrite to make the urls search engine friendly. http://www.phpfreaks.com/tutorials/23/0.php\n\n2. Create a 'conventional sitemap' on your site for the product pages and link to it from every page. http://www.seo-blog.com/sitemaps.php\n\n3. Use 'Google Sitemaps' to submit all your pages for crawling. https://www.google.com/webmasters/sitemaps/docs/en/about.html\n\nPerform the tasks in the order of 1, 2, and 3 and do not be tempted to leave one of them out :)
697962
YES -- Google can crawl dynamic pages but they may not always be successful. That is why submitting a sitemap is important to make sure that their bots do not miss anything\n\nFrom the Google Webmaster Help pages\nhttp://www.google.com/support/webmasters/bin/answer.py?answer=34431&query=dynamic&topic=&type=\n\n-------------------------------------------------------\nDoes Google index dynamic pages? \n\nYes, Google indexes dynamically generated webpages, including .asp pages, .php pages, and pages with question marks in their URLs. However, these pages can cause problems for our crawler and may be ignored. If you're concerned that your dynamically generated pages are being ignored, you may want to consider creating static copies of these pages for our crawler. If you do this, please be sure to include a robots.txt file that disallows the dynamic pages in order to ensure that these pages aren't seen as having duplicate content. In addition, you might consider creating and submitting a detailed Sitemap of your pages using the Sitemaps component of our webmaster tools. Our webmaster tools are an easy way for you to submit all your URLs to the Google index and get detailed reports about the visibility of your pages on Google. You can automatically keep us informed of all of your current pages and of any updates you make to those pages. Please note that submitting a Sitemap doesn't guarantee that all pages of your site will be crawled or included in our search results. To learn more about Google Sitemaps and webmaster tools, please visit our information page. \n-------------------------------------------------------
932
You could download the site using an HTML downloader to gain access to everypage that is linked up on the site.\n\nor if you want more diagnostics (such as broken links, etc.) you could run Xenu's Link Sleuth
95941
http://www.usq.edu.au/spider/info/guide.htm