internet is-x-legal
Does the robots exclusion standard have any legal weight?
<p>There is a standard through which websites communicate to the web crawlers upon which search engines are based which pages should be indexed and included in the search results.  This is called the <a href="https://en.wikipedia.org/wiki/Robots_exclusion_standard" rel="noreferrer">Robots exclusion standard</a>, is implemented in the robots.txt file that most web sites have, stack exchange's is <a href="https://stackexchange.com/robots.txt">here</a>.</p>
<p>If one was interested in the whole content of a site, for example one was researching the network structure of web sites, would it be legal to ignore the robots.txt and crawl the whole site (perhaps excluding /search[?/])?</p>

<p>The existence of a robots.txt file and the directory and file inclusions/exclusions in a robots.txt file do not constitute a legally binding contract for the use of the website by the visitor; if it exists, the Terms of Service would usually establish the contract for use of the site.</p>
<p>Many TOSs prohibit bots, crawlers and any automated means from crawling or scraping a site, and you are contractually obligated to honor those stipulations simply by visiting the site. Violating a TOS is typically a civil matter, depending on jurisdiction.</p>
<p>Simply crawling a site to &quot;map&quot; it, and crawling a site to &quot;scrape&quot; it and copy all content are two different things. Crawling could violate the TOS. Scraping content can violate the TOS and also be copyright infringement, depending on jurisdiction. See <a href="https://law.stackexchange.com/search?q=scraping">https://law.stackexchange.com/search?q=scraping</a></p>
<p>I suppose a TOS could state that users are legally bound to the restrictions in a robots.txt, but I've never seen that in a TOS.</p>
