Robots.txt is a file created by webmasters to instruct search engines on crawl behaviour, wether a directory or file type should be indexed. Robots.txt are used in many different scenarios, mainly to prevent indexing of protected content, images and much more. Search engines obey robots.txt, as this allows the webmaster to have selective preferences for website security and for privacy reasons.
Hundreds of thousands of websites are built and released online with no block on crawl permissions for various directories of a website. This prevents major security implications as search engines can list protected content unless instructed not too.
Robots.txt is also used for SEO purposes to prevent the indexing of various documents. Some webmasters prevent pages with duplicate content from being indexed, whilst others use robots.txt to stop Google from crawling internal links pages and much more.
Robots.txt is a raw text file placed on a webserver, typically on the root of the domain, so that search engines can access it directly. Search engines will obey the robots.txt file as it is a mandatory requirement. Over the years the robots.txt file has become of more use, and has many substantial benefits, even tailoring instructions to each search engine based on the crawlers name.
Search engines have bots that work on their behalf, to crawl and index content, these bots have names assigned to them, it is these names that can be used within robots.txt to either allow or block access to various documents. For example, you can prevent Yahoo's bot (Slurp) from indexing a particular part of the website, whilst allowing other bots such as Google's bot (Googlebot) to index it.
This is very useful when tailoring results as per search engine requirements. Some search engines for example, may not be able to index flash, as a result, you could allow Google and Yahoo to crawl it, whilst disallowing other search engines to prevent failed results.
This standard has become more widely used over the past 8 years as the internet and search engines have evolved, it is with this that more webmasters really need to utilise the power of robots.txt.
