Robots.txt

A text document used to give directives to search engine bots and spiders on how to crawl and index a website.
HomeSEOGlossary
Robots.txt