Which file guides search engine crawlers on which pages to crawl?

Explanation:

robots.txt tells search engines which pages they can or cannot crawl.