Robots.txt

A plain text file located in the root directory of the website (/robots.txt) that tells bots, including Googlebot, which pages or directories should not be crawled. Misconfigured robots.txt may cause the entire site to not be indexed. Important: URLs blocked by robots.txt cannot be indexed but may remain visible; For complete privacy use noindex.