Crawl Budget

The number of pages that search engine bots will visit your site in a given period of time. It is critical on large sites. URLs that waste crawl budget: parameter URLs, pages with session IDs, pagination errors, 404 responsive broken links. Blocking with robots.txt, concatenating with canonical, and providing fast TTFB increase crawl efficiency.