Concept v1.0, updated 2026-07-07

What crawl budget is

The finite attention a crawler gives your site, and how it gets wasted.

Crawl budget is the finite amount of fetching a crawler will do on your site in a given period. It is not unlimited. If a bot spends its budget on the wrong URLs it never reaches the ones that matter.

Budget is spent on redirects, error pages, pages set to noindex and duplicate URLs. A single page reachable at four addresses, with and without a trailing slash, with a tracking parameter, over HTTP and HTTPS, is four fetches for one page. A noindex page is a softer case than a dead one - a bot still fetches it, but a noindex,follow page passes its link equity onward, so it is not pure waste. On a login or utility screen a noindex is expected and correct.

You trim it by keeping internal links pointing at the final canonical URL, stripping tracking parameters, resolving redirect chains, and aligning robots.txt with your sitemap so you are not listing URLs you also block.

Kenovar finds this waste in the site audit and the log analyser: the redirects, the errors and the noindex pages a bot is spending time on.

Read next

Why AI cannot see content behind JavaScript

Non-rendering crawlers get the shell, not the content and how to fix it.

What internal PageRank is

How link equity flows through your own site, and why buried pages lose out.

What a redirect chain is

One URL that bounces through several before it resolves, and the cost.

All topics