Tutorial v1.0, updated 2026-07-07

Trim crawl waste on your site

Stop bots spending their budget on redirects, errors and duplicates.

The outcome: crawlers spend their time on your real pages, not on waste. Less waste means more of your real content gets crawled, indexed and answered from. See crawl budget.

  1. Run the Audit, or build your twin, to crawl the site and roll up the issues: redirects, errors, noindex pages, thin content and near duplicates.
  2. Run the Log analyser on a server access log to see what bots are actually fetching, and how much of it is redirects and errors. This is what waste looks like in practice.
  3. Fix the big ones. Point internal links at final URLs so nothing redirects. Strip tracking parameters. Collapse redirect chains. Set a canonical on duplicates, and use the header form for non HTML ones.
  4. Check robots.txt against your sitemap with the robots.txt tester. A URL listed in the sitemap and blocked in robots is a contradiction that wastes budget.
  5. Re-crawl and confirm the issue counts drop.

Read next

Why AI cannot see content behind JavaScript

Non-rendering crawlers get the shell, not the content and how to fix it.

What internal PageRank is

How link equity flows through your own site, and why buried pages lose out.

What crawl budget is

The finite attention a crawler gives your site, and how it gets wasted.

All topics