How to v1.0, updated 2026-07-07

Reading server logs to see what crawlers do

Your access log is the ground truth of how bots actually crawl you.

Every request to your server is recorded in an access log: the URL, the time, the status code and the user agent that asked. That log is the ground truth of how crawlers actually treat your site, not how you assume they do.

It answers questions nothing else can. Which bots visit, and how often. Which pages they crawl, and which they never touch. How much of their time goes on redirects, errors and pages you do not care about. Whether a bot you want, like GPTBot, is reaching your key pages at all.

To read one, filter to the user agents you care about, then look at the status codes and the URLs. A lot of 404s means bots are chasing dead links. A lot of 301s means they are hopping through redirects. A pile of requests to parameter URLs means crawl budget is leaking there.

Kenovar's log analyser takes an access log and does this for you: it groups by bot, counts the status codes and surfaces the waste. It is the honest counterpart to the audit. The audit tells you what should happen. The log tells you what does.

Read next

Why AI cannot see content behind JavaScript

Non-rendering crawlers get the shell, not the content and how to fix it.

What internal PageRank is

How link equity flows through your own site, and why buried pages lose out.

What crawl budget is

The finite attention a crawler gives your site, and how it gets wasted.

All topics