What Googlebot actually fetches: log files against the crawl

The crawl finds 4,000 indexable pages. Search Console reports 2,600 in the index, and nobody can say where the rest went. There is no shortage of theories — thin texts, too few internal links, crawl budget — but none of them can be evidenced from the crawl: it says what is on the site, not what a bot ever fetched of it. Spend three months lengthening texts while Googlebot burns half its requests on redirect chains and 404 pages, and you have opened the wrong building site.

How to go about it in JMX

1. Crawl first. Start in the Crawl tab. Without a loaded crawl the Status (crawl) column stays empty, and the question about never-fetched pages stays open.
2. The Log files tab, “Add files …”. Apache and nginx in combined and common format, IIS in W3C extended format; detection happens on the first lines, otherwise you choose manually. Rotated archives ending in .gz go straight in, several files make one analysis.
3. Set the switches under “Analysis”. “Verify bot identities by DNS” stays on. “Skip images, CSS and JavaScript” is preset. “Treat the query string as its own URL” is what you need for parameterised filter pages, “Join against the loaded crawl” for everything else. Then Analyse.
4. Read the lines above the table. They give entries and days, the share of bot requests without status 200, the URLs bots fetch that the crawl does not know, and the indexable pages with no bot visit at all.
5. Filter in the table. The columns are path, bot requests, other, status (bot), status (crawl) and last seen. The column filter on Status (bot) shows wasted budget with an address attached; where it differs from Status (crawl), the search engine is getting something other than your auditor.
6. The Tasks tab, the Structure tab. The log findings sit in the plan as prepared tasks — broken bot targets, orphaned bot URLs, never-fetched indexable pages, and from a hundred refuted bot claims onward, blocking the scrapers as well. Their impact stands at 0 points: the score is built over the crawl and knows nothing about crawl budget. In the structure diagram the bot requests colour the nodes: red means no confirmed bot was ever here.

What to watch out for

“Never fetched” is a statement about the period covered by the file, not about the site. Some pages Google fetches every few weeks; a log covering three days therefore produces hundreds of rows that mean only that nothing happened in those three days. The summary names the number of days for exactly this reason. Take 30 days, better more.

The second trap is the “other” column. If it says 0 almost everywhere, the analysis is rarely at fault: a CDN or a cache is serving the visitors, and the origin server’s log really does hold almost nothing but bots. For visitor numbers a server log is the weaker tool — that is what web analytics with Matomo is for; conversely, the behaviour of the search engines is only in the log, because a bot executes no JavaScript and appears in no analytics count.

Two limits belong with this. The DNS check resolves the IP backwards and the name it finds forwards again; what cannot be confirmed that way does not count as a bot request, even if it was one. And the analysis is a matter for the window: jmx-cli knows no switch for log files, so the nightly run does not contribute this part. Without a license the page table stops at 25 pages, while the line and bot counts stay complete.

After that, “Google is not indexing this” is no longer a theory but a list of paths with status code and date — and each one calls for a decision nobody has had to make so far.