One line of robots.txt locks the answer crawlers out

In the spring somebody reads an article about AI scrapers, copies a block list from a forum into the robots.txt and considers the matter closed. The rankings hold, Search Console says nothing, the log looks the way it always does. Six months later a prospect asks an answer engine for providers in the region, gets three names — and their own site is not among them, because the copied list carried OAI-SearchBot next to GPTBot, and that crawler does not collect training material but builds the index that gets quoted from. The loss announces itself nowhere.

How to go about it in JMX

1. Start a crawl. Enter the address, Start. JMX loads the robots.txt of every host involved; without a reachable file it later says “unknown” and not “fine”.
2. The GEO tab, “AI crawler access” card. For 22 known identities it gives the operator, the purpose and the access — Allowed or Blocked.
3. Read the Purpose column. It knows three values: model training, answer search, fetch on user request. Six identities count as answer-critical — OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, Amazonbot and YouBot.
4. The Technical tab. In the “crawler access” row, the technical sign-off separates search engines, answer engines and pure training crawlers and gives a number per group.
5. The Issues tab. The rules GEO.CRAWLER.BLOCKED and GEO.CRAWLER.PATH_BLOCKED name the identities concerned along with a recommendation; the Tasks tab carries the same work with an estimated effort.
6. Settings → Crawl → Custom robots.txt (draft). Paste the corrected version in and crawl again. The draft applies to the crawled site instead of its live file, foreign hosts keep theirs; the technical sign-off states that it ran against a draft.
7. After deploying, analyse the log files. Whether the answer crawlers come back is only in the server log — Apache, nginx and IIS, gzip-compressed too.

What to watch out for

“Blocked” is not an error. GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended collect training data; blocking them loses you not a single referral, with one exception — Google-Extended also controls grounding in the Gemini apps and in Vertex AI, and blocked, Gemini can no longer draw on the site as a source there, which is why JMX reports exactly that case one level above the pure training crawlers. And JMX explicitly does not count this as an error but calls its own message an inventory. Read the card as a traffic light and clear everything to green, and you give up a licensing decision you may well have made deliberately.

What is checked is a file, not behaviour. Whether a provider honors robots.txt is stated in no line of it, and the second block sits elsewhere: if a WAF or a CDN rule turns the user agent away with 403, nothing about that is in the file. That is what the user agent templates are for — JMX crawls as GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot and shows whether 200 or 403 comes back. The log analysis has a limit too: Googlebot, Bingbot, Applebot, YandexBot and Baiduspider are confirmed by JMX via reverse and forward DNS; for the AI crawlers no provider publishes anything checkable. An entry saying PerplexityBot remains the client’s claim.

The awkward special case is the path block. A crawler group of its own forbids /advice/, the site root stays open, the overview reports the crawler as allowed — that is what GEO.CRAWLER.PATH_BLOCKED is for, naming the pages concerned. If the * group blocks the path instead, JMX reports nothing here: that hits classic search engines just as much and is an SEO decision. And the list is a list. If a service launches a new bot next month, it is missing from the overview until somebody adds it.

Never re-read that one line and you learn about the exclusion not from a message but from the fact that, for months, somebody else has been standing in the answer to your own core question.