Changing robots.txt without locking pages out

One line is meant to go into the robots.txt. The admin directory should be blocked, an endless calendar is eating crawl budget, and the copied template holds three more Disallow lines nobody can explain any more. You deploy the file, four weeks later the impressions fall, and by the time somebody traces the cause to an old deployment, the affected pages have long since disappeared from the index and the rankings with them. The mistake is not expensive because it is hard to fix, but because weeks lie between cause and symptom.

How to go about it in JMX

1. Record the current state. Enter the address at the top, Start, save the run in a project file. This crawl is your baseline.
2. Store the draft. Under Settings → Crawl → Custom robots.txt (draft) paste the full text of the planned file; from then on it applies to the crawled site instead of its live file.
3. Run the crawl a second time. The draft is a crawl setting and not a filter over the loaded inventory. Without a second run, no number changes.
4. Open the History tab. The comparison of the two runs goes page by page: every page that loses its indexability stands in the list with the risk Critical, with the value before and after side by side. That is the answer to “what am I locking out” in rows rather than in totals.
5. Go through the Issues tab. Three rules belong here: SEO.ROBOTS.BLOCKED_BY_ROBOTS_TXT, SEO.ROBOTS.BLOCKED_BUT_IN_SITEMAP — the sitemap submits what the robots.txt forbids — and SEO.ROBOTS.BLOCKED_RESOURCE for your own blocked CSS and JS files.
6. The GEO tab, “AI crawler access” section. Every crawler tile names its purpose: model training, answer search, or fetch on user request. Blocking training costs no referrals; blocking the answer crawlers takes you out of the AI answers.
7. The Technical tab. The technical sign-off gathers crawler access, robots.txt and sitemaps on one page and states explicitly that it ran against a draft.

What to watch out for

A Disallow takes nothing out of the index. It forbids retrieval, not inclusion: a blocked URL can still appear in the results, then without a title and without a description. What gets awkward is the combination of both, because a noindex on a page no crawler may load any more is never read — which is why JMX derives indexability in a fixed order, and “blocked by robots” sits ahead of “noindex” there. If you want to be rid of a page, leave it retrievable.

The calculation runs against the crawl, not against Google’s index. What no link and no sitemap entry reaches is not in the inventory and therefore cannot appear as a loss either. The draft also applies only to the hosts of the start addresses; foreign hosts keep their own file, because a draft is the file of one host and anything else would be an invented finding. Without a license every analysis stops at 25 pages, and comparing two 25-page runs does not answer this question.

The most common special case is a line that blocks everything with “AI” in the name across the board and catches the answer crawlers with it. Just as widespread are path blocks from old templates that exclude whole system directories such as /wp-includes/ or /assets/ — a recommendation that has been obsolete since 2014, because Google renders pages before it evaluates them and, without the blocked stylesheets, gets to see a broken layout.

Check the draft beforehand and you move the discovery from the traffic drop to the afternoon before it — and in the deploy script jmx-cli --robots-file draft.txt example.com --fail-on Warning takes that over, its return code 2 keeping the file from going live at all.