A crawl every night: spotting changes with jmx-cli

A deployment goes live on Friday evening. Afterwards a template carries a noindex that does not belong there — not on one page but on 340. It is noticed when the clicks go missing, and that is two weeks later at the earliest, because Google has to re-crawl the affected pages before they fall out of the index. This is not a finding in the usual sense: the page answers 200, the title is right, the structure too. It was just different yesterday.

How to go about it in JMX

1. Run a first crawl in the interface and store it as a .jmx file via Save project. That file is the reference point: every further run attaches itself to it, and without it there is no predecessor to compare the new state against — both switches of the change detection need --project.
2. Describe the areas of the site under Settings → Segments, name and pattern each. “24 pages changed” then becomes “21 of them in the shop”.
3. Set the crawl limits and then leave them alone: --max-urls, --max-depth. Change them between two runs and you get hundreds of pages reported as “new” that are not.
4. Hook the nightly call into cron, a task scheduler or CI: jmx-cli example.com --project /var/jmx/example.jmx --changes /var/jmx/changes.xlsx --fail-on-change Critical --quiet. From the second run against the same project file, the command line works out the page-by-page comparison on its own.
5. Evaluate the return code, because that is how a scheduled run recognises its result: 0 nothing beyond the threshold, 1 the run itself failed, 2 something critical has moved. If you want the state and not just the movement to be armed, hang --fail-on Critical next to it.
6. In the morning, open the project file and go through the “What changed on the pages” section in the History tab. The table starts with the worst. A selected row shows the values before and after side by side, with the tally per segment below.

What to watch out for

The most common mistake sits in the script, not in the tool: treating 1 and 2 the same. An expired certificate, a full disk, a DNS outage — that is a 1 and not a statement about the site. The other way round is worse: check only for 2 and swallow the 1, and you have a night watch that has not run through for weeks because of a network problem and still looks green, because not a single critical finding was reported.

The comparison sees only what the crawl sees. jmx-cli does not render JavaScript: a page that sets its noindex only in the browser stands unchanged in both runs, because the served HTML is unchanged — that is what rendering in the interface is for. Search Console figures, rankings and Core Web Vitals are not fetched by the nightly run either. Two crawls are compared, nothing else.

The special case is the planned migration. On the night of a relaunch the comparison reports hundreds of critical rows and is right to — only that is not a fault but the migration. Run one crawl by hand beforehand as the new reference point. The limit of 25 pages without a valid license applies in the interface only — jmx-cli knows no license check and crawls as much as –max-urls allows.

Leave the threshold at Critical — a night watch that goes red three times a week over a lengthened title is, after a fortnight, a mail rule nobody reads any more.