The crawl does not get through: 403, timeout, login area

The crawl starts, and after forty addresses the Status column shows three 403s, two timeouts, and half the shop does not appear at all because it sits behind a login nobody mentioned. The damage does not come from the block but from the fact that an incomplete crawl does not look incomplete: it looks like a site with 180 pages, the metrics above it look plausible, and the report built from it describes a site that does not exist in that form. At first, nobody notices the gap.

How to go about it in JMX

1. Read what the run reported. In the Results tab the Status, Indexability and Reason columns say what came back; the robots.txt sub-tab lists the addresses the file blocked.
2. Throttle the speed before you change anything else. Under Settings → Crawl, Concurrent requests and Concurrent requests per host are 5 by default, Delay between requests (ms) is 0. On a 429 go down to 1 or 2 requests and 500 to 2000 ms; next to them sit Timeout (s) at 30 and Retries at 2.
3. Take crawl-delay seriously. Respect crawl-delay adopts the value from robots.txt, and the larger of the two applies — your delay is a lower bound. The Technical tab names the number of seconds: 10 seconds means more than 13 hours for 5,000 pages, on paper.
4. Check the identity. Under User agent and Template, JMX crawls as JMXBot/1.0 with a contact address by default. The identity also decides which robots.txt group applies: with the Googlebot template, the Googlebot rules apply — on someone else’s server, a false statement.
5. Store basic auth. HTTP basic authentication with Username and Password, or without a window via jmx-cli --user … --password ….
6. Crawl behind the login. Switch on Forms-based authentication, enter the Login page address, and if needed the Username field name (optional) and Password field name (optional). JMX fetches the page, fills in the form and sends hidden fields such as CSRF tokens along unchanged; if the response page shows a password field again, the run stops.
7. Self-signed certificate. That one sits in the Environments tab: Accept invalid certificates applies to the Test environment only, the live site is crawled untouched. On the command line, --insecure.

What to watch out for

The most common false conclusion has nothing to do with the server. Without a license JMX evaluates at most 25 pages per analysis, and the limit announces itself every time. A run that stops at 25 addresses is not bot protection.

Against bot protection, nothing in the settings helps anyway. Only what is temporary gets retried — timeouts, 429 and 5xx — and on a 429 JMX waits the time the server names; a 403 is settled after the first attempt. JMX does not switch IP addresses and does not solve CAPTCHAs. The Rendering tab is no way out either: it works on the loaded crawl, and on a page with 403 there is nothing to render. What helps sits on the other side: an exception in the firewall for your IP address or your identity.

Two special cases remain. A draft under Custom robots.txt (draft) applies only to the hosts of the start address; a test system on its own hostname keeps its live file. And switching off Respect robots.txt — --ignore-robots on the command line — is a decision for your own systems.

So record in every report how many addresses the run reached and how many the server refused: otherwise, at the next crawl, you are comparing against an inventory whose gap nobody can explain any more.