New pages not indexed: finding the technical cause

The new advice section has been online for seven weeks, 80 posts, linked from the main menu. For 46 of them Search Console reports “Discovered – currently not indexed”. The client asks, and the usual answers — patience, more internal links, better copy — are no answers at all as long as nobody has checked whether Google is even allowed to fetch these pages, and whether what a crawler gets there passes technically as an indexable page. Until then, paid work is lying around returning nothing.

How to go about it in JMX

1. Start a crawl, then the Results tab. The Status codes, Canonicals and Directives sub-tabs show what stands in the way of inclusion. JMX derives indexability per URL in a fixed order — no response, blocked by robots, 5xx, 4xx, redirect, noindex, non-HTML, foreign canonical, indexable — so that the stated reason is unambiguous. A canonical that points all 80 posts at the section page explains the case in one line.
2. The Technical tab. The technical sign-off gathers eight core questions on one page: crawler access, robots.txt, sitemaps, indexability, redirects, load times, language codes and security headers. What could not be checked stands there as “Not checkable” and not as “Fine”.
3. Results → orphan pages. Pages that not a single internal link points to. Alongside them the optional Link score column (0 to 100, internal PageRank): three links from the home page weigh more than thirty from a forgotten tag archive.
4. The Rendering view. Loads the pages in Chrome, Edge or Chromium and compares the result with the HTML as served. What is interesting is the difference: links that only exist after JavaScript, and the finding nobody looks for — a noindex that is only set in the browser.
5. The Search Console tab, Inspect URLs. Takes the indexable pages of the crawl and queries the real status for each. The quota is 2,000 requests per day and property; JMX counts along and stops before Google refuses.
6. The Log files view. Apache, nginx or IIS logs held against the crawl. You see which indexable pages no bot ever fetched — and only what reverse and forward DNS confirm counts as a bot.
7. Generate a sitemap, then the Sitemap check tab. Only internal, indexable HTML pages with status 200 go in. Comparing it with the file served live shows which posts are missing there and which appear twice in different spellings.

What to watch out for

“Indexable” is not “indexed”. The crawl only says that nothing stands in the way; whether Google took the page is answered by the URL inspection in Search Console alone. A site: query is an indication. It is not proof.

The limit of the method sits at the end: JMX submits nothing to Google. The IndexNow button in the migration view reaches Bing, Yandex, Seznam and Naver; that Google does not take part is stated in the success message. A button that pushes a page into the Google index exists in no tool.

Two special cases distort the analysis if you overlook them. Without a license every analysis stops at 25 pages — a section with 80 posts is never checked in full that way, and the status bar says why. And where no log file is loaded there is no statement about bots: open means open, not no.

If everything stays technically clear and Google still does not take the pages, the cause is no longer a technical one — then it is worth looking at the near-duplicate detection and at the Semantics view, which finds pairs of pages that mean the same thing and therefore compete for the same slot.