Getting filters, search pages and print views out of the index

A shop with 800 articles, and Search Console reports 41,000 indexed pages. The number is not a typo. The surplus consists of filter combinations, internal search results and print views. It gets expensive in two places: Googlebot fetches variants instead of finding new articles, and the signals of a real product page are spread across a dozen addresses that nobody linked and that differ only in the order of the results.

How to go about it in JMX

1. Crawl without blinkers. Under Settings → Crawl, empty the Exclude URLs field for this run and raise the maximum number of URLs. If you are looking for parameter addresses, you must not filter them out beforehand.
2. Read the order of magnitude. The File types card on the Dashboard breaks down what the URL count consists of — HTML pages, redirects, images, foreign hosts; every bar opens the list it counts.
3. Collect the candidates. In the Results tab, Internal sub-tab, set the search field with Regex ticked to [?&](sort|filter|q|print)=. The Directives sub-tab shows alongside it which of these addresses carry a noindex.
4. Search for the gap directly. The Text search tab also asks the other way round: “does not contain” noindex, in the source, restricted to the path area /suche/. Out comes the list of pages that should have carried the directive.
5. Read the findings. The Issues view carries SEO.URL.TOO_MANY_PARAMS (default: more than 3 parameters), SEO.DUPLICATE.TITLE, the near-duplicate detection SEO.DUPLICATE.CONTENT_NEAR up to Hamming distance 6, and SEO.SITEMAP.NONINDEXABLE_URL.
6. Assign. Under Settings → Segments, create a segment “Filters and search” via a regular expression; the breakdown with findings per page then sits above the rule list, including the entry for pages without a segment.
7. Hold it against the index. In the Search Console tab, Inspect URLs queries the real status for the indexable pages of the crawl; the quota is 2,000 requests per day and property.

What to watch out for

A block in robots.txt clears nothing out of the index. It prevents loading, not inclusion: Google can list a blocked address on incoming links alone, then without a title and without a description, and the noindex on the page goes unread because the page is never fetched. The order is therefore: allow first and set noindex, wait until the addresses are gone, then block. You check the draft beforehand under Settings → Crawl → Custom robots.txt (draft) against the full crawl.

JMX knows the crawl, not the index. What the spider reports is the set of reachable addresses, not the set of what Google has stored. The URL inspection in Search Console is the only authoritative answer. A site: query is an indication. The reverse holds just as much: filter URLs that only come into being when a form is submitted and that are linked nowhere are found by no crawler — but by Google via an external link. That is why the Migration view pulls an inventory together from several sources that contradict each other on purpose.

The most awkward special case sits in your own exclusion pattern. Keep \? permanently in “Exclude URLs” and you get a clean report about a site whose problem you have just hidden from yourself. A second one comes out of rendering: if a print view sets its noindex only via JavaScript, it looks indexable in the served HTML — that case is reported by JavaScript rendering.

Whether the clean-up is one small job or a project is not decided by the number of addresses: the Tasks tab puts the estimated effort next to the impact in points, and only that second value says whether 12,000 superfluous URLs are one line in a template or three person-days.