04 · Pillar
Crawl, Index, Rank
Three stages, in that order: a crawler fetches the page, an index stores it, a ranking system picks it. Most ranking problems are really crawl or index problems, and both are measurable on your own site.
Most of your bot log was never a crawlerMost of what looks like crawler waste is probing for files you never published. In my own logs it was 80.1%, and a keyword denylist put the same traffic at 21%.Crawl, Index, RankA redirect that works in your browser can still fail every crawlerA browser resolves a relative Location header and caches whatever it is told. A crawler reads the raw headers, and this site's own 301 is missing one of them.Crawl, Index, RankGoogle treats 404 and 410 the same, and the code that changes its behavior is 429Google's own docs say all 4xx errors except 429 are treated the same, so a 410 does not remove a page faster than a 404. The code that changes crawling is 429.Crawl, Index, RankA static site with no server logs can still run real log analysisA static site has no access log to analyze. You can collect one yourself, and mine showed 72% of this site's crawler log was a single credential scanner.Crawl, Index, RankThe most specific robots.txt rule wins, not the first one writtenWhen an Allow and a Disallow both match a URL, Google picks the longer rule path. Seven of 19 retail sites ship exactly that conflict today.Crawl, Index, RankEvery status in Google's Page Indexing report, translatedGoogle's Page Indexing report has 18 statuses in three tiers. On this site, 105 of 245 discoverable URLs are engineered never to be indexed, and that is correct.Crawl, Index, Rank