04 · Pillar
Crawl, Index, Rank
Three stages, in that order: a crawler fetches the page, an index stores it, a ranking system picks it. Most ranking problems are really crawl or index problems, and both are measurable on your own site.
Sitemap lastmod: my site sets it for every page, and it matched the last significant edit on 1 of 92 updated articlesGoogle uses sitemap lastmod if it is consistently accurate. My site sets it per page, and it matched the last significant edit on 1 of 92 edited articles.Crawl, Index, RankHow long does Google take to crawl a new page? On this site, a median of 3 days, and the waits came in batchesOn this site, Google's crawler first requested 13 of 15 new pages a median of 3 days after launch, in batches, and none on the day a page went live.Crawl, Index, RankDo AI crawlers fetch JavaScript? In 22 days of my log, 14 claimed crawlers split three waysDo AI crawlers execute JavaScript? My 22-day log shows seven claimed crawlers requesting scripts and six requesting none. A request is not a run.Crawl, Index, RankOrphan pages on this site still get crawled: six articles with zero internal links drew the same bot traffic as linked onesSeven articles on this site cannot be reached by clicking from the homepage, yet the six with no inbound links drew bot requests at the same rate.Crawl, Index, RankSitemap Couldn't Fetch in Search Console: Real CausesSearch Console's 'Couldn't fetch' status can sit on a sitemap that loads fine. Google ties one cause to site quality, and no fix on the file touches it.Crawl, Index, RankVideoObject Schema Required Properties: Google's 3 FieldsGoogle requires three VideoObject properties and recommends 11. On an article with a YouTube embed, only three of the 11 still need a decision.Crawl, Index, RankWhat does mobile-first indexing mean now? Two end datesWhat does mobile-first indexing mean now? Google finished it October 31, 2023, ended its desktop fallback July 5, 2024, and still sends desktop Googlebot.Crawl, Index, RankMeta name googlebot-image: why Google ignores itGoogle accepts only googlebot and googlebot-news in a robots meta tag, so meta name googlebot-image does nothing, while robots.txt accepts it.Crawl, Index, RankDoes a Video Transcript Help SEO? An Audit of 60 VideosA crawler that reads your page cannot watch the video on it. I checked all 60 of mine and none ships the words spoken in them, which is the only form a machine can read.Crawl, Index, RankDoes llms.txt Do Anything? A Study and One Broken FileCommon Crawl found most llms.txt files are unedited template output. Mine passed every check and still told crawlers a section of my site did not exist.Crawl, Index, RankHow to Verify Googlebot Is Real: The Two-Step CheckAny program can type Googlebot into its user agent. Google's real test is a reverse DNS lookup confirmed by a forward one, and I ran it live against five addresses.Crawl, Index, RankHow to Detect a Crawl Trap From Server LogsMost of what looks like crawler waste is probing for files you never published. In my own logs it was 80.1%, and a keyword denylist put the same traffic at 21%.Crawl, Index, RankRedirect Headers: What Your Browser Hides From YouA browser resolves a relative Location header and caches whatever it is told. A crawler reads the raw headers, and this site's own 301 is missing one of them.Crawl, Index, Rank404 vs 410: Google Treats Them the Same. 429 Is Different.Google's own docs say all 4xx errors except 429 are treated the same, so a 410 does not remove a page faster than a 404. The code that changes crawling is 429.Crawl, Index, RankLog Analysis With No Server Logs: The Free-Plan MethodA static site has no access log to analyze. You can collect one yourself, and mine showed 72% of this site's crawler log was a single credential scanner.Crawl, Index, RankRobots.txt: The Longest Rule Wins, Not the FirstWhen an Allow and a Disallow both match a URL, Google picks the longer rule path. Seven of 19 retail sites ship exactly that conflict today.Crawl, Index, RankEvery Google Page Indexing Status, TranslatedGoogle's Page Indexing report has 18 statuses in three tiers. On this site, 105 of 245 discoverable URLs are engineered never to be indexed, and that is correct.Crawl, Index, Rank