02 · Pillar
How Machines Read the Web
Agents and crawlers read structure, not pixels. Semantic HTML, server-rendered schema, and embeddable item data decide what gets understood and surfaced.
Agent Skills well-known index: more than 70 platforms adopted the format, and in 36 days one scanner asked for my site's indexMore than 70 platforms support Agent Skills, but the standard has no website path. My site's well-known skills index drew one scanner in 36 days of logs.How Machines Read the WebHow ChatGPT search finds pages: its own crawler, plus Cloudflare's view of the web. On my site, that crawler trailed Googlebot on 18 of 24 new articlesChatGPT search finds pages with OpenAI's crawler, and a Cloudflare pilot from July adds network signals. On my site it trailed Googlebot on 18 of 24.How Machines Read the WebFor about a week, ChatGPT stopped tagging its citation links, so GA4 filed those clicks as DirectFrom GPT-6 on October 7, 2026 until a fix reported October 15, many ChatGPT citation links lost their utm_source tag, so GA4 filed the clicks as Direct.How Machines Read the WebSearch Console's generative AI control: the closest property that changed it decides the settingIn Search Console's generative AI control, the closest parent property that stopped inheriting decides the setting, so a child property can override the domain.How Machines Read the WebDoes Google Pay Publishers for AI Overviews? Only in a PilotDoes Google pay publishers for AI Overviews? Only in a Search Console pilot, confirmed to one outlet, that pays for an event Google's AI report never counts.How Machines Read the WebLighthouse's AI catalog audit rewards publishing nothingLighthouse 13.5's AI catalog audit scored this site 4 of 4 with no catalog, and 4 of 5 with the catalog a content site would write first.How Machines Read the WebSearch Console AI Report: Impressions, No ClicksGoogle's generative AI report shipped worldwide with impressions and no clicks. Three separate changes hit AI citations in the same two weeks, and the report cannot price any of them.How Machines Read the WebWhat agent-readiness scanners actually checkA public scanner rated this site level 4 of 4 for AI agents while three of its machine-facing claims were false. Structural checks cannot see a wrong value.How Machines Read the WebGoogle's crawlers download JSON, they don't parse itGoogle now applies one pass of HTML unescaping to JSON-LD, and its crawlers do not parse JSON at all. A successful fetch says nothing about whether your data parsed.How Machines Read the WebGoogle goto Redirects and URL-Level AttributionGoogle confirmed google.com/goto redirects are rolling out. Domain-level rank tracking survives the change, and URL-level attribution is what degrades.How Machines Read the WebChatGPT answered 56% of queries without searchingAcross 270 alternatives queries, ChatGPT searched the live web on 44%. On the rest nothing you publish this quarter can change the answer.How Machines Read the WebBlocking AI crawlers did not stop AI citationsAround 75% of sites blocking OpenAI or Google AI bots were still cited. Their own table shows why: nobody blocked the retrieval bot, only the training bots.How Machines Read the WebCitation half-life and your content refresh queueA vendor now plots per-URL citation decay and says half of cited content is under 13 weeks old. The metric is useful. The figure has no published method.How Machines Read the WebCloudflare's AI agent checklist, tested on 15 retailersCloudflare published a real, checkable AI-agent specification. A 15-retailer test shows why the score placed beside it should not be trusted.How Machines Read the WebWhy most AI-driven site visits skip the referral tagA panel study finds AI mentions lift site visits 1.5 to 2.5x, but for ChatGPT, only about 2.5% of those visits carry a trackable referral tag.How Machines Read the WebGoogle Tests Sign-In Wall on Search Results After SerpApiGoogle is testing a sign-in wall on deep search results weeks after a DMCA loss to SerpApi, swapping an anonymous CAPTCHA for an identity check.How Machines Read the WebWhen an AI agent logs in, analytics see a humanThree AI agents now log into accounts and complete purchases, and this site's own crawler log proves no user-agent bucket can catch them.How Machines Read the WebAI Text Watermarking: What It Can and Can't ProveAnthropic now watermarks Claude's text. Google has watermarked Gemini's since May 2024 with no ranking signal to show for it. Here is what the mark proves.How Machines Read the WebCrawl Budget 304s: Why Most Retail Sites Still FailGoogle's crawl-budget guide now recommends serving 304 responses, but testing 13 live retail homepages found 11 still send the full page every time.How Machines Read the WebRobots.txt Disallow vs Noindex: What Each One Actually BlocksGoogle indexed Claude's blocked share pages via inbound links, then said noindex needs crawl access too. Same mechanism, opposite failure.How Machines Read the WebValidate Fix vs. Request Indexing in Search ConsoleValidate Fix samples a few pages before recrawling the rest, which only pays off for batch issues affecting many URLs.How Machines Read the WebChatGPT source selection: the editable policy behind itChatGPT's source selection runs on an editable server-side policy that OpenAI can rewrite at will, and the underlying fields already changed twice in July 2026.How Machines Read the WebGoogle's billions of clicks claim vs. the traffic dataGoogle says AI Search sends billions of clicks weekly and calls the number stable, but Cloudflare and Wikipedia both report reproducible traffic declines.How Machines Read the WebAnnotated Page Content (APC): what Gemini reads in ChromeChrome builds a structured, typed node tree from a page's render tree and hands that to Gemini when a tab is shared, per Dejan AI's teardown.How Machines Read the WebDoes AI prefer Reddit? OpenAI rejects it, Google keeps itOpenAI rejects 99.4% of the Reddit pages it retrieves, Anthropic cites zero, and only Google keeps Reddit near its baseline citation rate.How Machines Read the WebShopify's Hreflang Toggle Doesn't Fix the Real RiskShopify's new toggle turns off auto-generated hreflang tags, but it can't verify your replacement is correct or see CDN-level conflicts.How Machines Read the WebHow Brave Search finds pages (and cites via Claude)Brave has no Search Console, no sitemap ingestion, and no IndexNow. It rides on Googlebot-crawlability, and Claude's web search runs on Brave's index, not Bing's.How Machines Read the WebBot-Challenge Screens Can Get Your Real Page DeindexedGoogle's John Mueller confirmed bot-challenge screens served to Googlebot with a 200 status get indexed, and dedup can hand your canonical to a stranger's copy.How Machines Read the WebGoogle's nearest-seed patent: what it says vs what leakedA 2006 Google patent ranks pages by distance from a few trusted seed sites. It expires in 2026, but the leak shows the idea survived as PageRank-NearestSeeds.How Machines Read the WebEU AI Act Article 50: AI content disclosure rulesThe EU AI Act's Article 50 makes AI-content disclosure enforceable on August 2, 2026. Here is what gets marked, who has to disclose, and what it means for search.How Machines Read the WebWhat Is RAG (and When Not to Use It)RAG lets an AI look answers up instead of guessing from training. Here is how it works, what you can build with it, and when a simpler tool wins.How Machines Read the WebDoes Google use S-BERT to detect AI content? What the paper saysGoogle's S-CTS paper targets coordinated AI-slop channels on video platforms, not Search rankings. The real lesson is coordinated footprints, not your pages.How Machines Read the WebAI usage is a demand signal, not a jobs reportAnthropic's June 2026 Economic Index shows the unit of demand shifting from keyword to artifact. Read as a demand signal, it maps which intents AI is absorbing.How Machines Read the WebClaude Sonnet 5 vs Opus 4.8, and the agent cost curveClaude Sonnet 5 matches or comes within two points of Opus 4.8 on most agentic benchmarks, at a fraction of the price. For commerce, the story is the cost curve.How Machines Read the WebAgents read the accessibility tree firstProduction browser agents like Playwright MCP read the accessibility tree before pixels. For e-commerce, clean semantic HTML and schema decide machine readability.How Machines Read the WebSoft tokens: single embeddings for item representationGoogle is testing 'soft tokens', single embeddings that replace multi-token item descriptions, for cheaper representation with cold-start implications.How Machines Read the WebGoogle's crawler split: SSR vs. CSR and Merchant CenterThe Shopping crawler does not render JavaScript; it validates price and availability. SSR vs. CSR is the real determinant of Merchant Center compliance.How Machines Read the Web