# Cloudflare wrote a real AI agent checklist. Don't trust the score next to it.

> Cloudflare published a real, checkable AI-agent specification. A 15-retailer test shows why the score placed beside it should not be trusted.

Canonical: https://brandonlazovic.dev/articles/cloudflare-agent-readiness-checklist/  
Author: Brandon Lazovic  
Published: 2026-08-18

## The short version

- Cloudflare's agent-readiness checklist, organized into four named tiers, is a genuinely concrete answer to what a site must do for AI agents, but the AEO score placed beside it is graded against a category benchmark computed once and reused across every account in that category.
- A rerun of a 15-origin probe (14 large US retailers plus this site, run 2026-08-12) reached 14 of 15 origins, of which 12 name zero AI crawlers in robots.txt, none carry Cloudflare's own Content Signals directive, and only 2 serve a real llms.txt file. Kroger timed out and is unmeasured rather than counted as absent.
- Target's llms.txt is one of the two real ones found, and it documents its own URL grammar for products and categories, the single most useful artifact in the sample.
- Cloudflare's own Mention Rate versus Citation Rate split, not its overall score, is the metric worth acting on: a gap between the two means an assistant already knows a brand but is not yet citing its URL.

Cloudflare published a real specification for what a website has to do to work with AI agents, organized into four named tiers rather than a vague call to be "AI ready." [1] On 2026-08-12, 14 large US retail homepages plus this site were checked against the public, checkable parts of that specification. Among the 14 that responded, 12 name zero AI crawlers in their robots.txt file. Not one carries Cloudflare's own Content Signals directive. Only two serve a real llms.txt, and just one of those documents its own URL grammar for agents to parse. The checklist itself is the durable artifact here. From a category benchmark computed once and reused across every account in that category, the score Cloudflare places next to it is the part worth distrusting.

> Not one of the 14 reachable US retailer homepages declares Cloudflare's own Content Signals directive. Two serve a real llms.txt, and only one of those documents its own URL grammar for agents.

## What does Cloudflare's agent-readiness checklist actually ask a site to do?

Cloudflare's checklist groups roughly a dozen technical checks into four ranked tiers, ordered by effort rather than importance. [1] The bottom two tiers cover basics most sites can fix directly, while the top two ask for protocols most sites have not built. Separately, a fourth tier tracks payment standards and does not count toward the score. [1]

Quick wins covers the basics most sites are missing, including a crawler-readable robots.txt, an XML sitemap, AI-crawler-specific rules, and clean Markdown served to agents.

Technical groundwork adds Content Signals, a robots.txt directive stating whether AI training, AI input, or search indexing is allowed. Alongside that, it asks for an API catalog, link headers (metadata pointing to related resources), and agent login instructions.

Above that sits Advanced integration, covering five agent-native features.

- OAuth discovery, agent authentication before an API call
- The Model Context Protocol, letting an assistant call a site's tools directly
- Agent2Agent cards, describing what one agent can do for another
- A skills index, tasks an agent can run on the site
- Web Bot Auth, proof of which bot is knocking

Beyond those three, a fourth tier called Commerce covers four emerging agent-payment standards.

- x402, a payment protocol built on HTTP's 402 status code
- The Agent Commerce Protocol, a proposed standard for agent checkout
- The Universal Commerce Protocol, a rival proposal for the same step
- AP2, Google's Agent Payments Protocol

Cloudflare states plainly that this tier is informational only and does not count toward the score. [1]

## How ready are 15 major retail sites for that checklist right now?

Fifteen major retail sites are not ready for Cloudflare's checklist yet, and the shortfall concentrates in the newer tiers. On 2026-08-12, fourteen large US retailers plus this site were tested against `/robots.txt`, `/llms.txt`, and `/.well-known/mcp.json` for status code and content type rather than a bare 200.

The sample covered 14 large US retailers, including Walmart, Target, Kroger, Gap, Macy's, and Etsy, alongside brandonlazovic.dev. Because Kroger timed out on repeated attempts, a known flake for that host, robots.txt was reachable on only 14 of 15.

Of those 14, two name at least one of the six AI crawlers Cloudflare's checklist checks for. Gap names five, GPTBot, ClaudeBot, PerplexityBot, CCBot, and OAI-SearchBot, while Macy's names one, ClaudeBot. Across all 15, none carry a Content-Signal directive.

Sitemaps are close to universal. Walmart alone declares 34, though not every site clears the bar. Because Etsy's robots.txt disallows crawling its own internal sitemap help pages and declares no `Sitemap` directive at all, and Kroger's non-response leaves it unmeasured, the honest count is 13 of 15 confirmed, not the 15 of 15 first expected.

Not one origin returns a working `/.well-known/mcp.json`. Every attempt ends as a 404, a 500, or a timeout, a trust question this site has [covered before at the protocol level](/articles/mcp-servers-trust-question/).

![Table of three checks against 14 reachable retail homepages out of 15 tested (Kroger timed out and stays unmeasured, not counted as failing). 2 of 14 name at least one of six AI crawlers in robots.txt, Gap names five and Macy's names one. 0 of 14, highlighted here, carry Cloudflare's own Content Signals directive, zero adoption across the sample. 2 of 14 serve a real llms.txt file, and only Target's documents its own URL grammar for building a product or category page instead of only listing existing links.](/figures/cloudflare-agent-readiness-checklist/checklist-gap.png)

*A first-party probe run 2026-08-12 against robots.txt, llms.txt, and /.well-known/mcp.json across 14 large US retailers plus this site.*

## Does zero adoption of Content Signals mean retailers are failing, or that the standard hasn't won yet?

Zero adoption here reads more as a standard that has not won yet than as retailers failing an audit. In April 2026, Cloudflare scanned the 200,000 most-visited domains and found Content Signals adoption at roughly 4% of sites, a figure Cloudflare itself called early momentum rather than a mature standard. [2]

Against that 4% figure, zero of 15 in this narrower, retail-specific sample is not a contradiction. It is consistent with a standard that is early everywhere and later still in this vertical.

Content Signals also carries a structural limit that has nothing to do with adoption speed. Plainly, the specification states that "not all automated systems honor robots.txt files, and some may ignore these Content-Signal directives." [3] A directive a crawler can disregard functions as a stated preference, not an access control, closer to a licensing reservation than a firewall rule.

Cloudflare's own generator ships that directive with a clause reserving rights under Article 4 of the EU's 2019 copyright directive, a legal posture aimed at future enforcement. [3]

## What makes Target's llms.txt actually useful, unlike most llms.txt files?

Target's llms.txt does the one thing the llms.txt proposal was designed for, and that most implementations skip. It tells an agent how to construct a URL rather than only listing existing ones. On 2026-08-12, a direct fetch measured the file at 6,205 bytes with a genuine text/plain content type.

It opens with three grammar notes rather than a bare link list. Product pages follow `/p/{product slug}/-/A-{TCIN}`, category pages follow `/c/{category name}/-/N-{category node ID}`, and browse pages follow `/b/{browse name}/-/N-{browse node ID}`. [9]

From those three lines alone, an agent can construct a working product or category URL for Target without crawling a single page first, a meaningfully different artifact from the link directory most llms.txt files are.

Whether any assistant's retrieval pipeline reads llms.txt is separate from whether sites are publishing one, and nobody has confirmed the first half. The proposal's own author writes that OpenAI, Anthropic, and Google's Gemini publish llms.txt files for their own developer documentation, and that Chrome's Lighthouse tool now audits sites for one. [4]

In both cases, that is real momentum, not evidence that a live assistant fetches a stranger's llms.txt before answering a question about that brand, so this piece treats vendor consumption as unconfirmed.

## Should you trust the score Cloudflare puts next to the checklist?

Cloudflare's own score should not be read as an accuracy measurement, and Cloudflare's own post explains why. According to Cloudflare, building the score means querying assistants with realistic category prompts, building a benchmark, then reusing it across every account in the category. [1]

Cloudflare states this directly: "rather than re-querying models every time a site owner runs a scan, we run this panel once per category and reuse the baseline across all accounts in that domain." [1]

Where judgment is needed, Cloudflare's own Workers AI grades against exact text analysis rather than a model grading its own output, and each assistant is prompted several times to smooth answer variance. [1]

Since Cloudflare documents the mechanism rather than hiding it, none of that is disqualifying. The honest criticism is freshness and independence, not accuracy. A score built once per category will not move when a competitor ships a new page, and there is no outside way to check whether Workers AI's grading matches customer experience.

As this site has already argued elsewhere, engines discover pages through structurally different mechanisms, and [chasing per-engine optimization is a trap](/articles/brave-search-discovery/). A published checklist and a per-vendor visibility score are different objects, and the score inherits none of the checklist's transparency.

Currently, the AEO tab sits behind a request. Cloudflare's own post closes by asking readers to "request early access to AEO Visibility," so this piece treats the dashboard as unavailable. [1]

## What's the one metric from Cloudflare's tool actually worth keeping?

The split between Mention Rate and Citation Rate is worth keeping, not the composite score built on top of it. By Cloudflare's own definition, Citation Rate is the share of category answers that cite a site as a source, and Mention Rate is how often an assistant names the brand, cited or not. [1]

Cloudflare frames the gap this way: "read alongside your Citation Rate, it separates awareness from attribution: assistants naming you far more than they cite you means you're on their radar but not yet earning the citation, a specific, targetable gap." [1]

That is a different, actionable finding than a single blended score. Either assistants do not know a brand exists, or they know it and cite someone else's URL anyway, and the gap tells you which.

On 2026-08-03, Microsoft Clarity shipped a related but distinct split, adding branded-versus-non-branded segmentation to its AI Citations dashboard. That lets an account separate brand-name citations from generic category citations. [8]

Cloudflare's mention-versus-citation split and Clarity's branded-versus-non-branded split measure different things, and neither vendor built the other's feature. At bottom, both point at the same idea. A single AI-visibility number hides more than it shows, and vendors here are arriving at finer splits, not a better single score.

## What does this site's own crawler log show about automatic crawling versus live user requests?

This site's own log shows automatic crawling running far ahead of live, user-triggered fetches for two of three assistants, though it cannot prove those crawls drive citations. Within that split sit two distinct crawler identities every major assistant publishes: an automatic, unprompted crawl bot, and a separate user-agent string reserved for a live, user-triggered fetch. [5] [6] [7]

OpenAI documents ChatGPT-User as tied to "certain user actions," not automatic crawling. [5] In the same way, Perplexity documents Perplexity-User, adding that it "might visit a web page to help provide an accurate answer and include a link to the page in its response." [6] For Anthropic, the equivalent is Claude-User, which "supports Claude AI users" when someone asks a question. [7]

As of 2026-08-12, `analytics/ai-crawler-log.csv` held 882 rows and 14,166 total logged requests across 27 dates, 2026-07-16 through 2026-08-11. The crawl-to-user ratio varies by assistant rather than sitting at one universal skew.

OpenAI logs 717 automatic requests, GPTBot plus OAI-SearchBot, against 718 ChatGPT-User requests, essentially even. For Perplexity, the split runs 542 PerplexityBot requests against 14 Perplexity-User requests, roughly 39-to-1 toward the crawl-only bucket. Anthropic's ClaudeBot logs 539 requests against 16 Claude-User requests, roughly 34-to-1, much closer to Perplexity's skew than to OpenAI's near parity.

Because this log has no referral field, it cannot show whether these fetches sent a person back to the site, the same evidentiary gap this site named when [checking Google's own AI-traffic claims against site-level data](/articles/google-ai-clicks-open-web/). Still, one operator's pattern here is [a demand signal worth reading, not a verdict](/articles/ai-usage-is-a-demand-signal/). Two of three assistants run crawl-heavy relative to any live-user signal, and only OpenAI's ratio comes close to even.

## What can you check on your own site right now?

Run the checklist's quick-wins tier against your own origin with two commands.

The first checks robots.txt for the six crawlers Cloudflare's checklist looks for, plus a Content-Signal line, using `curl -s https://yoursite.com/robots.txt | grep -iE 'content-signal|gptbot|claudebot|perplexitybot|google-extended|ccbot|oai-searchbot'`. The second checks whether `/llms.txt` returns a real file rather than a disguised error page: `curl -sI https://yoursite.com/llms.txt | grep -i content-type` shows whether a 200 status marks a real file. There, a `content-type: text/html` response usually means your server returned its not-found page dressed up as a success, the soft-404 this probe had to screen for on every 200 it found.

Both real hits in this sample, Target and brandonlazovic.dev, cleared that check with a genuine `text/plain` body. The other 13 origins skipped the question by returning a 404, a 500, or nothing. Only `text/plain`, or an explicit markdown type, gives an agent something to parse instead of a page to strip first.

Cloudflare's checklist and its mechanics are documented well enough to check yourself, more than most vendor scoring tools offer. That transparency is why the checklist earns a place here, while the score next to it does not.

## Sources

1. Cloudflare: From ranking to recommended (Matthew Conroy and Jack Galilee, Aug 6, 2026) — https://blog.cloudflare.com/aeo/
2. Cloudflare: Introducing the Agent Readiness score (Andre Jesus and Vance Morrison, Apr 17, 2026) — https://blog.cloudflare.com/agent-readiness/
3. Content Signals (Cloudflare's specification page) — https://contentsignals.org/
4. The llms.txt file, v2 (Jeremy Howard) — https://llmstxt.org/
5. OpenAI: Overview of OpenAI Crawlers — https://developers.openai.com/api/docs/bots
6. Perplexity: Perplexity Crawlers — https://docs.perplexity.ai/docs/resources/perplexity-crawlers
7. Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? — https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
8. Microsoft Clarity: See Branded and Non-Branded AI Queries in the Clarity Citations Dashboard — https://clarity.microsoft.com/blog/branded-non-branded-queries/
9. Target.com: /llms.txt (fetched 2026-08-12) — https://www.target.com/llms.txt