Disallow Stops the Crawl. It Doesn't Stop the Index.

Google indexed shared Claude conversations in late July even though claude.ai’s robots.txt disallowed the path Googlebot needed to fetch them. Three days later, Google described the mirror-image mistake. On the Search Off the Record podcast, John Mueller and Martin Splitt said a noindex tag on a spam-abused search-results page does nothing if a robots.txt rule keeps the crawler from ever reading it. Both failures share one cause. In practice, Disallow controls what Googlebot fetches, not what Google is allowed to list, and noindex only works on a page the crawler can actually open. Block harder in either direction and the indexing problem gets worse, not better.

113 of 869 AI-crawler requests to this site over three days went to /robots.txt, 13.0% and the single most-requested path, ahead of the homepage at 90.

What happened when Google indexed Claude’s disallowed share pages?

Over the weekend of July 25 to 26, 2026, shared Claude conversations began surfacing in Google search results, discoverable through the site: search operator. Checking claude.ai’s crawl rules on July 27, Search Engine Journal’s Matt Southern found the share pages carried an X-Robots-Tag: none header, the noindex signal, sitting behind a robots.txt rule that disallowed the same path for the wildcard crawler group.¹

The noindex instruction was sitting right there, and Googlebot could not read it. By then, Disallow had already blocked the fetch.

TechCrunch and 404 Media reported finding medical material, internal company documents, and files carrying the names and phone numbers of primary-school-aged children surfaced this way, alongside a second exposure path through Claude’s shared Artifacts.¹ On July 26, a day earlier, independent IT consultant Daniel J. Glover had flagged the same conflict. Anthropic told TechCrunch that a share link only becomes discoverable if someone posts it somewhere a crawler can reach, which is accurate and does not touch the actual mechanism. Once a single link escapes onto any crawlable page, the index entry follows. Disallow cannot un-invite it.

Why does Google’s own documentation say Disallow can defeat a noindex tag?

Google states the requirement directly in its noindex documentation: “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler.”⁴ Published long before this incident, that sentence is a precise description of what happened to Claude’s share pages.

The same passage spells out the consequence. If the page is blocked, “the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it.”⁴

From the other direction, Google’s introduction to robots.txt carries the same warning: a page disallowed in robots.txt “can still be indexed if linked to from other sites,” and combining crawling and indexing rules “might cause some rules to counteract other rules.”³ Read together, the two documents describe one mechanism from opposite ends. What gets removed is Googlebot’s ability to read anything on the page at all, including the one tag meant to pull it from the index.

Should you disallow or noindex your internal search-results pages?

Disallow the page pattern to stop new junk from being crawled in the first place, and reach for noindex only on URLs that are already indexed and need to come out, because the two tools solve different problems and stacking them defeats the second one.

On the July 30 episode of Search Off the Record, Splitt and Mueller described how internal site search creates an “infinite crawl space,” an unbounded set of auto-generated URLs that strains crawl budget and server load.⁵ Beyond crawl budget, the episode names a security angle distinct from ordinary duplicate content. Bad actors query a site’s own search feature for pharmaceutical, adult, or casino terms, and the resulting auto-generated result pages get indexed carrying that spam under the site’s own domain authority.⁵ That is functionally the same failure as the Claude case, in my own read rather than language the episode uses: a page nobody chose to publish, landing in the index because a crawler could reach it and nothing stopped it once it was there. Fixing spam already indexed this way takes noindex, not disallow, because a removal instruction only works if the crawler revisits the page to read it, and a block would prevent exactly that revisit. On top of that, the show notes add two practical rules: one broad wildcard Disallow pattern beats a growing exception list, and manual configuration beats depending on Google’s automatic infinite-space detection.⁵

Why do practitioners keep treating robots.txt as an indexing control?

Practitioners over-trust robots.txt as an indexing control because crawlers actually read it, constantly, which makes the file feel authoritative even though it was never built to remove anything from an index. A file fetched more often than the front page starts to feel load-bearing for everything, including a job it never does.

The read frequency is not a guess. I keep a crawler log for this site, and across it for 2026-07-21 through 2026-07-23, out of 869 total path-level requests, 113 hit /robots.txt alone. That is 13.0% of everything I logged, and the single most-requested path on the site, ahead of the homepage’s 90 requests.

What did checking claude.ai’s robots.txt today turn up, and why doesn’t it match the news report?

On August 1, fetching claude.ai/robots.txt directly shows no explicit /share/* rule at all. The wildcard user-agent group currently disallows /new?, /chat/, /join/, /magic-link, /api/, /onboarding, /upgrade*, /lti/, /settings, and /task*, and nothing else.² On July 27, Search Engine Journal’s check found a /share/* disallow specifically.¹

The Wayback Machine narrows when that changed. The most recent archived capture of the file, taken July 30 at 20:23 UTC, already shows the same wildcard group with no /share/* rule in it.⁸ The rule came out within three days of the Search Engine Journal check, then. What replaced it is still unclear: Anthropic may have folded the share path under the broader /chat/* rule, or addressed the underlying noindex-behind-a-block conflict some other way entirely.

Since I cannot pin down which, the gap confirms the discipline this whole subject argues for instead. Within a single company’s control, a robots.txt file is a live document that can change the same day, and a screenshot from a week-old news story is already stale. Check it yourself before you cite it, and expect it to have moved.

Which directive is right for search pages, facets, cart, account, and staging?

Four e-commerce surfaces get this exact pair of directives wrong most often, and each one needs a different fix: disallow for auto-generated internal search results, allow plus canonical for faceted navigation, a case-by-case split for cart and account, and disallow plus a password wall for staging, never disallow alone where a stray inbound link is possible.

• Internal /search result pages. Disallow the pattern in robots.txt to stop new ones from being crawled and indexed, matching the broad-wildcard-rule guidance above. This is prevention. If spam-abused query pages are already indexed, disallow will not remove them; drop the block long enough for a noindex tag or header to be read, confirm removal in Search Console, then reapply the block.
• Faceted navigation parameter URLs (color, size, sort combinations). Allow the crawl and declare the parent category page canonical. Google’s canonicalization guidance lists “the results of sorting and filtering functions of a category page” as a standard duplicate-content case it expects to consolidate, and treats a stated canonical preference as “a hint, not a rule.”⁶ Unconfirmed by Google for canonical tags specifically, this next point is my own inference: a rel=canonical tag is delivered the same way noindex is, in the page head or an HTTP header, so a robots.txt block on facet URLs would hide the canonical signal from Googlebot exactly as it hid Claude’s noindex tag. Disallowing facets to save crawl budget defeats the consolidation the canonical tag was supposed to achieve.
• /cart and /account. Because session-scoped URLs rarely earn inbound links anywhere Google can see them, disallow is usually safe here. The exception is a platform feature that mints a persistent, shareable link, a “share this cart” tool or a guest-checkout confirmation URL. Once a URL like that gets linked from anywhere public, it is the Claude pattern, and it needs allow plus noindex instead of a bare disallow.
• Staging and preview environments. Disallow the whole environment, and do not stop there. Google’s robots.txt introduction is explicit that the file “is not a mechanism for keeping a web page out of Google” and recommends password protection for this exact case.³ A stray internal link or a canonical left pointing across environments can hand a disallowed staging URL an index entry with no snippet, the same failure running on a different environment.

How do you check whether your own site has this problem right now?

Run a site: search for the URL patterns you expect to be blocked, for example site:yourdomain.com inurl:cart or site:yourdomain.com/search?, and see what Google actually shows for them. Then open Search Console’s Page indexing report and look for the status labeled “Indexed, though blocked by robots.txt.”⁷ A nonzero count there is this failure running on your own domain.

Google’s own description of that status names the mechanism directly: “The page was indexed despite being blocked by your website’s robots.txt file… Google won’t request and crawl the page, but we can still index it, using the information from the page that links to your blocked page.”⁷

A nonzero count under that status is not a rounding error. Read plainly, it means at least one URL you tried to hide with Disallow is already sitting in Google’s index, usually as a bare link with no snippet, and the fix Google names in the same report is to remove the robots.txt block and add noindex instead.