Google Wants a 304. 11 of 13 Retail Homepages Send the Full Page Anyway.

Google updated its crawl-budget documentation on July 22, 2026, and it now tells sites to serve an HTTP 304 Not Modified response whenever a page has not changed since Googlebot’s last visit. The guide frames itself as advice for huge or fast-changing sites, which reads as advice for someone else’s catalog. One of its three qualifying triggers, a large share of pages stuck in “Discovered, currently not indexed,” carries no size floor at all, so a mid-size faceted storefront can trip it nowhere near a million pages. Against 13 reachable US retail homepages, a live test run for this piece found 11 still answer a repeat request with the full page instead of the 304 Google now asks for. Crawl budget resolves into three practical levers: what you let Google fetch, what you let Google index, and how cheaply each fetch resolves. Most sites, in this sample, work only one of the three.

11 of the 13 reachable retail homepages tested for this piece, this site’s own included, answered a repeat request with a full 200 instead of the 304 Google’s crawl-budget guide now recommends.

What did Google actually change in its crawl-budget documentation?

In this cycle, Google’s crawl-budget guide picked up two additions, confirmed by fetching the live page directly rather than trusting a screenshot. It now recommends supporting 304 responses, and for the first time it states that every Google crawler hitting a site draws from one shared capacity limit, so a spike from Googlebot-Image can crowd out Googlebot itself.¹ Search Engine Journal flagged both additions on August 3.²

The page’s own words, fetched the same day this was written: “Use HTTP caching: Support 304 (Not Modified) HTTP status codes. If a page hasn’t changed since Google last crawled it, returning a 304 code tells Google to reuse the cached version, saving your server bandwidth and resources.”¹ On shared capacity, the guide adds that “while each crawler has a different crawl demand, the crawl capacity limit is shared across all crawlers. This means that high demand from one crawler can reduce the capacity available for others.”¹ Its footer stamps a last-updated date of July 22, 2026 UTC. Following redirects, two URLs point at this guidance and both resolve to the same page; this piece cites the one at developers.google.com/crawling/docs/crawl-budget.

Does this guidance actually apply to a mid-size e-commerce catalog?

Google’s crawl-budget guide applies more often than its own framing suggests, because one of its three qualifying triggers carries no page-count floor. Google calls the page “an advanced guide intended primarily” for three kinds of sites: those with 1 million or more pages that change about weekly, sites of 10,000 or more pages that change daily, and any site “with a large portion of their total URLs classified by Search Console as Discovered, currently not indexed.”¹ The first two criteria carry explicit page-count floors that most mid-size retailers never reach. The third does not.

While Google never frames it this way itself, the gap between the headline scoping and the third bullet is my own read of the page. A faceted catalog throwing off duplicate parameter combinations, color, size, sort, can trip the third trigger at almost any scale. Google’s own list of demand factors names this directly: “perceived inventory,” the pile of duplicate or unwanted URLs Google crawls without guidance, is “the factor that you can positively control the most.”¹ A faceted navigation system is the textbook generator of that inventory, at 8,000 pages or 8 million.

What actually happens when you ask a real retail homepage for a conditional response?

Run against 20 origins on August 7, 2026 (19 large US retail homepages plus this site’s own), a conditional-request test split cleanly into three outcomes among the 13 origins that were reachable at all. Two answered correctly with a 304. Those were Macy’s and Walgreens. Four sent a validator, the ETag or Last-Modified header a server hands out so a client can later ask “has this changed since I last looked,” and then ignored it, retransmitting the full page anyway: Costco, Target, Nike, and Publix. The remaining seven sent no validator at all, so a 304 was never possible, not for Walmart, Lowe’s, CVS, Gap, Nordstrom, IKEA, or this site’s own homepage. In total, seven origins could not be tested. Best Buy, Kroger, and REI timed out, and Home Depot, Etsy, Kohl’s, and Wayfair returned a 403 or 429.

A live conditional-request probe of 20 origins run for this piece on August 7, 2026 found these three outcomes among the 13 that were reachable at all.

That distribution sits close to a pre-flight version of this same measurement, run against a partially overlapping list of 12 reachable hosts two weeks earlier. The ten retailers named in both runs reproduced their earlier outcome exactly; Macy’s, Costco, Target, and Nike behaved identically both times. Walgreens, added only for this wider run, was the single new site to honor the validator, nudging the honor rate from 1 of 12 to 2 of 13, sampling composition rather than a real shift.

Costco is the cleanest failure. It sent a stable ETag, the identical value across three requests spaced two seconds apart, then resent the full 3.8-megabyte homepage on the conditional request anyway. Google’s “support 304” line is written for exactly this case: the validator already works, and nothing on the origin is checking it.

Target and Publix are a different failure. On every repeat request, Target changed its ETag, and its byte count moved too, consistent with real per-request personalization rather than a caching bug. Publix’s ETag carries a visible monitoring-tool token, a Dynatrace agent identifier, baked into the value itself, so the field meant to answer “has this page changed” is partly answering a different question: whether a specific monitoring session touched this response. Nike sat in between, matching on two of three requests and drifting on the third, milder than the “changes every single request” description this measurement’s pre-flight version gave it two weeks earlier; retailer edge behavior moves between runs, and that is the honest caveat.

Across the largest group, seven of 13, no ETag or Last-Modified header appears at all, which makes the whole question moot. Nothing in the response gives Google anything to send back, so every request is a full 200 by construction. This site’s own homepage is one of them, confirmed directly on both requests: no ETag, no Last-Modified. Its /robots.txt does not share that problem. Fetched the same day, it returned a weak ETag and answered a matching conditional request with a clean 304 and an empty body. One origin, two different caching paths, and only one of them working.

On a small site, real demand shows why that gap still matters. This site’s own log, recomputed from analytics/ai-crawler-paths.csv for July 21 through August 6, 2026, counts 8,849 logged path requests across 619 distinct paths. The homepage alone took 4,006 of them, 45.3%; /robots.txt took another 629, 7.1%; two paths absorbed 52.4% of everything logged. Whether those 629 robots.txt hits carried a conditional header and got the cheap response, the log cannot say; it records paths and counts, not headers. But the homepage sitting at nearly half of all logged traffic with no way to answer cheaply is the “how cheaply the fetch resolves” lever failing on this site’s single most-requested URL.

Why do a stable-but-ignored validator and an unstable one need different fixes?

A stable validator an origin ignores and a validator that never settles are two different defects, even though both waste the fetch. For Costco’s pattern, a stable ETag the origin never checks, the fix is a server-side change to honor the header it already has, the literal case Google’s “support 304” line addresses. Target’s and Publix’s pattern, a validator that shifts between identical requests, has nothing to honor yet; it needs a deterministic validator before a 304 is possible. The seven-site majority sending no validator needs a caching-layer decision before either fix is relevant. Given that its guidance speaks only to the first problem, Google’s one line assumes the other two are already solved.

Why is Search Console the wrong instrument to watch this change land?

Because the report that would show the fetch-side change and the report that would show whether it mattered run on different clocks. Search Console’s Crawl Stats report bins every crawl response by type, and “Not modified (304)” is one of the listed categories, so a shift toward more 304s is checkable there.³ But the number that tells you whether you needed to care in the first place, your share of URLs marked “Discovered, currently not indexed” in the Page Indexing report, has been rendering in multi-week blocks rather than daily updates since June 11, 2026, three separate stretches of 18, 10, and then 14 days. For its full span, each stretch sat frozen, never moving day to day.⁴

The fix and the instrument that would confirm the fix worked run on different schedules, so watching one gives no fast read on the other. Call that a reconciliation trap. It works a bit like a bank balance that only refreshes weekly while purchases post daily; the spending is real and immediate, but the number meant to confirm its effect lags behind on its own clock. The analogy breaks where a bank differs from Search Console: nobody promises daily refresh here, so there’s no missed service commitment, only a genuinely slower instrument sitting behind a faster one. As it happens, the same dependency runs one layer deeper. Google’s Gary Illyes confirmed on LinkedIn that pushing an unavailable_after date forward on a page only takes effect once Googlebot actually recrawls that specific URL again; extend the date without a recrawl, and Google is still working from the stale one.⁵ Every lever this piece has named, the 304, the canonical, the 404, assumes Google comes back promptly enough to notice the change.

Which lever applies to faceted navigation, internal search, paginated categories, and expired PDPs?

Because each fails crawl budget for a different reason, faceted navigation, internal search, paginated categories, and expired product pages each need a different one of the same three fixes: what you let Google fetch, what you let Google index, or how cheaply the fetch resolves. Treating all four the same is the recurring mistake.

• Faceted navigation URLs (color, size, sort). Tangled with indexing, but mainly “what you let Google fetch.” Allow the crawl and canonicalize to the parent category rather than disallowing; a robots.txt block would hide the canonical signal the way it hides a noindex tag. The combinations still burn Google’s “perceived inventory” budget, the pile of duplicate URLs it crawls without guidance, even once canonicalized correctly.
• Internal /search result pages. The fetch lever, cleanly. Disallow the pattern so auto-generated query pages never enter the crawl queue, the surface the site’s prior disallow-versus-noindex teardown already covers in full.
• Paginated category pages. The “how cheaply the fetch resolves” lever. These get recrawled often and rarely change, so a working 304 is the entire fix; blocking them wastes real, indexable content.
• Expired or out-of-stock PDPs. The “what you let Google index” lever. Google’s own guidance: “Return a 404 or 410 status code for permanently removed pages,” and separately, eliminate soft-404s, pages that stay live with thin availability copy instead of a real status code.¹ An expired PDP indexed with “currently unavailable” text is that soft-404 pattern.

In audit work across several e-commerce catalogs, the shape behind a large “Discovered, currently not indexed” share tends to repeat, a practitioner observation rather than a measurement: a faceted navigation system generating far more crawlable URL combinations than there are pages actually worth indexing, most differing from a parent category page only by sort order or a single-result color filter.

How do you check whether your own site has this problem right now?

Run two curl requests. First, capture whatever validator your homepage sends:

curl -sI https://example.com/ | grep -i etag

Then send that exact value back as a conditional request:

curl -s -o /dev/null -w "%{http_code}\n" -H 'If-None-Match: "paste-the-etag-value-here"' https://example.com/

A working origin prints 304. If step one returned nothing, there is no validator to send, the failure this piece found on seven of 13 homepages. If step two prints 200 despite a real ETag, compare response sizes; an identical body means the origin ignored a validator it already had, Costco’s pattern, while a changed body means the page genuinely differs. On the Google side, check Crawl Stats for your own share of 304s, and check the Page Indexing report for your count under “Discovered, currently not indexed.” If that count is large and a bot-challenge misconfiguration isn’t the cause, the mechanics above are where the fix lives.