Confirmed
Google: Robots.txt Governs Crawling, Not What AI Does With Scraped Content
Google's Search Central YouTube series posted its second episode on October 21, in which Gary Illyes and Cherry Prommawin explained that robots.txt, per RFC 9309, controls which URLs a crawler may request, not what a system does with the content afterward. Blocking one AI crawler while allowing others requires targeting that crawler's specific user-agent token, and non-English or emoji paths should be percent-encoded for crawlers other than Google's.
In plain termsRobots.txt is the file that tells automated crawlers which pages they're allowed to fetch; it has no power over what happens to the words on a page once a crawler has already read them.
Why it mattersA disallow rule that keeps a crawler out of your logs says nothing about whether that crawler's owner trains on content it already has, so an AI-training opt-out needs a licensing or content agreement, not a robots.txt line.
Our takeI argued in robots-disallow-is-not-noindex that robots.txt only ever controls the crawl, never what happens downstream, and Illyes is stating the AI-training version of the same mechanism. Expect this distinction to keep resurfacing as sites try to use robots.txt as a blunt instrument against AI training and find it does nothing of the sort.
Confirmed
Google Tells Sites To Serve 503s, Not 404s, During PHP2Shell Emergency Patching
John Mueller posted on Bluesky on October 21 that sites taking PHP installations offline to patch the PHP2Shell remote-code-execution flaws should serve an HTTP 503 rather than let requests time out or return 404 or 500, and that the 503 can be served outside PHP itself through DNS or a CDN. Google's guidance separately warns that robots.txt itself must keep returning a normal response rather than a 503, since a robots.txt that goes dark for an extended period is what actually puts indexed URLs at risk.
In plain termsA 503 status code tells a visiting crawler 'come back later', while a 404 tells it 'this page is gone'; sending the wrong one during a short outage can get pages removed from search results instead of just delayed.
Why it mattersServing the wrong status code during an emergency patch is the difference between an outage search engines treat as temporary and one that starts dropping pages from the index, so the 503 instruction is worth having ready before the next forced patch window.
Our takeThe emergency timing here matters more than the advice itself, since most teams only remember the correct status code mid-incident, after pages have already started timing out. Worth turning into a documented runbook step rather than a one-off reminder to rediscover during the next forced patch.
Observed
Google Tests Local Knowledge Panel Layout Reorder, Moving Photos Below Name And Actions
Search consultant Ben Fisher spotted a Google test reorganizing the local Knowledge Panel on desktop, moving the business name, rating, description and action buttons above the photo gallery and tightening the layout to match Google's existing mobile design. Search Engine Roundtable reported the change on October 22, with no Google confirmation of its scope or permanence.
In plain termsThe Knowledge Panel is the information box that appears next to a business in Google Search and Maps, showing its name, hours, reviews and photos.
Why it mattersA desktop layout test that pushes photos below contact and action details could shift where searchers click to call or get directions, so local-heavy clients should watch panel click patterns rather than assume engagement drops mean weaker demand.
Our takeThree Ben Fisher sightings on this panel in about five weeks, an AI Overview insert, a Posts reorder, and now this, reads less like isolated tests and more like Google actively iterating on the panel's information hierarchy. Treat click and call-through data from this period as unstable until one layout sticks.
Model & AI releases
AI model launches, model API price changes and major assistant features, kept separate from the search news above. Every model story Pulse has covered is collected under LLM Models & Agents, and the monthly model comparison shows how they stack up.
Confirmed
IBM Releases Granite Speech 5.0, A Faster Open-Weight Speech Model Family
IBM published Granite Speech 5.0 on Hugging Face on October 21, a three-model update for transcription, translation and speaker attribution. The new granite-speech-5.0-2b succeeds the 4.1 2B model, and IBM reports it ranks second among open-source ASR models at a 5.06% average English word-error rate, while the updated Turbo CTC variant runs roughly 1.6x faster at 19,870 RTFx. All models ship under Apache 2.0.
In plain termsWord error rate is the percentage of words a speech-to-text model gets wrong when transcribing audio; a lower number means more accurate transcripts.
Why it mattersA faster, cheaper open-weight transcription model gives teams building call-center or video-indexing pipelines a lower-cost option to benchmark against whatever they're running today.
Our takeA second-place open-weight ranking at a meaningfully lower latency is the more interesting number than the ranking itself, since most production transcription pipelines are speed-constrained before they're accuracy-constrained. Worth a bake-off against whatever turbo-tier ASR model is already in production.