Cloudflare Changed Its Defaults on September 15: The AI Agents Shopping for Your Customers May Be Locked Out
One Switch Became Three in July
Cloudflare used to give you a single control for AI bots. Flip it, and everything with AI in the name got turned away together. That changed in July, when the control became three separately managed categories: Search, Agent, and Training. Each one has its own state, and you can block one without touching the other two.
Cloudflare’s own definitions, word for word. Search covers “crawlers that index your content so they can answer questions about it later.” Agent covers “automated activity acting in real time on a person’s behalf, such as chat fetch bots.” Training covers “crawlers that take your content to train or fine-tune a model.”
The new piece compared with the old training-versus-retrieval framing is Agent. Search and Agent sound like the same thing until you look at timing. Search reads your page ahead of any question and files it away. Agent fires because a person is talking to an assistant right now and the assistant is opening your URL as part of that conversation. Block Search and you drift out of the index over weeks. Block Agent and the shopper in front of the assistant loses your product page in that exact session.
| Category | Cloudflare’s definition | What the shopper is doing | September 15 default |
|---|---|---|---|
| Search | Crawlers that index your content so they can answer questions about it later | Nothing yet, the index is being built ahead of time | Allowed |
| Agent | Automated activity acting in real time on a person’s behalf, such as chat fetch bots | Mid-conversation with an assistant that is opening your page | Blocked on pages that display ads |
| Training | Crawlers that take your content to train or fine-tune a model | Nothing, the content goes into model weights | Blocked on pages that display ads |
Cloudflare also states that multi-purpose crawlers combining Search and Training are affected by the new defaults to block Training. Cloudflare stops there and names no specific crawler, so neither should anyone reading this.
The upside of splitting the control is that “keep my content out of training runs, but let a shopper’s assistant read anything it wants” is now a configuration you can actually express. The old single switch forced an all-or-nothing call, and plenty of brands allowed training they did not want simply to protect their visibility in AI answers. Those two decisions are now independent.
Who the September 15 Default Actually Touches
Two qualifiers travel with that default, and dropping either one changes what it means.
The first is newly onboarding. Domains that come onto Cloudflare from September 15 get the new default. Existing customers keep their current settings unless they opt in, and they could opt out at any time before September 15.
The second is pages that display ads. This is not a sitewide block. On ad-bearing pages, bots classified as Training or Agent are blocked, and Search stays allowed.
For a DTC brand, that narrows the risk to a few real situations. You changed hosting, moved DNS, or an agency re-onboarded your domain, which counts as new. Someone on your team accepted the new defaults in the dashboard. Or your blog, buying guides, and comparison posts carry ad slots or affiliate widgets while your product pages do not, which produces a store that is half open and half closed.
If none of those describe you, the setting still deserves five minutes. The July split and the September 15 default both landed inside two months, and what the dashboard toggle reports does not always match what the edge enforces.
Agent Traffic Sits Closest to Revenue
A shopper types “check whether this dress still comes in medium and what it costs” into ChatGPT, pastes your link, and the assistant opens the page to read price and stock. That fetch lands in the Agent category. Blocked, the shopper gets back some version of “I cannot open that page,” and the assistant moves to whoever it can read.
The failure mode is different from a Search block. Losing Search is slow. Mentions taper off, and you notice a quarter later. Losing Agent is immediate and invisible at the same time, because the lost sale never produces a pageview, a session, or a line in any analytics property you own. Conversion rate on the traffic you can see holds steady while the number of shoppers who ever reach you falls.
Which bots Cloudflare places in Agent is Cloudflare’s classification, not something you can read off a User-Agent string. Plenty of crawlers do not declare their purpose in the header at all. The only reliable way to know what sits in that bucket is to open the category in the dashboard and read the list Cloudflare shows there.
Agent traffic is also low volume by nature. One shopper in one conversation might generate three or four requests against your origin, which vanishes into ordinary traffic on any dashboard chart. No monitoring you already run will flag it, so you have to go and read the setting yourself and then test it against a live assistant.
The Audit: Where to Look and What to Check
Confirm the domain is behind Cloudflare and open AI Crawl Control. The deep link is the /:account/:zone/ai path on dash.cloudflare.com, which is faster than walking the menu tree. The feature is available on every Cloudflare plan, free included.
Read the state of all three switches and pay attention to Agent. If your domain onboarded after September 15, the new default is already live and the Agent state is probably not what you would have chosen.
Write down every bot name Cloudflare lists under the Agent category. That list is the authoritative answer for your configuration decisions. Do not build allow rules from assumptions about which bot belongs where.
Cross-check your WAF and firewall rules. AI Crawl Control is one layer; custom rules written by you or a previous contractor are another, and the two can disagree. A dashboard reading “allowed” while a custom rule drops the request at the edge is a common outcome, and reading the toggle alone will never surface it.
Inventory the pages carrying ad slots or affiliate components. The new default applies only there, so a clean product template tells you nothing about what happens on your highest-traffic blog post.
Testing Whether a Real Shopper’s Assistant Can Reach Your Product Page
Use the actual entry point. Paste a product URL into ChatGPT and into Perplexity, and ask each one for the price, the available sizes, or the stock status. If it returns the specific numbers printed on that page, Agent-class fetching is getting through. If it reports that it cannot open the link or retrieve the content, something is stopping it.
Spoofing a User-Agent with curl is a weak test here. Many bots operate from verified IP ranges, so a forged header sent from an ordinary machine can get a 403 because the source address does not match, not because the category is blocked. That produces a false alarm and sends you chasing a setting that was fine.
Server logs remain the primary evidence. Take the bot names you copied out of the Agent category, pull their recent requests, and check whether the status codes are 200 or 403. Logs report what actually reached your origin, which a toggle in a dashboard does not.
Run the test on an ad-bearing page and on a page without ads. The default applies only to the first kind, so testing product pages alone hides half the picture. Repeat a few days after any configuration change, since recrawling and reinclusion in AI answers both take time.
Related Articles
Google Shut Down the Content API: Migrate to Merchant API or Your Feed Starts Erroring in September
Content API for Shopping was sunset on August 18, 2026, and requests began experiencing progressive errors on September 1. Merchant API takes over product uploads, inventory, account management, and the Google Ads inventory link. Here is how to confirm in half an hour which API your stack calls, what goes dark when the feed stops, and how the extension request form works as an emergency lever.
Amazon Synthetic Performer Image Labeling: XMP Metadata Compliance Guide for Sellers
Starting July 22 2026, Amazon requires all product images and A+ Content featuring photorealistic AI-generated people to carry contains-synthetic-performer XMP metadata. This guide covers what needs tagging, how to batch-add metadata with ExifTool and Adobe Bridge, and what happens if you skip it.
AI Traffic Control Compared 2026: How Cloudflare, Akamai and Vercel Classify Bots Differently
Cloudflare split AI traffic into Search, Agent and Training in July 2026. Akamai sorts the same traffic into training crawlers, search crawlers and fetchers. Vercel's AI bots managed ruleset covers training, search and user-generated fetches. Three vendors, nearly identical splits, different names and different defaults. A selection guide with a comparison table, an honest read on robots.txt, and a recommended setup for three kinds of store.