In July, Cloudflare split its single AI-bot switch into three separately managed categories: Search, Agent, and Training. From September 15, domains newly onboarding get a default that blocks Training and Agent bots on pages that display ads while Search stays allowed. Here is what each category covers, who the new default actually touches, and the dashboard audit that takes half an hour.
Cloudflare split AI traffic into Search, Agent and Training in July 2026. Akamai sorts the same traffic into training crawlers, search crawlers and fetchers. Vercel's AI bots managed ruleset covers training, search and user-generated fetches. Three vendors, nearly identical splits, different names and different defaults. A selection guide with a comparison table, an honest read on robots.txt, and a recommended setup for three kinds of store.
Cloudflare blocks AI crawlers by default now and added Pay Per Crawl billing. Many stores never noticed GPTBot, ClaudeBot, and PerplexityBot got shut out, which quietly erases them from AI answers. Here's how to audit, which bots to allow, and the difference between training and retrieval crawlers.