Cloudflare’s AI Traffic Taxonomy: Search, Agent, and Training — What Every Site Owner Needs to Know

On July 1, 2026, Cloudflare retired its single “block AI bots” toggle and replaced it with something that looks a lot like a traffic constitution: three granular categories — Search, Agent, and Training — each with its own controls, its own default settings, and its own business implications.

The change is more than a product update. It is the moment the industry arrived at a shared vocabulary for AI traffic. If you are building a bot policy, operating an e-commerce site, or trying to understand what AI systems are actually doing when they arrive at your pages, the Search / Agent / Training frame is the one worth learning.

Here is what each category means, what changed on the dates that matter, and what any site owner needs to do in response.

The Three Categories, Defined

Cloudflare’s July 1 update introduced three distinct traffic classifications for automated AI systems:

Search. Crawlers that index web content and return referral traffic to the indexed sites. This is the historical crawler category: GoogleBot, BingBot, and the newer AI-native search crawlers — GPTBot (when used for search indexing), PerplexityBot, ClaudeBot — that power AI search results. You generally want Search crawlers to reach your site, because they are the mechanism through which you appear in AI-generated answers and traditional search results alike. Blocking them means opting out of discovery.

Agent. Real-time automated sessions acting on behalf of a specific user. A shopping agent comparing prices across twenty sites on your customer’s behalf. A research agent summarizing your product documentation for a purchasing decision. A workflow automation executing a recurring task with delegated access. Agent traffic is the fastest-growing category and the least understood: it converts differently from human traffic, it responds to different signals, and it carries a completely different risk profile from Search traffic.

Training. Crawlers collecting content to train or fine-tune AI models. These visit pages to harvest text, structure, and data — and return nothing. No referral traffic. No purchases. No conversions. The policy question for Training crawlers is not about business risk in the usual sense; it is about whether you want your content used to train AI models without compensation, and on what terms.

These three categories were always conceptually distinct. Cloudflare’s update is significant because it forced a choice: site owners must now actively select a policy per category instead of defaulting to all-or-nothing. That means understanding what each category actually does to your site.

What Changed on July 1 — and What Changes on September 15

Before July 1, Cloudflare’s AI bot controls were binary. You could block “AI crawlers” as a class or allow them. That control was blunt enough to be nearly useless: blocking all AI bots also blocked Search crawlers, which meant opting out of AI-native search discovery. Most site owners left the toggle off and had no meaningful policy at all.

The July 1 update replaced that toggle with per-category controls. Site owners can now allow Search, block Training, and make a deliberate decision about Agent traffic — or the reverse. Cloudflare also launched BotBase, a searchable directory of every bot the platform tracks, available to Enterprise Bot Management customers. For the first time, policy decisions can be made against named, identified bots rather than blunt categories.

The September 15 date matters specifically for new domains. Any site onboarding after September 15, 2026 will have Agent and Training crawlers blocked by default on ad-monetized pages. The logic is that ad-monetized pages depend on human attention for revenue, and Agent and Training crawlers do not generate that attention. The default is conservative — you can override it — but it signals where platform defaults are heading for all sites over time.

Why the Distinction Matters for Your Business

The three categories have completely different business implications, and treating them as a single mass produces the wrong policy in all three cases.

Search crawlers should, in almost every case, be allowed unrestricted access to your published content. The entire premise of Generative Engine Optimization is that AI-native search engines need to read and index your content to cite it in AI-generated answers. Blocking Search crawlers does not protect your content — it removes your site from AI-generated discovery. An llms.txt implementation and clean structured data are designed specifically to make your content more useful to Search crawlers, not less accessible.

Agent traffic is where the policy question is genuinely complex. QAIL’s analysis of 30M+ website visits found that AI bot traffic now represents 38–52% of all web traffic. A meaningful fraction of that is Agent traffic — real-time sessions executing user-delegated tasks. Some of that agent traffic converts to revenue: a shopping agent that finds what it needs and completes a purchase is a customer. Some does not: a research agent scraping your pricing page to feed a competitor’s catalog is a liability. A blanket policy in either direction gets the economics wrong. What you need is the ability to distinguish a verified, authorized shopping agent from an unidentified scraper — and respond to each appropriately.

Training crawlers are the most straightforward policy case for most sites: they take without giving. Blocking them does not affect your search presence or your agent-driven conversions. Whether to allow them is a content licensing question, not a traffic optimization question.

The Monitoring Gap That Makes Any Policy Fragile

Here is the problem with a three-category policy: it is only as good as your ability to correctly classify traffic into those categories.

A 2026 study by Iden found that only 47.1% of AI agents operating on enterprise sites are actively monitored or secured. The other 52.9% operate without oversight, logging, or identity controls. That means more than half of the AI agent traffic reaching most sites is effectively invisible — not classified, not logged, not subject to any policy at all.

The classification problem is hard because the signals most sites rely on are unreliable. User-agent strings are self-reported and trivially spoofed. IP reputation databases lag behind the actual population of AI agents by weeks or months. Behavioral signals — request timing, access patterns, session depth — can distinguish human from automated traffic in bulk, but not between a legitimate shopping agent and a malicious scraper traveling at the same speed.

The IETF is working on a structural solution: Web Bot Auth, a draft standard applying HTTP Message Signatures (RFC 9421) to automated traffic. Under Web Bot Auth, an automated agent signs each outbound request with an Ed25519 keypair and publishes its public key at a well-known directory endpoint. Any server can then verify the signature against the published key — proving not just that the agent is what it claims to be, but that the specific request was issued by that agent and has not been tampered with in transit.

Web Bot Auth is still a draft. But it represents the direction the industry is moving: away from heuristic classification based on unreliable signals, toward cryptographic identity that can be verified at the edge. The Know Your Agent framework anticipates exactly this shift — the question of who the agent is, who authorized it, and what it is permitted to do.

What User-Agent Strings Cannot Tell You

The three-category taxonomy is a policy framework. Enforcement requires classification, and classification today still depends heavily on user-agent strings — despite their well-documented unreliability.

Legitimate agents generally do declare themselves honestly. GPTBot uses GPTBot/1.0; ClaudeBot uses ClaudeBot/0.1; PerplexityBot declares PerplexityBot/1.0. Identifying these crawlers from their user-agent strings is feasible and well-documented.

The problem is that malicious or misclassified agents do not declare themselves honestly. An Agent-category scraper that wants to bypass your policy will set a Chrome user-agent and operate at human-speed timing. A Training crawler that has been blocked will cycle through residential proxy IPs and behave like a browsing user for long enough to collect what it came for.

Effective enforcement of a three-category policy requires combining user-agent identification with behavioral analysis, IP intelligence, and — where available — cryptographic verification. It also requires a tiered access control framework that can act on that classification in real time: serving different responses, applying different rate limits, or challenging unverified agents before they reach your content.

What to Do Now

The three-category framework is the right mental model regardless of whether you are on Cloudflare. Here is what to act on immediately:

Audit your current settings. If you are on Cloudflare, check your AI crawler controls and make deliberate choices per category. Do not leave Search crawlers blocked, do not leave Agent crawlers unmonitored, and make a considered decision about Training. If you are not on Cloudflare, translate the same three-way logic into your CDN’s or WAF’s controls — the categories are platform-agnostic even if the toggles are not.

Log agent sessions separately from human sessions. Your analytics stack almost certainly reports human and agent traffic into the same funnel, producing meaningless averages. Session duration, bounce rate, page depth, and conversion rate all mean different things for agent sessions than for human ones. Separating the two gives you actual signal instead of noise.

Start building toward cryptographic verification. Web Bot Auth is a draft, not a standard yet. But the major agent platforms — Visa’s Trusted Agent Protocol, Google’s Agent Pay Protocol, and the FIDO Agentic Authentication working group — are all moving toward signed agent identities. When cryptographic verification is available for the agents that matter most to your revenue, you will want to be ready to use it.

Write an agent policy page, not just a robots.txt. robots.txt communicates crawl permissions. An agent policy communicates what authorized agents can do on your site — what they can buy, what they can access, what they can store and transmit. As agent traffic becomes a commercial channel, the absence of an explicit policy is a gap.

The Taxonomy Is Infrastructure, Not a Setting

Cloudflare’s July 1 update will be remembered as the moment three-category AI traffic classification became the industry default. But the update itself is just a toggle panel. What it represents is the recognition that Search, Agent, and Training traffic are fundamentally different things that require fundamentally different policies — and that any site running a single undifferentiated policy is flying blind on at least two of the three.

The deeper work is building the infrastructure to classify traffic accurately, enforce policy in real time, and log what agents are doing at a level of granularity that is actually useful. That work is ongoing for most sites. The Cloudflare update made it urgent.

Start with your Agent Readiness Score to see how your site currently performs across the dimensions that matter for AI traffic — and where the gaps are. Run the free assessment at QAIL and get a baseline in thirty seconds.