Anti-Bot

Search, Agent, or Training? Cloudflare's New Bot Rules and Your Scraper

Cloudflare announced that Training and Agent bots would be blocked by default on ad-supported pages from September 15 — then quietly revised it six weeks later. Here is what the rule actually says now, and the 20-minute audit worth running on your pipeline.

Your scraper now has a purpose, and the network is checking it. Not your IP. Not your user agent. What you intend to do with the page after you read it.

That is the real change of 2026, and it is easy to miss underneath a season of blog posts announcing that Cloudflare "blocked AI crawlers on September 15." Cloudflare did something narrower and more consequential. It shipped a vocabulary — Search, Agent, Training — pushed it into robots.txt, wired it to cryptographic identity, and started making it the default for new domains.

It also quietly rewrote the September 15 rule six weeks before it landed, which almost nobody has noticed. We'll start there, because if you're reading a checklist built on the original announcement, you're planning against a policy that no longer exists.

What actually changed on September 15 (and what got revised)

On July 1, 2026, Cloudflare announced the coming defaults in plain terms: "For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default."

That sentence is the one that got quoted into a hundred posts. It is also no longer the policy.

On August 21, with the Bot Preference Sync announcement, Cloudflare described something meaningfully softer. Publishers self-identify at onboarding — "At the time of onboarding, such customers can select the option, 'I monetize from pages with ads on this domain', which will set Training to Disallow as the default." And for everyone else: "For the non-publisher case, new customers will not have any blocks or disallows added by default when they onboard a domain: the choice is up to the customer."

Three differences, and they all cut the same way:

Training only, not Training and Agent. The Agent category — the one most real products live in — is not in the revised default at all.
Opt-in, not automatic. It follows a checkbox the site owner ticks about their own business model, not a blanket rule applied to every new domain.
Nothing by default for non-publishers. If a new domain doesn't monetize with ads, no blocks are added.

Add the qualifier that survived both versions — new domains only, existing customers keep whatever they have — and the practical impact on your pipeline this week rounds to zero.

So if you were told your scrapers would break on the 15th, they didn't, and it's worth noticing who told you. The gap between the July announcement and the August revision is a decent filter for which sources actually read primary documentation.

The thing worth paying attention to was never the switch. It's the vocabulary the switch is written in — because that vocabulary is now the organizing principle for bot policy on one of the largest networks on the internet, and vocabularies outlive the settings built on top of them.

The three categories, and why the middle one is the problem

Cloudflare defines them like this:

Search — "any behavior that collects or indexes your content, so it can answer questions about it later."
Agent — "automated behavior that is acting, usually in real time, on a person's behalf, to get something done right now."
Training — "a crawler taking your content to train or fine-tune a model."

Training is the one everybody argues about and the one fewest developers are actually doing. Search is broadly tolerated; it has been the deal since 1996. You send traffic, you get to index.

Agent is where the ambiguity lives, and Agent is where most real work now sits.

Think about what you're building. A user asks your assistant to pull the last four quarterly filings for a company. A sales tool enriches a lead by fetching a company's about page. A research workflow reads thirty sources and returns a synthesis. None of that is training a model. None of it is a search index. It is software acting, in real time, on a person's behalf — which is the textbook definition of the category that now gets blocked by default on ad-supported pages.

The uncomfortable implication: the more your product looks like a useful agent, the more squarely it lands in the category with the least established norms and the most restrictive default.

And the taxonomy is broader than three. Cloudflare's verified-bot documentation classifies behavior across eleven categories — Search, Agent, Training, Transact, Data Collection, Security Testing, SEO, Ads Verification, Social/Link Preview, Feed Fetching, and Monitoring & Operations — crossed with an operational model of Direct ("operated by a single, narrow operator — usually on the operator's own infrastructure") versus Intermediary ("an agentic service that a wide range of end users can operate").

That second axis matters more than it looks. If you sell a tool that other people point at targets you don't choose, you are an Intermediary, and you inherit the behavior of your worst customer.

Your user agent is no longer your identity

For twenty years, declaring who you were meant setting a string. Anyone could set any string. The whole system ran on good manners.

Web Bot Auth replaces the manners with math.

The mechanism is HTTP Message Signatures over Ed25519. You generate a keypair, publish the public key as a JWK, and host a directory at /.well-known/http-message-signatures-directory served with Content-Type: application/http-message-signatures-directory+json. The thumbprint of that key becomes your identifier. Then you sign your requests:

Signature-Agent: "https://signature-agent.test"
Signature-Input: sig2=("@authority" "signature-agent")
 ;created=1735689600
 ;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U"
 ;alg="ed25519"
 ;expires=1735693200
 ;tag="web-bot-auth"
Signature: sig2=:jdq0SqOwHdyHr9+r5jw3iYZH6aNGKijYp/EstF4RQTQdi5N5YYKrD+mCT1HA==:

Three headers. Signature-Input declares what you signed and carries the metadata — note created and expires, which should span a short window so a captured signature can't be replayed an hour later. Signature is the signature itself. Signature-Agent points at your key directory so the verifier can fetch your public key and check the math.

You can test against Cloudflare's endpoint at https://crawltest.com/cdn-cgi/web-bot-auth — HTTP 200 if your signature verifies, 401 if the key is unknown, 400 if the message is malformed. That third code is the one you'll actually see first.

To become a Verified bot, Cloudflare asks two things. First, that you "declare who it is deterministically, through a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS." Second, that the bot "obey robots.txt and crawl directives, maintains reasonable request rates, and has not been observed evading website owner preferences."

Read the second one as what it is: a permanent record. Verification is not a certificate you earn once — it is a reputation you spend. Among the documented breaches of policy is the case where "the disclosed purpose of the service does not reflect on the traffic." In other words, saying one thing and doing another is itself the violation, independent of whether the thing you did was harmful.

Which brings us to the design decision this whole regime forces.

One crawler, one purpose

Most data pipelines grew the same way. One fetching layer, shared by everything, because that was the sane engineering choice. The retrieval path for your agent, the nightly enrichment job, the archive crawl — same client, same pool, same identity.

Under a purpose-based regime, that shared client is now the single riskiest component you own. It cannot declare its purpose honestly, because it has several. And a bot whose observed behavior doesn't match its declared purpose is precisely the failure mode the verification system is built to catch.

So: split the crawler by purpose, not by convenience.

Real-time fetches on behalf of a user → Agent. Separate client, separate identity, separate key.
Scheduled bulk collection into your own store → Data Collection or Training, depending on what the store feeds. Be honest about which.
Index building for later question-answering → Search.

This is more infrastructure than you had last week. It is also the only configuration that survives the next two years, and it has a side benefit worth naming: once each path has its own identity, rate limits, and declared purpose, you can finally tell which one is getting you blocked.

robots.txt grew a second language

The Content Signals Policy adds a directive that expresses permission by use rather than permission by path:

User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /

Three signals, plus an extension:

search — building a search index and providing search results
ai-input — feeding content into models to answer in real time
ai-train — training or fine-tuning models
use — immediate (interact, store nothing), reference (index, excerpt, link back), or full (summarize and reproduce)

When Cloudflare's managed robots.txt is on, the default is search=yes, ai-train=no, use=reference. And Bot Preference Sync — announced August 21 for all plan tiers "from Free tier to Enterprise," and on by default for new customers — generates that file automatically from a site's dashboard settings. Which means these directives are set to appear on a very large number of sites whose owners have never hand-edited a robots.txt in their lives.

Two things follow.

First, Allow: / and Content-signal now say different things, and both are binding on anyone claiming good faith. A page can be fully crawlable and still carry ai-train=no. If your parser reads the path directives and ignores the signals, you are non-compliant while believing you're clean.

Second, absence is not permission and it is not refusal. Cloudflare is explicit: "If the website operator does not include a content signal for a corresponding use, the website operator neither grants nor restricts permission via content signal with respect to the corresponding use." Silence stays silent. Don't read it as a yes, and don't read it as a no.

The practical instruction: parse Content-signal in your robots handler, store it alongside every record you collect, and make it queryable. The day a customer or a lawyer asks what permissions applied to a given row, you want that to be a database query and not an archaeology project.

The honest version of what this means

Let me argue against the urgency of my own headline for a moment, because you're going to read a lot of posts this month that won't.

Today, this changes almost nothing for an existing pipeline. The defaults touch new domains, they were narrowed to Training only, and they now depend on a publisher ticking a box about ad revenue. Your current targets are unaffected unless their owners go and change settings. Anyone selling you an emergency is selling you something.

Enforcement is uneven and will stay that way. Content Signals is a stated preference, not a technical control. It has no more teeth than robots.txt ever did, which is to say: it binds the compliant and is invisible to everyone else. Whether it means anything in a courtroom is untested.

The taxonomy is genuinely blurry at the edges. A retrieval-augmented system that caches a page and serves it to a thousand users — Agent, or Search? Cloudflare's definitions don't cleanly resolve it. You will have to make a defensible call and document your reasoning.

So why act now?

Because defaults are how the web actually changes. Not through bans — through the settings nobody touches. Every domain onboarded from September 15 carries this vocabulary, and the population of sites speaking it grows monotonically from here. The cost of building purpose-separated, signed, signal-aware collection is low today and rises sharply the day it stops being optional. That asymmetry is the entire argument.

And there is an upside that gets underplayed. A legible, verified, honestly-declared crawler is a crawler that can be allowed — explicitly, by name, in a firewall rule. Cloudflare exposes cf.bot_management.signed_agent as a targetable field, though the framing is already shifting: as of July 1, 2026, "the distinction between a Verified bot and a signed agent is expressed by a new metadata field tracked in BotBase: Direct versus Intermediary." Expect the plumbing to keep moving. The direction won't. In a web where unidentified automation is increasingly treated as hostile, being identifiable stops being a cost and starts being an advantage.

The 20-minute audit

Run this before you change any infrastructure.

1. Inventory by purpose, not by script. List every automated fetch your organization makes. For each: what triggers it, where the data lands, and what it ultimately feeds. Assign exactly one Cloudflare category. Anything you can't assign to one is the thing to fix first.

2. Find out who's behind Cloudflare. Check response headers on your target list for server: cloudflare and the cf-ray header. Sort your targets into behind-Cloudflare and not.

3. Cross-reference against ad inventory. The default only bites on ad-supported pages. Among your Cloudflare-fronted targets, which actually serve ads? For most B2B and public-data work, the honest answer is very few — and that should calm you down considerably.

4. Read the robots.txt you've been ignoring. Pull /robots.txt for every target and grep for Content-signal. Today you'll find it on a handful. Re-run it quarterly and watch the number move.

5. Decide your declared identity. Direct or Intermediary? One identity or several? If you operate infrastructure other people point at targets you don't choose, you are an Intermediary — plan for the accountability that comes with it.

Five steps, one afternoon, and you will know more about your exposure than most teams shipping agents today.

What this means if you're building agents

There's a specific trap here for anyone wiring web access into an LLM.

The instinct is to hand the model a generic fetch tool and let it roam. It's the fastest path to a demo. It is also, under this regime, an automated client with no declared purpose, no signature, and behavior that varies with whatever a user types — the exact profile that fails verification and gets classified as hostile.

The alternative is to put a named, signed, purpose-scoped retrieval service between your model and the web. The agent asks for data. The service decides how to get it, declares what it is, honors the signals, and returns structured results. Your model never touches a socket.

That boundary was always good architecture. It is now also the compliance surface, which is a rare case of the clean design and the defensible one being the same design.

Where ScrapeUp fits

This is the layer we build. ScrapeUp is a retrieval service you call with a URL and a description of what you want, and it returns structured data — handling the fetching, the rendering, and the extraction so your application doesn't hold that responsibility itself.

For the problem in this post, that separation is the point: one place where identity, rate discipline, and signal handling live, rather than a behavior scattered across every script in your organization. If you're running document-to-data work, research pipelines, or enrichment behind an agent, consolidating the fetch path is worth doing on architectural grounds alone — the regulatory tailwind is a bonus.

We are not going to tell you this solves compliance. Nobody's product does. The category you declare, the signals you honor, and the record you keep are decisions that stay yours.

Get started

The free tier includes 25,000 credits, no card required — enough to route a real pipeline through it and see what the extraction looks like on your own targets.

Start with 25,000 free credits

If you're working through purpose separation on a larger pipeline and want to talk through the architecture, reach us at sales@scrapeup.com.

Frequently asked questions

Did Cloudflare block all AI crawlers on September 15, 2026?

No. The July 1 announcement described Training and Agent being blocked by default on ad-supported pages for new domains. Cloudflare revised this on August 21: publishers who self-identify as monetizing with ads get Training set to Disallow, and non-publisher new domains get no default blocks at all. Existing customers keep their current settings.

Does this affect my existing scrapers?

Not directly. Every version of the default applies only to domains newly onboarding to Cloudflare. Sites you already collect from are unchanged unless their owners actively change settings. The reason to act now is the direction of travel, not an immediate break.

What is the difference between Search, Agent, and Training bots?

Search collects or indexes content to answer questions about it later. Agent acts in real time on a person's behalf. Training crawls content to train or fine-tune a model. Cloudflare's full classification extends to eleven behavior categories, including Data Collection and Transact.

Do I need Web Bot Auth to scrape the web?

Not today. It is the mechanism for becoming a Verified bot, which matters if you operate a named service at scale or want site owners to be able to allow you explicitly. For small internal pipelines it is optional — for now.

Is Content-signal in robots.txt legally binding?

Untested. It expresses a site owner's stated preference by use rather than by path. Treat it as a compliance signal you should honor and log, not as a technical control — and consult counsel for your specific situation.

What should I do if my scraper serves several purposes?

Split it. A client that can't declare one honest purpose is the highest-risk component in a purpose-based regime, and divergence between declared and observed behavior is documented grounds for losing verified status.