Web Bot Auth is here: what happens to agents that do not sign
On 15 September a default changes, and a lot of automation that works today will stop working.
Cloudflare is splitting bot traffic by purpose rather than by name. From that date, crawlers that mix purposes get blocked by default on any page that carries advertising. The change lands on new Cloudflare customers, on new sites added by existing customers, and on every existing free customer. That last group is the one people are missing. If you assumed this only affects sites signing up after the deadline, it does not.
Alongside that, Cloudflare has introduced Web Bot Auth and a new class called signed agents. It is worth understanding the two together, because they are the same decision viewed from opposite ends.
What Web Bot Auth actually is
An agent signs its requests. Not a shared secret, not an API key in a header, but a cryptographic signature over the HTTP message itself, using a private key the agent holds. The site verifies the signature against a public key the agent publishes. If it checks out, the site knows which agent is calling, with a confidence a user agent string has never offered.
Cloudflare now sorts automated traffic into a few groups. Verified bots is the old category for transparent crawlers run by a single company, the search engines being the obvious case. Signed agents is the new one, and the wording matters: it covers agents directed by an end user rather than by a single company, running through remote browsing infrastructure. ChatGPT agent, Goose, Browserbase and Anchor Browser sit in that bucket.
So there are now two ways for a site to decide about your traffic.
The two buckets
If you are signed, you are judged on identity. The site knows who you are before it looks at anything you did. Your reputation travels with your key.
If you are not signed, you are judged on behaviour. There is no identity to check, so the site scores the request: where it came from, what it looks like, how it moves, whether the shape of the session resembles a person. Nothing about that is new, and we wrote about the scorecard in getting past bot defences, the honest version. What is new is that the two paths are now explicitly separate, and the second one is where almost all commercial automation now lives.
That is the part worth saying plainly, because a proxy company has an obvious incentive not to.
Who can realistically get signed
Signing is the correct answer if you qualify for it. If you are building an agent product with a name, a company behind it and a public key you are willing to stand behind, sign your requests. It is a better foundation than anything else discussed here, and no proxy is a substitute for an identity.
Most commercial automation does not qualify, and not because it is doing anything wrong. Price monitoring for a retailer, checking your own adverts are being served correctly, tracking search results across markets, gathering public listings: these are ordinary business jobs, run by companies that are not in the business of publishing a bot identity and negotiating with every site individually. There is no signed agent programme for them to join. They fall into the behaviour bucket by default, and they will stay there.
If that is you, this is a change in what you should optimise for, not a change in whether you can operate.
What the behaviour bucket actually scores
Once identity is off the table, the decision comes down to signals. Roughly in order of how cheap they are for a site to check:
Address type. This is the first and cheapest filter. Traffic from a hosting or cloud network is trivially identifiable, because those ranges are published and easy to catalogue. It is also the signal a proxy genuinely addresses, which is the honest scope of what we do. We covered the distinction in datacenter or residential, and when each one is right.
Address history. Not just the type but the record. An address that has been used for abuse carries that with it. This is why IP checkers flag proxies even when the address itself is a real residential one.
Pacing. How fast you ask, and how regularly. Human traffic is uneven. A request every 1.8 seconds for four hours is not.
Consistency. The address, the language headers, the timezone and the browser fingerprint should agree with each other. A German address presenting an American locale and a UTC clock is a contradiction, and contradictions are cheap to spot. Your exit address can be perfect while the browser gives you away.
Session shape. Whether you arrive, take one thing and leave, or move through a site the way a person would.
A proxy moves the first two. It does nothing for the last three. Anyone telling you otherwise is selling.
What this means in practice, by what you are doing
If you run a browser based agent. You are already the heaviest user of bandwidth per task, because a real browser pulls images, fonts and scripts on every page, and the model decides what to load so the volume is unpredictable by design. That unpredictability is the thing metered pricing punishes hardest, and it is the argument for unmetered per port plans over paying by the gigabyte. We did that arithmetic in per gigabyte or per port.
If you run high volume collection with no sessions. Stateless crawling, retry tolerant, large payloads. This is the least affected by the change and the best served by unmetered capacity. Pace yourself and keep your headers coherent.
If you are hitting sites that have started answering with a 402. That is a different mechanism and no amount of address rotation changes it, because the decision was not made about your address. We wrote that up separately in paying to crawl.
If you are building an agent product. Sign. Then use a proxy for the parts signing does not cover, which is mostly geography and address type, not identity.
A short checklist for 15 September
If your jobs start failing next week, work through this before changing anything expensive:
- Check whether your targets sit behind Cloudflare and carry ads. That combination is what the default change actually targets. Plenty of sites are unaffected.
- Check what your crawler declares itself as. A crawler that blends search, agent and training behaviour is exactly what the new default blocks. If yours announces multiple purposes, separate them.
- Look at the response before you retry. A 403 from a bot decision, a 402 asking for payment and a 429 asking you to slow down are three different problems, and only one of them is helped by a different address.
- Check your exit type. If your traffic is leaving from a hosting network, that is the cheapest thing on this list to fix and the first thing a site checks.
- Check your pacing and your headers agree with your address. Free, and it moves more of the scorecard than most people expect.
- Only then look at capacity. If the fix is more parallelism or more addresses, size it against what the job actually needs rather than buying a number.
The honest summary
Cloudflare has made the decision legible, which is genuinely useful even when the answer is no. Signed agents get judged on who they are. Everyone else gets judged on how they behave, and behaviour is a scorecard where the address is one line of several.
We sell one line of that scorecard. It is an important line, and it is the first one a site checks, but it is one line. If you want the rest, the work is in pacing, consistency and session shape, and none of it is something you buy.
If you want to talk through where your own jobs sit, we are easy to reach.
ProxyOmega