Paying to crawl: what a 402 means for your pipeline

For most of the web’s history the answer to “may I fetch this page” has been binary. You either got the content or you got a block, and everything in between was a retry problem. That is changing. Publishers can now put a price on access and reply to an unpaid request with a status code that says, in effect, not for free.

The mechanism is HTTP 402 Payment Required — a status code reserved in the original spec and left unused for thirty years. It is being used now, and the practical question for anyone running a collection pipeline is what to do when one arrives.

What a 402 actually contains

A 402 in this context is not an error in the sense that a 500 is an error. Nothing went wrong. The server understood the request, is willing to serve it, and is quoting a price.

The response carries a header naming the cost of the resource. A crawler that wants the content signals payment intent on the retry and, if the arrangement is in place, gets a normal 200 with the body. A crawler that does not gets the 402 again.

Two things follow from that, and both matter more than they first appear.

The first is that a 402 is a deliberate, stable answer. Retrying it harder will not help. Rotating to a different address will not help either, because the decision was not made about your address. Backing off and retrying in an hour produces exactly the same response, having spent an hour.

The second is that this only applies to crawlers that identify themselves. The whole arrangement rests on the server knowing which crawler is asking, which means signed, declared requests. That is a real fork in the road, and it is worth being honest about where it leads.

The uncomfortable middle

If you declare yourself, you become addressable. You can be priced, rate-limited, allowed, or refused as a named party. For an organisation that wants a durable licensed relationship with a publisher, that is the point — it is the only route to content that is otherwise closed, and it comes with terms you can rely on.

If you do not declare yourself, none of this applies to you, and you are back in the older game of looking like ordinary traffic and living with whatever the site’s defences decide.

We are not going to pretend the second option stopped existing, because it did not. But it is worth being clear-eyed that these are now two different activities with different economics, rather than one activity with a difficulty slider. A pipeline that is fetching public pages at modest volume for price checks or availability is not in the same conversation as one ingesting a publisher’s archive for model training, and treating them identically is how teams end up with bills or blocks they did not plan for.

What this does not change

Plenty of what you already do is unaffected.

Public pages that no one has priced are still public pages. The overwhelming majority of the web has not put a price on access, and fetching a product listing, a search result or a review page works exactly as it did last year.

Geography still matters. If a page renders differently for a visitor in Berlin than one in Chicago, you still need to be in the right place to see the right version, and no payment arrangement substitutes for that.

Concurrency still matters. Sites still rate-limit per address, and spreading requests is still how you avoid tripping those limits.

The change is narrower than the headlines suggest: a specific class of high-volume, self-identifying crawler now has a paid lane available on a subset of sites. That is genuinely new. It is not the end of collecting public data.

Handling it in code

The concrete work is small, and worth doing before you meet one in the wild.

Treat 402 as terminal, not transient. Most HTTP clients bucket anything that is not a 2xx into “retry with backoff”. A 402 should exit the retry loop immediately, the same way you would treat a 404. Retrying it wastes your quota and produces nothing.

Log it separately. If 402s are folded into a general error count, you will see a success rate dip and start investigating your proxies, your headers, and your concurrency — none of which are the cause. A separate counter turns a mystery into a line item.

Record the price. The quoted cost is in the response. Capturing it tells you what the closed portion of your target list would actually cost, which is the number you need before anyone can decide whether to pay it.

Decide per source, not globally. A price that is absurd for a page you fetch daily may be trivial for one you need once a quarter.

The thirty-second test

You can see how your own stack behaves without waiting to encounter a live one. Point it at any endpoint that returns a 402 and watch what your client does:

curl -sS -o /dev/null -w "status=%{http_code}\n" https://httpbin.org/status/402

Then run the same URL through whatever wrapper your pipeline actually uses — your retry helper, your scraping framework, your queue worker. If it retries several times, sleeps, and reports a network failure, you have found the thing to fix. It should stop on the first response and record the status.

Where this leaves you

The web is not closing. It is acquiring a toll lane, and only some roads have one.

The teams that will be caught out are the ones whose pipelines silently treat a priced response as a broken one — burning retries, distorting their success metrics, and concluding their proxies are at fault. The fix costs an afternoon: recognise the status, stop retrying, count it separately, keep the price.

Everything else about collecting web data at scale is unchanged. You still need to be in the right country, spread across enough addresses to stay under rate limits, and honest with yourself about which requests actually matter. If you want the background on that last point, your success rate is probably measuring the wrong thing covers why most reported failure numbers are mostly noise, and choosing between rotating and sticky sessions covers how to spread requests without breaking the ones that need continuity.