Your scraper's success rate is measuring the wrong thing

A team comes to us with a number: “our success rate is barely fifty percent.” It is a good reason to be unhappy, and it is almost always wrong — not the arithmetic, the denominator. They are counting every request their browser makes. The pages they actually wanted came back fine, nearly every time. The failures were ad exchanges and tracking pixels that were never going to answer a proxy, on our network or anyone else’s.

This is the most common false alarm we see, and it sends people to fix things that were never broken. Here is what is really being counted, and how to measure the thing you care about.

One page is not one request

If you fetch a URL with a plain HTTP client, one page is one request and the accounting is simple. The moment you drive a real browser — Playwright, Puppeteer, Selenium, anything with a full rendering engine — that stops being true. Loading a single article can fire dozens to hundreds of requests: the document itself, then scripts, fonts, images, and a long tail of things the page brings along on its own.

That tail is the problem. It is ad exchanges, real-time-bidding and cookie-sync pixels, analytics beacons, consent managers, and background telemetry. You did not ask for any of it. The page did. And on a typical commercial page, the document you actually came for is a small minority of the requests your session generates — well under half, often far less.

The requests that fail are mostly not yours

Now put those two facts together. Advertising and measurement infrastructure actively refuses traffic it thinks is automated or proxied. That is not an accident or a gap in someone’s pool — it is the entire point of that infrastructure. Ad fraud is a real problem and bid-sync endpoints defend against it aggressively. They reject residential proxies, datacenter proxies, VPNs, and anything else that does not look like an ordinary person’s browser.

Across the traffic we handle, ad, tracking and telemetry endpoints account for well over half of all requests — and they fail several times more often than the pages our customers are actually trying to read. Names you will recognise in your own logs: adnxs.com, doubleclick.net, pubmatic.com, criteo.com, rubiconproject.com, openx.net, bidswitch.net, plus a steady drip of googleapis.com telemetry and app-update chatter that a full browser profile fires whether you want it or not.

So when you compute one blended number across everything, you are averaging your real target — which mostly succeeds — with a large pile of endpoints that are designed to refuse you. The blended number gets dragged toward the noise, and it tells you almost nothing about whether your job is working.

Why this matters more than it sounds

A wrong denominator does not just produce a sad number. It produces the wrong decision.

We have watched teams respond to a blended figure by switching providers, upgrading to mobile IPs, adding aggressive retry loops, or lowering concurrency until throughput collapsed. None of it moved the number, because none of it addressed the cause. The ad endpoints kept refusing them at exactly the same rate on the new network, because they refuse everyone. Meanwhile the extra retries hammered the target harder and made the real success rate slightly worse.

If you are going to act on a metric, the metric has to be about the thing you can change.

Measure the target, not the page

The fix is boring and it takes about ten minutes. Score only the requests to the host you set out to reach, and ignore everything else.

If you are scraping example.com, count requests whose host is example.com (plus its API subdomains). Everything else — every -sync., every pixel., every ad domain, every analytics beacon — is excluded from the numerator and the denominator both. Whatever number you get on that filtered set is your actual success rate, and it is the only one worth reporting.

In practice most people find they were already fine.

Better: stop requesting the noise at all

Filtering your logs fixes the measurement. Blocking the requests fixes the measurement and the bill, the latency, and a chunk of your failure count at the same time.

If you are already in a real browser, you can refuse the third-party traffic before it leaves. In Playwright:

const TARGET = 'example.com';

await page.route('**/*', (route) => {
  const url = new URL(route.request().url());
  const type = route.request().resourceType();

  // Anything that is not your target host is not your data.
  if (!url.hostname.endsWith(TARGET)) return route.abort();

  // Nor is most of what your target itself serves.
  if (['image', 'media', 'font', 'stylesheet'].includes(type)) return route.abort();

  return route.continue();
});

That one block does four things at once: your success rate becomes meaningful, pages load faster, you stop paying to download ad creative, and the failures that remain are all real. If you are on a metered plan, resource blocking is usually the single largest saving available to you.

The obvious caveat: some sites genuinely need their own scripts to render the content you want, so blocking by resourceType is a per-target decision. Blocking other hosts is nearly always safe. Start there.

The thirty-second version

Before you change providers, ask one question in a terminal — no browser, no third parties, nothing to muddy it. Swap in your own credentials, endpoint, and port:

curl -x "http://user123:[email protected]:10000" \
  -o /dev/null -s -w "%{http_code}\n" \
  https://example.com/the-page-you-actually-want

Run it twenty times and count the 200s.

• Mostly 200s — the network is doing its job. Your low number is a denominator problem, not a proxy problem. Filter your logs or block third parties and re-measure.
• A lot of 403s or challenges — that is a real block, and it is worth reading getting past anti-bot defenses rather than buying more IPs.
• 407 — credentials or a targeting typo, nothing to do with success rates. The error is telling you exactly that.

When the number is real

Sometimes it is. If your target host is genuinely failing, that is a signal worth acting on, and the causes are boring and fixable: the site is one of the strongly-defended ones and needs a cleaner network than a shared rotating pool; you are holding a session too long and it got flagged; your concurrency is high enough to look like an attack; or your client is fingerprintable in a way no IP can hide. Those are real problems with real fixes, and the anti-bot post and resilient scraping patterns cover most of them.

The point is not that failures do not exist. It is that you cannot see the ones that matter while they are buried under a pile of ad pixels that were never going to load. Clean the denominator first. Then, if there is still a problem, you will be able to see it — and fix the right thing.

There is a shorter version of this for your own reference in our docs.