When retrieval fails quietly: proxies for RAG pipelines

Most advice about collecting web data for AI systems treats it as one problem: get the pages, do not get blocked. That framing is fine right up until the pipeline is in production, and then it hides the distinction that actually matters.

Collecting a training corpus and running retrieval for a live system are not the same job. They tolerate failure completely differently. A page you missed while assembling a corpus is noise — the model never knew it existed, and the gap disappears into a dataset measured in millions of documents. A page you miss during retrieval becomes a specific wrong answer, delivered confidently, to one identifiable user.

Same fetch. Same failure. Wildly different blast radius. If you are running retrieval, that difference should drive how you route requests and, more importantly, what you alert on.

Two jobs with opposite tolerances

Bulk collection is a large, one-off sweep. It is throughput-shaped: you want a lot of pages cheaply, you can retry lazily, and you can afford to lose some. If a crawl of two million pages comes back with 1.96 million, the corpus is fine. Nobody can point at the missing forty thousand and show you the harm.

Retrieval is small, recurring and freshness-critical. The same handful of sources get re-fetched on a schedule, and the output goes straight into an answer. Here, losing two percent is not a rounding error — it is a subset of questions that get answered from stale or absent context, and the system has no way to tell the user that is what happened. It does not return an error. It returns an answer.

That is what makes retrieval failure so awkward: it does not look like failure. The pipeline reports success, the model produces fluent text, and the only symptom is that the content is wrong in a way nobody upstream can see.

We wrote about the related problem of agents acting on data they cannot verify in the trust budget of AI agents. This is the layer underneath it — if retrieval is quietly incomplete, everything above inherits the fault.

The failures that do not announce themselves

Three of these are worth designing against specifically, because none of them raises an exception.

The block that returns 200. A challenge page, a consent interstitial, or a “you may also like” fallback all come back with a perfectly healthy status code. Your fetcher stores it, your chunker chunks it, your index embeds it. You have now indexed the anti-bot page rather than the article, and the only way you find out is when someone asks about that document and gets a confident summary of a CAPTCHA.

The geographically wrong page. Retrieval against anything with regional variation — pricing, availability, legal text, product catalogues, local listings — depends on where the request appeared to come from. Fetch from an address in the wrong country and you get a real, valid, complete page containing the wrong facts. Nothing in the pipeline flags it, because nothing is broken. This is the failure most likely to survive all the way to a user.

The stale success. The fetch failed at 03:00, the retry queue was full, and the pipeline fell back to the copy from six days ago. Every dashboard is green. The answer is six days out of date, which for anything price- or availability-shaped is simply wrong.

Route by consequence, not by volume

Bulk collection sensibly optimises for cost per page. Retrieval should optimise for correctness of the pages that get used, which usually means the opposite trade.

A workable split:

• Route retrieval sources by how much a wrong answer costs. A handful of high-consequence sources justify better addresses and tighter geo control; the long tail of low-stakes reference material does not. Most pipelines have far fewer high-consequence sources than they assume — often a dozen or two.
• Pin geography deliberately for anything regional. Do not let the exit country be incidental. Set it explicitly, per source, and record which one you used alongside the fetched document. Targeting by country, city and ASN covers the mechanics; the discipline is treating location as part of the document’s identity rather than an accident of routing.
• Hold a session where the source expects continuity. Anything paginated or behind a soft session boundary should complete over one identity rather than hopping mid-sequence. Rotating or sticky sessions walks through when each applies.
• Keep bulk backfills on the cheap path. They genuinely are throughput problems, and spending retrieval-grade budget on them is waste.

The point is not that retrieval needs expensive addresses everywhere. It is that “which network” should follow from what a wrong answer costs, and for most of a corpus the answer is honestly not much.

Instrument the miss, not the error

Almost every pipeline logs errors. Very few log misses, and the miss is the thing that hurts.

Four fields per fetch, stored next to the document, will tell you more than any amount of dashboard staring:

• Fetch outcome — not just the status code, but whether the body passed a content assertion. A length floor and a required selector or phrase catches the block-that-returns-200 immediately.
• Exit country — recorded for every fetch, so a geo drift shows up as data rather than as a mystery.
• Fetched-at, and age at answer time — the second one matters more. A document can be fresh in your store and stale by the time it is served.
• Fallback flag — whether this answer used a live fetch or a cached copy, and how old that copy was.

Then alert on miss rate per source, not on aggregate error rate. Aggregates hide exactly the pattern you need: a pipeline at 99% overall can be at 40% on the one source that answers your most common question. The same reasoning applies here as in when your success rate is measuring the wrong thing — an average across dissimilar things tells you nothing about any of them.

The thirty-second test

Content assertions are the cheapest reliability work in the entire pipeline. Prove the point on one source before building anything:

# Fetch a known source and assert on the BODY, not just the status
curl -x "http://USERNAME:[email protected]:20228" \
     -s -w "\nstatus=%{http_code} bytes=%{size_download}\n" \
     "https://YOUR-SOURCE.example/article" \
  | tee /tmp/page.html | grep -qi "PHRASE_THAT_MUST_APPEAR" \
  && echo "CONTENT OK" || echo "MISS - fetched something, but not the article"

Run it a few dozen times. Any run that prints status=200 alongside MISS is a document your pipeline would have indexed and served. That single line — a required phrase and a byte floor — removes the most damaging class of retrieval failure, and it costs nothing.

Then run the same command with the exit country pinned two different ways and compare the bodies. If they differ, that source is regional, and geography is now part of its identity whether you were tracking it or not.

Where this leaves you

Bulk collection and retrieval look like the same engineering problem and are not. One is a throughput problem where losses vanish into the average. The other is a correctness problem where a single quiet miss becomes a confident, wrong, user-visible answer.

Sort your sources by what a wrong answer costs, pin geography where the content is regional, assert on the body rather than the status code, and alert per source rather than in aggregate. The proxy layer matters here — but it matters because of what it lets you control and record, not because of a headline success rate. If you want to talk through routing a specific set of sources, that is a conversation worth having before the pipeline is live rather than after.