Building a price-monitoring pipeline that does not fall over

Price and stock monitoring looks simple from the outside. Pick a list of product pages, check them every so often, write the numbers somewhere. The hard part is not the checking. It is that the list is long, the pages misbehave, and the whole thing has to keep running unattended for weeks without your attention.

A pipeline that falls over usually fails in one of two ways. Either one bad target stalls the whole run, or the bill grows in a way nobody predicted and someone quietly turns it off. Both are design problems, and both are avoidable if you decide the shape of the system before you write the loop.

Start from the run, not the request

The instinct is to make each individual check as reliable as possible. Retry hard, wait long, chase every failure until it succeeds. That is the wrong altitude for monitoring.

What you actually care about is coverage across the run. If you are tracking thousands of pages and forty of them fail on this pass, that is fine as long as they get picked up on the next one. Monitoring is a moving picture, not a single frame. A missed reading on one product at 2pm is invisible once you have the reading at 2:15.

So design for breadth, not precision. Set modest timeouts, cap retries low, and let the slow and broken targets drop out of the current pass rather than holding it hostage. The run finishing on time matters more than any one page inside it.

The question that shapes the whole architecture: if any single check fails right now, does the run still produce a useful picture? If the answer is yes, you can build something that survives contact with the real web.

Schedule for staleness, not for speed

Every product you monitor has a tolerance for how stale its data can be. A fast-moving marketplace listing might need checking every few minutes. A slow catalogue page is fine once or twice a day. Treat those as different tiers rather than running everything at the same cadence.

This matters because your total work per interval is what determines whether the pipeline keeps up. If you schedule everything at the tightest cadence any single target needs, you multiply your load for no benefit. Group targets by how fresh they need to be, give each group its own interval, and spread the work across the window so you are not firing everything in the same second.

Staggering also smooths your traffic. A pipeline that does all its work in a thirty-second burst and then sleeps is harder to reason about, and harder on the sites you are visiting, than one that paces itself evenly across the interval.

Dedupe your targets before you fetch them

Long target lists accumulate duplicates. The same product reached through two URLs, tracking parameters that make identical pages look distinct, variants that resolve to one canonical listing. Left alone, this quietly inflates every run.

Normalise before you schedule. Strip the parameters that do not change the response, collapse the aliases that point at the same page, and key your target list on the thing you actually care about rather than the raw URL. A clean, deduplicated target list is the single cheapest way to cut load, and it makes your coverage numbers honest.

It also prevents a subtle failure: if a duplicate is failing, you do not want it counted as two separate gaps in your data. One target, one reading.

Let partial failure be normal

Assume, from the first line of code, that some checks will fail on every run. Networks blip, pages change shape, a target is briefly unavailable. If a single failure can crash the run or corrupt your dataset, you have built something fragile.

The pattern is to isolate each check. Catch failures at the individual-target level, record them as a failed reading with a reason, and carry on. A run’s output should be the successful readings plus an honest account of what did not come back. Then your alerting can watch the failure rate over time rather than reacting to any one miss. A target that fails for six hours straight is a signal worth acting on. A target that fails once is just weather.

The same thinking applies to your writes. Never let a batch of results depend on all of them succeeding. Commit what you have, mark what you do not, and let the next pass fill the gaps.

Keep the bill the shape of the work

The quiet killer of monitoring pipelines is cost that scales with volume in a way you cannot forecast. Continuous checking means continuous requests, forever. If every request has a metered price attached, your monthly bill becomes a function of how thorough you were, and thoroughness starts to feel expensive. That is exactly the wrong incentive for a system whose whole value is breadth.

For continuous, high-volume, breadth-first work like this, a flat-rate pool fits the shape of the job. You want to run wide and run often without doing arithmetic on every page you visit. Premium Unlimited bills flat across a wide residential pool, which is a good match for bursty discovery work where the value is in the aggregate rather than any single request.

If your monitoring is steadier and more structured, running a known set of long-lived jobs against a stable list rather than sweeping broadly, Budget Unlimited gives you port-based residential access with the same predictable, flat monthly shape. Either way the principle holds: choose a bill you can predict so that running more never becomes a reason to run less.

A note on doing this well

Monitor public product data, keep your request rates reasonable, and respect the terms and robots rules of the sites you visit. A pipeline that paces itself and dedupes its targets is not just cheaper to run, it is also a better-behaved visitor. Restraint in the design tends to produce restraint in the traffic, which is good for everyone.

The system that survives is rarely the one that tries hardest on each request. It is the one that expects failure, schedules for freshness rather than raw speed, and runs on a bill that does not punish it for being thorough. Build for the run as a whole, and the individual checks are free to come and go without ever taking the pipeline down with them.