A resilient Python setup for scraping behind proxies

Most scraping code works fine the first hundred times you run it. The trouble starts on the run where a request stalls, a connection drops mid-body, or the same worker suddenly needs a fresh address. A resilient setup is not clever. It just assumes those days will come and refuses to fall over when they do.

This post is about the plumbing: how to attach the proxy, how to retry without making things worse, and how to decide whether each worker rotates or holds its address. The examples use placeholder credentials in the form user:pass@host:port so you can drop your own in.

Make the proxy part of the request, not a global

The most common mistake is setting the proxy once, at the top of the file, as if it were a constant. It works until you want two workers on two different addresses, or you want to swap the address after a failure. Then the global fights you.

Pass the proxy explicitly, close to where the request is made:

import requests

PROXY = "http://user:pass@host:port"

resp = requests.get(
    "https://example.com/data",
    proxies={"http": PROXY, "https": PROXY},
    timeout=(5, 20),
)

That timeout tuple matters more than it looks. The first number is how long to wait for the connection, the second is how long to wait for data once connected. A single scalar timeout covers the whole request and tends to be set too high out of caution, which means a stuck request hangs a worker for a minute when it should have given up in five seconds. Split them, keep the connect timeout short, and every request has a firm ceiling.

Retry with backoff, and only on the right failures

A retry loop that fires instantly on every error is a denial-of-service attack on yourself. The target sees a burst, the proxy sees a burst, and nothing improves. Space the retries out, and grow the gap each time.

import time
import random
import requests

def fetch(url, proxy, attempts=4):
    for i in range(attempts):
        try:
            resp = requests.get(
                url,
                proxies={"http": proxy, "https": proxy},
                timeout=(5, 20),
            )
            if resp.status_code in (429, 502, 503, 504):
                raise requests.exceptions.RetryError(resp.status_code)
            resp.raise_for_status()
            return resp
        except (requests.exceptions.RequestException,) as e:
            if i == attempts - 1:
                raise
            sleep = (2 ** i) + random.uniform(0, 1)
            time.sleep(sleep)

Two details are doing the work here. The exponent means the gaps grow — roughly one second, then two, then four — so a struggling target gets room to recover instead of a wall of retries. The random fraction is jitter: it keeps a set of workers from retrying in lockstep and hammering the same instant. Without jitter, everything that failed together retries together.

Be deliberate about what you retry. A timeout, a dropped connection, or a 503 is transient and worth another go. A 404 is an answer, not a failure, and retrying it just wastes budget. A 401 or 403 usually means something about your request is wrong, and repeating it unchanged will not fix it.

The question to ask of every failed request is not “how do I retry this” but “will retrying this unchanged ever succeed?” If the answer is no, stop and surface it.

Rotating versus session-pinned

There are two shapes of proxy client, and picking the wrong one quietly breaks jobs.

A rotating client wants a fresh address per request. It suits broad discovery work where each request stands alone and you would rather spread across a large pool than lean on any single address. You get this behaviour from a rotating endpoint — you keep pointing at the same host and each request lands on a different exit. Premium Unlimited is built for exactly this: a wide residential pool, billed flat, so a long discovery run does not turn into a per-request cost worry.

A session-pinned client wants the same address for a sequence of requests. Anything with a login, a cart, a multi-page flow, or server-side state needs to look like one consistent visitor. Bouncing addresses mid-session is what gets a flow rejected. For long-running jobs that want a stable, predictable address, Budget Unlimited is the port-based fit, and when a job needs one dedicated address that never moves, Static ISP gives you a single-tenant IP for the duration.

In httpx, the split is clean. A fresh client per request rotates; a long-lived Client that you reuse pins:

import httpx

# Pinned: reuse this client for a whole session's worth of requests
client = httpx.Client(
    proxies="http://user:pass@host:port",
    timeout=httpx.Timeout(20.0, connect=5.0),
)
r1 = client.get("https://example.com/login")
r2 = client.get("https://example.com/account")

The mental model is simple. If the requests belong to each other, share one client. If they are independent, let each go its own way.

One session per worker

When you scale out, the tempting shortcut is a single shared Session object across every thread. It saves a few lines and creates a subtle mess: connection pools get contended, and a header or cookie set by one job leaks into another. State bleeds sideways and the bugs are miserable to trace.

Give each worker its own session, built once when the worker starts:

import requests
from concurrent.futures import ThreadPoolExecutor

def make_session(proxy):
    s = requests.Session()
    s.proxies = {"http": proxy, "https": proxy}
    return s

def worker(url, proxy):
    session = make_session(proxy)
    return session.get(url, timeout=(5, 20)).text

with ThreadPoolExecutor(max_workers=8) as pool:
    results = pool.map(lambda u: worker(u, "http://user:pass@host:port"), urls)

Each worker owns its session, its connection pool, and its address. Nothing is shared, so nothing leaks. The reuse you want — keeping a connection warm across a worker’s own requests — still happens, because the session lives for the length of the worker, not for a single call.

Scrape like a good guest

None of this is a licence to be reckless. Resilient does not mean relentless. Pull public data, honour a site’s terms and its robots.txt where they apply, keep your request rate reasonable, and back off when a target signals it is under strain. The backoff and jitter above are as much about being a considerate visitor as they are about your own reliability — a scraper that eases off when a site is struggling is one that keeps working tomorrow.

The through-line is the same at every layer. Attach the proxy per request, give up quickly and retry slowly, pick rotating or pinned to match the work, and let each worker keep to itself. Do that and the hard days — the stalls, the drops, the occasional bad address — become a few extra log lines instead of a failed run.