How to fix and avoid 429 Too Many Requests

1. Honor Retry-After. If the header is present, wait exactly that long before retrying — hammering through the penalty window often lengthens it. Treat the header as authoritative over your own backoff schedule.
2. Add exponential backoff with jitter for 429s without Retry-After: wait 1s, 2s, 4s, 8s with a random offset, so parallel workers do not retry in lockstep and re-trigger the limit together.
3. Cut concurrency per target. Total throughput matters less than per-host pressure — cap simultaneous connections to any single host and add a small delay between requests to the same domain.
4. Identify the rate-limit key. Send one request from a fresh IP: if it succeeds instantly, the limit is per-IP and rotation is your lever; if it still 429s, the key rides your API token, account, or cookies.
5. Spread traffic across more IPs. For per-IP limits, distribute requests over a rotating pool so each address stays under the threshold. Residential IPs help further because per-IP budgets for consumer address space are typically more generous than for datacenter ranges.
6. Cache and deduplicate. The cheapest request is the one you do not send — cache stable responses, honor ETag/If-Modified-Since, and dedupe URL queues so retries and re-crawls do not double-spend quota.
7. Smooth bursts with a token-bucket limiter in your client. Most sites tolerate a steady drip far better than idle periods punctuated by bursts of hundreds of requests.
8. Check for shared-IP contention. On shared or datacenter proxies, other tenants may have already spent an IP's quota before your first request — cleaner pools mean your requests start from a lower counter.

import random
import time
import uuid
import requests

def proxied(session_id):
    user = f"USERNAME-country-us-session-{session_id}"
    p = f"http://{user}:[email protected]:10000"
    return {"http": p, "https": p}

def fetch(url, attempts=5):
    for i in range(attempts):
        r = requests.get(url, proxies=proxied(uuid.uuid4().hex[:8]), timeout=30)
        if r.status_code != 429:
            return r
        retry_after = r.headers.get("Retry-After")
        wait = float(retry_after) if retry_after else (2 ** i) + random.random()
        time.sleep(wait)
    return r

print(fetch("https://example.com/api/items").status_code)