Your proxy works locally and fails in Docker

A scraper runs on your laptop for weeks. You containerise it and change nothing else — same code, same credentials, same targets — and it hangs on startup. Then either a timeout, or a JSON parsing error thrown from inside your browser driver with no obvious connection to proxies at all.

The first instinct is to blame the pool. It is almost never the pool: in this failure the proxy carries none of your target requests, because the process dies before it makes one.

What HTTP_PROXY actually applies to

HTTP_PROXY, HTTPS_PROXY, ALL_PROXY and NO_PROXY are a convention, not a standard. There is no RFC, every client implements them slightly differently, and that inconsistency is the root of the confusion here.

The part people get wrong is what selects them. HTTP_PROXY does not mean “use a proxy for websites”. It means: for any request whose URL scheme is http://, send it to this proxy. The destination is irrelevant: it does not matter that the host is 127.0.0.1, or that the packet would never otherwise have left the machine.

So the moment it is set process-wide, all of these become proxied requests:

• http://127.0.0.1:9222/json/version — the Chrome DevTools Protocol endpoint your driver polls to discover the browser
• ws://127.0.0.1:9222/devtools/browser/<id> — the CDP websocket, if your websocket client applies proxy configuration to ws://
• http://127.0.0.1:9515/session — chromedriver
• http://localhost:4444/session — a Selenium Grid or standalone WebDriver endpoint
• http://127.0.0.1:8080/healthz — your own readiness check, or any sidecar call

None of these feel like “traffic”, but they are all HTTP requests, and a client that honours the environment cannot tell them apart from a page you meant to scrape.

Which clients do that is not obvious. It splits three ways:

• Honour the variables and will proxy loopback. curl and libcurl, Python’s requests, httpx and urllib, pip, and wget. These are the ones that break.
• Honour the variables but never proxy loopback. Go’s net/http. ProxyFromEnvironment returns no proxy at all when the host is localhost or any loopback address, so a Go control client is immune to this by design.
• Ignore the variables until asked. aiohttp, until you pass trust_env=True. Node’s built-in HTTP clients, until you set NODE_USE_ENV_PROXY=1 or pass --use-env-proxy. And Java, which never reads them at all — it has its own http.proxyHost and https.proxyHost system properties.

One curl detail makes debugging feel non-deterministic: libcurl reads http_proxy in lower case only, while every other variable in the family is read in either case. The reason is the CGI protocol, which turns an inbound Proxy: request header into an HTTP_PROXY environment variable for the script — the hijack published in 2016 as httpoxy. So a stack where curl behaves and Python does not is normal, and says nothing about your proxy.

The worked example: a JSON error that is really a proxy error

The exact chain behind the most common version of the bug:

1. You launch Chrome with --remote-debugging-port=9222.
2. Your driver polls http://127.0.0.1:9222/json/version to find the browser’s websocket URL.
3. The container has HTTP_PROXY=http://user:[email protected]:8080 set.
4. requests sees an http:// URL, finds no matching NO_PROXY entry, and opens a TCP connection to the proxy instead of to loopback. Because it is a plain http:// request, it is sent in absolute form — GET http://127.0.0.1:9222/json/version HTTP/1.1 — rather than as a CONNECT tunnel.
5. The proxy now reads 127.0.0.1 as an instruction to fetch from its own loopback interface, or rejects it as a non-routable destination.
6. What comes back is an HTML error page, or a 407 with an HTML body, or nothing.
7. Your driver calls .json() on it and you get:

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The stack trace points at JSON parsing. The cause is four steps upstream, in an environment variable you may not have set yourself.

The websocket half fails worse. A client that applies proxy configuration to the ws:// URL issues a CONNECT 127.0.0.1:9222 to the proxy, which never completes — and websocket clients frequently set no default timeout on that connect, so the process simply stops. No error, no CPU, no log line. It looks like a deadlock in your own code.

Why it looks nothing like a proxy failure

The signatures you will actually see:

• json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) on a control endpoint
• an indefinite hang on startup with no timeout firing
• net::ERR_PROXY_CONNECTION_FAILED or net::ERR_TUNNEL_CONNECTION_FAILED reported against a localhost URL
• requests.exceptions.ProxyError naming a host of 127.0.0.1
• Timeout 30000ms exceeded from Playwright while waiting for the DevTools endpoint
• WebDriverException: Message: Can not connect to the Service chromedriver

More useful are the things you will not see. No 407 Proxy Authentication Required, no 502, and no entry for the run in your provider’s usage dashboard, because no billable request was ever made. If a run failed and your bandwidth counter did not move at all, the proxy was not in the path of the failure.

Our support inbox sees this shape regularly, opened as “your proxy is down”. It resolves without anyone touching the pool, because the pool was never involved.

NO_PROXY alone is not enough

Adding NO_PROXY=localhost is the obvious fix and it is usually a partial one. Three reasons.

Entries are matched as literal host strings. localhost does not cover 127.0.0.1. Neither covers ::1, 0.0.0.0, host.docker.internal, or the service name of a container on a Compose network. If your driver connects by IP and your bypass list says localhost, the bypass does nothing.

Support varies. A bare * (bypass everything) and leading-dot suffixes like .internal are widely supported. Globs such as *.internal are not portable. CIDR entries work in requests for IPv4, and in curl 7.86.0 and later, and are ignored elsewhere. Ports attached to entries are honoured by some clients and dropped by others, so list bare hosts.

Case is read differently by different clients. Set both. A defensible value:

export NO_PROXY="localhost,127.0.0.1,0.0.0.0,::1,host.docker.internal,.internal,.svc,.cluster.local"
export no_proxy="$NO_PROXY"

NO_PROXY also only helps libraries that implement it, and cannot help a process that never inherited it: a supervisor started before you exported anything passes its own environment to every browser it spawns, and your correction reaches none of them.

Where the browser pushes loopback back through the proxy

There are two HTTP stacks in a browser automation process tree, and they have opposite defaults.

Chromium implicitly bypasses loopback and link-local addresses whenever a proxy is configured — localhost, *.localhost, [::1], 127.0.0.1/8, 169.254/16 and [FE80::]/10 behave as if they were already in the bypass list. The escape hatch that switches this off is the special rule <-loopback> in --proxy-bypass-list. Some automation recipes pass it to make a local test server reachable through the proxy; it also sends CDP traffic through the proxy. If it is in your launch arguments and you did not put it there deliberately, take it out. Firefox has an equivalent preference, network.proxy.allow_hijacking_localhost, false by default.

So the browser itself usually behaves. It is the driver sitting next to it, reading the environment, that does not — and configuring the proxy correctly inside the browser while leaving HTTP_PROXY exported for the driver is the most common route to this bug.

Why containers, WSL and cloud boxes are where it bites

Your laptop shell had no proxy variables. Several things set them for you elsewhere.

The Docker client injects them. A proxies block in ~/.docker/config.json applies to builds and containers alike, and it sets both cases — http_proxy and HTTP_PROXY, https_proxy and HTTPS_PROXY — so a client that reads only one of them is still caught. Nothing in your Dockerfile or Compose file mentions any of this. They are also predefined build arguments Docker accepts without an ARG line, so a CI system you did not write can pass them into a build.

The same string means two machines. HTTP_PROXY=http://localhost:3128 on the host points at the host’s egress proxy. Inside the container, localhost is the container. The variable copies across; the meaning does not. On Linux, host.docker.internal does not resolve at all without --add-host=host.docker.internal:host-gateway.

docker exec lies to you. It gives you a fresh shell with a fresh environment; the already-running supervisor may hold a different one. Read the real thing:

tr '\0' '\n' < /proc/1/environ | grep -i proxy

Replace 1 with the PID that actually spawns your browsers.

WSL inherits from Windows. WSLENV propagates selected Windows environment variables into the Linux side, so a proxy variable set for a Windows tool can appear in WSL without you exporting it. And a Chromium launched without --proxy-server reads the Windows system proxy configuration, so a machine-level setting applies with no environment variable involved at all.

Cloud boxes chain. Where an outbound corporate proxy is mandatory, your residential proxy has to be reached through it, and chaining fails in ways that read as credential errors — which sends people off to rotate a password that was never wrong.

The five-minute diagnostic

First, see what is actually set — in the process that matters, not in your shell:

env | grep -i proxy
tr '\0' '\n' < /proc/1/environ | grep -i proxy

Then run these two against your control endpoint:

curl -sv --max-time 10 http://127.0.0.1:9222/json/version 2>&1 | grep -E 'Uses proxy|Trying|HTTP/'
curl -s  --max-time 10 --noproxy '*' http://127.0.0.1:9222/json/version

If the first prints a line like * Uses proxy env variable http_proxy == 'http://...' and the second returns JSON, you are done. Your environment is sending loopback control traffic to a residential exit, and no amount of bandwidth, pool quality or credential rotation will change that.

Then confirm the other half, so you know the proxy is fine before you rule it out:

curl -s --max-time 20 -x "http://USERNAME:PASSWORD@HOST:PORT" https://api.ipify.org

An address that is not yours means the credentials and the pool are working. Two commands, two answers, no ticket needed.

The shape that does not break

Configure the proxy per request or per browser context, never as a process-wide environment variable. That single rule removes the whole class of failure, and it is what you want anyway once you are running more than one exit at a time.

# Python: opt out of the environment explicitly, then be deliberate.
s = requests.Session()
s.trust_env = False                       # ignores *_PROXY, and also .netrc
s.proxies = {"https": "http://USERNAME:PASSWORD@HOST:PORT"}

# httpx: same idea
client = httpx.Client(trust_env=False, proxy="http://USERNAME:PASSWORD@HOST:PORT")

# Playwright: per context, so the CDP transport is never involved
context = browser.new_context(proxy={
    "server":   "http://HOST:PORT",
    "username": "USERNAME",
    "password": "PASSWORD",
})

aiohttp ignores the environment until you pass trust_env=True, so simply do not pass it. In Go, build an explicit http.Transport{Proxy: http.ProxyURL(u)} rather than accepting ProxyFromEnvironment, so that one process can hold several exits. In shell scripts, always pass -x explicitly, and --noproxy '*' for anything local.

Checklist

1. Run env | grep -i proxy in the container, and read /proc/<pid>/environ for the process that actually spawns browsers.
2. Check ~/.docker/config.json for a proxies block, and your CI for http_proxy build arguments.
3. If a proxy variable is set process-wide, unset it and configure the client instead. That is the fix; the rest is for when you cannot.
4. If it must stay, set NO_PROXY and no_proxy covering localhost, 127.0.0.1, 0.0.0.0, ::1 and any container service names.
5. Remove <-loopback> from --proxy-bypass-list unless you know why it is there, and check network.proxy.allow_hijacking_localhost on Firefox.
6. Restart the supervisor, not just the worker. Environment changes do not reach children that already exist.
7. Prove the proxy separately with a single curl -x to an IP echo service before opening a ticket.

Once the request is leaving the machine correctly and still failing, the problem has moved a layer up: read the error, find the hop maps status codes to the four places a proxied request can break, and why IP checkers flag proxies covers requests that arrive and get treated as a bot. This post sits underneath both.