How to Use ProxyOmega Proxies with Scrapy
Scrapy is the most-used Python framework for large-scale crawling. Its async engine can fire thousands of requests per second, which means it hits anti-bot defences faster than any single IP can absorb. Pointing Scrapy at ProxyOmega takes a one-line settings change plus a per-request meta key.
Quick start
Use the Premium Unlimited endpoint below — it accepts the username-modifier syntax used throughout this guide. Other plans use the same code shape with a different host and port (see the endpoint table at the bottom).
# settings.py
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 750,
}
# my_spider.py
import scrapy
class IpSpider(scrapy.Spider):
name = 'ip'
start_urls = ['https://api.ipify.org?format=json']
def start_requests(self):
proxy = 'http://<USERNAME>:<PASSWORD>@premium.proxyomega.com:8000'
for url in self.start_urls:
yield scrapy.Request(url, meta={'proxy': proxy})
def parse(self, response):
self.logger.info('Exit IP: %s', response.text)Rotation: sticky session vs rotating
Append -session-<id>-ttl-<seconds> to your username to hold the same exit IP. Drop the suffix entirely for a fresh IP on every request.
import uuid
# Rotating — new IP every request
proxy = 'http://<USERNAME>:<PASSWORD>@premium.proxyomega.com:8000'
# Sticky — same IP for the full session lifetime (max 86400 s)
sid = uuid.uuid4().hex[:8]
sticky = f'http://<USERNAME>-session-{sid}-ttl-600:<PASSWORD>@premium.proxyomega.com:8000'
yield scrapy.Request(url, meta={'proxy': sticky}, dont_filter=True)Country targeting
Add -country-XX (ISO 3166 alpha-2) to your username. State and city modifiers are supported on Premium Unlimited, Platinum and Mobile.
# Append -country-XX (ISO 3166 alpha-2)
de = 'http://<USERNAME>-country-de:<PASSWORD>@premium.proxyomega.com:8000'
us_ca = 'http://<USERNAME>-country-us-state-CA:<PASSWORD>@premium.proxyomega.com:8000'
yield scrapy.Request('https://example.com', meta={'proxy': de})Authentication: username/password vs IP whitelist
Two ways to authenticate. Username/password is set on every request and is what every example in this guide uses. IP whitelist is configured in the ProxyOmega dashboard — once your runner IP is listed, drop the credentials from the proxy URL and the connection is authorised by source IP.
Scrapy passes credentials inside the proxy URL — no extra Proxy-Authorization header is needed. If you prefer IP whitelisting, add the spider host IP in the dashboard, then drop user:pass from the URL.
Common errors and fixes
Wrong username or password, or the credentials contain a special character that was not URL-encoded. Re-copy from the dashboard.
Either the runner IP is not on your whitelist (if you use IP auth) or your network blocks outbound on the proxy port. Test with curl first.
Residential exits sometimes have tail latency. Raise client timeout to 30–60 s and enable retry. If timeouts persist across many sessions, the target is rate-limiting your country — switch country.
You have a -session- suffix in your username, or HTTP keep-alive is reusing the connection. Drop the session suffix and disable keep-alive on the client to force new exits.
Recommended plan
All endpoints
| Plan | Host | Port | Notes |
|---|---|---|---|
| Budget Unlimited | residential.proxyomega.com | 10000–10099 | Per-port country / rotation set in the dashboard. No username suffixes. |
| Premium Unlimited | premium.proxyomega.com | 8000 | Username modifiers: -country-XX, -session-{id}-ttl-{seconds}. |
| Platinum | platinum.proxyomega.com | 20228 | ASN / state / city targeting via username modifiers. |
| Mobile | mobile.proxyomega.com | 20229 | Real 4G/5G carrier IPs. Same username modifier syntax. |
| IPv6 | ipv6.proxyomega.com | 9000 | IPv6 exits, supports UDP_ASSOCIATE. |
FAQ
Should I write my own proxy middleware?
No. The built-in HttpProxyMiddleware already reads request.meta["proxy"]. Custom middleware is only useful if you want to rotate inside a pool with retry-on-failure logic.
How do I rotate per request without sticky sessions?
Use the bare username (no -session- suffix). Every Scrapy request opens a fresh upstream connection, so each request will exit through a new IP.
CONCURRENT_REQUESTS keeps timing out, what next?
Lower CONCURRENT_REQUESTS_PER_DOMAIN, raise DOWNLOAD_TIMEOUT to 60, and enable RETRY_ENABLED. Most timeouts on rotating residential are tail-latency, not failure.
Does ProxyOmega work with scrapy-playwright?
Yes — pass the same proxy URL as a playwright launch option (see the Playwright guide).