Distribute load across a pool
Rotating residential IPs spread Scrapy's concurrent requests across many addresses, so no single IP trips per-IP rate limits. Long crawls survive that would otherwise stall after a few thousand pages.
Integration Scraping framework
Scrapy Proxy Integration
Scrapy crawls fast and concurrently, which is exactly what gets a single IP throttled or banned. Routing every request through ProxyOmega residential IPs spreads your traffic across millions of real addresses, unlocks geo-targeting, and keeps long crawls running. This guide gives you working downloader-middleware code and the session patterns that matter.
- 200+countries
- 1.5M+residential IPs
- SOCKS5on every port
Scrapy
What is Scrapy?
Scrapy is an open-source web-crawling framework written in Python and built on the Twisted asynchronous networking engine. Instead of fetching one page at a time, it schedules and processes many requests concurrently, which makes it the go-to tool for large, structured crawls: product catalogs, search results, marketplace listings, real-estate data, and price monitoring at scale.
The framework is organised around a few core concepts. Spiders define which URLs to start from, how to follow links, and how to parse responses into structured data. Item pipelines post-process that data (validation, deduplication, writing to a database or file). Downloader middlewares sit between the engine and the network layer, and this is the hook you use to attach a proxy to every outgoing request. Because proxy handling lives in a middleware rather than scattered through your spider code, you configure it once and every request inherits it.
Out of the box Scrapy gives you request scheduling, automatic retries, cookie handling, AutoThrottle, robots.txt handling, and exporters for JSON, CSV, and XML. What it does not give you is a way to avoid being fingerprinted and blocked by the sites you crawl. That is the job of a residential proxy layer.
Scrapy
Why Scrapy needs residential proxies
Scrapy's default concurrency (CONCURRENT_REQUESTS is 16, and many crawls raise it well beyond that) means a single origin IP can send a burst of requests to one host in seconds. That pattern is trivial for anti-bot systems to spot. From a datacenter IP the result is predictable: rate limits, CAPTCHA walls, empty responses, or an outright ban that stops the crawl.
ProxyOmega residential IPs are assigned to real consumer connections, so requests blend in with ordinary traffic. Spreading a crawl across a large rotating pool means no single address carries a suspicious volume, and geo-targeting lets you see the exact regional content, pricing, or availability a local visitor would see.
Geo-targeted results
Append a country to the proxy username and every request exits from that region. Essential for scraping localized pricing, search rankings, or catalog availability that changes by country.
Higher trust than datacenter
Residential addresses carry more trust with anti-bot vendors than datacenter ranges, so you see fewer CAPTCHAs and fewer soft-blocks (200 responses with missing content).
Session control
Pin an IP for a multi-step flow (login, pagination, cart) with a sticky session, or rotate freely for stateless page fetches. Both patterns attach to the username.
Scrapy
Using ProxyOmega with Scrapy
The cleanest way to attach ProxyOmega to Scrapy is a small downloader middleware that sets the proxy endpoint and a Proxy-Authorization header on every request. Keep Scrapy's built-in HttpProxyMiddleware enabled and run your own middleware just after it. Your ProxyOmega username carries the targeting parameters (here -country-us), and the password is the account API key from your dashboard.
For Budget Unlimited, connect to residential.proxyomega.com on any port in the 10000-10199 range. Every port serves HTTP, HTTPS, and SOCKS5 on that same port, so you never need a separate SOCKS5 port. The example below routes all traffic through US residential IPs; drop the -country-us suffix for the global pool, or swap in another country code.
# middlewares.py
from w3lib.http import basic_auth_header
class ProxyOmegaMiddleware:
"""Route every Scrapy request through ProxyOmega residential proxies."""
PROXY = "http://residential.proxyomega.com:10000"
def __init__(self, api_key):
self.api_key = api_key
@classmethod
def from_crawler(cls, crawler):
return cls(api_key=crawler.settings.get("PROXYOMEGA_API_KEY"))
def process_request(self, request, spider):
# Targeting params attach to the username, dash-separated.
username = "user123-country-us"
request.meta["proxy"] = self.PROXY
request.headers["Proxy-Authorization"] = basic_auth_header(
username, self.api_key
)
# settings.py
DOWNLOADER_MIDDLEWARES = {
"scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 750,
"myproject.middlewares.ProxyOmegaMiddleware": 760,
}
PROXYOMEGA_API_KEY = "your_dashboard_api_key" # account API key
# Be a good citizen so residential IPs last longer
CONCURRENT_REQUESTS = 16
DOWNLOAD_TIMEOUT = 30
RETRY_ENABLED = True
AUTOTHROTTLE_ENABLED = TrueScrapy
Rotating vs sticky sessions
Budget Unlimited rotates the IP on each port on an interval you set in the dashboard, so by default consecutive requests may exit from different addresses. That is ideal for stateless crawling where each page fetch is independent. When you need continuity across several requests, pin an IP with a sticky session.
A sticky session is created by adding -session-<id> to the username, optionally with -ttl-<minutes> to control how long that IP is held. Reuse the same session id and you keep the same exit IP; change the id and you get a fresh one. In Scrapy, generate a session id per logical unit of work (per login, per paginated sequence) and rotate it when you start the next one.
Rotating (default)
Leave the username without a session parameter. IPs cycle on the dashboard interval, giving each concurrent request a wide spread of addresses. Best for high-volume, stateless page fetches and broad crawls.
Sticky sessions
Add -session-abc123-ttl-10 to the username to hold one IP for ten minutes. Use it for login flows, multi-page navigation, or anything where the target ties state to the source IP. Rotate the id per session to get a new IP.
Scrapy
Which ProxyOmega plan to pick
Most Scrapy crawls run best on Budget Unlimited, where you pay by concurrency rather than bandwidth. Move up only when you need finer targeting or specific network types.
| Workload | Plan | Why |
|---|---|---|
| High-volume general crawling | Budget Unlimited (:10000) | Unlimited bandwidth, rotating pool, country targeting. Cost-effective for millions of pages. |
| Bandwidth-heavy or faster crawls | Premium Unlimited (:8000) | Dedicated 200Mbps-1Gbps throughput for large pages, images, or tight schedules. |
| Precise geo (state/city/ASN) | Platinum (:20228) | Pay-per-GB with the best targeting granularity for localized data down to city or ASN. |
| Mobile-only targets | Mobile (:20229) | Real 4G/5G IPs for sites that treat carrier traffic differently. |
| Fixed IP for allowlists | Static ISP (:30000) | One stable US (Dallas) IP for targets that require a consistent, whitelisted address. |
Scrapy
Troubleshooting
Most Scrapy proxy issues come down to credentials, targeting parameters, or IP allowlisting. Work through these before touching your spider logic.
407 Proxy Authentication Required
Your username or password is wrong, or a targeting parameter is invalid for the product. The password is the account API key from your dashboard, not your login password. A wrong param for a plan (for example a state parameter on Budget Unlimited) rejects the whole username, so double-check the suffix. Confirm the Proxy-Authorization header is actually being set by your middleware.
Connection reset / refused
If IP whitelisting is enabled on your account, requests from an address that is not on the list are dropped after authentication. Add your crawler's public IP in the dashboard, or remove whitelist entries you no longer use. Also confirm you are hitting a valid port in the 10000-10199 range.
Every request shows the same IP
You are probably reusing a sticky -session- id, or your dashboard rotation interval is long. Remove the session parameter for full rotation, or generate a new session id per logical crawl unit.
Timeouts and slow pages
Raise DOWNLOAD_TIMEOUT to 30-60s (residential hops add latency) and keep RETRY_ENABLED on so transient failures re-dispatch on a fresh IP. Lower CONCURRENT_REQUESTS if a target is aggressively rate limiting.
FAQ
Frequently asked questions
Does Scrapy support SOCKS5 proxies from ProxyOmega?
Yes. Every ProxyOmega port serves HTTP, HTTPS, and SOCKS5 on the same port, so there is no separate SOCKS5 port to quote. For SOCKS you would use a socks5h scheme in the proxy URL; for standard crawls the HTTP CONNECT method shown above handles HTTPS targets fine.
Where do I find my proxy password?
The proxy password is your account API key, available in the ProxyOmega dashboard. It is separate from your dashboard login password. Put it in the PROXYOMEGA_API_KEY setting rather than hardcoding it in your spider.
How do I target a specific country in Scrapy?
Append -country-<code> to the proxy username, for example user123-country-de for Germany. The parameter attaches to the username, so change it per request in your middleware if you need different regions across a single crawl.
How do I keep the same IP across a login and its follow-up pages?
Add -session-<id> to the username, optionally with -ttl-<minutes>. Reusing the same id keeps the same exit IP for the duration; generate a new id when you start an unrelated sequence.
Will proxies slow my crawl down?
Residential hops add some latency versus a direct connection, so raise DOWNLOAD_TIMEOUT and lean on AutoThrottle. For throughput-sensitive crawls, Premium Unlimited offers dedicated high-speed bandwidth.
Can I rotate IPs per request automatically?
Yes. Leave the session parameter off the username and the pool rotates on your dashboard interval, so concurrent requests spread across many IPs without extra code.
Scale your Scrapy crawls without bans Start routing today.
1.5M+ residential IPs, 200+ countries, SOCKS5 on every port.
ProxyOmega