Spread request volume
Rotating residential IPs distribute a scraper's requests across many addresses, so per-IP rate limits and 403 bans stop cutting jobs short.
Integration Scraping framework
BeautifulSoup Proxy Setup
BeautifulSoup parses HTML, but the fetching is done by requests, and that is where your IP is exposed. Sending those requests through ProxyOmega residential proxies gives each fetch a real consumer address, unlocks geo-targeting, and keeps scrapers alive across large jobs. This guide shows the exact proxies dict, session control, and error fixes.
- 200+countries
- 1.5M+residential IPs
- SOCKS5on every port
BeautifulSoup
What is BeautifulSoup?
BeautifulSoup is a Python library for parsing HTML and XML into a navigable tree you can search with tags, attributes, and CSS selectors. It is forgiving of messy markup and pairs with parsers like html.parser or lxml. It is one of the most popular tools for extracting data from web pages because the API is small and readable.
The key thing to understand is that BeautifulSoup does not fetch anything. It only parses HTML you already have. The fetching is almost always done by the requests library, and requests is where proxy configuration lives. So a BeautifulSoup scraper is really a requests-plus-BeautifulSoup scraper: requests downloads the page, BeautifulSoup turns it into structured data.
That split matters for proxying. You attach the proxy to the requests call, not to BeautifulSoup. Everything about IPs, authentication, rotation, and geo-targeting is handled at the requests layer, and BeautifulSoup simply parses whatever HTML comes back. This makes proxy integration straightforward: configure requests once and every fetch goes through your residential IPs.
BeautifulSoup
Why BeautifulSoup scrapers need residential proxies
A requests-based scraper is fast and lightweight, which means it can send a lot of requests from one IP in a short time. That volume from a single datacenter address is the classic signal anti-bot systems block, and the result is 403s, CAPTCHAs, or empty pages that leave BeautifulSoup with nothing to parse.
ProxyOmega routes each fetch through a real residential IP and rotates across a large pool, so no single address carries suspicious volume. Geo-targeting also lets requests retrieve the regional version of a page, so BeautifulSoup parses the content a local user would actually see.
Get the real content
Blocked requests return challenge pages or empty shells, leaving BeautifulSoup nothing useful. Residential IPs get full HTML back so your selectors actually match.
Localized pages
Geo-target the exit IP and requests fetches the local currency, language, and inventory version of a page for BeautifulSoup to parse.
Session continuity
Sticky sessions keep one IP across a login plus paginated fetches, so a multi-step scrape does not change address mid-flow.
BeautifulSoup
Using ProxyOmega with requests and BeautifulSoup
requests takes a proxies dict mapping the http and https schemes to your proxy URL, with credentials embedded as username:password in the URL. The username carries targeting parameters (here -country-us) and the password is your dashboard API key. BeautifulSoup then parses the returned text as normal.
For Budget Unlimited, use residential.proxyomega.com with any port from 10000 to 10199. Each port serves HTTP, HTTPS, and SOCKS5 on the same port. For SOCKS5, install requests[socks] and use a socks5h:// URL instead (the h makes the proxy resolve DNS, which is what you want for scraping). Use a requests.Session to reuse the proxy across many fetches efficiently.
import requests
from bs4 import BeautifulSoup
# Targeting params attach to the username, dash-separated.
username = "user123-country-us"
password = "your_dashboard_api_key" # account API key
endpoint = "residential.proxyomega.com:10000"
proxy_url = f"http://{username}:{password}@{endpoint}"
proxies = {"http": proxy_url, "https": proxy_url}
# A Session reuses the proxy config across many requests.
session = requests.Session()
session.proxies.update(proxies)
session.headers.update({"User-Agent": "Mozilla/5.0 (compatible; scraper/1.0)"})
resp = session.get("https://example.com", timeout=30)
resp.raise_for_status()
soup = BeautifulSoup(resp.text, "html.parser")
print(soup.title.string)
# Confirm the exit IP is residential:
ip = session.get("https://api.ipify.org", timeout=30).text
print("Exit IP:", ip)BeautifulSoup
Rotating vs sticky sessions
For stateless scraping (fetch a URL, parse it, move on) leave the session parameter off the username. Budget Unlimited rotates IPs on the dashboard interval, so requests spread naturally across the pool. This is the common case for BeautifulSoup jobs.
When you need continuity (a login followed by authenticated pages, or paginated results tied to a session) hold one IP with a sticky session. Add -session-<id> to the username, optionally with -ttl-<minutes>, and reuse it for that sequence of requests. Change the id to get a new IP for the next sequence.
Rotating (default)
Use the plain username with no session parameter. IPs rotate on the dashboard interval so consecutive fetches spread across many addresses. Best for scraping lists of independent URLs.
Sticky sessions
Add -session-page5-ttl-10 to the username to hold one IP for ten minutes. Ideal for login plus paginated fetches. Build the username string per session and rotate the id between sequences.
BeautifulSoup
Which ProxyOmega plan to pick
requests-based scraping is bandwidth-light, so Budget Unlimited fits most jobs. Move up only for precise targeting or specific network types.
| Workload | Plan | Why |
|---|---|---|
| High-volume HTML scraping | Budget Unlimited (:10000) | Unlimited bandwidth and a rotating pool with country targeting suit most requests jobs cost-effectively. |
| Large files or fast throughput | Premium Unlimited (:8000) | Dedicated 200Mbps-1Gbps for downloading big pages or many fetches quickly. |
| City/state/ASN precision | Platinum (:20228) | Pay-per-GB with the finest geo-targeting for localized data. |
| Mobile-only targets | Mobile (:20229) | Real 4G/5G IPs for sites that serve carrier traffic differently. |
| Fixed whitelisted IP | Static ISP (:30000) | One stable US (Dallas) address for targets requiring a consistent source IP. |
BeautifulSoup
Troubleshooting
requests surfaces proxy problems as status codes or exceptions. These are the common ones and their fixes.
407 Proxy Authentication Required
The username or password in the proxy URL is wrong, or a targeting parameter is invalid for the plan. The password is your dashboard API key, not your login password. A parameter that does not apply to the product (for example a city parameter on Budget Unlimited) rejects the entire username, so check the suffix.
ProxyError / connection reset
If IP whitelisting is enabled on your account, add the machine's public IP to the dashboard allowlist; otherwise requests are dropped after auth. Confirm the port is in the 10000-10199 range and the scheme is http:// for HTTP CONNECT.
SOCKS not working
SOCKS support needs the extra dependency: pip install requests[socks]. Then use a socks5h:// URL (the h resolves DNS through the proxy). The port is the same one you already use.
Timeouts or empty responses
Set an explicit timeout (30s is reasonable for residential hops), retry on failure so the next attempt lands on a fresh IP, and send a realistic User-Agent header so servers do not return stripped-down pages.
FAQ
Frequently asked questions
Do I configure the proxy on BeautifulSoup or requests?
On requests. BeautifulSoup only parses HTML you already fetched, so all proxy settings (IPs, auth, rotation, geo-targeting) live in the requests call. BeautifulSoup parses whatever comes back unchanged.
How do I use SOCKS5 with requests?
Install requests[socks] and use a socks5h:// URL in the proxies dict. The port is the same as HTTP, since every ProxyOmega port serves HTTP, HTTPS, and SOCKS5 together. The h makes the proxy resolve DNS.
How do I target a specific country?
Append -country-<code> to the proxy username, for example user123-country-jp. Rebuild the username string to switch regions between requests.
How do I keep the same IP across several requests?
Add -session-<id> to the username, optionally with -ttl-<minutes>, and reuse it across the requests that need continuity. Use a new id when you want a fresh IP.
Should I use a requests.Session?
Yes. A Session reuses the proxy config and connection pool across many fetches, which is faster and lets you keep cookies for a sticky-session flow.
What is the proxy password?
It is your account API key from the ProxyOmega dashboard, not your login password. Keep it in an environment variable rather than in source.
Fetch every page from a real IP Start routing today.
1.5M+ residential IPs, 200+ countries, SOCKS5 on every port.
ProxyOmega