What is User Agent?
A user agent is the client software that makes requests on a user's behalf — most often a web browser — and, by extension, the User-Agent HTTP header string that identifies that software to servers. The string encodes the browser family and version, the operating system, and the device class. Websites read it for content adaptation, analytics, and bot screening.
How user agents work
Every HTTP request carries a User-Agent header, set automatically by the browser or HTTP library. A typical desktop string looks like Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36. The odd structure is historical: tokens like Mozilla/5.0 and AppleWebKit persist for backward compatibility with decades of server-side parsing code, which is why nearly every browser claims to be several browsers at once.
Servers use the string for content negotiation (mobile versus desktop layouts), feature fallbacks, analytics, and traffic filtering. The ecosystem is also evolving: modern browsers are reducing the detail in the User-Agent string and moving structured client data into Client Hints headers (the Sec-CH-UA family). Browsers send a few low-entropy hints — brand, platform, and mobile status — by default, while higher-entropy details such as the full browser version or device model are sent only when a server explicitly requests them.
Why it matters for scraping and proxies
The user agent is the first and cheapest signal bot-detection systems check. Default library identifiers such as python-requests/2.31.0 or curl/8.4.0 announce automation outright, and many sites filter them with a single rule. Presenting a current, mainstream browser string removes that trivial tell and is standard practice in any scraping stack.
The string is also a consistency anchor. Detection systems cross-check the claimed browser against the TLS handshake fingerprint, the header set and its ordering, and JavaScript-visible properties such as navigator.platform. A Chrome user agent riding on a client with a non-Chrome TLS signature is a stronger bot signal than an honest library string. Rotating user agents therefore works best when the whole client profile rotates coherently — and when request origins are distributed across IPs as well.
Practical notes and misconceptions
Spoofing the user agent alone rarely defeats modern anti-bot systems; it is necessary but not sufficient. Use real strings taken from current browser releases rather than invented ones, retire outdated versions — an ancient browser build is itself suspicious — and keep companion headers such as Accept-Language and the Sec-CH-UA hints consistent with the claimed browser and platform.
Remember, too, that the header is entirely client-controlled and trivially forgeable, which is exactly why servers treat it as a hint rather than proof of identity.
User Agent, answered
Do I need to change my user agent for web scraping?
python-requests/2.31.0 or curl/8.4.0 identify automation immediately and are filtered by many sites. Send a current, real browser string instead, and keep the rest of your headers consistent with it. Remember that the user agent is only one signal — TLS and JavaScript fingerprints matter too.What does a user agent string look like?
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36. The Mozilla and WebKit tokens are historical compatibility artifacts; the meaningful parts are the operating system, browser family, and version. Mobile strings additionally include the device model and Android or iOS version.Related terms
Theory covered. Now route something. Start free.
Residential, ISP, mobile and IPv6 networks under one account — test the concepts on real infrastructure.
ProxyOmega