Blocking the IP is Not Enough—How to Stop Bots on Residential IPs
Detecting bad bots is a never-ending cat-and-mouse game, as bot operators continuously look for new ways to bypass bot detection systems. While first-generation bots were using different technologies than humans and were therefore reasonably easy to detect, today’s bad bot traffic is almost indistinguishable from legitimate human traffic.
Rudimentary bots that couldn’t execute JavaScript have been replaced by sophisticated programs that leverage advanced headless browsers, such as Headless Chrome. Bots also actively lie about their fingerprint to avoid detection.
But browser technologies and fingerprints aren’t the only things bots operators have changed. In the race to bypass detection systems, they are also moving away from data center IP addresses that make them too easy to identify.
To blend even more seamlessly in with human traffic, bad bots increasingly use residential IP addresses instead of data center IPs. But how prevalent is this phenomenon exactly?
To answer this question, we took a deep dive into the traffic data we collected during the busy Christmas shopping period, from 16 to 29 December 2019. Let’s see what we found!
Residential IPs Represented Nearly 30% of Bot Requests
As more and more websites and applications are setting up some form of protection against malicious automated traffic, bot developers are turning to residential IPs to camouflage their bots as legitimate traffic.
While residential IP addresses are more expensive than data center IPs, due to a more limited supply, they can be obtained easily enough through companies such as Geosurf or Luminati that provide residential proxies.
Out of the billions of bad bot requests we registered during the 2019 end-of-year holiday period, 29.55% were using a residential IP address. This means that nearly one in three bad bots requests would pass for human traffic if you were looking at the IP address only.
We also found that 20.55% of bad bots came from an organizational IP address. For the most part, these are probably infected devices that are exploited unbeknownst to the IP address owner. Poorly secured IoT devices, for example, are very popular among bad bot operators.
The takeaway: Security solutions that rely heavily on IP reputation are no longer a match for bad bot operators, and it’s increasingly harder to block bad bots like ticket buying bots.
Why a Bot IP Blocklist Isn’t Effective
Blocking unwanted traffic based on IP reputation is no longer a viable strategy, and neither is block listing bad user agents.
Bot developers who go to the trouble of paying for residential IPs proxies are careful and motivated, and will often modify the user agent as well as the HTTP headers sent by their bots to remain under the radar.
Among the top 5 most common user agents used by bots, none belong to known bots, such as Headless Chrome or PhantomJS:
- Mozilla/5.0 (malformed user-agent)
- Mozilla/5.0 (Windows NT 10.0; Win64; x64; WOW64; rv:67.0) Gecko/20100101 Firefox/67.0 (Firefox 67 on Windows 10)
- Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36 (Chrome 78 on Windows 10)
- Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36 (Chrome 79 on Windows 10)
- Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0 (Firefox 67 on Windows 7)
Except for the first user agent on the list—which is malformed and was used in a layer 7 DDoS attack against one of our customers’ websites—all the most common user agents that were used by the residential IP bots we detected are typical of human users.
The takeaway: Don’t trust the user-agents your visitors declare, and don’t base your anti-bot strategy on user agent block-listing.
Bad Bots Using Residential IPs Target Mobile Applications
Our original top fie list included two mobile application user agents, but since these user agents make it easy to identify the targeted application, we removed them from the ranking for our customers’ privacy.
However, the fact that two of the top five bot user agents were going after mobile applications highlights another shift in the way bots are acting: mobile API endpoints are increasingly being targeted because most companies have focused their bot management technology primarily on protecting their website APIs.
The modus operandi is simple enough: First, they analyze the traffic between the mobile application and the API server. Then, by forging the same user agent and HTTP headers as those used by the actual mobile application, they gain access to the mobile application APIs without being detected.
The takeaway: Anti-bot protection is just as important on your mobile application API as on your website API.
Bad Bots Use Residential IPs Worldwide
Next, let’s take a look at the geographic distribution of bots that exploit residential IP addresses worldwide. The map below shows the origin of the residential IP addresses used by the bots we detected during the holiday period.
While our numbers are skewed towards countries where there is a higher concentration of DataDome customers, the map perfectly illustrates our point: No place is safe.
Whenever we activate our bot protection solution on a website or application in a new country, we observe bad bot traffic routed through residential IP addresses in that country.
We also observe that bots tend to use residential IPs with the same geographic origin as the target, not their own country of origin. Bots that want to scrape content from American websites will use American residential IP addresses; bots that conduct credential stuffing attacks against Australian websites use Australian residential IP addresses.
The takeaway: Block-listing traffic from countries where you do not operate is ineffective. Not only do you run the risk of blocking legitimate users who are simply traveling or using a VPN for one reason or another; this strategy also leaves you unprotected against attacks conducted via infected residential IP addresses from your home country.
Residential IP Addresses Used by Bots Often Have Multiple Targets
The residential IPs that are used by bad bots are often provided as part of proxy services or bot-as-a-service (BaaS) solutions, which means that they typically have multiple users that are targeting a range of different websites and applications.
During the two-week period we are analyzing here, we detected more than 1.2 million residential IP addresses used by bad bots that made requests to two or more websites or applications. Of these, there were more than 105,000 IP addresses that attacked five or more different targets.
While the bad bots that leverage residential IP addresses are often used for scraping purposes, our data set includes more than 100 million credential stuffing attempts from these IPs, both on websites and mobile applications. Residential IP addresses are used in all kinds of bot attacks to help bots avoid notice.
The takeaway: Bot attacks are no longer conducted by script kiddies and other amateurs. Today’s bad bots are the products of a flourishing industry with considerable human, financial and technical resources at its disposal, and efficient protection must take that into account.
How to Block Bots That Are Using Residential Proxies
Bad bots are using more and more sophisticated methods to bypass bot detection systems. They use real browsers or headless browsers with modified fingerprints, lie about their user agent, and increasingly rely on residential IP addresses located in the same country as their target to blend in with humans. As much as a third of all bad bot requests now come from residential IP addresses.
As a consequence, security strategies that used to work—rate limiting, user agent block-listing, or blocking traffic from foreign countries—are not effective anymore. Without truly expert bot detection knowledge, it is nigh impossible to efficiently protect your applications against such advanced bots. Blocking bots that are using residential proxies involves gathering a lot of data about the request, comparing fingerprints to known and unknown threats, and utilizing machine learning to grow with increasingly well-designed bots.
In essence, behavioral detection is key—to block these bots, you need to be able to analyze how they are behaving.
How DataDome Blocks Bots With Behavioral Detection
To determine whether a request from a residential IP address comes from a real human user or a bot, the DataDome bot protection solution relies on a sophisticated detection engine that makes extensive use of artificial intelligence and machine learning.
Known bots are detected via server-side fingerprinting in less than 2 milliseconds, making server-side bot detection an excellent investment. However, the real challenge is new threats, which are identified via statistical and especially behavioral detection, using data from server-side fingerprints, a JS rendering engine, SDK inputs and session tracking—all in less than a second.
DataDome is used by high-profile websites worldwide, which benefits all our customers: whenever a new bot is detected on one of the domains we protect, our algorithm updates itself so that all our customers are automatically protected against the new threat in less than 50 milliseconds. Data from an attack we detect on a German website, for instance, will help protect an American mobile application against the same threat.
Ready to try? Start your free trial today (it takes 10 minutes and you don’t need a credit card), or contact us to request a demo.
Frequently Asked Questions
Do bots have IP addresses?
Any device that can connect to the internet is assigned an IP address, and software—like bots—running on a device will generally use the same address as the device when they make requests over the internet. Different IP addresses can be utilized with a proxy connection.
How do you find out if an IP address is a bot?
There is no one set of IP addresses guaranteed to only be used by bots. However, some IP addresses are more likely to be used by bots than others—particularly any IP address included in a (free or paid) proxy network, as well as some data center and residential proxies.
What is the IP address of Google bot?
Googlebot uses a wide range of IP addresses, available as a json list (https://developers.google.com/search/apis/ipranges/googlebot.json) to allow anyone to verify the IP address of a bot accessing their website, app, or API. Before allowing a Googlebot request, make sure its IP address is on the list.
How can you detect bot traffic?
Bot traffic appears in many different ways depending on the goal of the operator. However, look for the warning signs: abnormally high page views or bounce rate, very short or long session durations, traffic spikes from unknown locations, and junk conversions. Any of these may indicate the presence of bots.
Related posts
Attacks on Sports-Betting Platforms Ramp Up Amid 2026 FIFA World Cup
Tell me more
Libération Leverages DataDome + Arc XP to Neutralize Malicious AI Scraping in <2 Milliseconds
Tell me more
Introducing Proof of Browser: How DataDome Blocked 14 Million Bypass Attempts
Tell me more
The Forrester Wave™: Bot And Agent Trust Management Software, Q2 2026: Key Findings & DataDome's Recognition as a Leader
Tell me more
