Bot Protection Guide

How to Block Bad Bots on Your Website – 4 Mitigation Methods

Table of content

Bot Protection Guide

Newsletter

Stay on top of the latest threats.

Bots represent half of all web traffic. Although there are all kinds of internet bots for various purposes, a significant portion of bot traffic in Google Analytics comes from bad bots with malicious intent. That’s why excluding bot traffic for GA4 is important on top of protecting your website, app, and APIs.

While there are good bots that provide helpful services (e.g. Googlebot and Bingbot, which help get your site indexed by the two major search engines so potential customers can find you), bad bots can cause all sorts of damage to your site and business by:

Attempting a distributed denial of service (DDoS) layer 7 attack.
Scraping your site for private information that can be used illegally, such as to sell your users’ data.
Reposting your content on other sites, causing content duplication, price undercutting, and other issues.

Even good bots can potentially put an extra burden on your server resources when they are not managed properly, leading to an increased traffic load and slowing down your site’s speed. Managing and blocking bots, especially bad bots, is very important if you have a website and server. However, there are two main challenges:

We can’t simply block all bots, since there are good bots that can be beneficial.
We never want to accidentally block legitimate users by mistake.

This is why this guide explores how to block bots on your website and server effectively, as one of many bot mitigation methods.

What are bad bots?

Internet robots—or just “bots”—are automated software programs that are designed to perform relatively simple, repetitive actions over the internet. A key characteristic is that bots can perform tasks at a much faster speed than humans can, and a bot can operate 24/7 with no need for breaks or rest.

There are both good and bad bots. A good bot is typically owned by a legitimate company (e.g. Google or Facebook) and won’t hide its identity as a bot. Good bots follow the rules and policies of your website’s robots.txt file.

A bad bot, on the other hand, might try to disguise itself as a human to cause all sorts of problems.

For some types of bot attacks, such as DDoS, fraudsters may also use a botnets, or a group of devices (such as personal computers and IoT devices) that have been infected with malware and are now under the attacker’s control, essentially turning the infected devices into zombie devices.

Once a device is infected, it can infect other devices (by sending spam emails, for example), in order to grow the collection of bots in the botnet until it includes thousands, or even millions, of zombie devices.

Common Signs of Bad Bot Traffic

It is generally easier to notice the signs and symptoms of bad bots than to locate bad bots themselves. Signs include:

Sudden high traffic spikes. Bots tend to show up en masse, particularly for scraping and DDoS attacks, meaning you will see a sudden unexplained pageview spike.
Server performance issues. Because bots show up in such large volumes, your servers may not be able to handle the extra load, slowing down your website for all users. As soon as you add more server resources, more bots will be able to flood in, perpetuating the issue.
High bounce rate. Bots are programmed to achieve a goal. If that goal is met or found to be impossible, a bad bot will leave right away. Bots also operate by the millisecond, rather than the second.
Abnormal session durations. Humans tend to stay on a website for at least a few seconds, and don’t remain on one single page for more than a few minutes. Session lengths in the milliseconds or abnormally long ones may indicate bot traffic.

How Bad Bots Impact Your Websites & APIs

Bad bots are dangerous because they are specifically and carefully designed to perform malicious attacks, including brute force attacks, credential stuffing attacks, web scraping, and even large-scale DDoS attacks. Bot attacks can impact your business in many ways, including but not limited to:

Stealing Your Sensitive Data

Content scraper bots steal and reuse your content without your permission. They can also steal sensitive user data from your database if they gain access to it, which might expose you to legal penalties and impact your reputation in long term. Advanced bots can even harvest your users’ credit card information if it is not properly protected.

Slowing Down Your Site Speed

Bot activities on your site will put extra strain on your server’s resources, impacting performance and slowing down your site. Slow page speed might drive your visitors away and affect your site’s SEO performance.

Spamming Your Site With Fraud Links

Spambots can spam your forms, comment sections, and other areas on your website/platform that allow user-generated inputs. They often leave links to fraudulent/scam websites on your platform, which can easily ruin your business’ reputation, and can also get your site penalized by Google.

Skewing Your Analytics & Costs

Bots disrupt your site’s overall activities and might affect the cost of your advertising. Ad publishers might charge you a lot more for ad space because they assume you have increased traffic—even though it’s coming from bots. On the other hand, if you are a publisher and you don’t stop bad bots from reaching your site, your reputation can be damaged if your bot traffic reaches your advertisers.

Ruining Your Competitive Advantage

The harsh truth is, fraudsters might work on behalf of your competitors or sell hem your data, which can cause you to lose a valuable competitive advantage. This is especially common in industries where price and/or information can make or break the business, like ticketing e-commerce websites, hospitality websites, etc. For example, if you don’t block bad bots, they can steal your pricing data and sell it to your competitors, so your competitor can lower their prices even further to undercut your prices, eliminating your competitive advantage.

How to Block Bad Bots on Your Website, Server, & App

Today’s bots are extremely sophisticated, and hard to distinguish from real humans without proper bot detection techniques. Bad bots are not only behaving like a legitimate human visitor would, but can also use fingerprints/signatures typical of human users like a residential IP address, a consistent browser header and OS data, and other seemingly legitimate information.

Detecting Bad Bots vs. Humans & How to Tell The Difference

Based on how good they are at copying human behaviors, we can differentiate bad bots into four different groups:

Simple Bots: Access a site using automated scripts (not pretending to use a browser), and typically will only access a website from a single IP address (that is ISP-assigned). As a result, simple bots are very easy to detect with today’s anti-bot solutions.
Moderate-Level Bots: Typically make use of headless browsers (virtual software that simulates browsers) to make them look like legitimate visitors using real browsers.
Sophisticated Bots: Can imitate simple human behaviors like nonlinear mouse movements, random clicks, and so on. They also use headless browsers and/or browser automation software to fool bot management solutions.
Advanced Bots: these bots combine all the different technologies to imitate human behaviors, forge their user agents (UAs), and rotate through vast numbers of IP addresses.

However, no matter how good bots are at mimicking human traffic, they are not perfect at it. In general, we can use three main approaches to differentiate between humans and bots, and to stop bad bots:

Challenge-Based Approach: This method of blocking bad bots on your website relies on challenges and tests to filter bots from legitimate human users. CAPTCHAs are the most common examples of such tests, which were designed to be very easy for humans, but very hard if not impossible for bots to solve—although about half of bots today can bypass CAPTCHAs. Bot programmers have many tools they can use to bypass these challenges, like CAPTCHA farm services that allow hackers to pass the CAPTCHA to a human employee to solve before passing it back to the bot.
Static/Fingerprint-Based Approach: In this method, bot management software will analyze the client’s signatures and fingerprints and compare them with a known database. For example, bot management might check for OS and browser data, IP addresses, locations, and other information that can be cross-checked.
Dynamic/Behavioral-Based Approach: This method focuses on analyzing behaviors (what the bot is doing) rather than its fingerprints (what the bot is). For example, bot management will check the client’s mouse movements (human mouse movements tend to be more randomized), typing patterns, and overall activity.

A good bot management solution will combine all these approaches. In most cases, to detect and stop the most advanced bots, the detection engine will use AI and machine learning for the behavioral analysis.

Mitigating vs. Blocking Bad Bots

In managing bot activities, blocking the bot isn’t always the best approach, for two main reasons:

We want to avoid false positives (accidentally blocking human traffic) as much as possible, so we have to ensure unparalleled accuracy.
We may not want a bot to know that it has been detected and blocked.

Instead, we can use the following techniques for more granular mitigation:

1. Honey Trapping

You allow the bot to operate as usual, but you feed it with fake content/data to waste its resources and fool its operators. Alternatively, you can redirect the bot to another page that is similar visually but has thinner/fake content in it.

2. Challenging the Bot

You can challenge the bot with a CAPTCHA or with invisible tests like suddenly asking the client to move the mouse cursor in a certain way, which is going to be very difficult to solve by a bot.

3. Throttling & Rate-Limiting

You allow the bot to access the site, but slowing down its bandwidth allocation to make its operation much less efficient. The hope is that the operator will give up due to the very slow speed.

4. Blocking

There are attack vectors where blocking the bot activity altogether is the best approach—for example, if it’s very obviously spreading malware or performing a DDoS attack. Approach each bot on a case-by-case basis, and this is where having the right bot management solution can significantly help in stopping bots on your website.

Invest in a Bot Management Solution

Due to the sophistication of today’s malicious bots, having the right bot management solution that can perform behavior-based analysis is very important if you want to effectively block bots and online fraud on your website and server. DataDome utilizes AI and machine-learning technologies to detect bot activities in real-time and can mitigate malicious bot activities on autopilot.

Without a proper bot management solution for your business, detecting today’s sophisticated bot activities is extremely difficult—and expensive.

Start Protecting Your Website From Bots with DataDome

Investing in the right bot protection solution is the best approach for blocking and mitigating bots on your website, mobile app, and API. Effective bot protection will help:

Differentiate between legitimate human users and malicious bots mimicking humans to keep false positives as low as possible.
Identify the source of bot traffic and its reputation to prevent false positives.
Utilize AI technologies to analyze each bot’s behavior and make a case-by-case decision in managing these bot activities.
Allow good bots to access your site to provide their benefits, according to your preferences.

Blocking bots altogether might seem like a cost-efficient approach in many cases, but can actually be counterproductive in the long run. Persistent attackers will know right away when their bots are blocked by your security infrastructure, and might use the information to update the bot to make it even more effective at avoiding your security measures.

With bots getting more sophisticated than ever, having the right bot management strategy to block bad bots is no longer a luxury, but a necessity.