How to Fix Server Overload & Server Crashes Caused by Bad Bot Traffic [2023 Update]
Sometimes, servers crash, especially when events like Black Friday and Cyber Monday cause huge spikes in web traffic. Even the biggest e-commerce companies and enterprises—such as Lowe’s, H&M, Best Buy, and many others—struggle to keep up with major traffic spikes, and have suffered from server overloads that cost them millions of dollars in sales.
Heavy traffic is one of many issues that can cause a server to crash. In 2017, an AWS engineer accidentally mistyped a command while debugging an issue with Amazon’s S3 cloud storage service. Medium, Coursera, Quora, Slack, Docker, Expedia, and many other big web services went down for four hours. A more recent AWS outage in 2021 was caused by an issue during automated scaling, which knocked out sites like Disney+ and Netflix—as well as Internet of Things (IoT) devices like Ring doorbells and smart speakers.
But despite the ever-growing number of reasons servers can crash, customers have come to expect a seamless experience, even on the busiest online shopping days of the year. A slow website, browser time-out, or error code can quickly send your ready-to-spend customers elsewhere with their holiday list, while also impacting your brand reputation and user trust.
This article covers one particularly pernicious cause of server overload—bad bot traffic—and what you can do about it.
Bad Bots: An Often Overlooked Cause of Server Overload
Server overload is often a result of configuration issues, broken code, hosting errors, or legitimate traffic spikes during special occasions like flash sales. But a particularly nefarious cause of server overload that can be easy to miss or misdiagnose is bot traffic.
Bot traffic can adapt to look more and more like regular traffic, making it hard to spot. And while most successful online businesses have tools in place to identify and protect them against obvious DDoS attacks, today’s bots are not always obvious. Bots become more sophisticated by the second, bypassing your gen 1 defenses by mimicking human behavior. In fact, a significant portion of your traffic could be coming from bots that do nothing but scrape, steal, and cause damage.
For example, luxury homeware e-commerce site AMARA struggled with scaling their server resources to match the unpredictable traffic spikes caused by bad bots. The additional traffic slowed down their site, damaging the customer experience (CX). Once they installed DataDome, AMARA saw an overall 15% decrease in traffic—all from bots—and eliminated fraudulent traffic spikes. AMARA’s server resources are now dedicated fully to real customers, and the company spends much less time manually responding to bot traffic.
As another example, the Director of Operations and Co-Founder of Cairn.info said that malicious bot attacks on their websites were burdening their technical infrastructure, which had forced them to oversize one of their online portals. After the quick installation of DataDome, their problems were solved in less than three weeks.
How many malicious bots are part of your traffic? Most companies don’t know the exact answer. Bots are problem that increases over time, and popular cloud providers have autoscaling features. Therefore, the more traffic you get, the more servers cloud providers allocate you—and the more you pay.
Because it happens automatically, and because sophisticated bots are very hard to distinguish from real users, it’s easy to mistaken an increasing number of bots and a higher cloud bill as positive signs of user growth. But you would be mistaken.
How to Stop Bots From Overloading Your Servers
Most performance issues that are not related to bots can be resolved with a load balancer (such as AWS ELB), which will route traffic to the server where it will get the best performance while making sure no single server is overloaded with traffic. When your traffic grows, the load balancer will automatically incorporate new servers and distribute traffic appropriately.
While a load balancer can help with faster loading times and a better user experience, it does nothing to stop bots. You’ll still end up paying for more server capacity than you need.
A rules-based cybersecurity tool such as a web application firewall (WAF) will stop the most basic bots by allowing you to create request thresholds or set rules such as blocking all traffic from particular countries. But WAFs are no longer effective against today’s advanced bots. To fix a modern bot problem, you need a modern bot management solution.
DataDome’s solution is designed to detect even the most sophisticated bots. We analyze every request to your website in real time, and compare it with our massive database that processes 3 trillion signals per day. We detect malicious bots in less than 2 milliseconds and block them from accessing your online properties.
DataDome gives you tight control over which traffic you want to allow and which you want to block. For example, timeboxing is a DataDome feature that allows you to fine-tune when particular traffic is allowed and when to block it to ensure unnecessary traffic can’t disturb your endpoints during business hours.
Another feature is rate limiting, which gives you the ability to block selected traffic based on the number of hits it generates during a particular time period. Once traffic volume hits a particular threshold, it will either trigger a CAPTCHA or a hard block. This means your infrastructure resources are always reserved for people, not bots.
With DataDome, your servers will be stable, your websites will load faster and provide a better user experience (UX), and you won’t be paying for traffic you don’t want. According to Saint Gobain Distribution Bâtiment France, DataDome provided immediate ROI by saving them 12% of their infrastructure costs.
Malicious bots are an overlooked cause of server overload that leads to server instability, slower loading times, poor user experience, and a higher cloud bill. While load balancers and a good WAF can improve performance and stop the least sophisticated bots, only a good bot and online fraud protection solution will solve the problem of malicious bots altogether.
If you’d like to know more about how DataDome can help you with overloaded servers or a cloud bill that seems higher than it should be, click one of the buttons below to either start your free DataDome trial or request a demo.