Kurt Geiger: “What’s really cool about DataDome is that it’s fire-and-forget.”

The Problem: Scraper Bots Stealing Content & Slowing the Site

The first signs of bot activity on kurtgeiger.com appeared via various monitoring and alert systems that the company’s DevOps team had implemented.

“We were starting to see increasingly frequent activity from scraper bots,” says Mugurel Margarit, DevOps Manager at Kurt Geiger. “They were indexing our content, such as product descriptions, images, and prices. We also found out that some of them were scraping prices from our site as well as from competitor sites, in order to make price comparisons.”

Not only did the unwanted visitors steal content that was meant for humans; their activity also had a negative impact on the site’s performance.

“Certain bots were a bit too aggressive,” Mugurel explains. “Our infrastructure wasn’t quite scaled to handle that kind of volume of requests. While it never got to the point where the site was taken completely down, the scrapers would cause traffic spikes that temporarily overloaded our backend system and resulted in slowdowns.”

Mugurel and his team initially used the Nginx rate limiting feature to cap the number of requests from a single IP address. They analyzed the number of requests a regular user would make when navigating the website, did the math, and fixed a per-minute rate limit.

“It was efficient enough for the least sophisticated bots, but as you can probably guess, many of the web scrapers would attack from lots of different IP addresses,” Mugurel observes. “So they could run for quite some time before they were being blocked for making excessive requests.”

The Solution: SaaS-Based Bot Protection Seamlessly Integrated With Nginx

The team decided that it was time to beef up their protective measures, and started to research the market.

“We liked the fact that DataDome offered a completely self-managed setup, which enabled us to evaluate the solution without having to go through enterprise sales,” says Mugurel. “The seamless integration with Nginx was also a major advantage.”

The team installed the DataDome module for Nginx-based websites and launched the free trial. In free trial mode, DataDome bot detection software analyzes every request to the website and shows real-time traffic data in the user dashboard, but doesn’t block any traffic.

“Even if we knew there was a lot of bot activity on our site, seeing the real volume was quite a surprise,” Mugurel recalls. “So we quickly decided to go ahead and implement the solution, and got excellent assistance from the DataDome onboarding team.”

Before activating the protection, the DevOps team worked closely with the marketing department to establish an allow list of partners and tools that would need continued access to the site. Subsequently, the allow list is very easily maintained via the user dashboard, for example if an existing partner changes its IP address or a new partner needs to be added.

The Results: Optimized Bot Management With AI & Custom Rules

The moment the team activated the protection, they could observe massive volumes of blocked traffic in the dashboard. With all that traffic no longer reaching their servers, there were suddenly much fewer spikes and less pressure on the backend systems.

“My favorite feature is the AI detection,” Mugurel reveals. “It’s very interesting to see how it blocks bots I had no idea existed. Bots are getting sneakier and sneakier, and it’s nice to know that DataDome keeps up with new threats and prevents them from hammering our website.”

The DevOps team also uses the custom rules function to block certain ecosystem players with whom they have had bad experiences, and plans to use the rate limiting function to manage access conditions for certain partners as well. But otherwise, they mostly let the DataDome solution run on autopilot.

“From an operations perspective, what’s really cool about DataDome is that it’s fire-and-forget,” says Mugurel. “Once you’ve rolled it out, you don’t actually have to manage anything, other than maybe allow-listing a new SEO agency from time to time and things like that. It doesn’t require any other intervention from us, it just works.”

In closing, is there anything Mugurel would like to share with anyone considering the DataDome solution?

“I would like to say that support is always there for them if they need it,” he answers. “It’s working really well; in the couple of instances we’ve needed it, our issue was either resolved very fast or a new feature implemented quickly, so I’m really happy with the support so far.”