Carsales Optimizes Scraper Bot Protection with DataDome

The Challenge: Ensuring optimum protection against scraper bots.

As Australia’s #1 classified ads network, the Carsales team has always been keenly aware of the need to protect its listings from web scraping bots.

“Data is our main asset,” says Ilia Frenkel, SecDevOps Engineer at Carsales. “So we are very conscious of the scraper bots that are constantly trying to steal that data, and always had a solution in place to protect it”.

For several years, Carsales used a bundled CDN service which included bot management, alongside firewalls and other security features. But as their infrastructure evolved, the CDN solution no longer satisfied all the company’s needs.

“Our requirements changed when we moved to AWS,” Ilia explains. “For example, AWS provides a web application firewall as a service, so it was much cheaper and easier to use that service rather than a third party. As a consequence, we started to chop off bit by bit from our previous solution, and we eventually decided to challenge the bot management feature as well.”

The Solution: “Best in test” bot detection capabilities.

While the AWS platform includes efficient network layer security tools, it does not provide sophisticated web scraping protection or other bot management functionalities.

“With AWS, it’s very easy to see when someone is trying to scrape you or DDoS you from a single computer. It’s obvious and easy to handle,” says Ilia. “The problems arise when attackers are more sophisticated, when they use distributed networks and smart bots. So that’s where we knew we needed additional professional help.” 

Having identified and evaluated several bot management providers, both local Australian companies and overseas vendors, the team drilled the shortlist of potential candidates down to three.

The most important selection criteria was how well the respective solutions detected malicious requests.“We literally ran our existing bot management solution in parallel with the three contenders for easy comparison, and DataDome was the clear winner,” Ilia attests.

“To illustrate, let’s say our existing solution blocked 1,000 requests. We would expect competing solutions to block at least the same 1,000. DataDome did that, plus some additional bad requests that our existing solution had let through. The other two solutions were significantly less successful.”

The Results: Scraping protection on autopilot & easy access to useful analytics data.

Today, the DataDome solution protects Carsales’s classified ads from scraper bots in autopilot mode, with no need for regular interventions from Ilia and his team.

“It’s pretty much set it and forget it,” Ilia confirms. “In the implementation phase, we had some false positives, but figuring out why and unblocking the requests was much easier than with our previous solution. And after the initial period of setting up and fine-tuning things, there’s nothing for me to do really.”

He still keeps the DataDome dashboard permanently open to make sure everything runs smoothly and to keep an eye on the traffic data, which also serves as the basis for regular reports to the upper management and boards.

“The simplicity of the dashboard is impressive: it’s very easy to see at a glance what’s going on,” Ilia concludes.