Rue du Commerce (Carrefour): “User Experience Optimization Requires Bot Management.”

Website always available

No more price and product sheet scraping

Ultra-fast website, #1 in web performance ranking for French sites

Retail & e-commerce Scraping

Founded in 1999, Rue du Commerce is a pioneer of e-commerce in France. Initially a high-tech specialist and market leader, the founders later decided to turn the website into the first French marketplace. Today, the website offers more than 10 million different products.

Early in 2016, Rue du Commerce was acquired by the Carrefour group. The new team aimed to stand out from the competition through operational excellence and embarked on a major project to modernize the website.

The feat required a wide range of measures, ranging from architectural restructuring to careful image and CSS optimization. But even all these efforts did not improve the traffic quality, as bad bots proved to be a serious menace to the site.

“User experience is extremely important to us. As a generalist marketplace, it’s hard to differentiate ourselves through our products. But we can be the best from a technical standpoint,” says Jean-Michel Colas, CIO at Rue du Commerce.

The Problem: Servers saturated by useless, even harmful, traffic

Despite its robust and secure infrastructure, Rue du Commerce, like any e-commerce website, was regularly facing problems with malicious bots. These bots scan every available e-commerce application (website, mobile apps, APIs) in order to retrieve prices, product descriptions and inventory statuses, or even to gain access to user data and financial transactions.

While exposing publishers to major security risks, intense bad bot activity often leads to an unavailable website (DDoS attacks).

Rue du Commerce’s IT teams and their hosts were regularly called upon to deal with ongoing attacks to stop the site from being shut down. The constant burden on servers and on-call staff finally lead to a decision to get to the root of the problem: managing automated traffic.

The analysis showed that 30% of the site’s traffic was composed of malicious bots, which represents a significant impact on the use of technical resources, and the team decided to implement a bot protection solution.

The Solution: DataDome eliminates bad bot traffic

The implementation of the DataDome solution was managed directly by Rue du Commerce’s teams, supported by Boris Trehin, Onboarding Manager at DataDome: “We start by allow-listing all the partners, and by scrubbing the lists of bad bots and commercial bots to make sure no partners will be blocked. The next step is to customize the Captchas, and finally, we decide which endpoints are to be protected,” Boris explains.

DataDome also offers the possibility to create real-time custom rules, defined by the users in order to adapt the protection to their needs. Rue du Commerce’s commercial interests being mainly in France, the company has chosen to display a Captcha to any hit coming from outside of France—all other countries being considered suspicious by default.

The IT team has also created another rule in order to allow Fasterize, a partner, to perform intensive tests on the site without ever being blocked.

The Results: Record performance & peace of mind

As soon as the DataDome protection was activated, Rue du Commerce’s bandwidth consumption dropped by 30% to 40%, thereby freeing up resources for legitimate users. Unlike human users who will consult no more than fifteen pages on average, a scraper bot will crawl all of Rue du Commerce’s millions of URLs at each visit (sometimes multiple times per day), overloading the site. When bots are blocked before entering the website, the human user experience is greatly improved.

Rue du Commerce still experiences major traffic peaks, but they are now always a result of the site’s commercial activity, such as sales periods, Christmas, and Black Friday and Cyber Monday which are also becoming increasingly popular in France. In addition to traditional analytics tools, the infrastructure team now benefits from a real-time dashboard where they can adjust their custom rules in real time and allow or deny access to the site for specific partners.

Rue du Commerce has now risen to the top of the ranking for the fastest French mobile e-commerce sites. The achievement is the result of the hard work done by the web, UX and marketing teams on the HTML code, supported by the DataDome solution which eliminates all unwanted incoming traffic.