How to Allow Good Bots Through Bot Protection With DataDome

When you use a bot detection system, the goal is to:

1. Block bad bots.
2. Allow humans in.
3. Authenticate, allow, and manage good bots/partners.

Blocking bad bot traffic without impacting the human user experience (UX) is a common and frequently discussed challenge. However, properly handling good bots and partner traffic is also extremely important—since blocking good traffic sources can impact your business operation. At DataDome, we emphasize the importance of identifying good bots and partners vs. bad bots, and we ensure our solution is equipped to tell the difference.

How to Authenticate Good Bots

The most important thing to know when it comes to authenticated good bots is to use a safe authentication mechanism. 

Using only user-agent for authentication is unsafe because it exposes you to attackers, who have been known to forge user-agents and pretend to be good bots in order to access your site. For example, a lot of attackers forge their user-agent to match the one used by Googlebot in the hopes of having their request allowed.

You can use different techniques to safely authenticate good bots and your automated partner traffic. Examples of authentication techniques include a token in a header, a list of IP addresses or IP ranges, as well as reverse DNS.

How DataDome Authenticates Good Bots & Automated Partner Traffic

DataDome has a database of thousands of good bots, mostly marketing and SEO bots such as Googlebot, Bingbot, Semrush scraper, etc. During the free trial and throughout the onboarding phase, DataDome only labels and classifies your traffic—it doesn’t block or challenge it.

During onboarding, support will help you properly configure your different endpoints, and analysts will review your traffic to fine-tune detection if needed. We also use this period to identify partner and good bot traffic and ensure your partners are properly allowed by classifying them as good bot/automated partner traffic.

In the DataDome dashboard, you can see the response type for the good bots, partner bots, and marketing/SEO bots. Some of them are allowed by default, like Googlebot, which benefits websites by indexing their content.

Other SEO bots, like Semrush and Ahrefs, are initially blocked by default. However, if either of those is a partner or a tool you use, DataDome users can use your dashboard to switch your rules to allow Semrush and/or Ahrefs. 

Also, if you want to allow certain bots to reach your site only at certain times, you can use DataDome’s timeboxing feature, also available in the dashboard.

What if DataDome doesn’t know the tool/product I’m using?

If your tool is completely internal and unique, we can help you to create a custom rule to allow your traffic safely.

Otherwise—if it’s a generic tool—our cybersecurity analysts will add it to our known good bot database to automatically authenticate the bot safely, using the up-to-date IP address or reverse DNS.