AI bots can be blocked by adding their user-agent name to the disallow directive in the robots.txt file.

What Happens If I Don’t Have a Robots.txt?

Search engine web crawlers will index every page on your site. This can result in irrelevant content being indexed which can negatively impact your page rankings.

What is the Difference Between Robots.txt and Meta Tags?

Robots.txt controls access to your site at a directory level. Meta tags manage crawling and indexing behavior for individual pages.

Now Live: Monetize AI Traffic With DataDome & TollBit Integration

Agentic AI Integrations & partners

The DataDome & TollBit integration is now live, and DataDome customers can activate it in minutes to start monetizing AI traffic.

The pace of AI innovation continues unabated. In 2025, AI traffic has also skyrocketed, quadrupling as a share of bot traffic to our customers’ sites in less than 9 months. To give you a sense of the scale, we now see monthly traffic totals over 1.7 billion requests from OpenAI alone. And we’re clearly just at the beginning of the curve, as AI players introduce more powerful models with new use cases.

4x growth of LLM traffic

AI traffic as a percentage of all verified bot traffic across DataDome customers during January and August 2025.

Meanwhile, AI has changed the name of the game for publishers, community sites, and e-commerce businesses. They deliver compelling content fueling customer engagement—from breaking news, community insights, and product information— but now it is often scraped for free at scale by LLM crawlers to train their AI models and LLM fetchers that provide instant query responses without consent or compensation.

The existing business model is broken, and new business models are emerging fast.

Turn AI traffic into AI revenue

Now, there’s an opportunity to capitalize in a new way – TODAY. DataDome and Tollbit announced their joint solution last month, and we’re happy to share that our integration is now live in production. Together, DataDome and Tollbit enable enterprises to detect, control, and monetize AI traffic.

Customers are responding positively, including the VP of Corporate Business Development at Mansueto Ventures, publisher of notable outlets such as Inc. and Fast Company:

"To ensure we are in control of who accesses our content, and to unlock new revenue from AI traffic to our leading online business publications, we turned to DataDome and TollBit. DataDome detects and handles the most sophisticated AI traffic, boosting our ability to monetize. In concert, TollBit manages access rights to our content by AI providers, based on rates we define, and handles payment collection. These two partners worked closely with us to deploy an elegant joint solution.”

- Patrick Hainault

VP Corporate Business Development at Mansueto Ventures, publisher of Inc. and Fast Company

DataDome has been able to detect, classify, and show agentic AI and LLM crawler traffic in our threat dashboard since early this year. Now, we’re enabling customers to configure a granular monetization response policy by AI agent or LLM crawler and follow an easy workflow to select Tollbit and set pricing, terms, and access rules.

AI agents LLM traffic in DataDome dashboard

Compliant AI agents and LLM crawlers will gain access to customer sites and brands based on their defined and agreed-upon monetization policy. Non-compliant or unknown AI agents and LLM crawlers can be allowed or blocked in real-time, depending on the parameters customers set. With detailed reporting and analytics, customers have a full picture of what’s happening.

AI agent traffic in DataDome dashboard

How it works

Detection & classification
DataDome identifies and blocks malicious bots in real time while routing compliant AI bots for evaluation.
TollBit monetization layer
Compliant bots are directed to TollBit, where you set your own pricing and rules for AI access—whether it’s pay-per-crawl or subscription-based.
Full control & insights
Through one integration, you gain granular reporting on who’s accessing your content and how, turning previously lost traffic into measurable value.

In a world where hundreds of new AI players are consuming third-party content daily, businesses need visibility, control, and a sustainable way to protect and monetize their content assets. The open web doesn’t have to mean open season on publishers.

Get started today

AI-driven traffic is here to stay, but now it can work for you instead of against you. If you’re a DataDome customer, adding TollBit to your stack takes minutes.

Fill out the form found on this page to learn more about how to turn on DataDome + TollBit.