AI bots can be blocked by adding their user-agent name to the disallow directive in the robots.txt file.

What Happens If I Don’t Have a Robots.txt?

Search engine web crawlers will index every page on your site. This can result in irrelevant content being indexed which can negatively impact your page rankings.

What is the Difference Between Robots.txt and Meta Tags?

Robots.txt controls access to your site at a directory level. Meta tags manage crawling and indexing behavior for individual pages.

The Real Price of “Free” Bot Management: A Publisher’s $75,000 Lesson

Bot management Media & entertainment

When evaluating bot management solutions, it’s tempting to go with the cheapest option. Free CAPTCHAs and low-cost CDN add-ons look great on paper. But what you don’t see are the hidden costs that pile up fast.

A publisher came to DataDome after 18 months of fighting bot traffic with a budget-conscious approach. They had half a dozen digital properties to protect and chose what seemed like the smartest path: deploy the lowest-cost available solutions.

Their stack:

CAPTCHA on key forms
Third-party security provider as their primary defense
In-house rules management
Ad-hoc support from their security provider

The pain points quickly arose:

Bot spam flooded their forms
Newsletter engagement metrics became meaningless as bots skewed click rates and open tracking
Volumetric attacks on their homepage drove infrastructure costs through the roof
Performance degraded under sustained bot traffic and negatively impacted UX
Their small engineering team spent hours every week playing whack-a-mole

They quickly discovered that CAPTCHAs aren’t the deterrent they used to be. In this article, we’ll walk you through the true price of budget bot protection and its hidden costs.

CAPTCHA costs scale fast

Let’s start with the obvious cost: CAPTCHA licensing. This company had ~100 million monthly page views across their properties with CAPTCHA deployed on contact forms, newsletter signups, and comment submissions.

While a certain amount of their CAPTCHA use was free, they were quickly met with additional fees when their traffic scaled. Their annual CAPTCHA fees reached $19,000+ and failed to fix their bot problem. And that’s just one cost center.

While some CAPTCHA providers offer free tiers, free protection often means limited sophistication against evolving attack vectors.

Engineering hours quickly add up

This company had three engineers managing their security posture. Each spent an average of 3-4 hours per week on bot mitigation:

Tuning rules that weren’t working
Chasing support teams across multiple vendors
Bridging emergency calls during volumetric attacks
Troubleshooting false positives
Monitoring dashboards manually

The math adds up fast:

3 engineers × 3.5 hours/week = 10.5 hours weekly
10.5 hours × 4.33 weeks/month = 45.5 hours monthly
Average DevOps/Security Engineer rate: ~$67/hour

That’s nearly $3,048 a month in labor costs. That’s engineering time that could go toward product development, infrastructure optimization, or revenue-generating projects.

Bot traffic inflates infrastructure spend

This company had another hidden cost that wasn’t being tracked: bot traffic consuming expensive infrastructure resources.

Industry benchmarks put average infrastructure cost at $0.50 to $1.50 per thousand requests when you factor in bandwidth, compute, and database overhead.

Their reality:

100M monthly page views
Estimated 25-30% bot traffic = ~27M bot requests/month
Infrastructure cost per thousand requests: ~$1.20
Requests hitting origin servers (with database/compute overhead): ~1.35M/month (~5% of bot traffic)

That means their bot problem was costing them ~$1,620 a month in additional infrastructure costs.

That’s not a line item in anyone’s budget. After deploying DataDome, they blocked malicious bots that made up 40% of their traffic, recovering ~$650/month in infrastructure costs.

Legacy solutions struggle with sophisticated threats

Many traditional security providers excel at infrastructure delivery but face inherent challenges with advanced bot detection. These solutions often rely on:

Signature-based detection that sophisticated attackers easily bypass
Static rule sets that can’t adapt to rapidly evolving threats
Limited visibility into behavioral patterns across attack vectors
Reactive approaches that require manual intervention

Modern bot attacks use advanced evasion techniques—residential proxies, browser automation frameworks, and AI-powered tools—that legacy security approaches weren’t designed to handle. While these providers offer value in their core competencies, sophisticated bot mitigation requires specialized, behavior-based detection that adapts in real time.

Poor detection exposes the business

Beyond the above costs, insufficient bot protection exposed the publisher to even greater business risks that they couldn’t see—putting revenue, user trust, and data security on the line.

Here are some of the additional vulnerabilities they were exposed to:

Account takeovers: Weak credential stuffing protection left user accounts and sensitive data exposed to automated attacks.
Skewed analytics: Bot traffic distorted website metrics, making it impossible to understand real user behavior or make data-driven business decisions.
Ad fraud: Without effective bot detection, they were vulnerable to bots artificially inflating ad impressions and clicks, which drains ad budgets and erodes trust in ad inventory.
Poor user experience: Aggressive CAPTCHAs risked frustrating legitimate users, all while sophisticated bots could still slip through.
Revenue loss: Every gap in protection across the publisher’s endpoints created opportunities for bots to scrape content, commit fraud, or degrade performance—all direct threats to the business’s bottom line.

Proper bot protection ensures you’re protected against these threats before they can do damage to your business.

What changed with DataDome

Within the first month of switching to DataDome, the publisher saw the following results:

Eliminated bad bots that comprised 40% of total site traffic
Zero engineering hours spent on daily bot mitigation
All 6 properties fully protected with consistent policies
Sub-2ms detection latency with no performance impact
24/7 SOC & dedicated TAM

The publisher went from reactive whack-a-mole to proactive protection. Their DataDome technical account manager acts as an extension of their team, providing proactive monitoring, threat briefings, and strategic recommendations.

Plus, DataDome’s threat research team constantly builds new AI detection models as evasion techniques emerge. Threat intelligence aggregated and de-identified across DataDome’s customer base automatically strengthens detection for all customers, turning attack data into shared defense.

The real question isn’t price—it’s total cost

What might appear to be a cost-effective solution to your bot problem can quickly snowball into a large list of costs. The worst part? Low-cost, patchwork solutions can be ineffective—so you’re paying for a problem that’s not actually being solved.

The publisher’s total hidden costs:

Cost of ineffective solution:

CAPTCHA costs: $1,608/month ($19,296/year)
FTE time: $3,048/month ($36,576/year)
Infrastructure waste: $1,620/month ($19,440/year)
Total quantified cost: $6,276/month ($75,312/year)
Solution effectiveness: Insufficient for sophisticated attacks

DataDome:

Fully managed protection with guaranteed SLA
Zero FTE hours required for daily operations
Infrastructure costs reduced by ~$650/month ($7,800/year)
24/7 expert support with proactive threat monitoring
Transparent pricing based on your business needs
Solution effectiveness: 99.9% detection rate

What to consider before committing based on price

Bot protection is not optional, especially as bots and AI agents make up a growing share of internet traffic. But effective protection isn’t about blocking everything automated—good AI agents enable agentic commerce, while legitimate bots keep your site visible in search results.

The challenge is distinguishing these helpful tools from malicious bots and spoofed AI agents. You need a solution that gives you control through intent-based detection—analyzing the why behind each request to distinguish harmful threats in real time.

The difference between a “cheap” solution and an effective one isn’t just the invoice amount. It’s whether your team can focus on their actual jobs, whether your infrastructure costs stay predictable, and whether you’re actually protected when attackers evolve their tactics.

Ask yourself:

What’s the true cost when you add FTE time, infrastructure waste, and business risk?
Does this solution actually work against modern bypass techniques?
How quickly can you get expert help when something goes wrong?
How much time is your team spending on security instead of building?

Don’t be blind to the true cost. Do the math, factor in everything, and then make the call.