Bots, Web Crawlers and AI Agents list

Is your website experiencing resource drain, fake traffic, or content scraping? You're not alone. Many website owners face these challenges due to external bots, web crawlers and AI agents. To help you safeguard your online presence, we've compiled essential information about the most prevalent bots and how to identify and block them.

All Bots

Bot Name	Operator	Category
(Bing) MicrosoftPreview	Microsoft	Page Preview
Ahrefs Site Audit	Ahrefs	SEO & Analytics Bots
AhrefsBot	Ahrefs Pte Ltd	SEO & Analytics Bots
Amazon Route53 HealthCheck	Amazon Web Services, Inc.	Monitoring & Uptime Bots
Amazonbot	Amazon.com, Inc.	AI training crawlers
anthropic-ai	Anthropic	AI & LLM Crawlers
APIs-Google	Google LLC	Webhooks
Applebot	Apple Inc.	Search engine crawlers
babbar.tech	Babbar (Babbar SAS), France	SEO & Analytics Bots
Baidu	Baidu, Inc.	Search engine crawlers
Bingbot – BingPreview	Microsoft Corporation	Search engine crawlers
Botify	Botify SAS	SEO & Analytics Bots
Braintree	PayPal, Inc. (Braintree)	Webhooks
Brevo	Brevo SAS (formerly Sendinblue), Paris, France	Miscellaneous & unknown
BrightEdge Crawler	BrightEdge Technologies, Inc., San Mateo, California	SEO & Analytics Bots
Bytespider	ByteDance Ltd.	AI training crawlers
CCBot	CommonCrawl Foundation	AI training crawlers
ChatGPT browser extension	OpenAI	Agentic browsers
ChatGPT Operator	OpenAI	AI & LLM Crawlers
ChatGPT-User	OpenAI OpCo, LLC	AI assistants
Claude Desktop browser	Anthropic	Agentic browsers
Claude-User	Anthropic, PBC	AI assistants
ClaudeBot	Anthropic	AI training crawlers
Cloudflare Browser Rendering	Cloudflare, Inc.	Cloud & CDN Bots
Cloudflare Radar	Cloudflare, Inc.	Cloud & CDN Bots
CookieBot	Usercentrics GmbH	Legal & Compliance Bots
DeepSeek	DeepSeek	AI assistants
Detectify	Detectify AB	Security
Dots	OpenAI	Agentic browsers
DuckAssistBot	DuckDuckGo, Inc.	AI assistants
FeedFetcher-Google	Google Inc.	Content aggregators & rss
Ghost Inspector	Ghost Inspector, Inc.	Automation / Testing
Google API	Google LLC	Webhooks
Google StoreBot	Google Inc.	E-commerce & Price Scrapers Bots
Google-CloudVertexBot	Google LLC	AI search crawlers
GPTBot	OpenAI OpCo, LLC	AI training crawlers
HetrixTools	HetrixTools Inc	Monitoring & Uptime Bots
IBM Crawler	Unknown Author	Miscellaneous & unknown
ICC-Crawler	NICT	AI training crawlers
Iframely	Itteco Software Corp.	Content aggregators & rss
LinerBot	LINER	AI search crawlers
Meta-ExternalAgent	Meta Platforms	AI training crawlers
Meta-ExternalFetcher	Meta Platforms	AI assistants
MistralAI-Index	Mistral AI	AI search crawlers
MistralAI-Training	Mistral AI	AI training crawlers
MistralAI-User	Mistral AI	AI assistants
Muse	Meta	Agentic browsers
New Relic	New Relic, Inc.	Monitoring & Analytics
New York Times Newsgathering	The New York Times Company	Other
NodePing Service	NodePing LLC	Monitoring & Uptime Bots
OAI-AdsBot	OpenAI	AI & LLM Crawlers
OAI-SearchBot	OpenAI	AI search crawlers
OhDear uptime	Oh Dear BV	Monitoring & Analytics
Outbrain	Outbrain Inc.	Advertising & Marketing
Parse.ly	Parse.ly	Monitoring & Analytics
PerplexityBot	Perplexity AI, Inc.	AI search crawlers
PetalBot	Huawei Technologies Co., Ltd.	AI search crawlers
Pingdom	SolarWinds Worldwide, LLC	Monitoring & Uptime Bots
QualifiedBot	Qualified.com, Inc.	AI search crawlers
SEBot-WA	Unknown Author	Miscellaneous & unknown
Site Improve	Siteimprove A/S	SEO & Analytics Bots
Site24x7	Zoho Corporation Pvt. Ltd.	Monitoring & Uptime Bots
Siteimprove Crawl	Siteimprove A/S	Search Engine Optimization
SmartologyBot	Smartology Ltd	Advertising & Marketing
Splunk Synthetic Monitoring Tool	Splunk Inc.	Monitoring & Analytics
Splunk Synthetics	Splunk Inc.	Monitoring & Analytics
StatusCake	TrafficCake Limited	Monitoring & Analytics
StatusCake Pagespeed	StatusCake (TrafficCake Limited)	Monitoring & Uptime Bots
Stripe Webhook	Stripe, Inc.	Webhooks
TinEye	Idee Inc.	E-commerce & Price Scrapers Bots
Uptrends	ITRS Group	Monitoring & Analytics
YandexBot	Yandex LLC	Search engine crawlers
YandexMetrica	Yandex LLC	SEO & Analytics Bots
YouBot	SuSea, Inc.	AI & LLM Crawlers