Bots and Web Crawlers list

Is your website experiencing resource drain, fake traffic, or content scraping? You're not alone. Many website owners face these challenges due to external bots and web crawlers. To help you safeguard your online presence, we've compiled essential information about the most prevalent bots and how to identify and block them.

All Bots

Bot Name	Operator	Category
AhrefsBot	Ahrefs Pte Ltd	SEO & Analytics Bots
AhrefsSiteAudit	Ahrefs	SEO & Analytics Bots
Amazonbot	Amazon.com, Inc.	E-commerce & Price Scrapers Bots
anthropic-ai	Anthropic	AI & LLM Crawlers
Applebot	Apple Inc.	Search engine crawlers
Applebot-Extended	Apple Inc.	Search engine crawlers
Baidu	Baidu, Inc.	Search engine crawlers
bingbot	Microsoft Corporation	Search engine crawlers
botify	Botify SAS	SEO & Analytics Bots
Bytespider	ByteDance Ltd.	Search engine crawlers
CCBot	CommonCrawl Foundation	Miscellaneous & unknown
ChatGPT Operator	OpenAI	AI & LLM Crawlers
ChatGPT-User	OpenAI OpCo, LLC	AI & LLM Crawlers
Claude-Web	Anthropic	Miscellaneous & unknown
ClaudeBot	Anthropic	Miscellaneous & unknown
Cloudflare Browser Rendering	Cloudflare, Inc.	Cloud & CDN Bots
Cloudflare Radar	Cloudflare, Inc.	Cloud & CDN Bots
Cookiebot	Usercentrics GmbH	Legal & Compliance Bots
DeepSeek	DeepSeek	Search engine crawlers
Detectify	Detectify AB	Security
DuckAssistBot	DuckDuckGo, Inc.	AI & LLM Crawlers
Feedfetcher-Google	Google Inc.	Content aggregators & rss
Ghost Inspector	Ghost Inspector, Inc.	Automation / Testing
Google API	Google LLC	Webhooks
Google StoreBot	Google Inc.	E-commerce & Price Scrapers Bots
Google-CloudVertex	Google LLC	AI & LLM Crawlers
Google-Extended	Google	Search engine crawlers
GPTBot	OpenAI OpCo, LLC	AI & LLM Crawlers
HetrixTools	HetrixTools Inc	Monitoring & Uptime Bots
IBM Crawler	Unknown Author	Miscellaneous & unknown
ICC Crawler	NICT	AI & LLM Crawlers
Iframely	Itteco Software Corp.	Content aggregators & rss
LinerBot	LINER	AI & LLM Crawlers
Meta-ExternalAgent	Meta Platforms	Miscellaneous & unknown
Meta-ExternalFetcher	Meta Platforms	Social Media Bots
Microsoft Preview	Microsoft	Page Preview
MistralAI-User	Mistral AI	AI Assistant
New Relic	New Relic, Inc.	Monitoring & Analytics
New York Times Newsgathering	The New York Times Company	Miscellaneous & unknown
NodePing Service	NodePing LLC	Monitoring & Uptime Bots
OAI-SearchBot	OpenAI	AI & LLM Crawlers
OhDear uptime	Oh Dear BV	Monitoring & Analytics
Outbrain	Outbrain Inc.	Advertising & Marketing
Parse.ly	Parse.ly	Monitoring & Analytics
PerplexityBot	Perplexity AI, Inc.	AI & LLM Crawlers
PetalBot	Huawei Technologies Co., Ltd.	SEO & Analytics Bots
Pingdom	SolarWinds Worldwide, LLC	Monitoring & Uptime Bots
QualifiedBot	Qualified.com, Inc.	AI & LLM Crawlers
SEBot-WA	Unknown Author	Miscellaneous & unknown
Site Improve	Siteimprove A/S	SEO & Analytics Bots
Site24x7	Zoho Corporation Pvt. Ltd.	Monitoring & Uptime Bots
Siteimprove Crawl	Siteimprove A/S	SEO & Analytics Bots
SmartologyBot	Smartology Ltd	Advertising & Marketing
Splunk Synthetic Monitoring Tool	Splunk Inc.	Monitoring & Analytics
Splunk Synthetics	Splunk Inc.	Monitoring & Analytics
StatusCake	TrafficCake Limited	Monitoring & Analytics
statuscake bot	StatusCake (TrafficCake Limited)	Monitoring & Uptime Bots
Stripe	Stripe, Inc.	Webhooks
TinEye	Idee Inc.	E-commerce & Price Scrapers Bots
Uptrends	ITRS Group	Monitoring & Analytics
Yandex.Metrica	Yandex LLC	SEO & Analytics Bots
YandexBot	Yandex LLC	Search engine crawlers
YouBot	SuSea, Inc.	Miscellaneous & unknown