Ahrefs Site Audit

AhrefsSiteAudit is a web crawler used by Ahrefs’ SEO auditing tool to perform in-depth technical SEO analysis on websites.

It’s crawling your site because someone is auditing your domain using Ahrefs. It collects data on page speed, broken links, duplicate content, and SEO issues.

To block the AhrefsSiteAudit bot from accessing your website, you can implement several server-side configurations that effectively prevent the bot from crawling your site. Here are some effective methods:

1. Modifying .htaccess file:
For websites running on Apache servers, you can block the bot by configuring the .htaccess file. This method involves denying access based on the User-Agent string that identifies the bot. Add the following lines to your .htaccess file:


RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} AhrefsSiteAudit [NC]
RewriteRule .* - [F,L]


This configuration uses mod_rewrite to check the User-Agent header for the presence of “AhrefsSiteAudit” and returns a 403 Forbidden status, effectively blocking access.

2. Implementing server-side firewall rules:
If you have access to your server’s firewall, you can set up rules to block incoming HTTP requests based on their User-Agent strings. This method involves identifying the IP addresses or ranges used by AhrefsSiteAudit (if consistent) or blocking by User-Agent string directly at the firewall level.

3. Using server configuration files:
For Nginx users, similar to the .htaccess method for Apache, you can block the bot by adding conditions in your server configuration:


if ($http_user_agent ~* "AhrefsSiteAudit") {
return 403;
}


This snippet checks for “AhrefsSiteAudit” in the User-Agent string and returns a 403 status code if found.

Each of these methods has its strengths and limitations, and the choice of method may depend on your specific server environment and how critical it is to block this specific bot. Always ensure that changes are tested in a staging environment before applying them to live servers to avoid unintended disruptions.