Ghost Inspector

Ghost Inspector “crawler bot” is the cloud-based browser automation used by Ghost Inspector to execute recorded end-to-end tests and optionally crawl a site (via starting URLs or sitemaps) to run those tests across discovered pages. It uses real browsers (headless Chrome) with a distinct user agent, suitable for CI/CD and production monitoring.
 
Legitimate uses
– UI/functional regression testing across pages
– Transaction monitoring (login, checkout, forms)
– Pre/post-deploy smoke tests and uptime checks
– Cross-browser/device validation and visual diffs
– SLA monitoring with alerts and artifacts (screenshots, videos)
 
Misuse risks (high level)
– Automated content scraping at scale
– Probing authentication/flows for account takeover
– Card testing via scripted checkout flows
– Inventory hoarding/denial-of-inventory
– Abuse of ad/affiliate flows or click fraud
 
Mitigations
– Robust bot detection
– Rate limiting
– Behavioral analytics
– MFA
– Anomaly monitoring

Why it’s crawling
– A third party (or your own team/partner) is running automated UI tests/monitors against your site.
– Public URLs, sitemaps, emails, or leaked/staging links may be included in those tests.
 
Potential negative impacts
– Performance/load spikes and elevated concurrency on critical flows.
– Skewed analytics and A/B results; polluted funnels and attribution.
– Form/database pollution (test accounts, carts), accidental transactions, and transactional email/SMS bursts.
– API quota/RUM/third-party billing increases; webhook churn.
– WAF/IDS noise, alert fatigue, and rate-limit contention with real users.
– Inventory/coupon/reservation locks and session exhaustion.
– Training drift in bot/fraud models due to mislabeled “automation as human” traffic.

– User-Agent filtering: At your web server/reverse proxy, block requests whose User-Agent matches “Ghost Inspector” (and optionally “HeadlessChrome”); return 403 before app code runs.
 
– IP denylist: Block Ghost Inspector’s published test-runner IP ranges at your firewall/load balancer. Keep the list updated automatically to prevent bypass.
 
– Automation detection gate: On first load, require a short-lived, HMAC-signed token (set via JS) on subsequent requests. If navigator.webdriver is true, token missing/invalid, or integrity checks fail, return 403.
 
– Strong access control: For non-public/staging/admin areas, enforce mTLS or IP allowlists, or require authenticated sessions before serving any HTML/API. This prevents third-party test infrastructure from reaching pages entirely.
 
Notes:
– Combine methods for resilience (e.g., UA + IP + token).
– Log and monitor blocks to tune false positives and maintain rules.