If your site’s bot protection blocked our scan, create a verification header on your site’s page in BearScan and add one firewall rule that allows requests carrying it. We send it on every request for your site, and nobody else has it. A page we couldn’t reach is reported as blocked, never as a pass.
How BearScan identifies itself
- Page fetches, which find the pages to check, send the same Chrome User-Agent the browser does. robots.txt rules for
BearScanor*still apply to us. - Browser renders run Chrome in our own Cloudflare Container, one per scan, and present as the Chrome a visitor runs: the same User-Agent and client hints, so the report shows the page your visitors get. To pick them out of your traffic, use the verification header below.
- On your homepage the browser answers your cookie banner the way a visitor would: once refusing and, in a fresh browser, once accepting. It also loads the page with a
Sec-GPC: 1Global Privacy Control signal. That’s how we check what each choice changes. - With a verification header set, every request to your site also carries
X-BearScan-Verify: bsv_…. A rendered page’s requests to other hosts, such as analytics or a CDN, never get it. - We don’t publish IP addresses to allow. Our requests leave Cloudflare’s network from addresses that change, so an IP rule would break without warning.
1. Create the header
Sign in, open your site in BearScan, and under Scanner access choose Create header. Copy the value; it’s shown once. Treat it like a password: anyone who has it can get past the rule you’re about to add.
2. Allow it in your firewall
Allow BearScan in Cloudflare
In Security, add a custom rule. Expression: (http.request.headers["x-bearscan-verify"][0] eq "bsv_…"). Action: Skip, and choose the bot protections and managed rules that blocked the scan. Put it first in the list.
Allow BearScan in Vercel Firewall
Add a custom rule: if Request Header X-BearScan-Verify equals your value, then Bypass. Vercel lets verified bots through its bot protection by default, but a challenge-mode rule or attack mode can still stop us.
Allow BearScan in Akamai Bot Manager
Create a custom-defined bot whose condition is the request header X-BearScan-Verify matching your value, put it in a custom bot category, and set that category’s action to Allow.
Allow BearScan on Netlify and other hosts
If an edge function or WAF rule blocks bots, exempt requests whose X-BearScan-Verify header matches your value. The same idea works in any firewall that can match a request header.
3. Rerun the scan
Pages that were blocked are listed in the report’s coverage section. Rerun from the site page and they’re checked like any other.
How BearScan limits its crawl of your site
The crawler follows your robots.txt. We scan at most three pages at a time, so a scan looks like a few visitors, not a load test. To stop us entirely, remove the header or disallow BearScan in robots.txt.