The crawler identified as `OpenSEO-Audit/1.0` with almost no headers, which
naive bot filters and security plugins block outright.
- New crawl-request.ts: AUDIT_USER_AGENT (a current Chrome string),
buildAuditHeaders() (Accept, Accept-Language, Sec-Fetch-*, Sec-Ch-Ua,
Upgrade-Insecure-Requests), and fetchForAudit() — fetch + those headers +
one retry on a transient 429/503.
- Wired into the page crawl (site-audit-workflow-helpers), robots.txt +
sitemap discovery, and start-URL redirect probing.
Gets past the naive tier; still reported as "blocked" for JS/TLS challenges
(Cloudflare Managed Challenge, DataDome) — those need a real browser. Doc note
points operators at WAF IP/UA allowlisting for their own sites.
No env dependency (keeps the audit lib importable without a cloudflare:workers
mock). tsc / oxlint / knip clean; new crawl-request.test.ts (5); suite
otherwise unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
17dfca406f