* Site audit P0 (1/3): issue engine, incremental persistence, block detection
Server-side foundation of the P0 feature set from docs/site-audit-pm-research.md:
- Issue engine: shared registry of issue types (severity, explanation,
how-to-fix). Per-page reporters run inside crawl steps; cross-page checks
(duplicate titles/descriptions/content, broken internal links, redirect
chains/loops, orphan pages) run at finalize as SQL over the persisted crawl.
- New audit_links + audit_issues tables, audit_pages columns (depth, content
hash, header signals, fetch class, sitemap flag); audit tables moved to
src/db/{,pg/}audit.schema.ts; migrations 0029 (D1) / 0006 (PG).
- Incremental persistence: pages/links/issues written inside each crawl-batch
step with deterministic row ids + upserts (retry idempotent); slim step
state; robots.txt checkpointed as step state; merged progress steps keep a
10k-page crawl within the Workflows step budget.
- Crawler: manual redirect handling with inline follow of normalization-
equivalent redirects, response header capture (X-Robots-Tag, Link
rel=canonical), BFS depth, sitemap-last seeding, SSRF check on discovered
links, honest 'we were blocked' classification (403/429/cf-mitigated/
challenge).
- MCP: run_site_audit, get_audit_status, get_audit_issues, get_audit_pages;
limitTier resolved via shared AuditService.resolveAuditLimitTier.
- Lighthouse strategies reduced to auto/none (legacy all/manual map on read).
- Self-healing: getStatus reconciles audits whose workflow instance errored/
terminated without reaching mark-failed.
The Issues UI and the badseo.dev e2e fixture site stack on top of this PR.
Deploy notes: run db:migrate:prod (additive); terminate running audits before
deploying — the workflow step structure changed and in-flight instances cannot
replay under the new code (a finalize guard fails them loudly instead of
completing empty).
* Store only internal link edges in audit_links
Both consumers (broken-internal-link and orphan checks) filter on
isInternal; per-page external counts already live on audit_pages.
Dropping external rows cuts stored edges on outbound-heavy sites.
Column stays so P1 external-link checks can re-add rows without a
migration.
* Review fixes: failAudit CAS guard, dedupe hash helpers, cheaper checks
- failAudit only transitions running audits, so the getStatus reconciler
can't flip a just-completed audit to failed when it races finalize
- collapse the duplicate SHA-256 helper into audit/ids.ts
- finalize integrity guard uses a limit-1 existence probe instead of
fetching every page row
- get_audit_status MCP tool no longer reads the audit row twice when an
explicit auditId is given
The Worker now connects to Postgres exclusively through the HYPERDRIVE
binding. Local dev uses the binding's localConnectionString (committed in
wrangler.jsonc, pointing at the throwaway Docker Postgres from
docs/LOCAL_POSTGRES.md) instead of a POSTGRES_DATABASE_URL Worker var.
Closes the Codex finding about unpooled direct connections from deployed
Workers: the not-recommended direct-connection config is no longer possible.
POSTGRES_DATABASE_URL remains as a Node-side env var for drizzle-kit and
the D1->Postgres migration script only.
* docs: document DCR redirect-URI step for MCP behind Cloudflare Access
Enabling Managed OAuth alone leaves MCP clients unable to finish Dynamic
Client Registration when their redirect URI isn't allowlisted, so they log
in but expose no tools (issue #47). Document allowing localhost/loopback
clients and HTTPS web-connector redirect URIs, linking Cloudflare's docs.
* docs: fold DCR redirect-URI guidance into the setup steps
Move the Dynamic Client Registration guidance from a standalone paragraph
into step 6 of the numbered list, with sub-bullets for localhost/web
redirect URIs and a footnote explaining why (logs in but no tools).
* docs: replace DCR footnote with a sub-bullet
* refactor: move lighthouse audits to dataforseo (#43)
* refactor: move lighthouse audits to dataforseo
* chore: remove obsolete audit settings modal
* refactor: rename psi flows to lighthouse
* save
* refactor: simplify audit lighthouse storage flow
* fix: separate lighthouse metrics from actionable audits
* refactor: remove redundant audit project inputs
* feat: redesign lighthouse issues screen with score gauges and table layout
Replace flat score cards with circular SVG gauges, condense metrics into
a compact grid, and switch issue list from cards to an expandable table
with fixed column widths.
* test: harden lighthouse regression coverage
* fix: restore project-scoped audit inputs
* refactor: simplify lighthouse payload handling
* refactor: inline lighthouse server handlers
* refactor: share audit workflow types
* refactor: simplify lighthouse payload flows
* save
* refactor: drop project pagespeed api key
* fix: restore lighthouse issues loading with resilient project context
* fix: restore audit issues back navigation
* refactor: simplify project context and lighthouse error handling
* fix: tolerate DataForSEO lighthouse payload drift
* refactor: route audit lighthouse through dataforseo client
---------
* refactor: move DataForSEO response cache from KV to R2
KV TTL-based expiry is imprecise for cache freshness. Switch to R2 with
soft TTL via custom metadata (expiresAt) and a 7-day lifecycle rule for
cleanup. Cache objects live under the `dataforseo-cache/` prefix,
separate from durable audit payloads in `site-audit/`.
* refactor: use Workers crypto for R2 cache keys
Keep the cache helper aligned with the Cloudflare runtime and avoid pulling Worker bindings into Vitest. Also clarify that R2 lifecycle cleanup is optional but recommended to control storage growth.
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
