The crawler identified as `OpenSEO-Audit/1.0` with almost no headers, which
naive bot filters and security plugins block outright.
- New crawl-request.ts: AUDIT_USER_AGENT (a current Chrome string),
buildAuditHeaders() (Accept, Accept-Language, Sec-Fetch-*, Sec-Ch-Ua,
Upgrade-Insecure-Requests), and fetchForAudit() — fetch + those headers +
one retry on a transient 429/503.
- Wired into the page crawl (site-audit-workflow-helpers), robots.txt +
sitemap discovery, and start-URL redirect probing.
Gets past the naive tier; still reported as "blocked" for JS/TLS challenges
(Cloudflare Managed Challenge, DataDome) — those need a real browser. Doc note
points operators at WAF IP/UA allowlisting for their own sites.
No env dependency (keeps the audit lib importable without a cloudflare:workers
mock). tsc / oxlint / knip clean; new crawl-request.test.ts (5); suite
otherwise unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- activity_log table (sqlite + pg, structurally identical; schema-parity
covers it). Plain-text columns, no FKs — an append-only trail that must
outlive the projects/users it references, so target_label snapshots a
human-readable name at write time.
- ActivityRepository: record() (fire-and-forget, never breaks the caller) +
list() (org-scoped, actor/action filters, keyset pagination) + listActors().
- Recording wired into the mutations worth tracking: project
create/archive/restore/domain, audit start, team user create/remove/
password-reset, invitation sent.
- getActivityLog / getActivityActors server functions (owner/admin gated) +
Settings → Activity tab (ActivityLogView: filter by user & action, load
more).
- Migration: drizzle/0045_*, drizzle-pg/0023_*. The pipeline does not run
migrations — see docs/SELF_HOSTING_TEAM_MODE.md step 5 for the one-time
`drizzle-kit migrate` on the server. Writes fail silently until the table
exists.
tsc / oxlint / knip clean. New ActivityRepository.test.ts (4) + schema-parity
picks up the new table; suite otherwise unchanged (pre-existing samSkills
CRLF failure only).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Makes AUTH_MODE=team usable end to end.
- resolveTeamContext (middleware/ensure-user/team.ts): a session resolves to a
membership in the single shared workspace. No per-user fallback org — a
signed-in user with no membership is treated as signed out, so the owner can
actually remove people.
- teamProvisioning.ts: one path that writes user + credential account + member
together (hashPassword from better-auth/crypto). Shared by both entry points.
- /api/team-setup (raw route, outside auth middleware): GET reports whether an
owner is needed; POST creates the first owner + the shared org, then
self-disables once any user exists.
- /setup route + sign-in redirect: first run sends you to create the owner.
- teamUsers server functions (owner/admin-gated): list / create (with temp
password) / reset password / remove. Removal drops membership + sessions,
keeps the user row for historical attribution.
- Settings gains a "Users" tab in team mode (TeamUsers.tsx).
- docs/SELF_HOSTING_TEAM_MODE.md: activation runbook (.env, build, first owner).
No DB migration — all rows are existing better-auth tables. tsc / oxlint / knip
clean. New teamProvisioning.test.ts (4 cases) passes; suite otherwise unchanged
(pre-existing samSkills.test.ts CRLF failure only).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Site audit P0 (1/3): issue engine, incremental persistence, block detection
Server-side foundation of the P0 feature set from docs/site-audit-pm-research.md:
- Issue engine: shared registry of issue types (severity, explanation,
how-to-fix). Per-page reporters run inside crawl steps; cross-page checks
(duplicate titles/descriptions/content, broken internal links, redirect
chains/loops, orphan pages) run at finalize as SQL over the persisted crawl.
- New audit_links + audit_issues tables, audit_pages columns (depth, content
hash, header signals, fetch class, sitemap flag); audit tables moved to
src/db/{,pg/}audit.schema.ts; migrations 0029 (D1) / 0006 (PG).
- Incremental persistence: pages/links/issues written inside each crawl-batch
step with deterministic row ids + upserts (retry idempotent); slim step
state; robots.txt checkpointed as step state; merged progress steps keep a
10k-page crawl within the Workflows step budget.
- Crawler: manual redirect handling with inline follow of normalization-
equivalent redirects, response header capture (X-Robots-Tag, Link
rel=canonical), BFS depth, sitemap-last seeding, SSRF check on discovered
links, honest 'we were blocked' classification (403/429/cf-mitigated/
challenge).
- MCP: run_site_audit, get_audit_status, get_audit_issues, get_audit_pages;
limitTier resolved via shared AuditService.resolveAuditLimitTier.
- Lighthouse strategies reduced to auto/none (legacy all/manual map on read).
- Self-healing: getStatus reconciles audits whose workflow instance errored/
terminated without reaching mark-failed.
The Issues UI and the badseo.dev e2e fixture site stack on top of this PR.
Deploy notes: run db:migrate:prod (additive); terminate running audits before
deploying — the workflow step structure changed and in-flight instances cannot
replay under the new code (a finalize guard fails them loudly instead of
completing empty).
* Store only internal link edges in audit_links
Both consumers (broken-internal-link and orphan checks) filter on
isInternal; per-page external counts already live on audit_pages.
Dropping external rows cuts stored edges on outbound-heavy sites.
Column stays so P1 external-link checks can re-add rows without a
migration.
* Review fixes: failAudit CAS guard, dedupe hash helpers, cheaper checks
- failAudit only transitions running audits, so the getStatus reconciler
can't flip a just-completed audit to failed when it races finalize
- collapse the duplicate SHA-256 helper into audit/ids.ts
- finalize integrity guard uses a limit-1 existence probe instead of
fetching every page row
- get_audit_status MCP tool no longer reads the audit row twice when an
explicit auditId is given
The Worker now connects to Postgres exclusively through the HYPERDRIVE
binding. Local dev uses the binding's localConnectionString (committed in
wrangler.jsonc, pointing at the throwaway Docker Postgres from
docs/LOCAL_POSTGRES.md) instead of a POSTGRES_DATABASE_URL Worker var.
Closes the Codex finding about unpooled direct connections from deployed
Workers: the not-recommended direct-connection config is no longer possible.
POSTGRES_DATABASE_URL remains as a Node-side env var for drizzle-kit and
the D1->Postgres migration script only.
* docs: document DCR redirect-URI step for MCP behind Cloudflare Access
Enabling Managed OAuth alone leaves MCP clients unable to finish Dynamic
Client Registration when their redirect URI isn't allowlisted, so they log
in but expose no tools (issue #47). Document allowing localhost/loopback
clients and HTTPS web-connector redirect URIs, linking Cloudflare's docs.
* docs: fold DCR redirect-URI guidance into the setup steps
Move the Dynamic Client Registration guidance from a standalone paragraph
into step 6 of the numbered list, with sub-bullets for localhost/web
redirect URIs and a footnote explaining why (logs in but no tools).
* docs: replace DCR footnote with a sub-bullet
* refactor: move lighthouse audits to dataforseo (#43)
* refactor: move lighthouse audits to dataforseo
* chore: remove obsolete audit settings modal
* refactor: rename psi flows to lighthouse
* save
* refactor: simplify audit lighthouse storage flow
* fix: separate lighthouse metrics from actionable audits
* refactor: remove redundant audit project inputs
* feat: redesign lighthouse issues screen with score gauges and table layout
Replace flat score cards with circular SVG gauges, condense metrics into
a compact grid, and switch issue list from cards to an expandable table
with fixed column widths.
* test: harden lighthouse regression coverage
* fix: restore project-scoped audit inputs
* refactor: simplify lighthouse payload handling
* refactor: inline lighthouse server handlers
* refactor: share audit workflow types
* refactor: simplify lighthouse payload flows
* save
* refactor: drop project pagespeed api key
* fix: restore lighthouse issues loading with resilient project context
* fix: restore audit issues back navigation
* refactor: simplify project context and lighthouse error handling
* fix: tolerate DataForSEO lighthouse payload drift
* refactor: route audit lighthouse through dataforseo client
---------
* refactor: move DataForSEO response cache from KV to R2
KV TTL-based expiry is imprecise for cache freshness. Switch to R2 with
soft TTL via custom metadata (expiresAt) and a 7-day lifecycle rule for
cleanup. Cache objects live under the `dataforseo-cache/` prefix,
separate from durable audit payloads in `site-audit/`.
* refactor: use Workers crypto for R2 cache keys
Keep the cache helper aligned with the Cloudflare runtime and avoid pulling Worker bindings into Vitest. Also clarify that R2 lifecycle cleanup is optional but recommended to control storage growth.
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
