Explicit, user-triggered button ("Get report from DataForSEO") shown when
the native crawler is blocked — never an automatic retry. DataForSEO's
OnPage API crawls from its own infrastructure with JS rendering, which
clears blocks a plain fetch() from our server can't.
- audits.crawlSource ("native" | "dataforseo") + dataforseoTaskId columns.
- dataforseo/onpage.ts: task_post (JS rendering + store_raw_html) / summary
polling / pages listing / raw_html retrieval. Only task_post is billed
(~$0.00125/page); the rest are free reads of already-billed results, per
DataForSEO's pricing docs.
- DataForSeoAuditService: hard quota of 5 runs per project per calendar
month, enforced server-side via the activity log (audit.dataforseo_report)
before any DataForSEO spend — a rejected 6th run never reaches the API.
- Reuses the native pipeline instead of duplicating it: extracted
buildAnalyzedPageResult() out of crawlPage() so both sources feed the same
analyzeHtml -> runPageReporters -> runMultipageChecks -> auditPages/
auditIssues path. No Cloudflare Workflow backs these audits; the existing
getAuditStatus poll (already running every 3s while "running") drives an
advance step each call instead.
- Known gap: broken-internal-link and orphan-page checks are native-only
(they read the crawl's link graph from the AuditScratchpad Durable Object,
which only the native crawl populates).
- UI: page-limit picker (25-500) + live quota display on the existing
"blocked" screen.
Migration: drizzle/0046_*, drizzle-pg/0024_*.
tsc / oxlint / knip clean. New onpage.test.ts (7) + DataForSeoAuditService
.test.ts (5, including the quota-rejection path); full suite otherwise
unchanged (1195 pass, pre-existing samSkills Windows-CRLF failure only).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Audit crawl: retry chunks conservatively and stop re-learning the window per chunk
Production audits on heavy-page sites (500-700KB/page) died with
exceededMemory in the first crawl chunk, and the single step retry re-ran
the exact same profile and died again.
- claimChunk now reports isRetry (leftover leases from a dead attempt);
a retried chunk crawls under RETRY_CRAWL_WINDOW (start 3, max 5,
halved byte budget) instead of restarting at window 10.
- The adapted window carries across chunks via durable step results, so
every ~200 pages no longer re-spikes to the initial window.
- First persist sub-batch shrinks to 5 pages so the byte bound sees the
site's page weight before a full 25-page batch is in flight.
- In-flight HTML budget halved to 8 MiB (16 MiB never constrained the
observed ~650KB pages: bound was 25, above the 20 max); growth now
requires a full 25-page sample.
- Parse-time caps: 1,000 extracted links/images per page so mega-menu and
crawler-trap pages can't bloat retained persist batches.
* Guard endWindow for instances replaying pre-deploy step results
* ci: un-export internal-only interfaces (knip)
Cap crawled HTML bodies at 2 MiB before Cheerio parsing, bulk-insert
audit links on Postgres instead of per-row Hyperdrive round trips, and
raise the Worker CPU ceiling to the 5-minute paid max for link-heavy
crawl steps.
* Site audit P0 (1/3): issue engine, incremental persistence, block detection
Server-side foundation of the P0 feature set from docs/site-audit-pm-research.md:
- Issue engine: shared registry of issue types (severity, explanation,
how-to-fix). Per-page reporters run inside crawl steps; cross-page checks
(duplicate titles/descriptions/content, broken internal links, redirect
chains/loops, orphan pages) run at finalize as SQL over the persisted crawl.
- New audit_links + audit_issues tables, audit_pages columns (depth, content
hash, header signals, fetch class, sitemap flag); audit tables moved to
src/db/{,pg/}audit.schema.ts; migrations 0029 (D1) / 0006 (PG).
- Incremental persistence: pages/links/issues written inside each crawl-batch
step with deterministic row ids + upserts (retry idempotent); slim step
state; robots.txt checkpointed as step state; merged progress steps keep a
10k-page crawl within the Workflows step budget.
- Crawler: manual redirect handling with inline follow of normalization-
equivalent redirects, response header capture (X-Robots-Tag, Link
rel=canonical), BFS depth, sitemap-last seeding, SSRF check on discovered
links, honest 'we were blocked' classification (403/429/cf-mitigated/
challenge).
- MCP: run_site_audit, get_audit_status, get_audit_issues, get_audit_pages;
limitTier resolved via shared AuditService.resolveAuditLimitTier.
- Lighthouse strategies reduced to auto/none (legacy all/manual map on read).
- Self-healing: getStatus reconciles audits whose workflow instance errored/
terminated without reaching mark-failed.
The Issues UI and the badseo.dev e2e fixture site stack on top of this PR.
Deploy notes: run db:migrate:prod (additive); terminate running audits before
deploying — the workflow step structure changed and in-flight instances cannot
replay under the new code (a finalize guard fails them loudly instead of
completing empty).
* Store only internal link edges in audit_links
Both consumers (broken-internal-link and orphan checks) filter on
isInternal; per-page external counts already live on audit_pages.
Dropping external rows cuts stored edges on outbound-heavy sites.
Column stays so P1 external-link checks can re-add rows without a
migration.
* Review fixes: failAudit CAS guard, dedupe hash helpers, cheaper checks
- failAudit only transitions running audits, so the getStatus reconciler
can't flip a just-completed audit to failed when it races finalize
- collapse the duplicate SHA-256 helper into audit/ids.ts
- finalize integrity guard uses a limit-1 existence probe instead of
fetching every page row
- get_audit_status MCP tool no longer reads the audit row twice when an
explicit auditId is given
The callers of startAudit and beginRankCheckRun were forwarding the
incoming billingCustomer object directly into workflow.create params.
At runtime the object can be an EnsuredUserContext with extra fields
beyond BillingCustomerContext, which pollutes the workflow instance's
persisted params payload. Explicitly pick the four required fields so
only the intended data is serialized into the workflow.
* refactor: rename delegated auth user table
* feat: scaffold hosted better auth setup
* feat: add hosted auth flows
* refactor: scope project access to organizations
* fix: harden hosted auth entry points
* fix: stabilize org backfills and auth state
* refactor: simplify hosted organization setup
* fix: restore hosted auth signup flow
* fix: preserve hosted workspace access
* fix: preserve hosted auth redirects
* Improve hosted auth UX: auto-redirect to sign-up, hide header on auth pages, add form placeholders, and trust portless dev origins
- Auto-redirect unauthenticated users to /sign-up in hosted mode
- Hide top nav on /sign-in and /sign-up for a cleaner auth experience
- Add input placeholders across sign-in and sign-up forms
- Make name field optional on sign-up (falls back to email username)
- Update copy: remove 'hosted' from user-facing text, rename link to 'Create account'
- Trust *.open-seo.localhost:1355 in dev mode to fix Better Auth origin rejection with portless worktrees
* Simplify hosted auth flow and remove standalone PSI
Use TanStack Form for sign-in and sign-up, make hosted unauthenticated handling redirect-focused, and inline auth route errors. Remove the leftover standalone PSI route, services, and table so PSI only exists within site audits.
* Align project auth with Better Auth organizations
* Make server function auth middleware global
* Reduce auth server function boilerplate
* delete migrations
* fix regenerated migration data backfills
* Simplify hosted auth flow and project audit scoping
* Use active project context for audit actions
* Allow hosted session project updates
* Let agent dev server inherit auth mode
* Match hosted header to gateway account menu
* Scope project session updates to active project
* Inline authenticated server function setup
* Polish header project and account controls
* restore auth generate script
* Use explicit project access in server functions
Make project-scoped server functions take projectId input and enforce ownership through shared middleware instead of session-backed current project state. Document the tradeoffs in an ADR so future changes can follow the same boundary.
* fix ci dependency detection for auth tooling
* Harden project auth in server middleware
Authorize projectId automatically in authenticated server middleware and add a requireProject guard for project-scoped handlers. This makes the auth boundary harder to bypass and removes ad hoc non-null assertions from server functions.
* Inline project id input schemas
Remove tiny shared projectId schema helpers where they were adding indirection without reducing real complexity. Keep project-scoped validation explicit at each server function boundary.
* Skip hosted backlinks access checks
* Simplify auth mode helpers
* Avoid rerunning auth server middleware
* Simplify server function scoping ADR
* Fix backlinks project scoping in hosted auth
* Refine auth route foundations
* Simplify ensure user auth resolution
Split auth-mode context resolvers into focused modules so the middleware reads as request orchestration instead of implementation details. Reuse a shared ensured-user context type across server middleware.
* Simplify hosted organization bootstrap
Use Better Auth to own hosted organization creation and membership so hosted auth only needs to resolve a default active organization. Keep delegated-mode compatibility records isolated in a separate helper.
* Clarify hosted auth and backlinks behavior
Document the hosted AUTH_MODE deploy contract and explain why hosted deployments skip manual backlinks verification. This makes the platform-managed behavior explicit in the code paths that differ from self-serve mode.
* Document hosted org creation callback
Explain why auth.ts injects createOrganization into the hosted org helper. This makes the dependency direction explicit and avoids future import cycles while keeping the helper reusable.
* Fix CI check failures
* Fix nav link prop forwarding
* save
* refactor: simplify keyword source selection flow
* fix: improve stacked keyword page layout
* fix: keep SERP tied to searched keyword
Stop silently falling back SERP lookups to related keywords and show a keyword-specific empty message so missing coverage is explicit to users.
* refactor: extract keyword controller and switch SERP to live
Move keyword research orchestration out of the route, call DataForSEO organic live SERP, and harden response normalization with runtime validation for safer typed handling.
* refactor: split keyword research page state into focused hooks
* fix: aggregate auto keyword fallback and reuse cached results
* scope keyword metrics to projectId
* refactor: modularize SEO routes and harden parsing for CI checks
* refactor: adopt zod json codecs at parse boundaries
* refactor: adopt papaparse, tldts, and zod error enums
* refactor: reorganize client and server feature layout
Group keyword UI/controller files and server repositories/services by feature while keeping serverFunctions as the stable API surface. This improves navigation and maintainability without changing runtime behavior.
* refactor: move PSI business logic into service layer
Keep serverFunctions/psi as thin transport handlers by delegating orchestration, export shaping, and source resolution to PsiAuditService. This aligns PSI with the existing service-first backend pattern without changing endpoint behavior.
* refactor: enforce 350/120 size limits across modules
* move self host
* fix: show dynamic range for search trends
Replace the static 'Past 12 months' subtitle with a computed range from the actual plotted trend points so the UI reflects data lag correctly.
* fix: bind audit workflow writes to workflow instance
* chore: make ci checks pass
