* Site audit P0: issue engine, incremental persistence, block detection
Implements the P0 feature set from docs/site-audit-pm-research.md:
- Issue engine: 24 issue types (shared registry with severity,
explanation, how-to-fix). Per-page reporters run inside crawl steps;
cross-page checks (duplicate titles/descriptions/content, broken
internal links, redirect chains/loops, orphan pages) run at finalize
as SQL over the persisted crawl.
- New audit_links + audit_issues tables, audit_pages columns (depth,
content hash, header signals, fetch class, sitemap flag); audit
tables moved to src/db/audit.schema.ts.
- Incremental persistence: pages/links/issues written to D1 inside
each crawl-batch step with deterministic row ids + upserts (retry
idempotent); slim step state; robots.txt checkpointed as step state
for deterministic replay; merged progress steps keep a 10k-page
crawl within the Workflows step budget.
- Crawler: manual redirect handling with inline follow of
normalization-equivalent redirects (slash-canonical sites), response
header capture (X-Robots-Tag, Link rel=canonical), BFS depth,
sitemap-last seeding, SSRF check on discovered links, honest
"we were blocked" classification (403/429/cf-mitigated/challenge).
- UI: Issues tab (default) with severity grouping, per-type
explanations, drill-down, CSV/JSON/Sheets export, blocked banner.
- MCP: run_site_audit, get_audit_status, get_audit_issues (severity-
sorted, how_to_fix per issue), get_audit_pages.
- Lighthouse strategies reduced to auto/none (legacy all/manual map on
read); auto stays 10 URLs x 2 = 20 checks.
- Self-healing: getStatus reconciles audits whose workflow instance
errored/terminated without reaching mark-failed.
Deploy notes: run db:migrate:prod (additive migration 0022); terminate
running audits before deploying - the workflow step structure changed
and in-flight instances cannot replay under the new code (a finalize
guard fails them loudly instead of completing empty).
* feat(onboarding): hide agent chat step; subscribe after intro steps (#312)
* feat(onboarding): hide agent chat step; subscribe after intro steps
Remove the hosted-only strategy-chat diversion from the onboarding
sequence. After the three intro questions, hosted users now hit the
subscribe paywall directly, then return to the GSC and MCP connect
steps. The chat route and components stay in place but unlinked, to be
revisited later. Preserve the post-payment 'You're in!' interstitial by
carrying checkout=success through validateSearch.
* fix(onboarding): set checkout=success from subscribe route, not speculatively
The previous redirect baked checkout=success into the onboarding return
URL at the point needsSubscription is true — i.e. before the user had
paid. It only worked because the subscribe route gates its redirect on
actual access. Move the marker to the subscribe route's redirect-to-app
path, where checkoutCompleted reflects a real returned-from-Stripe
payment, so the 'You're in!' screen can never show pre-payment.
* website: change link
* fix(rank-tracking): unarchive config when re-adding an archived domain (#313)
* Unify dual-backend DB layer (D1 default + Postgres opt-in) (#238)
* D1 → Postgres data migration (ETL + runbook) (#274)
* Fix Postgres-only rank-tracking & site-audit workflow failures (#317)
* rank-tracking: raise per-project config limit from 20 to 100 (#318)
The cap was only a soft guard against runaway scheduled DataForSEO
workload, not a hard product constraint. Bump it to 100 so projects
tracking many domain/location combos aren't blocked.
Co-authored-by: Claude <noreply@anthropic.com>
* fix(db): add missing indexes and drop redundant ones (#319)
Postgres advisor flagged seq-scans and redundant indexes across both
backends (D1 + Postgres):
- add projects(organization_id) — org-scoped project listings seq-scanned
- add account(account_id, provider_id) — better-auth sign-in lookup
- add verification(expires_at) — expired-token cleanup range scan
- drop saved_keyword_tag_assignments_keyword_idx — covered by unique
(saved_keyword_id, tag_id) prefix
- drop rank_snapshots_run_idx — covered by unique
(run_id, tracking_keyword_id, device) prefix
Mirrored in both schema dialects + parity-test required-index guard.
* refactor(keywords): unify keyword-metric fetching behind one helper (#320)
* Fix production errors: onboarding crash hardening + DataForSEO spend/noise cleanup (#282)
* fix(ai-search): use valid Claude model_name and fail fast on unknown ones (#323)
DataForSEO dropped the Claude Sonnet 4.0 family from its llm_responses
catalog, so model_name=claude-sonnet-4-0 was rejected with 'Invalid
Field: model_name' while still billing the failed task. Point Claude at
claude-sonnet-4-5 and validate every model_name against DataForSEO's
accepted catalog before dispatching the paid call.
* fix(mcp): 405 the standalone GET SSE stream to stop /mcp OOM (#325)
The stateless MCP server returns JSON on POST (enableJsonResponse) and
pushes no server-initiated messages, so the optional standalone GET SSE
stream serves no purpose. Left enabled, each GET holds an SSE stream open
indefinitely (25s keepalive, no eventStore) and pins a fresh per-request
McpServer (~5MB of tools + Zod schemas); a few dozen concurrent connected
clients exceed the 128MB isolate limit. This was 100% of the /mcp
exceededMemory OOMs (GET only; POST never OOMed).
Return 405 (spec-compliant 'no standalone stream') before building the
server, so GET allocates nothing. Also removes the bulk of the elevated
GET canceled / responseStreamDisconnected outcomes.
* Re-add free plan as the floor; remove subscribe gate (#321)
* Pin production to Postgres via committed Hyperdrive binding (#329)
* Add Cloudflare Turnstile captcha on email signup (#326)
* Triage production log errors: audit crash, Autumn webhook FK, PostHog capture, auth rate-limit IP, log noise (#327)
* Add badseo.dev: a test site of deliberate SEO mistakes
An open-source Cloudflare Worker that serves ~27 pages, each breaking one
common technical-SEO rule (missing title, redirect loop, orphan page, thin
content, and so on). It doubles as the end-to-end fixture for the OpenSEO
site audit: every page declares the audit issues it should trigger, and
scripts/run-audit.ts drives the real audit engine against a running copy to
check that it does (36/36 checks, 25/25 issue types).
Styled to match the OpenSEO marketing site (web/). Maintained-by-OpenSEO
badge links back to openseo.so.
* badseo.dev: logo in pill, footer/hover polish, SEO-optimized titles
- Use the OpenSEO pine-tree logo (downscaled, base64-embedded, served at
/openseo-logo.png) in a light chip inside the badge, replacing the ◎ glyph.
- Footer band now fills to the bottom of the page (dropped the mismatched
body padding strip) with room for the floating badge.
- Index rows: remove the stray full-row underline and the stark white hover
box; hover is now a soft cream tint with the name underlined.
- Drop the "Maintained by OpenSEO" hero eyebrow; new H1 "A website
demonstrating common technical SEO problems" and a cleaner subtitle.
- Optimize homepage + catalog <title>/meta around real keywords from OpenSEO
keyword research (technical seo issues KD25/vol170; technical seo checklist
KD16/vol390), keeping meta lengths within limits.
* Site audit P0 (1/3): issue engine, incremental persistence, block detection
Server-side foundation of the P0 feature set from docs/site-audit-pm-research.md:
- Issue engine: shared registry of issue types (severity, explanation,
how-to-fix). Per-page reporters run inside crawl steps; cross-page checks
(duplicate titles/descriptions/content, broken internal links, redirect
chains/loops, orphan pages) run at finalize as SQL over the persisted crawl.
- New audit_links + audit_issues tables, audit_pages columns (depth, content
hash, header signals, fetch class, sitemap flag); audit tables moved to
src/db/{,pg/}audit.schema.ts; migrations 0029 (D1) / 0006 (PG).
- Incremental persistence: pages/links/issues written inside each crawl-batch
step with deterministic row ids + upserts (retry idempotent); slim step
state; robots.txt checkpointed as step state; merged progress steps keep a
10k-page crawl within the Workflows step budget.
- Crawler: manual redirect handling with inline follow of normalization-
equivalent redirects, response header capture (X-Robots-Tag, Link
rel=canonical), BFS depth, sitemap-last seeding, SSRF check on discovered
links, honest 'we were blocked' classification (403/429/cf-mitigated/
challenge).
- MCP: run_site_audit, get_audit_status, get_audit_issues, get_audit_pages;
limitTier resolved via shared AuditService.resolveAuditLimitTier.
- Lighthouse strategies reduced to auto/none (legacy all/manual map on read).
- Self-healing: getStatus reconciles audits whose workflow instance errored/
terminated without reaching mark-failed.
The Issues UI and the badseo.dev e2e fixture site stack on top of this PR.
Deploy notes: run db:migrate:prod (additive); terminate running audits before
deploying — the workflow step structure changed and in-flight instances cannot
replay under the new code (a finalize guard fails them loudly instead of
completing empty).
* Site audit P0 (2/3): Issues tab UI
- Issues tab (new default) with severity grouping, per-type explanations and
how-to-fix, drill-down to affected pages, CSV/JSON/Sheets export, and the
'we were blocked' banner when the crawl was challenged.
- Tabs always render (Issues/Pages, Performance when Lighthouse ran);
audit route search schema gains the issues tab and defaults to it.
Stacks on claude/audit-p0-server (issue engine + persistence).
* badseo.dev: render the badge logo as a white tree, no chip
The silver source logo was invisible on the dark pill, so it sat in a white
chip. Render it white via a CSS filter instead, so the tree fills the pill
with no backing background.
* badseo.dev: add build (typecheck) step before deploy
- Add 'build'/'typecheck' scripts (tsc --noEmit); 'deploy' now runs the build
before wrangler deploy.
- Scope the tsconfig typecheck to the Worker source (src/); the e2e harness in
scripts/ imports the main app and is run with tsx from the repo root.
- Document the deploy flow and first-time custom-domain setup in the README.
* badseo.dev: add trailing-slash redirect-cycle fixture + regression test
Reproduces the 508 "Loop Detected" class of bug from every-app/open-seo#61: a
CMS-style page whose canonical URL ends in a trailing slash, with the non-slash
form 301-redirecting to it. A crawler that strips trailing slashes turns the
canonical /foo/ back into /foo, follows the 301 to /foo/, strips it again, and
loops.
- New fixture at /redirect/trailing-slash: the non-slash form (intercepted in
index.ts on the raw path) 301s to the slash form, which is served as the
canonical 200.
- Harness asserts the page is crawled exactly once as a 200 with NO redirect
loop, plus a dedicated "Trailing-slash cycle -> 200, no loop" guard.
Verified the guard bites: temporarily disabling crawlPage's slash-canonical
inline-follow makes both checks fail (redirect-loop, status 301); with it in
place the harness is 38/38, 25/25 issue types.
* Add webapp-testing skill (installed via /reload-skills)
Vendors the anthropics/skills webapp-testing toolkit: real files under
.agents/skills/webapp-testing, a symlink from .claude/skills/, and skills-lock.json
pinning the source + hash. Matches how the other project skills are tracked.
* Site audit: redesign issues tab as grouped table + calmer page header
- Issues: single bordered table with severity sections (Critical/Warning/Info
headers carry the counts), dot indicators instead of filled pills, plain
right-aligned page counts, all rows collapsed by default; expanded rows get
a severity-colored left rule
- Removed the dead severity-count chips (they looked like filters but were
inert spans)
- Header: audited hostname is now the H1 with the status badge inline
- Blocked banner: compact tinted panel instead of a full-size alert
- Stats: hairline strip instead of four separate cards; issues stat shows a
severity breakdown, Lighthouse tile hidden when no tests ran, dropped the
orange issues-count coloring
* audit: fix trailing-slash redirect cycle at the root (preserve slashes)
Replaces the crawlPage inline-follow workaround with the root-cause fix, so we
don't carry two fixes for the same bug (every-app/open-seo#61).
- normalizeUrl: stop stripping trailing slashes. A trailing slash is the
canonical form on most CMSes, which 301 the non-slash version to it. Stripping
rewrote the canonical URL into its own redirect source and looped (508). Now
/path and /path/ are distinct and the redirect resolves normally.
- crawlPage: remove the isSelfAfterNormalization inline-follow (+ now-unused
resolveRawUrl). With slashes preserved it's dead code; a trailing-slash
redirect is recorded as an ordinary hop.
- add canonicalUrlKey (www/http/https-tolerant) and use it for the Lighthouse
homepage match, which had the same redirect-mismatch vulnerability.
- tests: preserve-trailing-slash + canonicalUrlKey unit tests; badseo harness
guard is now fix-agnostic (canonical resolves to 200, no loop/error).
Verified: 36 audit unit tests pass, tsc clean, badseo e2e 38/38. Reintroducing
stripping makes the trailing-slash guard fail (redirect-loop), confirming the
regression guard bites.
* Audit: add no-outgoing-links + meta-description-too-short checks, catch empty H1s
Two checks Ahrefs covers that we didn't, plus a fix: <h1></h1> now counts
as missing. badseo.dev gains fixtures for all three (41 checks, 27/27
issue types covered).
* Audit pages table: honest redirect/non-HTML rows, wrapped titles
- 3xx rows show their redirect target (dim →) instead of a red 'missing'
title, and dash out H1/Words/Images since nothing was analyzed
- red 'missing' only when the engine actually flagged missing-title, so
200 non-HTML files (security.txt) read as blank, not broken
- URL cells include the host when it differs from the audited site's, so
apex→www redirect sources no longer render identically to their target
- titles wrap to two lines (line-clamp) in a wider column instead of
truncating at 220px; PagesTable moved to its own file (lint max-lines)
* Audit pages table: canonical-host display, URL default sort, full title wrap
- host prefix now compares against the site's predominant 2xx host, not
the typed start URL — auditing apex 12port.com no longer prefixes every
www row with the host
- default sort by URL so the table opens as a site inventory instead of
leading with redirects on error-free sites
- titles wrap fully instead of clamping at two lines; long titles are the
thing being audited, so their tails shouldn't be hidden
* ci: exclude vendored skills from prettier; format test file
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Site audit P0 (1/3): issue engine, incremental persistence, block detection
Server-side foundation of the P0 feature set from docs/site-audit-pm-research.md:
- Issue engine: shared registry of issue types (severity, explanation,
how-to-fix). Per-page reporters run inside crawl steps; cross-page checks
(duplicate titles/descriptions/content, broken internal links, redirect
chains/loops, orphan pages) run at finalize as SQL over the persisted crawl.
- New audit_links + audit_issues tables, audit_pages columns (depth, content
hash, header signals, fetch class, sitemap flag); audit tables moved to
src/db/{,pg/}audit.schema.ts; migrations 0029 (D1) / 0006 (PG).
- Incremental persistence: pages/links/issues written inside each crawl-batch
step with deterministic row ids + upserts (retry idempotent); slim step
state; robots.txt checkpointed as step state; merged progress steps keep a
10k-page crawl within the Workflows step budget.
- Crawler: manual redirect handling with inline follow of normalization-
equivalent redirects, response header capture (X-Robots-Tag, Link
rel=canonical), BFS depth, sitemap-last seeding, SSRF check on discovered
links, honest 'we were blocked' classification (403/429/cf-mitigated/
challenge).
- MCP: run_site_audit, get_audit_status, get_audit_issues, get_audit_pages;
limitTier resolved via shared AuditService.resolveAuditLimitTier.
- Lighthouse strategies reduced to auto/none (legacy all/manual map on read).
- Self-healing: getStatus reconciles audits whose workflow instance errored/
terminated without reaching mark-failed.
The Issues UI and the badseo.dev e2e fixture site stack on top of this PR.
Deploy notes: run db:migrate:prod (additive); terminate running audits before
deploying — the workflow step structure changed and in-flight instances cannot
replay under the new code (a finalize guard fails them loudly instead of
completing empty).
* Store only internal link edges in audit_links
Both consumers (broken-internal-link and orphan checks) filter on
isInternal; per-page external counts already live on audit_pages.
Dropping external rows cuts stored edges on outbound-heavy sites.
Column stays so P1 external-link checks can re-add rows without a
migration.
* Review fixes: failAudit CAS guard, dedupe hash helpers, cheaper checks
- failAudit only transitions running audits, so the getStatus reconciler
can't flip a just-completed audit to failed when it races finalize
- collapse the duplicate SHA-256 helper into audit/ids.ts
- finalize integrity guard uses a limit-1 existence probe instead of
fetching every page row
- get_audit_status MCP tool no longer reads the audit row twice when an
explicit auditId is given
* Fix: don't mask DataForSEO 40501 validation errors as empty results
isNoResultsTask classified any task with status_code 40501 as a
successful empty result, but 40501 is not unique to "No Search
Results" — DataForSEO also returns it for validation rejections
like "Invalid Field: 'target'.".
After the Business Listings and Q&A endpoints opted into
treatNoResultsAsEmpty, invalid-field 40501 responses (reachable via
the MCP search_local_businesses categories input, which accepts
arbitrary strings) were silently masked as empty successes — billed
and tracked, but hiding the real provider validation error.
Match on the status message ("no search results") instead of the
ambiguous code so charged validation failures flow through to
DataforseoChargedTaskError with the diagnostic message, while genuine
no-results responses still return as empty successes. Fixes all call
sites uniformly (Business, SERP live, SERP task_get polling).
* Format isNoResultsTask per prettier
Production OOM triage: every 'Worker exceeded memory limit' burst hits
unrelated cheap routes right after a deploy — the 128MB limit is per
isolate, and the main worker's eagerly-evaluated module graph is what
crowds it, not any single request.
- Alias just-bash to a throwing stub (worker never uses it): removes
just-bash + turndown + @mixmark-io/domino from the bundle. The chain
was pulled in eagerly by @cloudflare/think via the SamChatAgent
re-export in src/server.ts; SAM only uses its own MCP tools.
- Disable Think's workspace bash tool on SamChatAgent so the stub is
unreachable at runtime.
- Dynamic-import page-analyzer (cheerio) in the site-audit crawl step
so it evaluates only when an audit runs, not in every isolate.
- Drop the turndown CJS alias workaround (#339): turndown is no longer
in the graph at all.
Main eager server chunk: 14,434 kB -> 11,712 kB (-19%); cheerio's
503 kB now a lazy chunk.
DataForSEO dropped the Claude Sonnet 4.0 family from its llm_responses
catalog, so model_name=claude-sonnet-4-0 was rejected with 'Invalid
Field: model_name' while still billing the failed task. Point Claude at
claude-sonnet-4-5 and validate every model_name against DataForSEO's
accepted catalog before dispatching the paid call.
* Add onboarding agent v1 product spec
* Add onboarding agent implementation plan (Project Think)
* Update onboarding plan: chat + seed function (drop Think/Workflows)
* feat(onboarding): data + metering foundation, Project Context store, MCP tool
* feat(onboarding): site read + DataForSEO signal + OpenRouter strategy seed
* feat(onboarding): strategy + streaming chat UI with update_project_context tool
* fix(onboarding): address review — bound free runs, cap chat, share auth+error helpers, harden scrape
* fix(onboarding): use canonical keyword-locations list, not a separate country list
* Improve onboarding strategy chat
* feat(onboarding): refine upgrade rail UI + fact-checked copy
- Rebuild upgrade sidebar: drop the nested card so the rail itself is the
container (header / plan / features / CTA / progress footer with dividers)
- Remove the 'Free preview' badge + headline pitch; header now reads
'Previewing OpenSEO' with the site domain beneath
- Tighten copy against the fact sheet: fix monthly-vs-top-up credit wording,
drop 'live' rank tracking, add money-back + open-source trust signals,
unify CTAs to 'Upgrade to continue', cut cross-panel feature redundancy
- Replace off-strategy suggested question; add progress bar counter
- FORCE_FREE_PREVIEW flag to always show the preview/limit UI while testing
* feat(onboarding): add 'What do you recommend' strategy chip; revert suggested questions
- Add a highlighted suggestion chip that prompts Sam for the strategy, shown
only when the user hasn't already used the welcome 'Show my strategy' CTA
- Track strategyRequested so the chip isn't re-offered after use
- Restore the original four suggested questions
* feat(onboarding): add OpenSEO Discord CTA + fact-sheet entry
- Discord link in the upgrade sidebar
- Fact-sheet community entry + system-prompt guidance so Sam can point
users to the Discord for community/second-opinion help
* chore(merge-ready): round 1 fixes
- scrape.ts SSRF: validate the initial domain via audit/url-policy
(normalizeAndValidateStartUrl) and re-validate each redirect hop with
redirect:"manual" (one hop, blocked/private/metadata hosts + DoH rebinding).
Replace the content-length-only guard with a bounded streaming read so
chunked/CDN responses can't buffer past MAX_RESPONSE_BYTES. Remove the
unguarded normalizeDomainToUrl helper. Add scrape.test.ts.
- http-errors.ts: map PAYMENT_REQUIRED AppError to HTTP 402 (was 500), so the
onboarding chat paywall backstop surfaces correctly.
- OnboardingStrategyChat: replace the hardcoded FORCE_FREE_PREVIEW=true debug
flag (which forced paid users into the free-preview/paywall UI) with a
safe-by-default ?preview=1 URL override.
- onboardingStrategy.ts: delete the dead generateOnboardingStrategy export
(knip) and its now-unused imports; the chat tool path uses runOnboardingSeed.
- chat.ts: rename inner runOnboardingSeed result to fix no-shadow.
- Extract presentational chat sub-components into OnboardingStrategyChatParts
to satisfy max-lines; reformat Markdown.tsx for prettier.
* chore(merge-ready): round 2 fixes
- chat.ts: validate message role in schema + count total messages (not just user-role) so the free-question gate can't be bypassed with mislabelled roles
- OnboardingStrategyChat.tsx: surface useChat error state with a paywall-aware notice; branch 'Ask about OpenSEO' message text on isPaid
- OnboardingStrategyChatParts.tsx: guard free-preview welcome copy behind !isPaid (paid variant for subscribers)
- onboardingStrategy.ts: reset onboardingRunStatus/onboardingRunAt when the domain changes so a corrected domain can get a fresh free seed
* chore(merge-ready): round 3 fixes
- onboarding chat: count only user-role messages for free-question paywall to match client gate (was counting all messages, firing ~3 turns early)
- ProjectContextStore: drop unused return value/type from saveProjectContextVersion, inline latest-version query into getCurrentProjectContextMarkdown, remove dead toVersion helper and ProjectContextVersion type
- onboarding chat UI: replace 'Why is OpenSEO better than Claude?' suggested chip with 'How does OpenSEO work with Claude?' (Claude is an MCP client, not a competitor)
* feat(onboarding): route post-upgrade to GSC step; drop isPaid from preview chat
- Checkout successUrl now returns to /onboarding?step=3 (GSC connect) instead
of the strategy chat, with a 'You're in!' success banner introducing the
remaining GSC + MCP setup steps. Fixes the post-Stripe 'stuck on paywall'
race since the user leaves the chat entirely.
- The strategy chat is now purely the pre-upgrade free preview: removed the
managed-access query, the isPaid branching, and the ?preview override. The
7-question cap always applies (kept as a conversion funnel).
* feat(onboarding): show post-upgrade success as its own step screen
Instead of a banner stacked above the GSC step, render a 'You're in!' screen
using the standard step layout (logo, title, card, Continue) in place of the
GSC step when ?checkout=success is present. Continue drops the param to reveal
the actual GSC step.
* refactor(nav): remove Project settings from account dropdown
Project settings is now reachable only via the project switcher's 'Manage
projects' → /projects → per-project settings. Drops the dead
projectSettingsLinkOptions helper and the now-unused AccountMenu projectId prop.
* refactor(onboarding): remove project-context persistence + MCP tool
Defers the Project Context store to a later PR to simplify this one.
- Delete ProjectContextStore, the get_project_context MCP tool (+ registration),
the project_context_versions table (schema + migration 0024 + snapshot), and
the update_project_context chat tool.
- generate_initial_strategy now returns the synthesized strategy to the chat
without persisting it; claimRun still gates paid spend (one free run).
- Chat system prompt no longer injects saved context; it just grounds Sam with
the project's domain.
- getOnboardingStrategyState returns only { projectId, domain }.
- Move the agent fact sheet out of docs/ (human docs) to
src/server/features/onboarding/openseo-fact-sheet.md.
* refactor(routing): move /strategy to /onboarding/chat
Rename the onboarding strategy chat route from /strategy to /onboarding/chat.
_authenticated.onboarding.tsx becomes the index route; the chat is a sibling,
so TanStack auto-creates the shared /onboarding parent (Outlet).
* chore(onboarding): clean up leftovers from the persistence removal
- Drop the chat's onFinish=invalidateStrategyState refetch: now that the
strategy state is just { projectId, domain } (no persisted markdown), the
chat can't mutate it, so the post-turn refetch was dead work.
- Fix a stale 'Project Context' doc comment in synthesis.ts.
- Note in specs 0005/0006 that strategy persistence + the MCP read tool were
deferred, so the docs don't contradict the shipped code.
* feat(onboarding): meter LLM (OpenRouter) spend via Autumn track_tokens
Mirror the DataForSEO metering pattern for LLM cost: a best-effort
trackLlmUsage helper emits a PostHog usage event and records token usage on
Autumn's token-tracking endpoint (REST; not in the autumn-js SDK yet), priced
from the model slug. Wired into the chat stream (onFinish) and the strategy
synthesis call. getOnboardingModel now also returns the resolved model slug.
* feat(onboarding): step-styled site form + account menu on chat page
- Restyle the website/country form to match the onboarding step layout (logo,
title, helper) and explain why we ask (read the site + pick the search market).
- Extract OnboardingAccountMenu to a shared component and render it on the
onboarding chat page so signed-in users can reach account actions there too.
* fix(auth): keep verify-email on 'check your inbox' after email sign-up
The post-sign-up redirect always passes ?email=; key the waiting state off it
so a just-signed-up user sees check-your-inbox + resend instead of the sign-in
CTA the verification gate would immediately block, even while the session is
still resolving.
* fix(onboarding): meter total LLM usage across all stream steps
Review caught that streamText onFinish 'usage' is only the last step; with
stopWhen=4 + the strategy tool, multi-step runs under-metered. Use 'totalUsage'
and await the metering so it fires before the stream closes. Also drop the
in-flux cache/reasoning token fields (negligible here).
* feat(onboarding): adopt the 'chat with tools' architecture from agent-onboarding-2
Replace the deterministic seed + synthesis pipeline (and the
onboarding_run_status/run_at columns) with two on-demand tools Sam calls —
read_website and get_seo_metrics — and have Sam write the strategy itself
in-stream, so a mid-stream refresh re-runs cache-backed tools instead of
dead-ending on a 'complete' status. Rename OnboardingStrategy* -> OnboardingChat*.
Preserved from this branch: LLM metering (now via the chat onFinish totalUsage,
covering the in-stream strategy), the account menu on the chat page, the
verify-email fix, and the step-styled site form. Drop columns via migration 0025.
* docs(onboarding): correct spend-bound + stale synthesis comments
Clarify that get_seo_metrics spend is bounded by the question cap + one project
per un-upgraded account (not solely caching, which doesn't cover no-data sites),
and drop 'synthesis' from comments now that Sam writes the strategy in-stream.
* refactor(onboarding): metered LLM via Autumn AI-SDK adapter; drop skipBalanceAssert
Now that every org gets an onboarding_plan with usage credits, onboarding spend
draws down the normal balance — no bypass needed.
- LLM metering: use Autumn's official @useautumn/gateway adapter (withLlmMetering
wraps the model; correct token-pool pricing for cached/reasoning tokens),
replacing the hand-rolled onFinish/track_tokens REST plumbing. Point it at the
existing 'llm_usage' feature (backed by usage_credits + topup_credits) rather
than a to-be-created 'ai_credits' feature.
- DataForSEO: remove skipBalanceAssert end-to-end (chat metering object, the
meter() plumbing in dataforseo/client.ts, and the DomainService override type);
onboarding now asserts balance like every other caller. Kept the email-verified
+ Labs-location gate on get_seo_metrics as the anti-farming bound.
Co-authored with a parallel agent's LLM-metering refactor.
* docs(onboarding): fix stale metering comment + diverged-architecture specs
- DomainService MeteringOverrides comment no longer claims a balance-gate bypass
(skipBalanceAssert + the onboarding seed are gone).
- specs 0005/0006: correct the update notes — the seed/synthesis pipeline,
claimRun, and skipBalanceAssert were replaced by the chat-with-tools design;
flag the bodies as the superseded plan.
- Document the pinned Autumn track_tokens API version.
* feat(onboarding): gate the chat turn on credit balance (LLM included)
Now that every org gets onboarding_plan trial credits and LLM tokens draw from
the same usage/topup balance, assert that balance before streaming — not just
track it. Extract the DataForSEO balance check into subscription.ts
(getUsageCreditsRemaining / assertUsageCreditsAvailable) and reuse it; the chat
throws a friendly PAYMENT_REQUIRED when credits are gone (client shows the
upgrade copy).
* feat(onboarding): make the strategy chat hosted-only
The chat needs the managed LLM + trial credits, so self-hosted has no business
there. Gate the step-2 navigation on hosted mode and add a beforeLoad redirect
on /onboarding/chat so self-hosted lands back in the wizard.
* feat(onboarding): site-form + welcome copy; drop open-source badge
- Site form: 'Tell us about your website.' title, short input labels, no extra
helper descriptions.
- Welcome message: lead with the upgrade ask + a Discord/email escape hatch.
- Remove the 'Open source — self-host for free anytime' badge from the rail.
* fix(onboarding): show typing indicator during the submitted wait
showTyping gated on the last message lacking assistant text, but right after
send the last message is the user's own (which has text), so nothing showed
until the assistant message appeared. Show it whenever busy and the last
message isn't assistant-text-yet.
* refactor(onboarding): drop redundant email-verified gate on get_seo_metrics
The route guard already requires a verified email to reach the chat in hosted
mode, and the trial-credit balance bounds spend — so the in-tool emailVerified
check was redundant for real users and blocked local/bypass testing. Keep the
Labs-location check (functional).
* feat(billing): meter onboarding LLM spend into the shared credit pool
Both DataForSEO and onboarding-LLM now draw from the same usage_credits/
topup_credits pool via one helper, instead of LLM needing a separate Autumn
ai_credit_system.
- Extract trackUsageCreditSpend (markup -> credits -> monthly/topup split ->
autumn.track + usage:credits_consume) into subscription.ts; DataForSEO's
trackDataforseoCost now delegates to it (behavior unchanged, tests pass).
- Enable OpenRouter usage accounting; the chat onFinish sums the real per-step
cost OpenRouter reports and deducts it through the same helper.
- Drop the @useautumn/gateway adapter, llm-metering.ts, track_tokens, and the
AUTUMN_LLM_USAGE_FEATURE_ID constant — no ai_credit_system feature needed.
* feat(onboarding): persist the strategy chat in a Durable Object (AIChatAgent)
Move the onboarding chat from a stateless streamText route to an Agents SDK
AIChatAgent Durable Object, so the conversation persists (DO SQLite) and
survives reloads — one instance per project.
- OnboardingChatAgent.onChatMessage ports the system prompt, read_website +
get_seo_metrics tools, the credit-balance/free-question gate, and the
OpenRouter cost metering. Billing gates surface as a normal assistant message
(staticAssistantResponse) rather than an HTTP 402.
- The Worker authorizes every /agents/* connection (resolve session + verify the
caller's org owns the projectId) before it reaches the DO; the DO derives org
/domain from the project it is named after. Auth stays on the proven path.
- Client swaps useChat -> useAgent + useAgentChat (WebSocket), keyed by projectId.
- Adds the DO binding + new_sqlite_classes migration; pins @cloudflare/ai-chat
0.6.1 to match agents 0.12.3.
* chore(onboarding): bump agents+ai-chat to latest; fix review findings
- Bump agents 0.12.3 -> 0.15.0 and @cloudflare/ai-chat -> 0.8.4 (the supported
pairing; verified MCP, the DO, and the build still compile).
- Thread the per-turn abortSignal into streamText so a user aborting mid-stream
cancels the billable LLM call (was leaking sub-cent cost on abort).
- Ensure the org's Autumn customer exists in the Worker authorize step before
the DO checks the credit balance, avoiding a false 'out of credits' gate on a
brand-new org's first message.
* chore: remove stray reservation-booker-seo-report.html
* chore: drop stale @useautumn/gateway minimumReleaseAge exclusion
The package was removed when LLM metering moved to the shared credit pool.
* perf(onboarding): fetch get_seo_metrics signals in parallel; clarify question-cap
- get_seo_metrics now fetches the domain overview and ranked keywords
concurrently instead of in series (faster tool turn). Trade-off: it always
issues the metered ranked-keywords call now, including for no-ranking sites.
- Correct the FREE_ONBOARDING_QUESTION_LIMIT comment: the server re-check counts
client-supplied history, so the cap is a conversion nudge, not a security
boundary — the credit balance is the real spend bound.
* track core product analytics flows
Track auth, search, export, audit, and credit-consumption events with canonical route IDs so PostHog funnels and usage dashboards stay low-noise and privacy-safe.
* fix: keep auth actions usable after session loss
* refactor: simplify analytics and auth helpers
- Replace isRecord/getActiveOrganizationId type guards with simple cast
- Refactor getAnalyticsRouteContext from if/return chain to route tables
- Replace toVerificationIssueType switch with zod enum
- Merge duplicate credits_consume events into single event per API call
- Merge two PostHogBootstrap useEffects into one
* refactor: add projectId to middleware context to reduce boilerplate
The requireProjectContext middleware now includes projectId directly,
eliminating repeated manual construction of BillingCustomerContext
objects across all server function handlers.
* remove unused BILLING_* env var fallbacks from cost profile script
* remove before_send event enrichment to preserve native PostHog URL tracking
The before_send hook was stripping $pathname, $current_url, $referrer and
other URL properties, which breaks PostHog web analytics dashboards, paths
analysis, session replay, and attribution. The route_id/route_group injection
it provided is unnecessary since PostHog already captures $pathname natively.
* remove route mapping layer, pass raw redirect paths to analytics events
The route ID registry (STATIC_ROUTES, PROJECT_ROUTES, getAnalyticsRouteContext,
getRedirectRouteId) duplicated what PostHog already captures via $pathname.
Replace redirect_route_id with redirect_to containing the raw path, and remove
~80 lines of route mapping infrastructure.
* clean up analytics events: drop redundant submit events and derived properties
- Remove search_submit events for keywords, domain overview, and backlinks
(the search_complete events capture the meaningful outcome data)
- Remove target_type from backlinks events (derived 1:1 from search_scope)
- Remove result_limit from keyword research (requested limit, not useful
alongside actual result_count)
- Remove export_format from data:export events (always "csv")
* refactor: inline wrappers, colocate helpers, deduplicate getActiveOrganizationId
- Inline toVerificationIssueType into verify-email.tsx (single-use wrapper)
- Move mapDataforseoPathToCreditFeature into dataforseoClient.ts (only consumer)
- Extract shared getActiveOrganizationId into lib/auth-session.ts (was
duplicated in __root.tsx and middleware/ensure-user/hosted.ts)
- Rename shared/analytics.ts → shared/internal-user.ts (only email helpers
remain after removing route mapping, verification, and dataforseo helpers)
* remove internal user tracking and email domain properties
Drop is_internal_user super property, email_domain person property, and all
supporting code (shared/internal-user.ts, getEmailDomain, isInternalUserEmail).
Simplifies initPostHog and identifyAnalyticsUser signatures.
* remove backlinks:search_complete effect-based tracking
The reactive useEffect + useRef dedup pattern added ~30 lines of plumbing
inside a data hook for a single analytics event. Not worth the complexity.
* simplify: replace manual type guards with zod, deduplicate posthog and sign-out helpers
- Replace hand-rolled typeof checks in getActiveOrganizationId and
isAuthenticatedServerFunctionContext with zod safeParse
- Extract withPostHogClient helper to deduplicate client posthog wrapper
- Move apiKey guard into getServerPostHogClient factory
- Extract signOutAndRedirect to avoid duplicated sign-out logic
- Drop derivable has_results from analytics events
- Remove unnecessary path normalization in mapDataforseoPathToCreditFeature
* fix: strip email from pageview URLs, restore sign-out guard, harden server posthog, fix path mapper
- Sanitize $current_url on pageviews to remove email query param (PII)
- Restore onSuccess for sign-out redirect to avoid bounce-back on failure
- Swallow shutdown() errors so PostHog outages can't fail billed work
- Rewrite mapDataforseoPathToCreditFeature to match real API path structure
(path[1] = module, path[3] = endpoint) instead of scanning all segments
* simplify: remove redundant refs in verify-email, infer middleware context type
- Remove unnecessary useRef guards in verify-email effects (deps already prevent re-firing)
- Use z.ZodType<EnsuredUserContext> annotation to infer return type instead of casting
- Add comment explaining one-shot PostHog client on Workers
* fix: reset PostHog identity on sign-out before redirect
* fix: require POSTHOG_HOST env var instead of defaulting to us.i.posthog.com
* fix: annotate url as unknown to satisfy no-unsafe-assignment
* format
* fix: accept empty DataForSEO task results
Treat successful tasks with null items as empty payloads so empty ranked keyword responses do not fail billing validation.
* refactor: simplify DataForSEO null result handling
Add .nullable() to the existing structured result schema instead of
loosening the type to unknown[] and re-parsing in parseTaskItems.
* refactor: simplify backlinks zod parsing with structured result schema
Same pattern as the dataforseoSchemas fix: give taskSchema.result a
structured type with .items instead of z.unknown(), removing the
intermediate resultItemsSchema and two-step parsing in parseItems.
* refactor: replace manual type guards with zod schemas
- progress-kv.ts: replace isCrawledUrlEntry type guard with a zod
schema and use jsonCodec(z.array(...)) instead of parsing unknown
then filtering
- helpers.ts: tighten normalizeIntent param from unknown to
string | null | undefined to match actual call sites
- dataforseoBacklinksSupport.ts: allow null result elements to match
API responses where result contains [null]
* refactor: filter null result elements at the source
Filter out null elements from task.result in postBacklinks so
downstream functions receive clean BacklinksTaskResult[] instead
of (BacklinksTaskResult | null)[].
* save
Treat successful DataForSEO keyword responses with null items as empty results so auto mode can fall back cleanly. Simplify the no-results state by removing dead-end actions and top-aligning the empty card.
* fix: use backlinks history for default trends
* simplify backlinks: remove filters, always use history endpoint
Remove the filter UI (status, subdomains, indirect links, exclude internal)
and hardcode defaults across the stack. Replace the conditional
timeseries_summary + timeseries_new_lost_summary fallback with a single
backlinks/history/live call for trend data. This reduces the overview from
5 parallel API calls to 3 and removes all conditional branching.
* improve charts
* fix: refresh backlinks cost docs
