Cap crawled HTML bodies at 2 MiB before Cheerio parsing, bulk-insert
audit links on Postgres instead of per-row Hyperdrive round trips, and
raise the Worker CPU ceiling to the 5-minute paid max for link-heavy
crawl steps.
Remove the RankMovementCard (setup pitch + rank movement stats) from the
dashboard grid until the rank tracking experience gets more polish. The
overview endpoint still returns the rank summary, so restoring the card
is a small revert of this commit.
Add a multi-select intent filter (Informational, Commercial,
Transactional, Navigational, Unknown) to the Keyword Research table on
both the desktop and mobile layouts (closes#67).
Selected intents are stored as a comma-separated string in a new
`intents` filter field, so they fit the existing all-strings filter
shape and get active-filter counting, persistence, and clear-all for
free. Sorting, pagination, save, and export already derive from the
filtered row set, so they respect the intent filter automatically. The
persistence schema defaults `intents` to "" so filter blobs saved before
this change still load. Intent classification and URL params are
untouched.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
void setCached() leaves the R2 put unregistered, and workerd cancels
unregistered pending I/O once the response is sent — the write never
lands, so the caches behind domain overview, domain keyword/page pages,
and SERP research are silently re-fetched (and re-charged) on every
call. Register the writes with waitUntil, matching the pattern already
used in instrumentation.ts and brandLookup.ts.
* Add MCP activation + usage PostHog events and North Star dashboard prompt
- mcp:authorize_success (server) on OAuth consent completion
- mcp:tool_call (server) on every MCP tool invocation, with tool/success/
error_code/client_id and mcp_client vs in_app_agent source
- mcp:consent_viewed / mcp:consent_denied on the OAuth consent page
- mcp:setup_url_copy / mcp:setup_command_copy intent events on /ai
- docs/posthog-north-star-dashboard.md: prompt to configure the PostHog
core-actions dashboard (activation rate, MCP setup funnel, daily MCP users)
* Fix codex review findings: count schema-rejected MCP calls as failures, wrap audit tools
- mcp:tool_call now fires after output validation; schema mismatches report
success:false with MCP_OUTPUT_VALIDATION (the SDK surfaces them to the
client as JSON-RPC errors)
- run_site_audit/get_audit_status/get_audit_issues/get_audit_pages were
registered without instrumentMcpToolHandler, so their usage and failures
were invisible
* Remove dashboard prompt doc
* Site audit P0: issue engine, incremental persistence, block detection
Implements the P0 feature set from docs/site-audit-pm-research.md:
- Issue engine: 24 issue types (shared registry with severity,
explanation, how-to-fix). Per-page reporters run inside crawl steps;
cross-page checks (duplicate titles/descriptions/content, broken
internal links, redirect chains/loops, orphan pages) run at finalize
as SQL over the persisted crawl.
- New audit_links + audit_issues tables, audit_pages columns (depth,
content hash, header signals, fetch class, sitemap flag); audit
tables moved to src/db/audit.schema.ts.
- Incremental persistence: pages/links/issues written to D1 inside
each crawl-batch step with deterministic row ids + upserts (retry
idempotent); slim step state; robots.txt checkpointed as step state
for deterministic replay; merged progress steps keep a 10k-page
crawl within the Workflows step budget.
- Crawler: manual redirect handling with inline follow of
normalization-equivalent redirects (slash-canonical sites), response
header capture (X-Robots-Tag, Link rel=canonical), BFS depth,
sitemap-last seeding, SSRF check on discovered links, honest
"we were blocked" classification (403/429/cf-mitigated/challenge).
- UI: Issues tab (default) with severity grouping, per-type
explanations, drill-down, CSV/JSON/Sheets export, blocked banner.
- MCP: run_site_audit, get_audit_status, get_audit_issues (severity-
sorted, how_to_fix per issue), get_audit_pages.
- Lighthouse strategies reduced to auto/none (legacy all/manual map on
read); auto stays 10 URLs x 2 = 20 checks.
- Self-healing: getStatus reconciles audits whose workflow instance
errored/terminated without reaching mark-failed.
Deploy notes: run db:migrate:prod (additive migration 0022); terminate
running audits before deploying - the workflow step structure changed
and in-flight instances cannot replay under the new code (a finalize
guard fails them loudly instead of completing empty).
* feat(onboarding): hide agent chat step; subscribe after intro steps (#312)
* feat(onboarding): hide agent chat step; subscribe after intro steps
Remove the hosted-only strategy-chat diversion from the onboarding
sequence. After the three intro questions, hosted users now hit the
subscribe paywall directly, then return to the GSC and MCP connect
steps. The chat route and components stay in place but unlinked, to be
revisited later. Preserve the post-payment 'You're in!' interstitial by
carrying checkout=success through validateSearch.
* fix(onboarding): set checkout=success from subscribe route, not speculatively
The previous redirect baked checkout=success into the onboarding return
URL at the point needsSubscription is true — i.e. before the user had
paid. It only worked because the subscribe route gates its redirect on
actual access. Move the marker to the subscribe route's redirect-to-app
path, where checkoutCompleted reflects a real returned-from-Stripe
payment, so the 'You're in!' screen can never show pre-payment.
* website: change link
* fix(rank-tracking): unarchive config when re-adding an archived domain (#313)
* Unify dual-backend DB layer (D1 default + Postgres opt-in) (#238)
* D1 → Postgres data migration (ETL + runbook) (#274)
* Fix Postgres-only rank-tracking & site-audit workflow failures (#317)
* rank-tracking: raise per-project config limit from 20 to 100 (#318)
The cap was only a soft guard against runaway scheduled DataForSEO
workload, not a hard product constraint. Bump it to 100 so projects
tracking many domain/location combos aren't blocked.
Co-authored-by: Claude <noreply@anthropic.com>
* fix(db): add missing indexes and drop redundant ones (#319)
Postgres advisor flagged seq-scans and redundant indexes across both
backends (D1 + Postgres):
- add projects(organization_id) — org-scoped project listings seq-scanned
- add account(account_id, provider_id) — better-auth sign-in lookup
- add verification(expires_at) — expired-token cleanup range scan
- drop saved_keyword_tag_assignments_keyword_idx — covered by unique
(saved_keyword_id, tag_id) prefix
- drop rank_snapshots_run_idx — covered by unique
(run_id, tracking_keyword_id, device) prefix
Mirrored in both schema dialects + parity-test required-index guard.
* refactor(keywords): unify keyword-metric fetching behind one helper (#320)
* Fix production errors: onboarding crash hardening + DataForSEO spend/noise cleanup (#282)
* fix(ai-search): use valid Claude model_name and fail fast on unknown ones (#323)
DataForSEO dropped the Claude Sonnet 4.0 family from its llm_responses
catalog, so model_name=claude-sonnet-4-0 was rejected with 'Invalid
Field: model_name' while still billing the failed task. Point Claude at
claude-sonnet-4-5 and validate every model_name against DataForSEO's
accepted catalog before dispatching the paid call.
* fix(mcp): 405 the standalone GET SSE stream to stop /mcp OOM (#325)
The stateless MCP server returns JSON on POST (enableJsonResponse) and
pushes no server-initiated messages, so the optional standalone GET SSE
stream serves no purpose. Left enabled, each GET holds an SSE stream open
indefinitely (25s keepalive, no eventStore) and pins a fresh per-request
McpServer (~5MB of tools + Zod schemas); a few dozen concurrent connected
clients exceed the 128MB isolate limit. This was 100% of the /mcp
exceededMemory OOMs (GET only; POST never OOMed).
Return 405 (spec-compliant 'no standalone stream') before building the
server, so GET allocates nothing. Also removes the bulk of the elevated
GET canceled / responseStreamDisconnected outcomes.
* Re-add free plan as the floor; remove subscribe gate (#321)
* Pin production to Postgres via committed Hyperdrive binding (#329)
* Add Cloudflare Turnstile captcha on email signup (#326)
* Triage production log errors: audit crash, Autumn webhook FK, PostHog capture, auth rate-limit IP, log noise (#327)
* Add badseo.dev: a test site of deliberate SEO mistakes
An open-source Cloudflare Worker that serves ~27 pages, each breaking one
common technical-SEO rule (missing title, redirect loop, orphan page, thin
content, and so on). It doubles as the end-to-end fixture for the OpenSEO
site audit: every page declares the audit issues it should trigger, and
scripts/run-audit.ts drives the real audit engine against a running copy to
check that it does (36/36 checks, 25/25 issue types).
Styled to match the OpenSEO marketing site (web/). Maintained-by-OpenSEO
badge links back to openseo.so.
* badseo.dev: logo in pill, footer/hover polish, SEO-optimized titles
- Use the OpenSEO pine-tree logo (downscaled, base64-embedded, served at
/openseo-logo.png) in a light chip inside the badge, replacing the ◎ glyph.
- Footer band now fills to the bottom of the page (dropped the mismatched
body padding strip) with room for the floating badge.
- Index rows: remove the stray full-row underline and the stark white hover
box; hover is now a soft cream tint with the name underlined.
- Drop the "Maintained by OpenSEO" hero eyebrow; new H1 "A website
demonstrating common technical SEO problems" and a cleaner subtitle.
- Optimize homepage + catalog <title>/meta around real keywords from OpenSEO
keyword research (technical seo issues KD25/vol170; technical seo checklist
KD16/vol390), keeping meta lengths within limits.
* Site audit P0 (1/3): issue engine, incremental persistence, block detection
Server-side foundation of the P0 feature set from docs/site-audit-pm-research.md:
- Issue engine: shared registry of issue types (severity, explanation,
how-to-fix). Per-page reporters run inside crawl steps; cross-page checks
(duplicate titles/descriptions/content, broken internal links, redirect
chains/loops, orphan pages) run at finalize as SQL over the persisted crawl.
- New audit_links + audit_issues tables, audit_pages columns (depth, content
hash, header signals, fetch class, sitemap flag); audit tables moved to
src/db/{,pg/}audit.schema.ts; migrations 0029 (D1) / 0006 (PG).
- Incremental persistence: pages/links/issues written inside each crawl-batch
step with deterministic row ids + upserts (retry idempotent); slim step
state; robots.txt checkpointed as step state; merged progress steps keep a
10k-page crawl within the Workflows step budget.
- Crawler: manual redirect handling with inline follow of normalization-
equivalent redirects, response header capture (X-Robots-Tag, Link
rel=canonical), BFS depth, sitemap-last seeding, SSRF check on discovered
links, honest 'we were blocked' classification (403/429/cf-mitigated/
challenge).
- MCP: run_site_audit, get_audit_status, get_audit_issues, get_audit_pages;
limitTier resolved via shared AuditService.resolveAuditLimitTier.
- Lighthouse strategies reduced to auto/none (legacy all/manual map on read).
- Self-healing: getStatus reconciles audits whose workflow instance errored/
terminated without reaching mark-failed.
The Issues UI and the badseo.dev e2e fixture site stack on top of this PR.
Deploy notes: run db:migrate:prod (additive); terminate running audits before
deploying — the workflow step structure changed and in-flight instances cannot
replay under the new code (a finalize guard fails them loudly instead of
completing empty).
* Site audit P0 (2/3): Issues tab UI
- Issues tab (new default) with severity grouping, per-type explanations and
how-to-fix, drill-down to affected pages, CSV/JSON/Sheets export, and the
'we were blocked' banner when the crawl was challenged.
- Tabs always render (Issues/Pages, Performance when Lighthouse ran);
audit route search schema gains the issues tab and defaults to it.
Stacks on claude/audit-p0-server (issue engine + persistence).
* badseo.dev: render the badge logo as a white tree, no chip
The silver source logo was invisible on the dark pill, so it sat in a white
chip. Render it white via a CSS filter instead, so the tree fills the pill
with no backing background.
* badseo.dev: add build (typecheck) step before deploy
- Add 'build'/'typecheck' scripts (tsc --noEmit); 'deploy' now runs the build
before wrangler deploy.
- Scope the tsconfig typecheck to the Worker source (src/); the e2e harness in
scripts/ imports the main app and is run with tsx from the repo root.
- Document the deploy flow and first-time custom-domain setup in the README.
* badseo.dev: add trailing-slash redirect-cycle fixture + regression test
Reproduces the 508 "Loop Detected" class of bug from every-app/open-seo#61: a
CMS-style page whose canonical URL ends in a trailing slash, with the non-slash
form 301-redirecting to it. A crawler that strips trailing slashes turns the
canonical /foo/ back into /foo, follows the 301 to /foo/, strips it again, and
loops.
- New fixture at /redirect/trailing-slash: the non-slash form (intercepted in
index.ts on the raw path) 301s to the slash form, which is served as the
canonical 200.
- Harness asserts the page is crawled exactly once as a 200 with NO redirect
loop, plus a dedicated "Trailing-slash cycle -> 200, no loop" guard.
Verified the guard bites: temporarily disabling crawlPage's slash-canonical
inline-follow makes both checks fail (redirect-loop, status 301); with it in
place the harness is 38/38, 25/25 issue types.
* Add webapp-testing skill (installed via /reload-skills)
Vendors the anthropics/skills webapp-testing toolkit: real files under
.agents/skills/webapp-testing, a symlink from .claude/skills/, and skills-lock.json
pinning the source + hash. Matches how the other project skills are tracked.
* Site audit: redesign issues tab as grouped table + calmer page header
- Issues: single bordered table with severity sections (Critical/Warning/Info
headers carry the counts), dot indicators instead of filled pills, plain
right-aligned page counts, all rows collapsed by default; expanded rows get
a severity-colored left rule
- Removed the dead severity-count chips (they looked like filters but were
inert spans)
- Header: audited hostname is now the H1 with the status badge inline
- Blocked banner: compact tinted panel instead of a full-size alert
- Stats: hairline strip instead of four separate cards; issues stat shows a
severity breakdown, Lighthouse tile hidden when no tests ran, dropped the
orange issues-count coloring
* audit: fix trailing-slash redirect cycle at the root (preserve slashes)
Replaces the crawlPage inline-follow workaround with the root-cause fix, so we
don't carry two fixes for the same bug (every-app/open-seo#61).
- normalizeUrl: stop stripping trailing slashes. A trailing slash is the
canonical form on most CMSes, which 301 the non-slash version to it. Stripping
rewrote the canonical URL into its own redirect source and looped (508). Now
/path and /path/ are distinct and the redirect resolves normally.
- crawlPage: remove the isSelfAfterNormalization inline-follow (+ now-unused
resolveRawUrl). With slashes preserved it's dead code; a trailing-slash
redirect is recorded as an ordinary hop.
- add canonicalUrlKey (www/http/https-tolerant) and use it for the Lighthouse
homepage match, which had the same redirect-mismatch vulnerability.
- tests: preserve-trailing-slash + canonicalUrlKey unit tests; badseo harness
guard is now fix-agnostic (canonical resolves to 200, no loop/error).
Verified: 36 audit unit tests pass, tsc clean, badseo e2e 38/38. Reintroducing
stripping makes the trailing-slash guard fail (redirect-loop), confirming the
regression guard bites.
* Audit: add no-outgoing-links + meta-description-too-short checks, catch empty H1s
Two checks Ahrefs covers that we didn't, plus a fix: <h1></h1> now counts
as missing. badseo.dev gains fixtures for all three (41 checks, 27/27
issue types covered).
* Audit pages table: honest redirect/non-HTML rows, wrapped titles
- 3xx rows show their redirect target (dim →) instead of a red 'missing'
title, and dash out H1/Words/Images since nothing was analyzed
- red 'missing' only when the engine actually flagged missing-title, so
200 non-HTML files (security.txt) read as blank, not broken
- URL cells include the host when it differs from the audited site's, so
apex→www redirect sources no longer render identically to their target
- titles wrap to two lines (line-clamp) in a wider column instead of
truncating at 220px; PagesTable moved to its own file (lint max-lines)
* Audit pages table: canonical-host display, URL default sort, full title wrap
- host prefix now compares against the site's predominant 2xx host, not
the typed start URL — auditing apex 12port.com no longer prefixes every
www row with the host
- default sort by URL so the table opens as a site inventory instead of
leading with redirects on error-free sites
- titles wrap fully instead of clamping at two lines; long titles are the
thing being audited, so their tails shouldn't be hidden
* ci: exclude vendored skills from prettier; format test file
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Site audit P0 (1/3): issue engine, incremental persistence, block detection
Server-side foundation of the P0 feature set from docs/site-audit-pm-research.md:
- Issue engine: shared registry of issue types (severity, explanation,
how-to-fix). Per-page reporters run inside crawl steps; cross-page checks
(duplicate titles/descriptions/content, broken internal links, redirect
chains/loops, orphan pages) run at finalize as SQL over the persisted crawl.
- New audit_links + audit_issues tables, audit_pages columns (depth, content
hash, header signals, fetch class, sitemap flag); audit tables moved to
src/db/{,pg/}audit.schema.ts; migrations 0029 (D1) / 0006 (PG).
- Incremental persistence: pages/links/issues written inside each crawl-batch
step with deterministic row ids + upserts (retry idempotent); slim step
state; robots.txt checkpointed as step state; merged progress steps keep a
10k-page crawl within the Workflows step budget.
- Crawler: manual redirect handling with inline follow of normalization-
equivalent redirects, response header capture (X-Robots-Tag, Link
rel=canonical), BFS depth, sitemap-last seeding, SSRF check on discovered
links, honest 'we were blocked' classification (403/429/cf-mitigated/
challenge).
- MCP: run_site_audit, get_audit_status, get_audit_issues, get_audit_pages;
limitTier resolved via shared AuditService.resolveAuditLimitTier.
- Lighthouse strategies reduced to auto/none (legacy all/manual map on read).
- Self-healing: getStatus reconciles audits whose workflow instance errored/
terminated without reaching mark-failed.
The Issues UI and the badseo.dev e2e fixture site stack on top of this PR.
Deploy notes: run db:migrate:prod (additive); terminate running audits before
deploying — the workflow step structure changed and in-flight instances cannot
replay under the new code (a finalize guard fails them loudly instead of
completing empty).
* Store only internal link edges in audit_links
Both consumers (broken-internal-link and orphan checks) filter on
isInternal; per-page external counts already live on audit_pages.
Dropping external rows cuts stored edges on outbound-heavy sites.
Column stays so P1 external-link checks can re-add rows without a
migration.
* Review fixes: failAudit CAS guard, dedupe hash helpers, cheaper checks
- failAudit only transitions running audits, so the getStatus reconciler
can't flip a just-completed audit to failed when it races finalize
- collapse the duplicate SHA-256 helper into audit/ids.ts
- finalize integrity guard uses a limit-1 existence probe instead of
fetching every page row
- get_audit_status MCP tool no longer reads the audit row twice when an
explicit auditId is given
