54 Commits

Author SHA1 Message Date
Ben Senescu
022a7f1944
SERP analysis: depth 20 by default, load top 100 on demand; classify DataForSEO timeouts (#523) 2026-08-26 11:17:18 -04:00
Ben Senescu
a5a953e478
Project memory + SAM skills: shared per-project AI context (spec 0010) (#493) 2026-08-19 09:46:07 -04:00
Ben Senescu
c8b3eb9b0a
Add exact URL / subfolder / domain / subdomain research scopes (EVE-56) (#487) 2026-08-14 21:52:32 -04:00
Ben Senescu
db45a0dd6d
Fix MCP saved keyword metrics (#483) 2026-08-13 20:16:20 -04:00
Ben Senescu
13ada5b441
fix(sam): out-of-credits CoT leak, streaming scroll lock, rank-tracking language gap (#470)
* fix(sam): stream canned refusals without a provider call

Out-of-credits and session-gone refusals ran a real LLM turn with a
200-token cap; MiniMax M3 could spend the whole budget on reasoning
tokens, leaving the user a raw truncated chain-of-thought (which also
named the backing model) and no reply. Refusal turns now swap in a
static LanguageModelV3 that streams the refusal text through Think's
normal pipeline — rendered and persisted like any assistant message,
with no provider request at all.

Fixes every-app/open-seo#161

* fix(chat): stop pinning the transcript to the bottom while the user scrolls up

Both chat surfaces (SAM and onboarding) forced scrollTop to the bottom
on every streamed chunk, so scrolling up mid-reply was undone within
milliseconds. A shared stick-to-bottom hook now tracks pinned-ness from
real scroll events: scrolling away releases the pin, returning to the
bottom (or sending a message) re-arms it.

Fixes every-app/open-seo#160

* fix(rank-tracking): allow any SERP language for any country

The Add Domain modal restricted the language picker to the Labs
per-country subset and disabled it when only one option existed, so
e.g. tracking English searches in Czechia was impossible — even though
rank tracking runs against the SERP API, which serves every supported
language in every country. The picker now offers the full SERP language
list; create/update schemas validate codes against the master list so
unknown codes still fail before DataForSEO charges for them; and
keyword-metrics refreshes resolve a Labs-served language so an
unserved pair never reaches a charged Labs call.

Fixes every-app/open-seo#183

* fix(chat): surface silent turn failures in Workers logs

A provider stream dying mid-turn (chat:request:failed) and a DO restart
whose recovery gives up (chat:recovery:exhausted) leave the user a
replayed "Something went wrong" banner and a half-streamed message, but
never reach the onChatError hook — their only signal is the agents:chat
diagnostics channel, which was unsubscribed, so the chat agents' most
common failure modes produced zero log lines. A module-level
subscription now logs both for every chat DO.

SamChatAgent.onChatError also returns the error now: Think uses the
return value as the stored chat-terminal body that reconnecting clients
replay, and returning void stored the literal string "undefined".
2026-08-08 18:47:41 -04:00
Ben Senescu
b0e850db2b
Backlinks tab: default sort by most recent (firstSeen) instead of rank (#432) 2026-07-29 22:41:31 -04:00
Ben Senescu
eda5e1e354
Project dashboard: onboarding checklist + domain overview (#398) 2026-07-18 21:52:35 -04:00
bookingseo
0f08437cd0
feat: configurable default market for location/language fallbacks (#72) 2026-07-16 10:46:49 -04:00
Ben Senescu
67265a0046 Site audit P0: Issues tab UI, badseo.dev e2e harness, new checks + pages-table polish (#367)
* Site audit P0: issue engine, incremental persistence, block detection

Implements the P0 feature set from docs/site-audit-pm-research.md:

- Issue engine: 24 issue types (shared registry with severity,
  explanation, how-to-fix). Per-page reporters run inside crawl steps;
  cross-page checks (duplicate titles/descriptions/content, broken
  internal links, redirect chains/loops, orphan pages) run at finalize
  as SQL over the persisted crawl.
- New audit_links + audit_issues tables, audit_pages columns (depth,
  content hash, header signals, fetch class, sitemap flag); audit
  tables moved to src/db/audit.schema.ts.
- Incremental persistence: pages/links/issues written to D1 inside
  each crawl-batch step with deterministic row ids + upserts (retry
  idempotent); slim step state; robots.txt checkpointed as step state
  for deterministic replay; merged progress steps keep a 10k-page
  crawl within the Workflows step budget.
- Crawler: manual redirect handling with inline follow of
  normalization-equivalent redirects (slash-canonical sites), response
  header capture (X-Robots-Tag, Link rel=canonical), BFS depth,
  sitemap-last seeding, SSRF check on discovered links, honest
  "we were blocked" classification (403/429/cf-mitigated/challenge).
- UI: Issues tab (default) with severity grouping, per-type
  explanations, drill-down, CSV/JSON/Sheets export, blocked banner.
- MCP: run_site_audit, get_audit_status, get_audit_issues (severity-
  sorted, how_to_fix per issue), get_audit_pages.
- Lighthouse strategies reduced to auto/none (legacy all/manual map on
  read); auto stays 10 URLs x 2 = 20 checks.
- Self-healing: getStatus reconciles audits whose workflow instance
  errored/terminated without reaching mark-failed.

Deploy notes: run db:migrate:prod (additive migration 0022); terminate
running audits before deploying - the workflow step structure changed
and in-flight instances cannot replay under the new code (a finalize
guard fails them loudly instead of completing empty).

* feat(onboarding): hide agent chat step; subscribe after intro steps (#312)

* feat(onboarding): hide agent chat step; subscribe after intro steps

Remove the hosted-only strategy-chat diversion from the onboarding
sequence. After the three intro questions, hosted users now hit the
subscribe paywall directly, then return to the GSC and MCP connect
steps. The chat route and components stay in place but unlinked, to be
revisited later. Preserve the post-payment 'You're in!' interstitial by
carrying checkout=success through validateSearch.

* fix(onboarding): set checkout=success from subscribe route, not speculatively

The previous redirect baked checkout=success into the onboarding return
URL at the point needsSubscription is true — i.e. before the user had
paid. It only worked because the subscribe route gates its redirect on
actual access. Move the marker to the subscribe route's redirect-to-app
path, where checkoutCompleted reflects a real returned-from-Stripe
payment, so the 'You're in!' screen can never show pre-payment.

* website: change link

* fix(rank-tracking): unarchive config when re-adding an archived domain (#313)

* Unify dual-backend DB layer (D1 default + Postgres opt-in) (#238)

* D1 → Postgres data migration (ETL + runbook) (#274)

* Fix Postgres-only rank-tracking & site-audit workflow failures (#317)

* rank-tracking: raise per-project config limit from 20 to 100 (#318)

The cap was only a soft guard against runaway scheduled DataForSEO
workload, not a hard product constraint. Bump it to 100 so projects
tracking many domain/location combos aren't blocked.

Co-authored-by: Claude <noreply@anthropic.com>

* fix(db): add missing indexes and drop redundant ones (#319)

Postgres advisor flagged seq-scans and redundant indexes across both
backends (D1 + Postgres):

- add projects(organization_id) — org-scoped project listings seq-scanned
- add account(account_id, provider_id) — better-auth sign-in lookup
- add verification(expires_at) — expired-token cleanup range scan
- drop saved_keyword_tag_assignments_keyword_idx — covered by unique
  (saved_keyword_id, tag_id) prefix
- drop rank_snapshots_run_idx — covered by unique
  (run_id, tracking_keyword_id, device) prefix

Mirrored in both schema dialects + parity-test required-index guard.

* refactor(keywords): unify keyword-metric fetching behind one helper (#320)

* Fix production errors: onboarding crash hardening + DataForSEO spend/noise cleanup (#282)

* fix(ai-search): use valid Claude model_name and fail fast on unknown ones (#323)

DataForSEO dropped the Claude Sonnet 4.0 family from its llm_responses
catalog, so model_name=claude-sonnet-4-0 was rejected with 'Invalid
Field: model_name' while still billing the failed task. Point Claude at
claude-sonnet-4-5 and validate every model_name against DataForSEO's
accepted catalog before dispatching the paid call.

* fix(mcp): 405 the standalone GET SSE stream to stop /mcp OOM (#325)

The stateless MCP server returns JSON on POST (enableJsonResponse) and
pushes no server-initiated messages, so the optional standalone GET SSE
stream serves no purpose. Left enabled, each GET holds an SSE stream open
indefinitely (25s keepalive, no eventStore) and pins a fresh per-request
McpServer (~5MB of tools + Zod schemas); a few dozen concurrent connected
clients exceed the 128MB isolate limit. This was 100% of the /mcp
exceededMemory OOMs (GET only; POST never OOMed).

Return 405 (spec-compliant 'no standalone stream') before building the
server, so GET allocates nothing. Also removes the bulk of the elevated
GET canceled / responseStreamDisconnected outcomes.

* Re-add free plan as the floor; remove subscribe gate (#321)

* Pin production to Postgres via committed Hyperdrive binding (#329)

* Add Cloudflare Turnstile captcha on email signup (#326)

* Triage production log errors: audit crash, Autumn webhook FK, PostHog capture, auth rate-limit IP, log noise (#327)

* Add badseo.dev: a test site of deliberate SEO mistakes

An open-source Cloudflare Worker that serves ~27 pages, each breaking one
common technical-SEO rule (missing title, redirect loop, orphan page, thin
content, and so on). It doubles as the end-to-end fixture for the OpenSEO
site audit: every page declares the audit issues it should trigger, and
scripts/run-audit.ts drives the real audit engine against a running copy to
check that it does (36/36 checks, 25/25 issue types).

Styled to match the OpenSEO marketing site (web/). Maintained-by-OpenSEO
badge links back to openseo.so.

* badseo.dev: logo in pill, footer/hover polish, SEO-optimized titles

- Use the OpenSEO pine-tree logo (downscaled, base64-embedded, served at
  /openseo-logo.png) in a light chip inside the badge, replacing the ◎ glyph.
- Footer band now fills to the bottom of the page (dropped the mismatched
  body padding strip) with room for the floating badge.
- Index rows: remove the stray full-row underline and the stark white hover
  box; hover is now a soft cream tint with the name underlined.
- Drop the "Maintained by OpenSEO" hero eyebrow; new H1 "A website
  demonstrating common technical SEO problems" and a cleaner subtitle.
- Optimize homepage + catalog <title>/meta around real keywords from OpenSEO
  keyword research (technical seo issues KD25/vol170; technical seo checklist
  KD16/vol390), keeping meta lengths within limits.

* Site audit P0 (1/3): issue engine, incremental persistence, block detection

Server-side foundation of the P0 feature set from docs/site-audit-pm-research.md:

- Issue engine: shared registry of issue types (severity, explanation,
  how-to-fix). Per-page reporters run inside crawl steps; cross-page checks
  (duplicate titles/descriptions/content, broken internal links, redirect
  chains/loops, orphan pages) run at finalize as SQL over the persisted crawl.
- New audit_links + audit_issues tables, audit_pages columns (depth, content
  hash, header signals, fetch class, sitemap flag); audit tables moved to
  src/db/{,pg/}audit.schema.ts; migrations 0029 (D1) / 0006 (PG).
- Incremental persistence: pages/links/issues written inside each crawl-batch
  step with deterministic row ids + upserts (retry idempotent); slim step
  state; robots.txt checkpointed as step state; merged progress steps keep a
  10k-page crawl within the Workflows step budget.
- Crawler: manual redirect handling with inline follow of normalization-
  equivalent redirects, response header capture (X-Robots-Tag, Link
  rel=canonical), BFS depth, sitemap-last seeding, SSRF check on discovered
  links, honest 'we were blocked' classification (403/429/cf-mitigated/
  challenge).
- MCP: run_site_audit, get_audit_status, get_audit_issues, get_audit_pages;
  limitTier resolved via shared AuditService.resolveAuditLimitTier.
- Lighthouse strategies reduced to auto/none (legacy all/manual map on read).
- Self-healing: getStatus reconciles audits whose workflow instance errored/
  terminated without reaching mark-failed.

The Issues UI and the badseo.dev e2e fixture site stack on top of this PR.

Deploy notes: run db:migrate:prod (additive); terminate running audits before
deploying — the workflow step structure changed and in-flight instances cannot
replay under the new code (a finalize guard fails them loudly instead of
completing empty).

* Site audit P0 (2/3): Issues tab UI

- Issues tab (new default) with severity grouping, per-type explanations and
  how-to-fix, drill-down to affected pages, CSV/JSON/Sheets export, and the
  'we were blocked' banner when the crawl was challenged.
- Tabs always render (Issues/Pages, Performance when Lighthouse ran);
  audit route search schema gains the issues tab and defaults to it.

Stacks on claude/audit-p0-server (issue engine + persistence).

* badseo.dev: render the badge logo as a white tree, no chip

The silver source logo was invisible on the dark pill, so it sat in a white
chip. Render it white via a CSS filter instead, so the tree fills the pill
with no backing background.

* badseo.dev: add build (typecheck) step before deploy

- Add 'build'/'typecheck' scripts (tsc --noEmit); 'deploy' now runs the build
  before wrangler deploy.
- Scope the tsconfig typecheck to the Worker source (src/); the e2e harness in
  scripts/ imports the main app and is run with tsx from the repo root.
- Document the deploy flow and first-time custom-domain setup in the README.

* badseo.dev: add trailing-slash redirect-cycle fixture + regression test

Reproduces the 508 "Loop Detected" class of bug from every-app/open-seo#61: a
CMS-style page whose canonical URL ends in a trailing slash, with the non-slash
form 301-redirecting to it. A crawler that strips trailing slashes turns the
canonical /foo/ back into /foo, follows the 301 to /foo/, strips it again, and
loops.

- New fixture at /redirect/trailing-slash: the non-slash form (intercepted in
  index.ts on the raw path) 301s to the slash form, which is served as the
  canonical 200.
- Harness asserts the page is crawled exactly once as a 200 with NO redirect
  loop, plus a dedicated "Trailing-slash cycle -> 200, no loop" guard.

Verified the guard bites: temporarily disabling crawlPage's slash-canonical
inline-follow makes both checks fail (redirect-loop, status 301); with it in
place the harness is 38/38, 25/25 issue types.

* Add webapp-testing skill (installed via /reload-skills)

Vendors the anthropics/skills webapp-testing toolkit: real files under
.agents/skills/webapp-testing, a symlink from .claude/skills/, and skills-lock.json
pinning the source + hash. Matches how the other project skills are tracked.

* Site audit: redesign issues tab as grouped table + calmer page header

- Issues: single bordered table with severity sections (Critical/Warning/Info
  headers carry the counts), dot indicators instead of filled pills, plain
  right-aligned page counts, all rows collapsed by default; expanded rows get
  a severity-colored left rule
- Removed the dead severity-count chips (they looked like filters but were
  inert spans)
- Header: audited hostname is now the H1 with the status badge inline
- Blocked banner: compact tinted panel instead of a full-size alert
- Stats: hairline strip instead of four separate cards; issues stat shows a
  severity breakdown, Lighthouse tile hidden when no tests ran, dropped the
  orange issues-count coloring

* audit: fix trailing-slash redirect cycle at the root (preserve slashes)

Replaces the crawlPage inline-follow workaround with the root-cause fix, so we
don't carry two fixes for the same bug (every-app/open-seo#61).

- normalizeUrl: stop stripping trailing slashes. A trailing slash is the
  canonical form on most CMSes, which 301 the non-slash version to it. Stripping
  rewrote the canonical URL into its own redirect source and looped (508). Now
  /path and /path/ are distinct and the redirect resolves normally.
- crawlPage: remove the isSelfAfterNormalization inline-follow (+ now-unused
  resolveRawUrl). With slashes preserved it's dead code; a trailing-slash
  redirect is recorded as an ordinary hop.
- add canonicalUrlKey (www/http/https-tolerant) and use it for the Lighthouse
  homepage match, which had the same redirect-mismatch vulnerability.
- tests: preserve-trailing-slash + canonicalUrlKey unit tests; badseo harness
  guard is now fix-agnostic (canonical resolves to 200, no loop/error).

Verified: 36 audit unit tests pass, tsc clean, badseo e2e 38/38. Reintroducing
stripping makes the trailing-slash guard fail (redirect-loop), confirming the
regression guard bites.

* Audit: add no-outgoing-links + meta-description-too-short checks, catch empty H1s

Two checks Ahrefs covers that we didn't, plus a fix: <h1></h1> now counts
as missing. badseo.dev gains fixtures for all three (41 checks, 27/27
issue types covered).

* Audit pages table: honest redirect/non-HTML rows, wrapped titles

- 3xx rows show their redirect target (dim →) instead of a red 'missing'
  title, and dash out H1/Words/Images since nothing was analyzed
- red 'missing' only when the engine actually flagged missing-title, so
  200 non-HTML files (security.txt) read as blank, not broken
- URL cells include the host when it differs from the audited site's, so
  apex→www redirect sources no longer render identically to their target
- titles wrap to two lines (line-clamp) in a wider column instead of
  truncating at 220px; PagesTable moved to its own file (lint max-lines)

* Audit pages table: canonical-host display, URL default sort, full title wrap

- host prefix now compares against the site's predominant 2xx host, not
  the typed start URL — auditing apex 12port.com no longer prefixes every
  www row with the host
- default sort by URL so the table opens as a site inventory instead of
  leading with redirects on error-free sites
- titles wrap fully instead of clamping at two lines; long titles are the
  thing being audited, so their tails shouldn't be hidden

* ci: exclude vendored skills from prettier; format test file

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-07 22:08:14 -04:00
Ben Senescu
1c74fded7b Site audit P0 (1/3): issue engine, incremental persistence, block detection (#362)
* Site audit P0 (1/3): issue engine, incremental persistence, block detection

Server-side foundation of the P0 feature set from docs/site-audit-pm-research.md:

- Issue engine: shared registry of issue types (severity, explanation,
  how-to-fix). Per-page reporters run inside crawl steps; cross-page checks
  (duplicate titles/descriptions/content, broken internal links, redirect
  chains/loops, orphan pages) run at finalize as SQL over the persisted crawl.
- New audit_links + audit_issues tables, audit_pages columns (depth, content
  hash, header signals, fetch class, sitemap flag); audit tables moved to
  src/db/{,pg/}audit.schema.ts; migrations 0029 (D1) / 0006 (PG).
- Incremental persistence: pages/links/issues written inside each crawl-batch
  step with deterministic row ids + upserts (retry idempotent); slim step
  state; robots.txt checkpointed as step state; merged progress steps keep a
  10k-page crawl within the Workflows step budget.
- Crawler: manual redirect handling with inline follow of normalization-
  equivalent redirects, response header capture (X-Robots-Tag, Link
  rel=canonical), BFS depth, sitemap-last seeding, SSRF check on discovered
  links, honest 'we were blocked' classification (403/429/cf-mitigated/
  challenge).
- MCP: run_site_audit, get_audit_status, get_audit_issues, get_audit_pages;
  limitTier resolved via shared AuditService.resolveAuditLimitTier.
- Lighthouse strategies reduced to auto/none (legacy all/manual map on read).
- Self-healing: getStatus reconciles audits whose workflow instance errored/
  terminated without reaching mark-failed.

The Issues UI and the badseo.dev e2e fixture site stack on top of this PR.

Deploy notes: run db:migrate:prod (additive); terminate running audits before
deploying — the workflow step structure changed and in-flight instances cannot
replay under the new code (a finalize guard fails them loudly instead of
completing empty).

* Store only internal link edges in audit_links

Both consumers (broken-internal-link and orphan checks) filter on
isInternal; per-page external counts already live on audit_pages.
Dropping external rows cuts stored edges on outbound-heavy sites.
Column stays so P1 external-link checks can re-add rows without a
migration.

* Review fixes: failAudit CAS guard, dedupe hash helpers, cheaper checks

- failAudit only transitions running audits, so the getStatus reconciler
  can't flip a just-completed audit to failed when it races finalize
- collapse the duplicate SHA-256 helper into audit/ids.ts
- finalize integrity guard uses a limit-1 existence probe instead of
  fetching every page row
- get_audit_status MCP tool no longer reads the audit row twice when an
  explicit auditId is given
2026-07-07 22:08:14 -04:00
RDeemer63
ffa6ec5e70
feat(rank-tracking): add city/region-level location targeting (#62) 2026-07-07 21:44:19 -04:00
Ben Senescu
c3caf2009f
Remove backlinks + LLM-mentions access gates 2026-07-05 23:25:33 -04:00
Ben Senescu
3c7e704b4a
Triage active PostHog errors: validator noise, workflow output cap, Autumn retries, deploy-reset noise (#361) 2026-07-05 23:22:44 -04:00
Ben Senescu
2645750671
Free-plan audit limits: 50 pages, one at a time; remove 'all' lighthouse strategy (#352) 2026-07-05 18:56:19 -04:00
Ben Senescu
c5cbe84ce6 Unify dual-backend DB layer (D1 default + Postgres opt-in) (#238) 2026-07-02 18:15:46 -04:00
mattmacrocket
b8f47a51cc
feat: GSC UI + Striking Distance 2026-07-02 17:37:36 -04:00
Eugene
752561b0ac
feat: add refresh metrics action to Saved Keywords page (#51)
* feat: add refresh metrics action to Saved Keywords page

Adds an Actions dropdown to the Saved Keywords header with an
"Update keyword stats" option. Fetches fresh volume, CPC, competition,
difficulty, and intent from DataForSEO for all saved keywords in the
project, grouped by location/language. Mirrors the existing refresh
pattern from Rank Tracking.

Closes #49

* fix(keywords): preserve full metric shape on saved-keyword refresh

Align refreshSavedKeywordMetrics with the research/save persistence shape
so a refresh never degrades stored data:

- Persist real monthly_searches trend instead of writing "[]"
- Derive Google Ads competition from competition_index/100 instead of null
- Normalize intent via normalizeIntent (Labs) / "unknown" (Ads) to match
  mapKeywordDataItems / mapAdsKeywordItems

Also extract the per-batch fetch+map into fetchBatchMetrics to resolve the
oxlint max-depth violation, and apply prettier formatting so ci:check passes.

---------

Co-authored-by: Ben Senescu <bensenescu@gmail.com>
2026-06-30 15:52:12 -04:00
Matt Van Horn
1dabbc88a6
feat: add paginated get_backlinks_profile MCP tool (#38)
* feat: add paginated get_backlinks_profile MCP tool

Exposes detailed per-link backlink rows over MCP, reusing BacklinksService
paginated row fetching. Closes #36.

* fix(test): use type-narrowing text assertion to satisfy oxlint

expect.stringContaining inside toMatchObject tripped
typescript-eslint(no-unsafe-assignment), breaking ci:check. Match the
.toContain() pattern used elsewhere in the MCP tool tests.

* refactor(mcp): simplify get_backlinks_profile handler + fix cost estimate

- Drop the redundant backlinksRowsPageRequestSchema.parse re-validation in
  the handler; the MCP SDK already validates args against inputSchema. Build
  the service request straight from args, removing the duplicated defaults
  that could silently diverge.
- Preserve the target length cap by adding .max(2048) to the input schema
  (previously enforced only via the re-parse).
- Correct the credit estimate in the tool description: measured cost is
  ~30 credits/page, not the ~200-500 copied from get_backlinks_overview.
- Minor: type the statuses accumulator.

* fix(mcp): correct get_backlinks_overview credit estimate

Measured real cost: summary (~26 credits) + history for domain scope
(~29 credits) = ~50 per domain, ~25 for a single page. The previous
~200-500 figure was a ~5-10x overestimate (same one get_backlinks_profile
inherited).

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Ben Senescu <bensenescu@gmail.com>
2026-06-28 18:47:24 -04:00
Ben Senescu
b809f87b5e
Hybrid keyword data: Google Ads coverage for non-Labs countries + opt-in clickstream volumes (#262) 2026-06-12 14:08:11 -04:00
Ben Senescu
16b92ccea3
Backlinks: server-side pagination, sorting, filters, and a one-per-domain view (#258) 2026-06-10 23:27:04 -04:00
Ben Senescu
977c69c3de
Replace project deletion with archive/restore (soft delete) (#257)
Project deletion cascaded through saved keywords, rank tracking, and
audits, and the endpoint had no role check (Codex security finding).
Instead of gating a destructive delete, remove it: archiving stamps
archived_at, hides the project everywhere (lists, project context,
rank-check cron), and preserves all data. Archived projects can be
restored from the Manage projects page.

The restore input is named archivedProjectId because the global
ensureUserMiddleware resolves any projectId in input data against
active projects and would 404 before the handler runs.
2026-06-10 15:42:55 -04:00
Ben Senescu
8a85b7b794
AI Visibility: competitor Share of Voice + clarity fixes for AI citations (#248) 2026-06-09 22:33:11 -04:00
Ben Senescu
383531c5cd
Rank tracking: trends & data exploration (#247) 2026-06-09 21:06:50 -04:00
Ben Senescu
076a7fb6d7
Multi-project support per organization (#246) 2026-06-09 20:12:09 -04:00
Ben Senescu
1495660631
fix errors and onboarding problems (#242) 2026-06-05 15:22:32 -04:00
Ben Senescu
8b74c7da61
Improve onboarding questions and sign-up routing (#225) 2026-05-26 23:05:20 -04:00
Ben Senescu
f95c66b7f2
Switch domain sorting defaults and rank-tracking suggestions to traffic (#223)
* Default domain keyword sorting to traffic

* Align keyword suggestion preselection with traffic sort
2026-05-26 18:11:14 -04:00
Ben Senescu
2484f93ce6
Refactor pages to improve performance and consistency (#205) 2026-05-20 17:23:09 -04:00
Ben Senescu
ae2031bfad
Harden domain and backlinks validation before external calls (#183)
* Validate backlinks targets before calling DataForSEO

* Validate backlinks targets before provider calls
2026-05-12 19:15:51 -04:00
Ben Senescu
570fe2aa62
Add keyword tags, server-side saved filtering, and filtered exports (#179) 2026-05-12 00:02:43 -04:00
Ben Senescu
def9390cea
feat: Implement server-side pagination and filtering for domain keywords (#143) 2026-05-04 15:26:36 -04:00
Ben Senescu
c1a9c1f57d
Replace mutation-based search with URL-driven state management (#145)
* fix: preserve cmd+click on tracked domain rows

Tracked-domain rows used <div role="button" onClick={navigate(...)}>,
which prevented standard browser open-in-new-tab behavior (cmd+click,
middle-click, right-click → open). Use TanStack Router's <Link> as a
stretched overlay so the row is a real <a href> while the archive
button stays interactive.

* fix: use <Link> for tab toggles and history navigation

Replace onClick={() => navigate(...)} / setSearchParams patterns with
TanStack Router's <Link> across surfaces that change the URL on click.
<Link> renders a real <a href> and only intercepts plain left-clicks,
so cmd/ctrl+click, middle-click, and right-click → open-in-new-tab
all work natively.

- Audit: history "View" button + Pages/Performance tab toggles.
- Domain overview: Top Keywords / Top Pages tab toggles. The keyword-
  only sort fallback now happens in the Link's search updater.
- Backlinks: history items, Backlinks/Domains/Pages tab toggles, and
  "Recent searches" back-link. Removes now-unused
  navigateToBacklinksHistory / navigateToBacklinksTab helpers.

Keyword research and domain history items, and AI search histories,
are not converted: those pages don't trigger their data fetch from
URL params alone, so a plain link target wouldn't reproduce the
current click behavior without a deeper refactor.

* refactor: unify search-page state around URL-driven fetching

Drive Keyword Research, Brand Lookup, and Prompt Explorer from URL
search params so a search is reproducible from a link alone. With
that, all three history surfaces become <Link>s and cmd+click /
right-click → "open in new tab" work natively.

- Shared SearchHistorySection now takes a renderItemLink slot so
  callers wrap history items in a <Link> with the right destination.
- Prompt Explorer: added URL search params (q, models, web, cc, hb)
  via promptExplorerSearchSchema; switched the explore mutation to
  useQuery keyed on the URL params; addSearch now fires from a
  success effect; "Recent searches" back-button is a <Link>.
- Brand Lookup: history items + "Recent searches" back-button are
  <Link>s; local form state stays in sync with URL via an effect.
- Keyword Research: form submit still navigates+kicks off a search
  for the same-URL re-submit case, but the controller also runs an
  URL-driven search trigger (with a dedup ref against the form path).
  Direct URLs, cmd+click on history, and browser back/forward all
  reproduce the same fetch. "Recent searches" back-button and the
  history items are <Link>s; the bespoke resetView path is gone.

Also drops now-unused clearKeywordSearchParams,
navigateToBacklinksHistory/Tab helpers' last consumers, and the
PromptExplorerPage's onQueryChange/onSelectHistoryItem callbacks.

* format

* refactor: replace useMutation with manual state in keyword research

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-05-04 13:00:37 -04:00
Ben Senescu
1108cdd9a7
feat: gate llm mentions in self hosted product (#134) 2026-04-23 18:02:24 -04:00
Ben Senescu
faf7a294a3
feat: AI Visibility — Brand Lookup and Prompt Explorer (#128) 2026-04-23 10:48:38 -04:00
Ben Senescu
ed65d6976c
feat: support all 94 DataForSEO countries across features (#127) 2026-04-20 15:23:25 -04:00
Ben Senescu
5c2b71d340
feat: bulk delete for saved keywords (#122) 2026-04-16 19:14:50 -04:00
Ben Senescu
9a9a9a2a6c
feat: add keyword metrics (volume, difficulty, CPC) to rank tracking (#118) 2026-04-16 10:41:31 -04:00
Ben Senescu
e78ef5e3dd
fix: accept full URLs in rank tracking domain input (#117)
Replace DOMAIN_REGEX with URL-based normalization so users can paste
full URLs (e.g. https://example.com/path) into the domain config form
without getting a ZodError. The URL class handles protocol stripping,
hostname extraction, and validation natively.
2026-04-16 02:58:34 -04:00
Ben Senescu
aeafa5649d
feat: polish rank tracking UI (#115) 2026-04-16 00:42:10 -04:00
Ben Senescu
53a83996ef feat: Rank Tracking MVP (#91) 2026-04-14 20:29:38 -04:00
Ben Senescu
93c95dc8f7 feat: add structured filters to backlinks tables (#94)
* feat: add structured filters to backlinks tables

Add per-tab filter panels (Backlinks, Referring Domains, Top Pages) with
include/exclude text search, numeric range filters, link type selector,
and visibility toggles. Filter values persist in localStorage across
searches and page reloads.

* fix: improve backlinks filter layout and hide domain overview filters on pages tab

- Add w-full to text filter inputs so they fill their grid columns
- Restructure backlinks results card to use border-separated sections
  matching the domain overview pattern
- Hide filter button and panel on domain overview's Top Pages tab since
  keyword filters don't apply there

* fix: move backlinks filter button to its own toolbar row below tabs

Match the domain overview layout: tabs in their own header row, then a
separate toolbar row with the Filters button underneath.

* style: fix prettier formatting in backlinks filter files

* fix: resolve all ci:check issues

- Remove unused BacklinksResultsHeader.tsx and useBacklinksSpamPreferences.ts
- Un-export DEFAULT_BACKLINKS_SPAM_THRESHOLD and normalizeBacklinksSpamThreshold
  (only used internally)
- Remove unused BacklinksAllFilterValues type
- Fix oxlint unsafe type assertions in useBacklinksFilters.ts

* fix: wire spam filter options through to server and remove spam filters from non-backlinks tabs

The client was not passing hideSpam/spamThreshold to the server, so the
server always defaulted to hideSpam:true with threshold 40. This made
the new client-side spam score filters on the backlinks tab silently
filter an already-truncated dataset. Now the client sends hideSpam:false
so the server returns all rows and client-side filtering works correctly.

Removed spam score filters from the Referring Domains tab since spam
filtering is not needed there.

* fix: restore backlinks spam filtering and safe filter hydration

* simplify: remove server-side spam filter wiring, keep client-side only

All backlinks filtering (including spam score) is now done client-side.
Server-side spam filtering adds complexity without meaningful benefit
at current data volumes. This removes the bridging code added in the
last two commits and simplifies filter hydration.
2026-04-08 14:09:02 -04:00
Ben Senescu
a10f4d82a0 feat: hide spam backlinks by default (#89)
* feat: hide spam backlinks by default

* feat: add configurable backlinks spam filtering

* refactor: simplify backlinks spam filtering flow

Reduce duplicated spam-filter state and normalization so the backlinks page has one source of truth and client/server caching stays aligned.

* refactor: simplify backlinks spam filter semantics

* fix: keep spam filtering scoped to backlink rows

* feat: show backlinks spam score
2026-04-08 14:09:02 -04:00
Ben Senescu
e5206ec07b fix: batch saved keyword inserts for D1 (#85) 2026-04-08 14:09:02 -04:00
Ben Senescu
d773105e8d fix: use backlinks history for default trends (#69)
* fix: use backlinks history for default trends

* simplify backlinks: remove filters, always use history endpoint

Remove the filter UI (status, subdomains, indirect links, exclude internal)
and hardcode defaults across the stack. Replace the conditional
timeseries_summary + timeseries_new_lost_summary fallback with a single
backlinks/history/live call for trend data. This reduces the overview from
5 parallel API calls to 3 and removes all conditional branching.

* improve charts

* fix: refresh backlinks cost docs
2026-04-08 14:09:02 -04:00
Ben Senescu
180657fe33 feat: add filtering and sorting to domain overview (#74)
* feat: add filtering and sorting to domain overview

Add numeric range filters (traffic, volume, CPC, KD/score, rank) and
text include/exclude filters to the domain overview page. Make CPC and
Score columns sortable. Filters persist across searches and auto-apply
to new results.

* fix: match domain filter panel layout to keyword research pattern

Restructure DomainResultsCard to use border-separated sections instead
of card-body gap layout. Filters button now sits in a toolbar row with
keyword count and search, matching the keyword research page's design.

* fix: make filter text inputs full-width so labels stack above

* fix: prettier formatting

* fix: extract domain filtering logic to fix lint errors

Move filterAndSortKeywords into domainFiltering.ts to bring
controllerInternals under 350 lines. Use explicit key array
in useDomainFilters to avoid unsafe type assertion.

* feat: add optional helpText tooltip to SortableHeader

Reuses HeaderHelpLabel from keywords components to show hover
tooltips on CPC and Score column headers.

* save
2026-04-08 14:09:02 -04:00
Ben Senescu
b15b62cae6 Fix project-route auth gaps for domain and SERP lookups (#65) 2026-04-08 14:09:01 -04:00
Ben Senescu
638f5a6602 refactor: move lighthouse audits to dataforseo (#43)
* refactor: move lighthouse audits to dataforseo

* chore: remove obsolete audit settings modal

* refactor: rename psi flows to lighthouse

* save

* refactor: simplify audit lighthouse storage flow

* fix: separate lighthouse metrics from actionable audits

* refactor: remove redundant audit project inputs

* feat: redesign lighthouse issues screen with score gauges and table layout

Replace flat score cards with circular SVG gauges, condense metrics into
a compact grid, and switch issue list from cards to an expandable table
with fixed column widths.

* test: harden lighthouse regression coverage

* fix: restore project-scoped audit inputs

* refactor: simplify lighthouse payload handling

* refactor: inline lighthouse server handlers

* refactor: share audit workflow types

* refactor: simplify lighthouse payload flows

* save

* refactor: drop project pagespeed api key

* fix: restore lighthouse issues loading with resilient project context

* fix: restore audit issues back navigation

* refactor: simplify project context and lighthouse error handling

* fix: tolerate DataForSEO lighthouse payload drift

* refactor: route audit lighthouse through dataforseo client
2026-03-26 20:25:27 -04:00
Ben Senescu
4040a854a7
feat: Add better auth (#24)
* refactor: rename delegated auth user table

* feat: scaffold hosted better auth setup

* feat: add hosted auth flows

* refactor: scope project access to organizations

* fix: harden hosted auth entry points

* fix: stabilize org backfills and auth state

* refactor: simplify hosted organization setup

* fix: restore hosted auth signup flow

* fix: preserve hosted workspace access

* fix: preserve hosted auth redirects

* Improve hosted auth UX: auto-redirect to sign-up, hide header on auth pages, add form placeholders, and trust portless dev origins

- Auto-redirect unauthenticated users to /sign-up in hosted mode
- Hide top nav on /sign-in and /sign-up for a cleaner auth experience
- Add input placeholders across sign-in and sign-up forms
- Make name field optional on sign-up (falls back to email username)
- Update copy: remove 'hosted' from user-facing text, rename link to 'Create account'
- Trust *.open-seo.localhost:1355 in dev mode to fix Better Auth origin rejection with portless worktrees

* Simplify hosted auth flow and remove standalone PSI

Use TanStack Form for sign-in and sign-up, make hosted unauthenticated handling redirect-focused, and inline auth route errors. Remove the leftover standalone PSI route, services, and table so PSI only exists within site audits.

* Align project auth with Better Auth organizations

* Make server function auth middleware global

* Reduce auth server function boilerplate

* delete migrations

* fix regenerated migration data backfills

* Simplify hosted auth flow and project audit scoping

* Use active project context for audit actions

* Allow hosted session project updates

* Let agent dev server inherit auth mode

* Match hosted header to gateway account menu

* Scope project session updates to active project

* Inline authenticated server function setup

* Polish header project and account controls

* restore auth generate script

* Use explicit project access in server functions

Make project-scoped server functions take projectId input and enforce ownership through shared middleware instead of session-backed current project state. Document the tradeoffs in an ADR so future changes can follow the same boundary.

* fix ci dependency detection for auth tooling

* Harden project auth in server middleware

Authorize projectId automatically in authenticated server middleware and add a requireProject guard for project-scoped handlers. This makes the auth boundary harder to bypass and removes ad hoc non-null assertions from server functions.

* Inline project id input schemas

Remove tiny shared projectId schema helpers where they were adding indirection without reducing real complexity. Keep project-scoped validation explicit at each server function boundary.

* Skip hosted backlinks access checks

* Simplify auth mode helpers

* Avoid rerunning auth server middleware

* Simplify server function scoping ADR

* Fix backlinks project scoping in hosted auth

* Refine auth route foundations

* Simplify ensure user auth resolution

Split auth-mode context resolvers into focused modules so the middleware reads as request orchestration instead of implementation details. Reuse a shared ensured-user context type across server middleware.

* Simplify hosted organization bootstrap

Use Better Auth to own hosted organization creation and membership so hosted auth only needs to resolve a default active organization. Keep delegated-mode compatibility records isolated in a separate helper.

* Clarify hosted auth and backlinks behavior

Document the hosted AUTH_MODE deploy contract and explain why hosted deployments skip manual backlinks verification. This makes the platform-managed behavior explicit in the code paths that differ from self-serve mode.

* Document hosted org creation callback

Explain why auth.ts injects createOrganization into the hosted org helper. This makes the dependency direction explicit and avoids future import cycles while keeping the helper reusable.

* Fix CI check failures

* Fix nav link prop forwarding

* save
2026-03-19 19:24:34 -04:00
Ben Senescu
8405564925
feat: Add Backlinks page (#23)
* feat: add backlinks setup and overview workflow

* save

* fix backlinks timeseries and refine overview layout

* update backlinks table guidance

* refactor backlinks CI cleanup

* refactor backlinks page state colocation

* fix backlinks access error handling

Differentiate DataForSEO subscription access failures from billing issues and preserve exact page hosts for page-level backlink lookups.

* fix backlinks project scoping and error states

* simplify backlinks billing and lazy loading

* harden backlinks profiling and access flow

* refine backlinks access status typing

* update backlinks scope selection and access handling

* fix backlinks scope fallback and cache scoping

* fix backlinks ci checks

* docs: simplify backlinks pricing

* refine backlinks search: move scope toggle below input, remove redundant description

* refine backlinks summaries and help text

* refine backlinks table sorting

* fix backlinks invalid target handling

* delete docs

* fix backlinks scope inference and domain normalization

* docs: mark backlinks as a supported workflow

* fix backlinks links badge wrapping

* test: split backlinks test suites
2026-03-16 17:48:13 -04:00
Ben Senescu
b9c1061934
chore: clean up knip.jsonc (#20)
* Add website

* chore: clean knip config and remove dead exports
2026-03-11 23:36:54 -04:00