* Site audit P0 (1/3): issue engine, incremental persistence, block detection
Server-side foundation of the P0 feature set from docs/site-audit-pm-research.md:
- Issue engine: shared registry of issue types (severity, explanation,
how-to-fix). Per-page reporters run inside crawl steps; cross-page checks
(duplicate titles/descriptions/content, broken internal links, redirect
chains/loops, orphan pages) run at finalize as SQL over the persisted crawl.
- New audit_links + audit_issues tables, audit_pages columns (depth, content
hash, header signals, fetch class, sitemap flag); audit tables moved to
src/db/{,pg/}audit.schema.ts; migrations 0029 (D1) / 0006 (PG).
- Incremental persistence: pages/links/issues written inside each crawl-batch
step with deterministic row ids + upserts (retry idempotent); slim step
state; robots.txt checkpointed as step state; merged progress steps keep a
10k-page crawl within the Workflows step budget.
- Crawler: manual redirect handling with inline follow of normalization-
equivalent redirects, response header capture (X-Robots-Tag, Link
rel=canonical), BFS depth, sitemap-last seeding, SSRF check on discovered
links, honest 'we were blocked' classification (403/429/cf-mitigated/
challenge).
- MCP: run_site_audit, get_audit_status, get_audit_issues, get_audit_pages;
limitTier resolved via shared AuditService.resolveAuditLimitTier.
- Lighthouse strategies reduced to auto/none (legacy all/manual map on read).
- Self-healing: getStatus reconciles audits whose workflow instance errored/
terminated without reaching mark-failed.
The Issues UI and the badseo.dev e2e fixture site stack on top of this PR.
Deploy notes: run db:migrate:prod (additive); terminate running audits before
deploying — the workflow step structure changed and in-flight instances cannot
replay under the new code (a finalize guard fails them loudly instead of
completing empty).
* Store only internal link edges in audit_links
Both consumers (broken-internal-link and orphan checks) filter on
isInternal; per-page external counts already live on audit_pages.
Dropping external rows cuts stored edges on outbound-heavy sites.
Column stays so P1 external-link checks can re-add rows without a
migration.
* Review fixes: failAudit CAS guard, dedupe hash helpers, cheaper checks
- failAudit only transitions running audits, so the getStatus reconciler
can't flip a just-completed audit to failed when it races finalize
- collapse the duplicate SHA-256 helper into audit/ids.ts
- finalize integrity guard uses a limit-1 existence probe instead of
fetching every page row
- get_audit_status MCP tool no longer reads the audit row twice when an
explicit auditId is given
* track core product analytics flows
Track auth, search, export, audit, and credit-consumption events with canonical route IDs so PostHog funnels and usage dashboards stay low-noise and privacy-safe.
* fix: keep auth actions usable after session loss
* refactor: simplify analytics and auth helpers
- Replace isRecord/getActiveOrganizationId type guards with simple cast
- Refactor getAnalyticsRouteContext from if/return chain to route tables
- Replace toVerificationIssueType switch with zod enum
- Merge duplicate credits_consume events into single event per API call
- Merge two PostHogBootstrap useEffects into one
* refactor: add projectId to middleware context to reduce boilerplate
The requireProjectContext middleware now includes projectId directly,
eliminating repeated manual construction of BillingCustomerContext
objects across all server function handlers.
* remove unused BILLING_* env var fallbacks from cost profile script
* remove before_send event enrichment to preserve native PostHog URL tracking
The before_send hook was stripping $pathname, $current_url, $referrer and
other URL properties, which breaks PostHog web analytics dashboards, paths
analysis, session replay, and attribution. The route_id/route_group injection
it provided is unnecessary since PostHog already captures $pathname natively.
* remove route mapping layer, pass raw redirect paths to analytics events
The route ID registry (STATIC_ROUTES, PROJECT_ROUTES, getAnalyticsRouteContext,
getRedirectRouteId) duplicated what PostHog already captures via $pathname.
Replace redirect_route_id with redirect_to containing the raw path, and remove
~80 lines of route mapping infrastructure.
* clean up analytics events: drop redundant submit events and derived properties
- Remove search_submit events for keywords, domain overview, and backlinks
(the search_complete events capture the meaningful outcome data)
- Remove target_type from backlinks events (derived 1:1 from search_scope)
- Remove result_limit from keyword research (requested limit, not useful
alongside actual result_count)
- Remove export_format from data:export events (always "csv")
* refactor: inline wrappers, colocate helpers, deduplicate getActiveOrganizationId
- Inline toVerificationIssueType into verify-email.tsx (single-use wrapper)
- Move mapDataforseoPathToCreditFeature into dataforseoClient.ts (only consumer)
- Extract shared getActiveOrganizationId into lib/auth-session.ts (was
duplicated in __root.tsx and middleware/ensure-user/hosted.ts)
- Rename shared/analytics.ts → shared/internal-user.ts (only email helpers
remain after removing route mapping, verification, and dataforseo helpers)
* remove internal user tracking and email domain properties
Drop is_internal_user super property, email_domain person property, and all
supporting code (shared/internal-user.ts, getEmailDomain, isInternalUserEmail).
Simplifies initPostHog and identifyAnalyticsUser signatures.
* remove backlinks:search_complete effect-based tracking
The reactive useEffect + useRef dedup pattern added ~30 lines of plumbing
inside a data hook for a single analytics event. Not worth the complexity.
* simplify: replace manual type guards with zod, deduplicate posthog and sign-out helpers
- Replace hand-rolled typeof checks in getActiveOrganizationId and
isAuthenticatedServerFunctionContext with zod safeParse
- Extract withPostHogClient helper to deduplicate client posthog wrapper
- Move apiKey guard into getServerPostHogClient factory
- Extract signOutAndRedirect to avoid duplicated sign-out logic
- Drop derivable has_results from analytics events
- Remove unnecessary path normalization in mapDataforseoPathToCreditFeature
* fix: strip email from pageview URLs, restore sign-out guard, harden server posthog, fix path mapper
- Sanitize $current_url on pageviews to remove email query param (PII)
- Restore onSuccess for sign-out redirect to avoid bounce-back on failure
- Swallow shutdown() errors so PostHog outages can't fail billed work
- Rewrite mapDataforseoPathToCreditFeature to match real API path structure
(path[1] = module, path[3] = endpoint) instead of scanning all segments
* simplify: remove redundant refs in verify-email, infer middleware context type
- Remove unnecessary useRef guards in verify-email effects (deps already prevent re-firing)
- Use z.ZodType<EnsuredUserContext> annotation to infer return type instead of casting
- Add comment explaining one-shot PostHog client on Workers
* fix: reset PostHog identity on sign-out before redirect
* fix: require POSTHOG_HOST env var instead of defaulting to us.i.posthog.com
* fix: annotate url as unknown to satisfy no-unsafe-assignment
* format
* refactor: simplify keyword source selection flow
* fix: improve stacked keyword page layout
* fix: keep SERP tied to searched keyword
Stop silently falling back SERP lookups to related keywords and show a keyword-specific empty message so missing coverage is explicit to users.
* refactor: extract keyword controller and switch SERP to live
Move keyword research orchestration out of the route, call DataForSEO organic live SERP, and harden response normalization with runtime validation for safer typed handling.
* refactor: split keyword research page state into focused hooks
* fix: aggregate auto keyword fallback and reuse cached results
* scope keyword metrics to projectId
* refactor: modularize SEO routes and harden parsing for CI checks
* refactor: adopt zod json codecs at parse boundaries
* refactor: adopt papaparse, tldts, and zod error enums
* refactor: reorganize client and server feature layout
Group keyword UI/controller files and server repositories/services by feature while keeping serverFunctions as the stable API surface. This improves navigation and maintainability without changing runtime behavior.
* refactor: move PSI business logic into service layer
Keep serverFunctions/psi as thin transport handlers by delegating orchestration, export shaping, and source resolution to PsiAuditService. This aligns PSI with the existing service-first backend pattern without changing endpoint behavior.
* refactor: enforce 350/120 size limits across modules
* move self host
* fix: show dynamic range for search trends
Replace the static 'Past 12 months' subtitle with a computed range from the actual plotted trend points so the UI reflects data lag correctly.
* fix: bind audit workflow writes to workflow instance
* chore: make ci checks pass
