Three launch blockers from the Oct 17 direct-Stripe billing work:
- verifyStripeCheckoutSession defaulted an unconfirmed checkout session
to "active"/isPaying:true, and had no billing permission check, so an
abandoned (unpaid) checkout marked the org as a paying subscriber for
free. Now defaults to "incomplete" and only trusts a real subscription
status or a confirmed paid payment_status; also pins org id to the
caller's own organization instead of trusting session metadata.
- The Stripe webhook fell back to parsing an unsigned request body as a
real event whenever STRIPE_WEBHOOK_SECRET was unset, letting anyone POST
a fake checkout.session.completed and grant any org a paid plan.
STRIPE_WEBHOOK_SECRET is now required; unsigned requests are refused.
Also maps Stripe's subscription.status properly (trialing counted as
active) instead of collapsing anything but "active" to "canceled".
- meterDataforseoCall and every credit-check in subscription.ts skipped
all usage metering/capping whenever AUTUMN_SECRET_KEY was unset — which
is exactly the mode the Oct 17 work introduced for self-host direct
Stripe billing. Added a local per-org monthly usage ledger
(usage_ledger table, sqlite + pg) checked against each plan's monthly
data budget (src/shared/crawlerxPlanBudgets.ts, from the PRD's unit
economics table) so DataForSEO spend is capped even without Autumn.
Also: documented STRIPE_SECRET_KEY/STRIPE_WEBHOOK_SECRET/AUTUMN_SECRET_KEY
in .env.example, and published the CrawlerX product demo as a static page
at /demo.html (self-contained, no backend dependency).
Verified: tsc --noEmit clean, schema-parity test green, full vitest suite
green except two pre-existing unrelated failures (confirmed via stash),
knip/oxlint show no new findings over the pre-change baseline.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per-project overview pulling together GSC performance totals/top queries,
GA4 organic overview metrics, and the latest completed site audit summary
into a single page under /p/$projectId/report. Each section degrades
independently (not connected / error / ok) so a missing integration never
blocks the rest of the report. Includes a "Download PDF" button that uses
window.print() against print-aware layout classes added to AppShell so the
app chrome is hidden and content flows naturally when printed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Explicit, user-triggered button ("Get report from DataForSEO") shown when
the native crawler is blocked — never an automatic retry. DataForSEO's
OnPage API crawls from its own infrastructure with JS rendering, which
clears blocks a plain fetch() from our server can't.
- audits.crawlSource ("native" | "dataforseo") + dataforseoTaskId columns.
- dataforseo/onpage.ts: task_post (JS rendering + store_raw_html) / summary
polling / pages listing / raw_html retrieval. Only task_post is billed
(~$0.00125/page); the rest are free reads of already-billed results, per
DataForSEO's pricing docs.
- DataForSeoAuditService: hard quota of 5 runs per project per calendar
month, enforced server-side via the activity log (audit.dataforseo_report)
before any DataForSEO spend — a rejected 6th run never reaches the API.
- Reuses the native pipeline instead of duplicating it: extracted
buildAnalyzedPageResult() out of crawlPage() so both sources feed the same
analyzeHtml -> runPageReporters -> runMultipageChecks -> auditPages/
auditIssues path. No Cloudflare Workflow backs these audits; the existing
getAuditStatus poll (already running every 3s while "running") drives an
advance step each call instead.
- Known gap: broken-internal-link and orphan-page checks are native-only
(they read the crawl's link graph from the AuditScratchpad Durable Object,
which only the native crawl populates).
- UI: page-limit picker (25-500) + live quota display on the existing
"blocked" screen.
Migration: drizzle/0046_*, drizzle-pg/0024_*.
tsc / oxlint / knip clean. New onpage.test.ts (7) + DataForSeoAuditService
.test.ts (5, including the quota-rejection path); full suite otherwise
unchanged (1195 pass, pre-existing samSkills Windows-CRLF failure only).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The crawler identified as `OpenSEO-Audit/1.0` with almost no headers, which
naive bot filters and security plugins block outright.
- New crawl-request.ts: AUDIT_USER_AGENT (a current Chrome string),
buildAuditHeaders() (Accept, Accept-Language, Sec-Fetch-*, Sec-Ch-Ua,
Upgrade-Insecure-Requests), and fetchForAudit() — fetch + those headers +
one retry on a transient 429/503.
- Wired into the page crawl (site-audit-workflow-helpers), robots.txt +
sitemap discovery, and start-URL redirect probing.
Gets past the naive tier; still reported as "blocked" for JS/TLS challenges
(Cloudflare Managed Challenge, DataDome) — those need a real browser. Doc note
points operators at WAF IP/UA allowlisting for their own sites.
No env dependency (keeps the audit lib importable without a cloudflare:workers
mock). tsc / oxlint / knip clean; new crawl-request.test.ts (5); suite
otherwise unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- activity_log table (sqlite + pg, structurally identical; schema-parity
covers it). Plain-text columns, no FKs — an append-only trail that must
outlive the projects/users it references, so target_label snapshots a
human-readable name at write time.
- ActivityRepository: record() (fire-and-forget, never breaks the caller) +
list() (org-scoped, actor/action filters, keyset pagination) + listActors().
- Recording wired into the mutations worth tracking: project
create/archive/restore/domain, audit start, team user create/remove/
password-reset, invitation sent.
- getActivityLog / getActivityActors server functions (owner/admin gated) +
Settings → Activity tab (ActivityLogView: filter by user & action, load
more).
- Migration: drizzle/0045_*, drizzle-pg/0023_*. The pipeline does not run
migrations — see docs/SELF_HOSTING_TEAM_MODE.md step 5 for the one-time
`drizzle-kit migrate` on the server. Writes fail silently until the table
exists.
tsc / oxlint / knip clean. New ActivityRepository.test.ts (4) + schema-parity
picks up the new table; suite otherwise unchanged (pre-existing samSkills
CRLF failure only).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- _auth.setup.tsx: the "owner already exists?" check ran in beforeLoad, which
executes during SSR where fetchTeamSetupStatus's relative fetch to
/api/team-setup fails — so it always concluded an owner existed and
redirected to /sign-in, making the create-owner screen unreachable. Move the
check into a client-side effect with a loading state.
- setup-status.ts: add BETTER_AUTH_URL to CHECK_ENV_VARS so /api/health stops
falsely reporting "team mode requires BETTER_AUTH_URL" when it is set (the
Docker preflight already saw it; only the runtime health check's env
allowlist was missing it).
Verified locally end to end against a D1 build in AUTH_MODE=team: owner
bootstrap, self-disable + 409 on repeat, Better Auth sign-in issues a session
cookie, and get-session resolves the shared organization.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Makes AUTH_MODE=team usable end to end.
- resolveTeamContext (middleware/ensure-user/team.ts): a session resolves to a
membership in the single shared workspace. No per-user fallback org — a
signed-in user with no membership is treated as signed out, so the owner can
actually remove people.
- teamProvisioning.ts: one path that writes user + credential account + member
together (hashPassword from better-auth/crypto). Shared by both entry points.
- /api/team-setup (raw route, outside auth middleware): GET reports whether an
owner is needed; POST creates the first owner + the shared org, then
self-disables once any user exists.
- /setup route + sign-in redirect: first run sends you to create the owner.
- teamUsers server functions (owner/admin-gated): list / create (with temp
password) / reset password / remove. Removal drops membership + sessions,
keeps the user row for historical attribution.
- Settings gains a "Users" tab in team mode (TeamUsers.tsx).
- docs/SELF_HOSTING_TEAM_MODE.md: activation runbook (.env, build, first owner).
No DB migration — all rows are existing better-auth tables. tsc / oxlint / knip
clean. New teamProvisioning.test.ts (4 cases) passes; suite otherwise unchanged
(pre-existing samSkills.test.ts CRLF failure only).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Introduces a fourth AUTH_MODE, `team`: Better Auth email/password with the
existing organization/member/role/invitation stack, but none of the hosted
SaaS coupling (no Autumn billing, Turnstile, Loops email, Google social
login, onboarding chat, PostHog, disposable-email block, dub referrals).
- auth-mode.ts: add `team`; add isTeamAuthMode / isSessionAuthMode /
isSessionClientAuthMode ("is there a login session?" vs isHostedAuthMode's
"is this the billed product?").
- auth.ts: createAuth() builds a valid instance for `team` — verification
off, self-serve signup disabled, no captcha/Loops/social. hasTeamAuthConfig
(BETTER_AUTH_URL + BETTER_AUTH_SECRET only) + hasSessionAuthConfig.
- ensure-user: resolve.ts routes `team` through resolveHostedContext;
requireHostedSession + selfHostedOAuth callback accept any session mode.
- api/auth/$.ts: mount the Better Auth handler for `team` too.
- Client: route guards, sidebar account menu / sign-out, settings
Organization tab, invitation accept, and error cards switch from
isHostedClientAuthMode to isSessionClientAuthMode where they mean "has a
session". Sign-in goes straight to the email form (no Google button);
sign-up shows an invite-only notice.
- selfhost-preflight: validate `team` (requires BETTER_AUTH_URL +
BETTER_AUTH_SECRET >= 32 chars).
- .env.example: document `team`.
Ships inert: AUTH_MODE stays local_noauth. tsc / oxlint / knip clean;
test suite unchanged (1164 pass, 1 pre-existing Windows-CRLF failure in
samSkills.test.ts).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Audit crawl: retry chunks conservatively and stop re-learning the window per chunk
Production audits on heavy-page sites (500-700KB/page) died with
exceededMemory in the first crawl chunk, and the single step retry re-ran
the exact same profile and died again.
- claimChunk now reports isRetry (leftover leases from a dead attempt);
a retried chunk crawls under RETRY_CRAWL_WINDOW (start 3, max 5,
halved byte budget) instead of restarting at window 10.
- The adapted window carries across chunks via durable step results, so
every ~200 pages no longer re-spikes to the initial window.
- First persist sub-batch shrinks to 5 pages so the byte bound sees the
site's page weight before a full 25-page batch is in flight.
- In-flight HTML budget halved to 8 MiB (16 MiB never constrained the
observed ~650KB pages: bound was 25, above the 20 max); growth now
requires a full 25-page sample.
- Parse-time caps: 1,000 extracted links/images per page so mega-menu and
crawler-trap pages can't bloat retained persist batches.
* Guard endWindow for instances replaying pre-deploy step results
* ci: un-export internal-only interfaces (knip)
