process.env inside defineConfig() only sees whatever the OS process
environment already has — it does not automatically contain a project's
.env file contents just because Vite loads that file elsewhere (envPrefix
only reaches import.meta.env in app code, a separate mechanism). The
previous commit's PORT/ALLOWED_HOST read from process.env worked only by
coincidence of how a process happened to be launched, and silently did
nothing when it wasn't. loadEnv() is Vite's own documented API for reading
.env files inside the config itself, independent of the launch method.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both were documented self-host knobs (SELF_HOSTING_DOCKER.md, compose.yaml)
but hardcoded to port 3001 and seo.thedomainnest.com instead, so any second
instance on a different port/domain got either a port clash or Vite's
preview host-check ("Blocked request... not allowed"). Defaults preserve
exact current behavior for deployments that don't set these.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both were documented self-host knobs (SELF_HOSTING_DOCKER.md, compose.yaml)
but hardcoded to port 3001 and seo.thedomainnest.com instead, so any second
instance on a different port/domain got either a port clash or Vite's
preview host-check ("Blocked request... not allowed"). Defaults preserve
exact current behavior for deployments that don't set these.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three launch blockers from the Oct 17 direct-Stripe billing work:
- verifyStripeCheckoutSession defaulted an unconfirmed checkout session
to "active"/isPaying:true, and had no billing permission check, so an
abandoned (unpaid) checkout marked the org as a paying subscriber for
free. Now defaults to "incomplete" and only trusts a real subscription
status or a confirmed paid payment_status; also pins org id to the
caller's own organization instead of trusting session metadata.
- The Stripe webhook fell back to parsing an unsigned request body as a
real event whenever STRIPE_WEBHOOK_SECRET was unset, letting anyone POST
a fake checkout.session.completed and grant any org a paid plan.
STRIPE_WEBHOOK_SECRET is now required; unsigned requests are refused.
Also maps Stripe's subscription.status properly (trialing counted as
active) instead of collapsing anything but "active" to "canceled".
- meterDataforseoCall and every credit-check in subscription.ts skipped
all usage metering/capping whenever AUTUMN_SECRET_KEY was unset — which
is exactly the mode the Oct 17 work introduced for self-host direct
Stripe billing. Added a local per-org monthly usage ledger
(usage_ledger table, sqlite + pg) checked against each plan's monthly
data budget (src/shared/crawlerxPlanBudgets.ts, from the PRD's unit
economics table) so DataForSEO spend is capped even without Autumn.
Also: documented STRIPE_SECRET_KEY/STRIPE_WEBHOOK_SECRET/AUTUMN_SECRET_KEY
in .env.example, and published the CrawlerX product demo as a static page
at /demo.html (self-contained, no backend dependency).
Verified: tsc --noEmit clean, schema-parity test green, full vitest suite
green except two pre-existing unrelated failures (confirmed via stash),
knip/oxlint show no new findings over the pre-change baseline.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
16-week plan for taking the CrawlerX fork to external agency sales:
decisions, phased tasks with code pointers, risks, and a handover
checklist. Provided as Markdown and a standalone HTML page.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per-project overview pulling together GSC performance totals/top queries,
GA4 organic overview metrics, and the latest completed site audit summary
into a single page under /p/$projectId/report. Each section degrades
independently (not connected / error / ok) so a missing integration never
blocks the rest of the report. Includes a "Download PDF" button that uses
window.print() against print-aware layout classes added to AppShell so the
app chrome is hidden and content flows naturally when printed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Explicit, user-triggered button ("Get report from DataForSEO") shown when
the native crawler is blocked — never an automatic retry. DataForSEO's
OnPage API crawls from its own infrastructure with JS rendering, which
clears blocks a plain fetch() from our server can't.
- audits.crawlSource ("native" | "dataforseo") + dataforseoTaskId columns.
- dataforseo/onpage.ts: task_post (JS rendering + store_raw_html) / summary
polling / pages listing / raw_html retrieval. Only task_post is billed
(~$0.00125/page); the rest are free reads of already-billed results, per
DataForSEO's pricing docs.
- DataForSeoAuditService: hard quota of 5 runs per project per calendar
month, enforced server-side via the activity log (audit.dataforseo_report)
before any DataForSEO spend — a rejected 6th run never reaches the API.
- Reuses the native pipeline instead of duplicating it: extracted
buildAnalyzedPageResult() out of crawlPage() so both sources feed the same
analyzeHtml -> runPageReporters -> runMultipageChecks -> auditPages/
auditIssues path. No Cloudflare Workflow backs these audits; the existing
getAuditStatus poll (already running every 3s while "running") drives an
advance step each call instead.
- Known gap: broken-internal-link and orphan-page checks are native-only
(they read the crawl's link graph from the AuditScratchpad Durable Object,
which only the native crawl populates).
- UI: page-limit picker (25-500) + live quota display on the existing
"blocked" screen.
Migration: drizzle/0046_*, drizzle-pg/0024_*.
tsc / oxlint / knip clean. New onpage.test.ts (7) + DataForSeoAuditService
.test.ts (5, including the quota-rejection path); full suite otherwise
unchanged (1195 pass, pre-existing samSkills Windows-CRLF failure only).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The crawler identified as `OpenSEO-Audit/1.0` with almost no headers, which
naive bot filters and security plugins block outright.
- New crawl-request.ts: AUDIT_USER_AGENT (a current Chrome string),
buildAuditHeaders() (Accept, Accept-Language, Sec-Fetch-*, Sec-Ch-Ua,
Upgrade-Insecure-Requests), and fetchForAudit() — fetch + those headers +
one retry on a transient 429/503.
- Wired into the page crawl (site-audit-workflow-helpers), robots.txt +
sitemap discovery, and start-URL redirect probing.
Gets past the naive tier; still reported as "blocked" for JS/TLS challenges
(Cloudflare Managed Challenge, DataDome) — those need a real browser. Doc note
points operators at WAF IP/UA allowlisting for their own sites.
No env dependency (keeps the audit lib importable without a cloudflare:workers
mock). tsc / oxlint / knip clean; new crawl-request.test.ts (5); suite
otherwise unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- activity_log table (sqlite + pg, structurally identical; schema-parity
covers it). Plain-text columns, no FKs — an append-only trail that must
outlive the projects/users it references, so target_label snapshots a
human-readable name at write time.
- ActivityRepository: record() (fire-and-forget, never breaks the caller) +
list() (org-scoped, actor/action filters, keyset pagination) + listActors().
- Recording wired into the mutations worth tracking: project
create/archive/restore/domain, audit start, team user create/remove/
password-reset, invitation sent.
- getActivityLog / getActivityActors server functions (owner/admin gated) +
Settings → Activity tab (ActivityLogView: filter by user & action, load
more).
- Migration: drizzle/0045_*, drizzle-pg/0023_*. The pipeline does not run
migrations — see docs/SELF_HOSTING_TEAM_MODE.md step 5 for the one-time
`drizzle-kit migrate` on the server. Writes fail silently until the table
exists.
tsc / oxlint / knip clean. New ActivityRepository.test.ts (4) + schema-parity
picks up the new table; suite otherwise unchanged (pre-existing samSkills
CRLF failure only).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- _auth.setup.tsx: the "owner already exists?" check ran in beforeLoad, which
executes during SSR where fetchTeamSetupStatus's relative fetch to
/api/team-setup fails — so it always concluded an owner existed and
redirected to /sign-in, making the create-owner screen unreachable. Move the
check into a client-side effect with a loading state.
- setup-status.ts: add BETTER_AUTH_URL to CHECK_ENV_VARS so /api/health stops
falsely reporting "team mode requires BETTER_AUTH_URL" when it is set (the
Docker preflight already saw it; only the runtime health check's env
allowlist was missing it).
Verified locally end to end against a D1 build in AUTH_MODE=team: owner
bootstrap, self-disable + 409 on repeat, Better Auth sign-in issues a session
cookie, and get-session resolves the shared organization.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Makes AUTH_MODE=team usable end to end.
- resolveTeamContext (middleware/ensure-user/team.ts): a session resolves to a
membership in the single shared workspace. No per-user fallback org — a
signed-in user with no membership is treated as signed out, so the owner can
actually remove people.
- teamProvisioning.ts: one path that writes user + credential account + member
together (hashPassword from better-auth/crypto). Shared by both entry points.
- /api/team-setup (raw route, outside auth middleware): GET reports whether an
owner is needed; POST creates the first owner + the shared org, then
self-disables once any user exists.
- /setup route + sign-in redirect: first run sends you to create the owner.
- teamUsers server functions (owner/admin-gated): list / create (with temp
password) / reset password / remove. Removal drops membership + sessions,
keeps the user row for historical attribution.
- Settings gains a "Users" tab in team mode (TeamUsers.tsx).
- docs/SELF_HOSTING_TEAM_MODE.md: activation runbook (.env, build, first owner).
No DB migration — all rows are existing better-auth tables. tsc / oxlint / knip
clean. New teamProvisioning.test.ts (4 cases) passes; suite otherwise unchanged
(pre-existing samSkills.test.ts CRLF failure only).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Introduces a fourth AUTH_MODE, `team`: Better Auth email/password with the
existing organization/member/role/invitation stack, but none of the hosted
SaaS coupling (no Autumn billing, Turnstile, Loops email, Google social
login, onboarding chat, PostHog, disposable-email block, dub referrals).
- auth-mode.ts: add `team`; add isTeamAuthMode / isSessionAuthMode /
isSessionClientAuthMode ("is there a login session?" vs isHostedAuthMode's
"is this the billed product?").
- auth.ts: createAuth() builds a valid instance for `team` — verification
off, self-serve signup disabled, no captcha/Loops/social. hasTeamAuthConfig
(BETTER_AUTH_URL + BETTER_AUTH_SECRET only) + hasSessionAuthConfig.
- ensure-user: resolve.ts routes `team` through resolveHostedContext;
requireHostedSession + selfHostedOAuth callback accept any session mode.
- api/auth/$.ts: mount the Better Auth handler for `team` too.
- Client: route guards, sidebar account menu / sign-out, settings
Organization tab, invitation accept, and error cards switch from
isHostedClientAuthMode to isSessionClientAuthMode where they mean "has a
session". Sign-in goes straight to the email form (no Google button);
sign-up shows an invite-only notice.
- selfhost-preflight: validate `team` (requires BETTER_AUTH_URL +
BETTER_AUTH_SECRET >= 32 chars).
- .env.example: document `team`.
Ships inert: AUTH_MODE stays local_noauth. tsc / oxlint / knip clean;
test suite unchanged (1164 pass, 1 pre-existing Windows-CRLF failure in
samSkills.test.ts).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- DaisyUI themes openseo/openseo-dark -> crawlerx/crawlerx-dark on the
MetatronCube palette (primary #3779b9, secondary #0f1d34, accent #4faede)
- Load Inter as the app font; add --brand-gradient / .brand-gradient
- Cube mark + "CrawlerX" wordmark in the sidebar and mobile top bar
- Add public/crawlerx-mark.png; point transparent-logo.png at the cube mark
- <title>, site.webmanifest, and all user-facing "OpenSEO" copy across
src/client and src/routes -> "CrawlerX"
Deferred: favicons (need resizing), agent-facing identity (MCP server
name, SAM/onboarding prompts, fact sheet), web/ marketing site.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
