The marketing site's "Run Live Audit" widget was entirely fake: it looked
up a canned preset or a generic templated result for any domain typed in,
after a scripted delay. This adds /api/public/quick-audit, which crawls
the submitted URL with the same crawler + HTML analyzer the authenticated
product uses, and reports one real finding (or an honest error if the
site is unreachable/blocked) instead of ever fabricating one. Per-IP and
global daily KV caps bound the cost of anonymous abuse.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The previous commit pointed every <img> at crawlerx-mark.png instead of
transparent-logo.png, but both files are byte-for-byte the same
MetatronCube glass-cube artwork — the Oct-era rebrand renamed the file,
it never replaced the image. So the logo never actually changed anywhere
in the real app, despite every component referencing the "right" filename.
Added the real CrawlerX mark (the teal/amber check-and-arrow symbol
already used on the marketing site) as public/crawlerx-logo.svg and
repointed every logo reference at it.
Also: the favicons and apple-touch-icon were a literal pine tree (unrelated
leftover asset, not MetatronCube or CrawlerX), and the webmanifest's
theme_color was still MetatronCube blue. Regenerated all icon sizes from
the real mark and fixed the manifest color.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The Oct-era rebrand commit swapped the sidebar/app-shell logo and most
daisyUI component colors, but missed several spots:
- app.css's "crawlerx" / "crawlerx-dark" theme tokens were still the old
MetatronCube blue (#3779b9) instead of the brand guide's teal (#0E6E6A).
Every daisyUI-token-based component (most of the app) now gets the
correct color with this one change.
- AuthPage.tsx (sign-in/sign-up) was untouched by the token system: a
mountain-climber stock photo background, "Metatroncube / Solutions"
wordmark, blue/cyan gradients, generic value props, and a stray
alt="Metatroncube" on the logo. Replaced with the brand gradient, the
CrawlerX mark, on-brand copy and value props, and the real color tokens
instead of hardcoded Tailwind blue/cyan classes.
- Four more components (onboarding chat, post-signup onboarding, the
OAuth consent screen, the subscribe page) still pointed their logo
<img> at /transparent-logo.png instead of /crawlerx-mark.png.
- PricingGraphics.tsx (the decorative cube/chart illustration on the
subscribe page) had ~17 hardcoded MetatronCube-blue hex values; remapped
the whole palette to the CrawlerX teal family.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
autumn-js's client hooks call response.json() on every response regardless
of status. The plain-text "Not found" body from the previous commit's guard
made that throw a SyntaxError client-side ("Unexpected token 'N'...") in
hosted mode once Autumn-dependent UI actually mounted and called it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two call sites still hard-required AUTUMN_SECRET_KEY regardless of billing
mode: the /api/autumn/$ route (autumn-js's client-side hooks hit it after
sign-up/on billing screens) and getBillingUsageEvents (Autumn's per-event
usage history). Both now degrade gracefully — 404 and an empty list,
respectively — when Autumn isn't configured, the same way subscription.ts's
credit checks already do, instead of autumn-js throwing "secret key not
found" once a request actually reaches it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Verification, invitation, and password-reset emails always said
"OpenSEO" regardless of which product/brand is actually deployed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
process.env inside defineConfig() only sees whatever the OS process
environment already has — it does not automatically contain a project's
.env file contents just because Vite loads that file elsewhere (envPrefix
only reaches import.meta.env in app code, a separate mechanism). The
previous commit's PORT/ALLOWED_HOST read from process.env worked only by
coincidence of how a process happened to be launched, and silently did
nothing when it wasn't. loadEnv() is Vite's own documented API for reading
.env files inside the config itself, independent of the launch method.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both were documented self-host knobs (SELF_HOSTING_DOCKER.md, compose.yaml)
but hardcoded to port 3001 and seo.thedomainnest.com instead, so any second
instance on a different port/domain got either a port clash or Vite's
preview host-check ("Blocked request... not allowed"). Defaults preserve
exact current behavior for deployments that don't set these.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both were documented self-host knobs (SELF_HOSTING_DOCKER.md, compose.yaml)
but hardcoded to port 3001 and seo.thedomainnest.com instead, so any second
instance on a different port/domain got either a port clash or Vite's
preview host-check ("Blocked request... not allowed"). Defaults preserve
exact current behavior for deployments that don't set these.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three launch blockers from the Oct 17 direct-Stripe billing work:
- verifyStripeCheckoutSession defaulted an unconfirmed checkout session
to "active"/isPaying:true, and had no billing permission check, so an
abandoned (unpaid) checkout marked the org as a paying subscriber for
free. Now defaults to "incomplete" and only trusts a real subscription
status or a confirmed paid payment_status; also pins org id to the
caller's own organization instead of trusting session metadata.
- The Stripe webhook fell back to parsing an unsigned request body as a
real event whenever STRIPE_WEBHOOK_SECRET was unset, letting anyone POST
a fake checkout.session.completed and grant any org a paid plan.
STRIPE_WEBHOOK_SECRET is now required; unsigned requests are refused.
Also maps Stripe's subscription.status properly (trialing counted as
active) instead of collapsing anything but "active" to "canceled".
- meterDataforseoCall and every credit-check in subscription.ts skipped
all usage metering/capping whenever AUTUMN_SECRET_KEY was unset — which
is exactly the mode the Oct 17 work introduced for self-host direct
Stripe billing. Added a local per-org monthly usage ledger
(usage_ledger table, sqlite + pg) checked against each plan's monthly
data budget (src/shared/crawlerxPlanBudgets.ts, from the PRD's unit
economics table) so DataForSEO spend is capped even without Autumn.
Also: documented STRIPE_SECRET_KEY/STRIPE_WEBHOOK_SECRET/AUTUMN_SECRET_KEY
in .env.example, and published the CrawlerX product demo as a static page
at /demo.html (self-contained, no backend dependency).
Verified: tsc --noEmit clean, schema-parity test green, full vitest suite
green except two pre-existing unrelated failures (confirmed via stash),
knip/oxlint show no new findings over the pre-change baseline.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
16-week plan for taking the CrawlerX fork to external agency sales:
decisions, phased tasks with code pointers, risks, and a handover
checklist. Provided as Markdown and a standalone HTML page.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per-project overview pulling together GSC performance totals/top queries,
GA4 organic overview metrics, and the latest completed site audit summary
into a single page under /p/$projectId/report. Each section degrades
independently (not connected / error / ok) so a missing integration never
blocks the rest of the report. Includes a "Download PDF" button that uses
window.print() against print-aware layout classes added to AppShell so the
app chrome is hidden and content flows naturally when printed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Explicit, user-triggered button ("Get report from DataForSEO") shown when
the native crawler is blocked — never an automatic retry. DataForSEO's
OnPage API crawls from its own infrastructure with JS rendering, which
clears blocks a plain fetch() from our server can't.
- audits.crawlSource ("native" | "dataforseo") + dataforseoTaskId columns.
- dataforseo/onpage.ts: task_post (JS rendering + store_raw_html) / summary
polling / pages listing / raw_html retrieval. Only task_post is billed
(~$0.00125/page); the rest are free reads of already-billed results, per
DataForSEO's pricing docs.
- DataForSeoAuditService: hard quota of 5 runs per project per calendar
month, enforced server-side via the activity log (audit.dataforseo_report)
before any DataForSEO spend — a rejected 6th run never reaches the API.
- Reuses the native pipeline instead of duplicating it: extracted
buildAnalyzedPageResult() out of crawlPage() so both sources feed the same
analyzeHtml -> runPageReporters -> runMultipageChecks -> auditPages/
auditIssues path. No Cloudflare Workflow backs these audits; the existing
getAuditStatus poll (already running every 3s while "running") drives an
advance step each call instead.
- Known gap: broken-internal-link and orphan-page checks are native-only
(they read the crawl's link graph from the AuditScratchpad Durable Object,
which only the native crawl populates).
- UI: page-limit picker (25-500) + live quota display on the existing
"blocked" screen.
Migration: drizzle/0046_*, drizzle-pg/0024_*.
tsc / oxlint / knip clean. New onpage.test.ts (7) + DataForSeoAuditService
.test.ts (5, including the quota-rejection path); full suite otherwise
unchanged (1195 pass, pre-existing samSkills Windows-CRLF failure only).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The crawler identified as `OpenSEO-Audit/1.0` with almost no headers, which
naive bot filters and security plugins block outright.
- New crawl-request.ts: AUDIT_USER_AGENT (a current Chrome string),
buildAuditHeaders() (Accept, Accept-Language, Sec-Fetch-*, Sec-Ch-Ua,
Upgrade-Insecure-Requests), and fetchForAudit() — fetch + those headers +
one retry on a transient 429/503.
- Wired into the page crawl (site-audit-workflow-helpers), robots.txt +
sitemap discovery, and start-URL redirect probing.
Gets past the naive tier; still reported as "blocked" for JS/TLS challenges
(Cloudflare Managed Challenge, DataDome) — those need a real browser. Doc note
points operators at WAF IP/UA allowlisting for their own sites.
No env dependency (keeps the audit lib importable without a cloudflare:workers
mock). tsc / oxlint / knip clean; new crawl-request.test.ts (5); suite
otherwise unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- activity_log table (sqlite + pg, structurally identical; schema-parity
covers it). Plain-text columns, no FKs — an append-only trail that must
outlive the projects/users it references, so target_label snapshots a
human-readable name at write time.
- ActivityRepository: record() (fire-and-forget, never breaks the caller) +
list() (org-scoped, actor/action filters, keyset pagination) + listActors().
- Recording wired into the mutations worth tracking: project
create/archive/restore/domain, audit start, team user create/remove/
password-reset, invitation sent.
- getActivityLog / getActivityActors server functions (owner/admin gated) +
Settings → Activity tab (ActivityLogView: filter by user & action, load
more).
- Migration: drizzle/0045_*, drizzle-pg/0023_*. The pipeline does not run
migrations — see docs/SELF_HOSTING_TEAM_MODE.md step 5 for the one-time
`drizzle-kit migrate` on the server. Writes fail silently until the table
exists.
tsc / oxlint / knip clean. New ActivityRepository.test.ts (4) + schema-parity
picks up the new table; suite otherwise unchanged (pre-existing samSkills
CRLF failure only).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- _auth.setup.tsx: the "owner already exists?" check ran in beforeLoad, which
executes during SSR where fetchTeamSetupStatus's relative fetch to
/api/team-setup fails — so it always concluded an owner existed and
redirected to /sign-in, making the create-owner screen unreachable. Move the
check into a client-side effect with a loading state.
- setup-status.ts: add BETTER_AUTH_URL to CHECK_ENV_VARS so /api/health stops
falsely reporting "team mode requires BETTER_AUTH_URL" when it is set (the
Docker preflight already saw it; only the runtime health check's env
allowlist was missing it).
Verified locally end to end against a D1 build in AUTH_MODE=team: owner
bootstrap, self-disable + 409 on repeat, Better Auth sign-in issues a session
cookie, and get-session resolves the shared organization.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Makes AUTH_MODE=team usable end to end.
- resolveTeamContext (middleware/ensure-user/team.ts): a session resolves to a
membership in the single shared workspace. No per-user fallback org — a
signed-in user with no membership is treated as signed out, so the owner can
actually remove people.
- teamProvisioning.ts: one path that writes user + credential account + member
together (hashPassword from better-auth/crypto). Shared by both entry points.
- /api/team-setup (raw route, outside auth middleware): GET reports whether an
owner is needed; POST creates the first owner + the shared org, then
self-disables once any user exists.
- /setup route + sign-in redirect: first run sends you to create the owner.
- teamUsers server functions (owner/admin-gated): list / create (with temp
password) / reset password / remove. Removal drops membership + sessions,
keeps the user row for historical attribution.
- Settings gains a "Users" tab in team mode (TeamUsers.tsx).
- docs/SELF_HOSTING_TEAM_MODE.md: activation runbook (.env, build, first owner).
No DB migration — all rows are existing better-auth tables. tsc / oxlint / knip
clean. New teamProvisioning.test.ts (4 cases) passes; suite otherwise unchanged
(pre-existing samSkills.test.ts CRLF failure only).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Introduces a fourth AUTH_MODE, `team`: Better Auth email/password with the
existing organization/member/role/invitation stack, but none of the hosted
SaaS coupling (no Autumn billing, Turnstile, Loops email, Google social
login, onboarding chat, PostHog, disposable-email block, dub referrals).
- auth-mode.ts: add `team`; add isTeamAuthMode / isSessionAuthMode /
isSessionClientAuthMode ("is there a login session?" vs isHostedAuthMode's
"is this the billed product?").
- auth.ts: createAuth() builds a valid instance for `team` — verification
off, self-serve signup disabled, no captcha/Loops/social. hasTeamAuthConfig
(BETTER_AUTH_URL + BETTER_AUTH_SECRET only) + hasSessionAuthConfig.
- ensure-user: resolve.ts routes `team` through resolveHostedContext;
requireHostedSession + selfHostedOAuth callback accept any session mode.
- api/auth/$.ts: mount the Better Auth handler for `team` too.
- Client: route guards, sidebar account menu / sign-out, settings
Organization tab, invitation accept, and error cards switch from
isHostedClientAuthMode to isSessionClientAuthMode where they mean "has a
session". Sign-in goes straight to the email form (no Google button);
sign-up shows an invite-only notice.
- selfhost-preflight: validate `team` (requires BETTER_AUTH_URL +
BETTER_AUTH_SECRET >= 32 chars).
- .env.example: document `team`.
Ships inert: AUTH_MODE stays local_noauth. tsc / oxlint / knip clean;
test suite unchanged (1164 pass, 1 pre-existing Windows-CRLF failure in
samSkills.test.ts).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- DaisyUI themes openseo/openseo-dark -> crawlerx/crawlerx-dark on the
MetatronCube palette (primary #3779b9, secondary #0f1d34, accent #4faede)
- Load Inter as the app font; add --brand-gradient / .brand-gradient
- Cube mark + "CrawlerX" wordmark in the sidebar and mobile top bar
- Add public/crawlerx-mark.png; point transparent-logo.png at the cube mark
- <title>, site.webmanifest, and all user-facing "OpenSEO" copy across
src/client and src/routes -> "CrawlerX"
Deferred: favicons (need resizing), agent-facing identity (MCP server
name, SAM/onboarding prompts, fact sheet), web/ marketing site.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
