Explicit, user-triggered button ("Get report from DataForSEO") shown when
the native crawler is blocked — never an automatic retry. DataForSEO's
OnPage API crawls from its own infrastructure with JS rendering, which
clears blocks a plain fetch() from our server can't.
- audits.crawlSource ("native" | "dataforseo") + dataforseoTaskId columns.
- dataforseo/onpage.ts: task_post (JS rendering + store_raw_html) / summary
polling / pages listing / raw_html retrieval. Only task_post is billed
(~$0.00125/page); the rest are free reads of already-billed results, per
DataForSEO's pricing docs.
- DataForSeoAuditService: hard quota of 5 runs per project per calendar
month, enforced server-side via the activity log (audit.dataforseo_report)
before any DataForSEO spend — a rejected 6th run never reaches the API.
- Reuses the native pipeline instead of duplicating it: extracted
buildAnalyzedPageResult() out of crawlPage() so both sources feed the same
analyzeHtml -> runPageReporters -> runMultipageChecks -> auditPages/
auditIssues path. No Cloudflare Workflow backs these audits; the existing
getAuditStatus poll (already running every 3s while "running") drives an
advance step each call instead.
- Known gap: broken-internal-link and orphan-page checks are native-only
(they read the crawl's link graph from the AuditScratchpad Durable Object,
which only the native crawl populates).
- UI: page-limit picker (25-500) + live quota display on the existing
"blocked" screen.
Migration: drizzle/0046_*, drizzle-pg/0024_*.
tsc / oxlint / knip clean. New onpage.test.ts (7) + DataForSeoAuditService
.test.ts (5, including the quota-rejection path); full suite otherwise
unchanged (1195 pass, pre-existing samSkills Windows-CRLF failure only).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The crawler identified as `OpenSEO-Audit/1.0` with almost no headers, which
naive bot filters and security plugins block outright.
- New crawl-request.ts: AUDIT_USER_AGENT (a current Chrome string),
buildAuditHeaders() (Accept, Accept-Language, Sec-Fetch-*, Sec-Ch-Ua,
Upgrade-Insecure-Requests), and fetchForAudit() — fetch + those headers +
one retry on a transient 429/503.
- Wired into the page crawl (site-audit-workflow-helpers), robots.txt +
sitemap discovery, and start-URL redirect probing.
Gets past the naive tier; still reported as "blocked" for JS/TLS challenges
(Cloudflare Managed Challenge, DataDome) — those need a real browser. Doc note
points operators at WAF IP/UA allowlisting for their own sites.
No env dependency (keeps the audit lib importable without a cloudflare:workers
mock). tsc / oxlint / knip clean; new crawl-request.test.ts (5); suite
otherwise unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- activity_log table (sqlite + pg, structurally identical; schema-parity
covers it). Plain-text columns, no FKs — an append-only trail that must
outlive the projects/users it references, so target_label snapshots a
human-readable name at write time.
- ActivityRepository: record() (fire-and-forget, never breaks the caller) +
list() (org-scoped, actor/action filters, keyset pagination) + listActors().
- Recording wired into the mutations worth tracking: project
create/archive/restore/domain, audit start, team user create/remove/
password-reset, invitation sent.
- getActivityLog / getActivityActors server functions (owner/admin gated) +
Settings → Activity tab (ActivityLogView: filter by user & action, load
more).
- Migration: drizzle/0045_*, drizzle-pg/0023_*. The pipeline does not run
migrations — see docs/SELF_HOSTING_TEAM_MODE.md step 5 for the one-time
`drizzle-kit migrate` on the server. Writes fail silently until the table
exists.
tsc / oxlint / knip clean. New ActivityRepository.test.ts (4) + schema-parity
picks up the new table; suite otherwise unchanged (pre-existing samSkills
CRLF failure only).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- _auth.setup.tsx: the "owner already exists?" check ran in beforeLoad, which
executes during SSR where fetchTeamSetupStatus's relative fetch to
/api/team-setup fails — so it always concluded an owner existed and
redirected to /sign-in, making the create-owner screen unreachable. Move the
check into a client-side effect with a loading state.
- setup-status.ts: add BETTER_AUTH_URL to CHECK_ENV_VARS so /api/health stops
falsely reporting "team mode requires BETTER_AUTH_URL" when it is set (the
Docker preflight already saw it; only the runtime health check's env
allowlist was missing it).
Verified locally end to end against a D1 build in AUTH_MODE=team: owner
bootstrap, self-disable + 409 on repeat, Better Auth sign-in issues a session
cookie, and get-session resolves the shared organization.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Makes AUTH_MODE=team usable end to end.
- resolveTeamContext (middleware/ensure-user/team.ts): a session resolves to a
membership in the single shared workspace. No per-user fallback org — a
signed-in user with no membership is treated as signed out, so the owner can
actually remove people.
- teamProvisioning.ts: one path that writes user + credential account + member
together (hashPassword from better-auth/crypto). Shared by both entry points.
- /api/team-setup (raw route, outside auth middleware): GET reports whether an
owner is needed; POST creates the first owner + the shared org, then
self-disables once any user exists.
- /setup route + sign-in redirect: first run sends you to create the owner.
- teamUsers server functions (owner/admin-gated): list / create (with temp
password) / reset password / remove. Removal drops membership + sessions,
keeps the user row for historical attribution.
- Settings gains a "Users" tab in team mode (TeamUsers.tsx).
- docs/SELF_HOSTING_TEAM_MODE.md: activation runbook (.env, build, first owner).
No DB migration — all rows are existing better-auth tables. tsc / oxlint / knip
clean. New teamProvisioning.test.ts (4 cases) passes; suite otherwise unchanged
(pre-existing samSkills.test.ts CRLF failure only).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Introduces a fourth AUTH_MODE, `team`: Better Auth email/password with the
existing organization/member/role/invitation stack, but none of the hosted
SaaS coupling (no Autumn billing, Turnstile, Loops email, Google social
login, onboarding chat, PostHog, disposable-email block, dub referrals).
- auth-mode.ts: add `team`; add isTeamAuthMode / isSessionAuthMode /
isSessionClientAuthMode ("is there a login session?" vs isHostedAuthMode's
"is this the billed product?").
- auth.ts: createAuth() builds a valid instance for `team` — verification
off, self-serve signup disabled, no captcha/Loops/social. hasTeamAuthConfig
(BETTER_AUTH_URL + BETTER_AUTH_SECRET only) + hasSessionAuthConfig.
- ensure-user: resolve.ts routes `team` through resolveHostedContext;
requireHostedSession + selfHostedOAuth callback accept any session mode.
- api/auth/$.ts: mount the Better Auth handler for `team` too.
- Client: route guards, sidebar account menu / sign-out, settings
Organization tab, invitation accept, and error cards switch from
isHostedClientAuthMode to isSessionClientAuthMode where they mean "has a
session". Sign-in goes straight to the email form (no Google button);
sign-up shows an invite-only notice.
- selfhost-preflight: validate `team` (requires BETTER_AUTH_URL +
BETTER_AUTH_SECRET >= 32 chars).
- .env.example: document `team`.
Ships inert: AUTH_MODE stays local_noauth. tsc / oxlint / knip clean;
test suite unchanged (1164 pass, 1 pre-existing Windows-CRLF failure in
samSkills.test.ts).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- DaisyUI themes openseo/openseo-dark -> crawlerx/crawlerx-dark on the
MetatronCube palette (primary #3779b9, secondary #0f1d34, accent #4faede)
- Load Inter as the app font; add --brand-gradient / .brand-gradient
- Cube mark + "CrawlerX" wordmark in the sidebar and mobile top bar
- Add public/crawlerx-mark.png; point transparent-logo.png at the cube mark
- <title>, site.webmanifest, and all user-facing "OpenSEO" copy across
src/client and src/routes -> "CrawlerX"
Deferred: favicons (need resizing), agent-facing identity (MCP server
name, SAM/onboarding prompts, fact sheet), web/ marketing site.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace leftover 'personal, noncommercial' boilerplate with a license
covering internal business use and client-services work, permit using
generated Outputs in client deliverables, and carve Outputs out of the
Section 2.2 commercial-exploitation restriction.
