diff --git a/docs/AGENCY_EDITION_PLAN.html b/docs/AGENCY_EDITION_PLAN.html new file mode 100644 index 0000000..ce2e67f --- /dev/null +++ b/docs/AGENCY_EDITION_PLAN.html @@ -0,0 +1,676 @@ + + + + + +CrawlerX Agency Handover + + + + + + + +
+
+
Handover · September 2026
+

CrawlerX for agencies: build plan

+

Turn our internal fork of OpenSEO into a product agencies buy instead of Semrush: client workspaces, white-label reports, usage-based billing. This page is the working plan for the team taking over.

+
+ Base OpenSEO 0.1.7 (MIT) + Stack TanStack Start · Drizzle (SQLite + Postgres) · Better Auth · DataForSEO + Horizon 16 weeks, pilot at week 8 +
+
+
+ + + +
+ +
+

Start here

+

What the product is, where it stands, and the one architectural fact that shapes everything.

+ +
+
+

What works today

+
    +
  • Keyword research, saved keywords, scheduled rank tracking (with local locations)
    • +
  • Domain and competitor overview, backlinks
    • +
  • Site audit (own crawler, DataForSEO OnPage fallback, Lighthouse)
    • +
  • Search Console and GA4 integrations
    • +
  • AI visibility: brand lookup, prompt explorer
    • +
  • SAM in-app agent, MCP server and agent skills (hosted mode only)
    • +
+
+
+

What we added to the fork

+
    +
  • CrawlerX / MetatronCube rebrand (theme in src/client/styles/app.css)
    • +
  • AUTH_MODE=team: owner-provisioned users, roles, shared workspace
    • +
  • Activity log (src/server/features/activity/)
    • +
  • Browser-like crawler headers and DataForSEO OnPage fallback for bot-blocked sites
    • +
  • Combined SEO report: GSC + GA4 + audit (/p/$projectId/report)
    • +
+
+
+ +
+

The fact that shapes the plan

+

The codebase has four auth modes, and none is "an agency with many clients". team is a single shared workspace with no billing and no self-serve signup. hosted has orgs, roles, invitations and per-org credit billing, but it is wired to the upstream author's vendors (Autumn and Stripe, Loops email, PostHog, Cloudflare KV). The plan is to build the agency product on top of hosted mode, running it on our own accounts, and to run team-mode instances for pilot agencies while that work happens.

+
+ +

Read first: CLAUDE.md (engineering rules), specs/0002 (DataForSEO metering), specs/0011 (multi-user orgs), docs/SELF_HOSTING_TEAM_MODE.md, docs/LOCAL_DEVELOPMENT.md.

+
+ +
+

Decisions to confirm before Phase 1

+

The plan below assumes the recommended answer. If any changes, the affected phase is named.

+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
DecisionAssumed answerIf it changes
Architecturerecommended Multi-tenant hosted mode, deployed to Cloudflare with Postgres. Per-agency team-mode instances only for the pilot.Per-agency instances forever: drop P2.4–P2.6, add instance provisioning automation.
Branding depthAgencies white-label the reports and client view. The app shell stays CrawlerX.Full white-label (custom domain, app shell): add P3.1.
UpstreamHard fork. Cherry-pick security fixes and DataForSEO changes only.Track upstream: do P0.5 first and keep every change additive.
PricingPlatform fee per agency, client seats, plus credit packs for data usage.Pure usage: skip client seats in P2.4.
Team sizePlanning assumption: two full-stack engineers, one part-time devops/QA.Scale the timeline; the phase order does not change.
+
+
+ +
+

Timeline

+

Long-lead items (amber) start in week 0 because they gate launch, not because they are big.

+
+ +
+ Long lead time + Build + Pilot agencies onboard + Later +
+
+
+ + +
+

Phase 0: De-risk and clean

Weeks 1–2
+

Nothing customer-facing. Remove the things that would hurt us the moment a second customer appears.

+
Exit criteria: no upstream branding, contact details or telemetry reach a customer. Hosting decision written down. Google verification and DataForSEO terms are in progress. Tenant isolation is tested.
+ +
+
+
P0.1

Turn off upstream telemetry today

blockerdo firstS
+

self-host-telemetry.ts holds a hard-coded PostHog key for the upstream project and skips sending only in hosted mode. Our team-mode instance appears to send anonymous heartbeats to them unless opted out.

+
+
Do
Set OPENSEO_TELEMETRY_DISABLED=1 in the production .env and restart. Then remove the module and its wiring, or replace the key with our own PostHog project.
+
Where
src/server/lib/self-host-telemetry.ts
+
Done when
Network capture on a running instance shows no calls to us.i.posthog.com with the upstream key.
+
+
+ +
+
P0.2

Scrub upstream identity, and centralise brand constants

blockerM
+

Customer support requests currently point at the upstream author's email. Links send users to the upstream product, Discord and docs. Create one src/shared/brand.ts (product name, support email, docs and terms URLs, crawler contact URL) and make every reference read from it.

+
+
Where
+ client/features/audit/shared.tsx routes/_app/ai.tsx routes/_app/support.tsx routes/_authenticated.subscribe.tsx routes/_auth.sign-up.tsx client/features/onboarding/OnboardingChatParts.tsx client/features/settings/ApiKeySettings.tsx client/components/AuthConfigErrorCard.tsx shared/ga4.ts shared/gsc.ts server/lib/scrape.ts server/mcp/server.ts server/features/sam/SamChatAgent.ts server/features/rank-tracking/services/scheduledRankChecks.ts server/features/onboarding/OnboardingChatAgent.ts server/referrals/dub.ts +
+
Also
Skills install commands in ai.tsx point at every-app/open-seo. Publish our own skills package or hide that card. Remove the Dub referral code (upstream's partner program).
+
Done when
grep -ri "openseo.so\|every-app\|ben@" over src returns only tests, comments, and the licence notice. Add that grep to CI.
+
+
+ +
+
P0.3

Hosted-mode spike and hosting decision

blockerM
+

Production today appears to run vite preview under pm2 (the start script), which emulates the Cloudflare runtime locally. That is fine for an internal tool, not for paying customers. package.json already has deploy:postgres (Alchemy, hosted-prod stage, Postgres). Prove it works end to end on our own Cloudflare account, in a staging stage.

+
+
Check
Sign-up, org creation, invite, project create, keyword research, site audit workflow, rank check cron, GSC connect, MCP OAuth. Note every Cloudflare binding used: KV (invitation-send-limit.ts, oauth-provider.ts, progress-kv.ts, serp-locations.ts), R2 (r2.ts, r2-cache.ts), Workflows and Durable Objects (SiteAuditWorkflow.ts, chat agents).
+
Deliverable
One-page decision: deploy to Cloudflare (assumed) or port off it, with the list of breakages and their cost.
+
Done when
A staging environment runs hosted mode with our own domain. A second signed-up user gets an isolated workspace.
+
+
+ +
+
P0.4

Google OAuth verification and DataForSEO terms

blockerlong leadS to start
+

Search Console and GA4 scopes are sensitive. Until Google verifies the app, external users see an unverified-app warning and are capped. Verification takes weeks and needs a live privacy policy, a homepage on our domain and a demo video. Separately, get written confirmation that DataForSEO allows us to resell their data in a paid product.

+
+
Do
Create a dedicated Google Cloud project for production. Write the privacy policy and terms (replace the upstream links in _auth.sign-up.tsx). Submit the consent screen. Email DataForSEO sales about reseller terms and volume pricing.
+
Done when
Verification submitted with no open questions. DataForSEO reply saved in the repo's docs/ folder.
+
+
+ +
+
P0.5

Tenant isolation review and tests

blockerM
+

Every server function must scope by the caller's organisation (see specs/0001). Run /security-review, then write the test that would catch a leak: two orgs, and each server function called with the other org's projectId must fail.

+
+
Where
src/serverFunctions/*.ts src/server/mcp/project-auth.ts src/middleware/ensure-user/
+
Done when
A single parameterised test covers every project-scoped server function and MCP tool. Findings from the review are fixed or ticketed.
+
+
+ +
+
P0.6

Repo hygiene and upstream policy

S
+
+
Do
Delete the empty root file open-seo@0.1.7 and vite.config.ts.backup. Two files show as modified from line endings only: add a .gitattributes. Tag the fork point. Add an upstream remote and record the sync policy in docs/MAINTAINERS.md. Remove the internal hostname and server paths from docs/SELF_HOSTING_TEAM_MODE.md. Keep LICENSE and add a NOTICE crediting OpenSEO.
+
Done when
pnpm ci:check passes on a clean checkout.
+
+
+
+
+ + +
+

Phase 1: Pilot MVP

Weeks 2–8
+

Enough for three to five design-partner agencies to put their own clients on it. MetatronCube itself is design partner zero.

+
Exit criteria: an agency can send a client a branded PDF or link with real data in it, and see all their clients in one place. Pilots are invoiced by hand.
+ +
+
+
P1.1

Organisation branding

M
+

New table for per-organisation branding: display name, logo, primary colour, footer text. Logo stored in R2. Applied to reports and to the client view, not to the app shell.

+
+
Where
New src/db/branding.schema.ts and the Postgres twin under src/db/pg/. Follow server function → service → repository. Run npm run db:generate and keep schema-parity.test.ts green (SQLite and Postgres must both work).
+
Done when
An owner uploads a logo in Settings and it appears on the report. Only owner and admin can edit it.
+
+
+ +
+
P1.2

Report v2 and PDF export

top purchase driverL
+

ProjectReportService covers GSC, GA4 and the latest audit. Add rank tracking movement, a backlinks summary and AI visibility. Each section already degrades on its own when an integration is missing; keep that. Export: start with a print-optimised route (@media print, A4) since it is the cheapest path. Move to server-side rendering (Playwright is already a dev dependency) only if print output is not good enough.

+
+
Where
src/server/features/reports/services/ProjectReportService.ts src/client/features/reports/ReportView.tsx src/routes/_project/p/$projectId/report.tsx
+
Done when
A 4–6 page A4 PDF with the agency's logo and colours, no CrawlerX or upstream branding, and no API credits spent on render (reuse stored data).
+
+
+ +
+
P1.3

Shareable read-only report link

M
+

Store a snapshot of the report JSON at share time so a client sees stable numbers and cannot trigger spend. Public route addressed by a random token (store only its hash), with expiry and revoke. Log create and revoke in the activity log.

+
+
Where
New report_share table (projectId, tokenHash, snapshot, expiresAt, revokedAt). New public route outside _authenticated. Add actions to ACTIVITY_ACTIONS.
+
Done when
A revoked or expired link returns 404. The page exposes nothing beyond the snapshot.
+
+
+ +
+
P1.4

Agency dashboard across clients

M
+

One screen listing every client project with the few numbers an account manager scans: rank movement over 7 and 28 days, latest audit health, GSC clicks change, integration status (connected, expired, missing). Sort by "needs attention".

+
+
Where
src/server/features/dashboard/services/DashboardService.ts and src/routes/_app/projects.tsx
+
Done when
Loads in under 2 seconds with 50 projects on seeded data (pnpm seed:projects).
+
+
+ +
+
P1.5

Own transactional email

S–M
+

Invitations, password reset and verification currently go through Loops templates that belong to the upstream account. Put a small provider interface in src/server/email and implement it with Resend or SES. Replace billing/loops-sync.ts contact sync (remove, or point at our own tool).

+
+
Done when
Invite, reset and verify emails arrive from our domain (SPF, DKIM set). No Loops keys required in hosted mode.
+
+
+ +
+
P1.6

Crawler identity and allowlist guide

S
+

Many client sites sit behind Cloudflare Managed Challenge or DataDome, which the crawler cannot pass. Give agencies something to allowlist: a stable egress IP for the crawler and a user-agent that names us and links to a page explaining it. Publish a short help page with per-WAF instructions. Keep the DataForSEO OnPage fallback as the paid option.

+
+
Where
src/server/lib/audit/crawl-request.ts (AUDIT_USER_AGENT)
+
Done when
The audit failure card on a blocked site links to the allowlist guide and to the paid fallback.
+
+
+ +
+
P1.7

Pilot operations

M
+

Until self-serve exists, provision each pilot by script: create org and owner, apply branding, set an invoice reference. Set up production basics: nightly Postgres backup with a tested restore, error tracking, uptime check and a status page, and a shared support inbox.

+
+
Where
New script beside scripts/seed-projects.ts; runbooks in runbooks/.
+
Done when
A new pilot agency is live in under 30 minutes by following the runbook, and a restore has been rehearsed once.
+
+
+
+
+ + +
+

Phase 2: Agency product

Weeks 8–16
+

What turns a pilot into something agencies pay for monthly without hand-holding.

+
Exit criteria: an agency signs up, pays, adds clients, gives a client read-only access, and gets scheduled reports and alerts, with usage attributed per client.
+ +
+
+
P2.1

Client viewer role with per-project access

L
+

The member role exists but is not invitable: spec 0011 blocks it in beforeCreateInvitation, and every invitee is an admin. Add a client role that is read-only and limited to named projects. Model access as a project_member table (userId, projectId), not a JSON list. Enforce it in the project-auth middleware and the MCP layer.

+
+
Where
src/lib/org-permissions.ts src/server/mcp/project-auth.ts src/serverFunctions/organization.ts
+
Done when
A client user sees only assigned projects, cannot run credit-spending actions, and cannot see billing. Covered by the P0.5 isolation test.
+
+
+ +
+
P2.2

Scheduled report emails

M
+

Per-project schedule (weekly or monthly), recipients, and whether to attach the PDF or send the share link. A cron creates a snapshot (P1.3) and sends it. Cron hooks already exist in the scheduled handler in src/server.ts and wrangler.jsonc.

+
+
Done when
A failed send is retried and visible in the activity log. Unsubscribe link works.
+
+
+ +
+
P2.3

Alerts

M
+

Notify on rank drops above a threshold, audit score regressions, GSC clicks falling, and expired Google connections. Hook into scheduledRankChecks.ts after each run. Preferences table per user and project. Email first, in-app feed second.

+
+
Done when
No duplicate alert for the same condition within 24 hours.
+
+
+ +
+
P2.4

Per-client usage attribution and caps

L
+

Credits are pooled per organisation. Agencies need to know what each client costs, and to stop one client burning the pool. Record projectId on every usage event, show usage by client on the billing page, and add an optional monthly cap per project checked in the same preflight as the balance check.

+
+
Where
src/server/lib/dataforseoClient.ts (the single entry point per spec 0002), src/server/billing/subscription.ts
+
Done when
A capped project fails with a clear "client cap reached" error. Usage-by-client totals match the org total.
+
+
+ +
+
P2.5

Our own billing setup and agency plans

blocker for launchL
+

Create our own Autumn and Stripe accounts and define the agency plans (platform fee, client seats, credit packs). The 1.28 markup is a constant in src/shared/billing.ts: make it configuration, not a code edit, so pricing can change without a deploy. Stripe checkout already collects business name and tax ID.

+
+
Where
src/shared/billing.ts src/server/billing/ src/routes/_authenticated.subscribe.tsx
+
Done when
Test-mode subscribe, top-up, cancel and invoice all work. Cost estimates in the UI match what is billed.
+
+
+ +
+
P2.6

Self-serve signup, onboarding, launch pages

M
+

Signup flow adapted for agencies (agency name, first client project, connect Google), a free-trial policy with a spend ceiling so trials cannot burn DataForSEO money, terms and privacy pages, a pricing page. Rework the onboarding chat copy from "personal SEO tool" to "agency workspace".

+
+
Done when
A stranger can sign up, add a client, run a first audit and receive a report without talking to us. Trial spend per organisation is capped and monitored.
+
+
+
+
+ + +
+

Phase 3: After launch

Order by what pilots ask for
+

Do not start these until paying agencies have asked. Each is a real project.

+
+ + + + + + + + + +
IDItemNote
P3.1Custom domain and full white-labelPer-org hostname (Cloudflare for SaaS), branded login and emails.
P3.2Agency MCP and APIMCP and skills exist in hosted mode. Publish our own skill pack and per-org API keys.
P3.3Looker Studio connector or CSV exportAgencies keep their own dashboards. Today only export to Google Sheets exists.
P3.4Security hardening2FA, SSO for larger agencies, audit-log export, DPA template.
P3.5Local SEO depthLocal rank grids and Business Profile data, if agencies serve local businesses.
+
+
+ +
+

Quality bar for every task

+

These come from CLAUDE.md. New work is reviewed against them.

+
+
+

Code

+
    +
  • Backend shape: TanStack server function → service → repository.
    • +
  • Every schema change works on SQLite and Postgres; run npm run db:generate and keep the parity test green.
    • +
  • Relations are real tables and foreign keys, not JSON.
    • +
  • Validate untrusted input with Zod at the boundary.
    • +
  • All DataForSEO calls go through the metered client. A raw fetch*Raw call in feature code is a billing bypass.
    • +
+
+
+

Tests and CI

+
    +
  • A test must protect real behaviour or a real edge case. No tests for their own sake.
    • +
  • Test at the public entry point; do not mock ORM builder chains.
    • +
  • Merge gate: pnpm ci:check (prettier, knip, tsc, oxlint) and pnpm test.
    • +
  • Log small repo friction in .agents/PAPERCUTS.md.
    • +
  • Changes to .greptile/, AGENTS.md, CLAUDE.md and .github/ need maintainer review.
    • +
+
+
+
+ +
+

Risks

+
+ + + + + + + + + + + +
RiskWhy it mattersMitigation
DataForSEO cost overrunWe pay raw cost. Trials, rank tracking and large audits can burn money faster than revenue.P2.4 caps, trial spend ceiling in P2.6, alert on account balance, weekly review of pnpm billing:usage.
Google verification delayBlocks external GSC and GA4, which the report depends on.Submit in week 1 (P0.4). Pilots can use their own Google Cloud client in the meantime.
Reseller termsIf DataForSEO disallows resale, the model needs a different arrangement.Ask before building billing (P0.4).
Cross-tenant data leakFatal for trust, with agency clients' data.P0.5 test suite, and a review of every new server function.
Blocked crawlsSite audit fails on many client sites.P1.6 allowlist and identity, paid OnPage fallback.
Upstream driftUpstream ships almost daily. Our rebrand touched 42 files.Hard fork; brand constants in one file (P0.2); cherry-pick security fixes only.
Runtime not production-gradeInternal deployment runs a preview server under pm2.P0.3 decision; do not put paying agencies on it.
+
+
+ +
+

Handover checklist

+

Access the new team needs on day one, and what they should read.

+
+
+

Accounts and access

+
    +
  • Gitea repository and CI
    • +
  • Production and staging servers (SSH, pm2)
    • +
  • Postgres credentials and backup location
    • +
  • DataForSEO account and billing alerts
    • +
  • Google Cloud project (OAuth client, consent screen)
    • +
  • Cloudflare account and DNS for the product domain
    • +
  • Stripe and Autumn accounts (create if not yet)
    • +
  • Email provider, error tracking and PostHog project
    • +
  • Shared support inbox
    • +
+
+
+

Day-one tasks for each engineer

+
    +
  • Run the app locally (docs/LOCAL_DEVELOPMENT.md)
    • +
  • Read CLAUDE.md, specs/0001, 0002, 0011
    • +
  • Run pnpm test and pnpm ci:check green
    • +
  • Create a team-mode instance and click through every module
    • +
  • Do P0.1 (telemetry off) as the first small change
    • +
  • Skim .agents/PAPERCUTS.md for known friction
    • +
+
+
+ +
+ +
+ + diff --git a/docs/AGENCY_EDITION_PLAN.md b/docs/AGENCY_EDITION_PLAN.md new file mode 100644 index 0000000..9d7922c --- /dev/null +++ b/docs/AGENCY_EDITION_PLAN.md @@ -0,0 +1,325 @@ +# CrawlerX for agencies: build plan + +Handover document, September 2026. Turn our internal fork of OpenSEO into a product agencies buy instead of Semrush: client workspaces, white-label reports, usage-based billing. + +- **Base:** OpenSEO 0.1.7 (MIT) +- **Stack:** TanStack Start, Drizzle (SQLite + Postgres), Better Auth, DataForSEO +- **Horizon:** 16 weeks, pilot agencies onboard at week 8 + +Sources: repository review of the README, specs, docs, schema, billing code and git history on 21 September 2026. The app and tests were not run for this review, and tenant isolation has not been audited. P0.1 rests on reading `self-host-telemetry.ts`; confirm it with a network capture. + +--- + +## 1. Start here + +### What works today + +- Keyword research, saved keywords, scheduled rank tracking (with local locations) +- Domain and competitor overview, backlinks +- Site audit (own crawler, DataForSEO OnPage fallback, Lighthouse) +- Search Console and GA4 integrations +- AI visibility: brand lookup, prompt explorer +- SAM in-app agent, MCP server and agent skills (hosted mode only) + +### What we added to the fork + +- CrawlerX / MetatronCube rebrand (theme in `src/client/styles/app.css`) +- `AUTH_MODE=team`: owner-provisioned users, roles, shared workspace +- Activity log (`src/server/features/activity/`) +- Browser-like crawler headers and DataForSEO OnPage fallback for bot-blocked sites +- Combined SEO report: GSC + GA4 + audit (`/p/$projectId/report`) + +### The fact that shapes the plan + +The codebase has four auth modes, and none is "an agency with many clients". + +- `team` is a single shared workspace with no billing and no self-serve signup. +- `hosted` has orgs, roles, invitations and per-org credit billing, but it is wired to the upstream author's vendors (Autumn and Stripe, Loops email, PostHog, Cloudflare KV). + +The plan is to **build the agency product on top of `hosted` mode**, running it on our own accounts, and to run `team`-mode instances for pilot agencies while that work happens. + +**Read first:** `CLAUDE.md` (engineering rules), `specs/0002` (DataForSEO metering), `specs/0011` (multi-user orgs), `docs/SELF_HOSTING_TEAM_MODE.md`, `docs/LOCAL_DEVELOPMENT.md`. + +--- + +## 2. Decisions to confirm before Phase 1 + +The plan assumes the recommended answer. If any changes, the affected phase is named. + +| Decision | Assumed answer | If it changes | +|---|---|---| +| Architecture | **Recommended:** multi-tenant hosted mode, deployed to Cloudflare with Postgres. Per-agency team-mode instances only for the pilot. | Per-agency instances forever: drop P2.4–P2.6, add instance provisioning automation. | +| Branding depth | Agencies white-label the reports and client view. The app shell stays CrawlerX. | Full white-label (custom domain, app shell): add P3.1. | +| Upstream | Hard fork. Cherry-pick security fixes and DataForSEO changes only. | Track upstream: do P0.6 first and keep every change additive. | +| Pricing | Platform fee per agency, client seats, plus credit packs for data usage. | Pure usage: skip client seats in P2.4. | +| Team size | Planning assumption: two full-stack engineers, one part-time devops/QA. | Scale the timeline; the phase order does not change. | + +--- + +## 3. Timeline (weeks 1–16) + +Long-lead items start in week 0 because they gate launch, not because they are big. + +| Workstream | Weeks | +|---|---| +| Google verification, vendor terms (long lead) | 1–10 | +| Cleanup, scrub, security review | 1–2 | +| Hosted-mode platform on Cloudflare | 2–6 | +| Branding, PDF, share links | 3–8 | +| Agency dashboard, pilot ops (**pilot agencies onboard at week 8**) | 5–8 | +| Client role, project access | 9–11 | +| Scheduled reports, alerts | 11–13 | +| Usage caps, agency billing | 9–16 | +| Self-serve signup, launch | 14–16 | + +--- + +## 4. Phase 0: De-risk and clean (weeks 1–2) + +Nothing customer-facing. Remove the things that would hurt us the moment a second customer appears. + +**Exit criteria:** no upstream branding, contact details or telemetry reach a customer. Hosting decision written down. Google verification and DataForSEO terms are in progress. Tenant isolation is tested. + +### P0.1 Turn off upstream telemetry today `blocker` `do first` `S` + +`self-host-telemetry.ts` holds a hard-coded PostHog key for the upstream project and skips sending only in `hosted` mode. Our `team`-mode instance appears to send anonymous heartbeats to them unless opted out. + +- **Do:** set `OPENSEO_TELEMETRY_DISABLED=1` in the production `.env` and restart. Then remove the module and its wiring, or replace the key with our own PostHog project. +- **Where:** `src/server/lib/self-host-telemetry.ts` +- **Done when:** a network capture on a running instance shows no calls to `us.i.posthog.com` with the upstream key. + +### P0.2 Scrub upstream identity, and centralise brand constants `blocker` `M` + +Customer support requests currently point at the upstream author's email. Links send users to the upstream product, Discord and docs. Create one `src/shared/brand.ts` (product name, support email, docs and terms URLs, crawler contact URL) and make every reference read from it. + +- **Where:** + - `src/client/features/audit/shared.tsx` + - `src/routes/_app/ai.tsx` + - `src/routes/_app/support.tsx` + - `src/routes/_authenticated.subscribe.tsx` + - `src/routes/_auth.sign-up.tsx` + - `src/client/features/onboarding/OnboardingChatParts.tsx` + - `src/client/features/settings/ApiKeySettings.tsx` + - `src/client/components/AuthConfigErrorCard.tsx` + - `src/shared/ga4.ts`, `src/shared/gsc.ts` + - `src/server/lib/scrape.ts` + - `src/server/mcp/server.ts` + - `src/server/features/sam/SamChatAgent.ts` + - `src/server/features/rank-tracking/services/scheduledRankChecks.ts` + - `src/server/features/onboarding/OnboardingChatAgent.ts` + - `src/server/referrals/dub.ts` +- **Also:** the skills install commands in `ai.tsx` point at `every-app/open-seo`. Publish our own skills package or hide that card. Remove the Dub referral code (upstream's partner program). +- **Done when:** `grep -ri "openseo.so\|every-app\|ben@"` over `src` returns only tests, comments, and the licence notice. Add that grep to CI. + +### P0.3 Hosted-mode spike and hosting decision `blocker` `M` + +Production today appears to run `vite preview` under pm2 (the `start` script), which emulates the Cloudflare runtime locally. That is fine for an internal tool, not for paying customers. `package.json` already has `deploy:postgres` (Alchemy, hosted-prod stage, Postgres). Prove it works end to end on our own Cloudflare account, in a staging stage. + +- **Check:** sign-up, org creation, invite, project create, keyword research, site audit workflow, rank check cron, GSC connect, MCP OAuth. Note every Cloudflare binding used: + - KV: `invitation-send-limit.ts`, `oauth-provider.ts`, `progress-kv.ts`, `serp-locations.ts` + - R2: `r2.ts`, `r2-cache.ts` + - Workflows and Durable Objects: `SiteAuditWorkflow.ts`, chat agents +- **Deliverable:** a one-page decision: deploy to Cloudflare (assumed) or port off it, with the list of breakages and their cost. +- **Done when:** a staging environment runs hosted mode with our own domain. A second signed-up user gets an isolated workspace. + +### P0.4 Google OAuth verification and DataForSEO terms `blocker` `long lead` `S to start` + +Search Console and GA4 scopes are sensitive. Until Google verifies the app, external users see an unverified-app warning and are capped. Verification takes weeks and needs a live privacy policy, a homepage on our domain and a demo video. Separately, get written confirmation that DataForSEO allows us to resell their data in a paid product. + +- **Do:** create a dedicated Google Cloud project for production. Write the privacy policy and terms (replace the upstream links in `_auth.sign-up.tsx`). Submit the consent screen. Email DataForSEO sales about reseller terms and volume pricing. +- **Done when:** verification is submitted with no open questions. The DataForSEO reply is saved in `docs/`. + +### P0.5 Tenant isolation review and tests `blocker` `M` + +Every server function must scope by the caller's organisation (see `specs/0001`). Run `/security-review`, then write the test that would catch a leak: two orgs, and each server function called with the other org's `projectId` must fail. + +- **Where:** `src/serverFunctions/*.ts`, `src/server/mcp/project-auth.ts`, `src/middleware/ensure-user/` +- **Done when:** a single parameterised test covers every project-scoped server function and MCP tool. Findings from the review are fixed or ticketed. + +### P0.6 Repo hygiene and upstream policy `S` + +- **Do:** + - Delete the empty root file `open-seo@0.1.7` and `vite.config.ts.backup`. + - Two files show as modified from line endings only: add a `.gitattributes`. + - Tag the fork point. Add an `upstream` remote and record the sync policy in `docs/MAINTAINERS.md`. + - Remove the internal hostname and server paths from `docs/SELF_HOSTING_TEAM_MODE.md`. + - Keep `LICENSE` and add a `NOTICE` crediting OpenSEO. +- **Done when:** `pnpm ci:check` passes on a clean checkout. + +--- + +## 5. Phase 1: Pilot MVP (weeks 2–8) + +Enough for three to five design-partner agencies to put their own clients on it. MetatronCube itself is design partner zero. + +**Exit criteria:** an agency can send a client a branded PDF or link with real data in it, and see all their clients in one place. Pilots are invoiced by hand. + +### P1.1 Organisation branding `M` + +New table for per-organisation branding: display name, logo, primary colour, footer text. Logo stored in R2. Applied to reports and to the client view, not to the app shell. + +- **Where:** new `src/db/branding.schema.ts` and the Postgres twin under `src/db/pg/`. Follow server function → service → repository. Run `npm run db:generate` and keep `schema-parity.test.ts` green (SQLite and Postgres must both work). +- **Done when:** an owner uploads a logo in Settings and it appears on the report. Only owner and admin can edit it. + +### P1.2 Report v2 and PDF export `top purchase driver` `L` + +`ProjectReportService` covers GSC, GA4 and the latest audit. Add rank tracking movement, a backlinks summary and AI visibility. Each section already degrades on its own when an integration is missing; keep that. Export: start with a print-optimised route (`@media print`, A4) since it is the cheapest path. Move to server-side rendering (Playwright is already a dev dependency) only if print output is not good enough. + +- **Where:** `src/server/features/reports/services/ProjectReportService.ts`, `src/client/features/reports/ReportView.tsx`, `src/routes/_project/p/$projectId/report.tsx` +- **Done when:** a 4–6 page A4 PDF with the agency's logo and colours, no CrawlerX or upstream branding, and no API credits spent on render (reuse stored data). + +### P1.3 Shareable read-only report link `M` + +Store a snapshot of the report JSON at share time so a client sees stable numbers and cannot trigger spend. Public route addressed by a random token (store only its hash), with expiry and revoke. Log create and revoke in the activity log. + +- **Where:** new `report_share` table (projectId, tokenHash, snapshot, expiresAt, revokedAt). New public route outside `_authenticated`. Add actions to `ACTIVITY_ACTIONS`. +- **Done when:** a revoked or expired link returns 404. The page exposes nothing beyond the snapshot. + +### P1.4 Agency dashboard across clients `M` + +One screen listing every client project with the few numbers an account manager scans: rank movement over 7 and 28 days, latest audit health, GSC clicks change, integration status (connected, expired, missing). Sort by "needs attention". + +- **Where:** `src/server/features/dashboard/services/DashboardService.ts` and `src/routes/_app/projects.tsx` +- **Done when:** loads in under 2 seconds with 50 projects on seeded data (`pnpm seed:projects`). + +### P1.5 Own transactional email `S–M` + +Invitations, password reset and verification currently go through Loops templates that belong to the upstream account. Put a small provider interface in `src/server/email` and implement it with Resend or SES. Replace `billing/loops-sync.ts` contact sync (remove, or point at our own tool). + +- **Done when:** invite, reset and verify emails arrive from our domain (SPF, DKIM set). No Loops keys are required in hosted mode. + +### P1.6 Crawler identity and allowlist guide `S` + +Many client sites sit behind Cloudflare Managed Challenge or DataDome, which the crawler cannot pass. Give agencies something to allowlist: a stable egress IP for the crawler and a user-agent that names us and links to a page explaining it. Publish a short help page with per-WAF instructions. Keep the DataForSEO OnPage fallback as the paid option. + +- **Where:** `src/server/lib/audit/crawl-request.ts` (`AUDIT_USER_AGENT`) +- **Done when:** the audit failure card on a blocked site links to the allowlist guide and to the paid fallback. + +### P1.7 Pilot operations `M` + +Until self-serve exists, provision each pilot by script: create org and owner, apply branding, set an invoice reference. Set up production basics: nightly Postgres backup with a tested restore, error tracking, uptime check and a status page, and a shared support inbox. + +- **Where:** new script beside `scripts/seed-projects.ts`; runbooks in `runbooks/`. +- **Done when:** a new pilot agency is live in under 30 minutes by following the runbook, and a restore has been rehearsed once. + +--- + +## 6. Phase 2: Agency product (weeks 8–16) + +What turns a pilot into something agencies pay for monthly without hand-holding. + +**Exit criteria:** an agency signs up, pays, adds clients, gives a client read-only access, and gets scheduled reports and alerts, with usage attributed per client. + +### P2.1 Client viewer role with per-project access `L` + +The `member` role exists but is not invitable: spec 0011 blocks it in `beforeCreateInvitation`, and every invitee is an admin. Add a `client` role that is read-only and limited to named projects. Model access as a `project_member` table (userId, projectId), not a JSON list. Enforce it in the project-auth middleware and the MCP layer. + +- **Where:** `src/lib/org-permissions.ts`, `src/server/mcp/project-auth.ts`, `src/serverFunctions/organization.ts` +- **Done when:** a client user sees only assigned projects, cannot run credit-spending actions, and cannot see billing. Covered by the P0.5 isolation test. + +### P2.2 Scheduled report emails `M` + +Per-project schedule (weekly or monthly), recipients, and whether to attach the PDF or send the share link. A cron creates a snapshot (P1.3) and sends it. Cron hooks already exist in the `scheduled` handler in `src/server.ts` and `wrangler.jsonc`. + +- **Done when:** a failed send is retried and visible in the activity log. The unsubscribe link works. + +### P2.3 Alerts `M` + +Notify on rank drops above a threshold, audit score regressions, GSC clicks falling, and expired Google connections. Hook into `scheduledRankChecks.ts` after each run. Preferences table per user and project. Email first, in-app feed second. + +- **Done when:** no duplicate alert for the same condition within 24 hours. + +### P2.4 Per-client usage attribution and caps `L` + +Credits are pooled per organisation. Agencies need to know what each client costs, and to stop one client burning the pool. Record `projectId` on every usage event, show usage by client on the billing page, and add an optional monthly cap per project checked in the same preflight as the balance check. + +- **Where:** `src/server/lib/dataforseoClient.ts` (the single entry point per spec 0002), `src/server/billing/subscription.ts` +- **Done when:** a capped project fails with a clear "client cap reached" error. Usage-by-client totals match the org total. + +### P2.5 Our own billing setup and agency plans `blocker for launch` `L` + +Create our own Autumn and Stripe accounts and define the agency plans (platform fee, client seats, credit packs). The 1.28 markup is a constant in `src/shared/billing.ts`: make it configuration, not a code edit, so pricing can change without a deploy. Stripe checkout already collects business name and tax ID. + +- **Where:** `src/shared/billing.ts`, `src/server/billing/`, `src/routes/_authenticated.subscribe.tsx` +- **Done when:** test-mode subscribe, top-up, cancel and invoice all work. Cost estimates in the UI match what is billed. + +### P2.6 Self-serve signup, onboarding, launch pages `M` + +Signup flow adapted for agencies (agency name, first client project, connect Google), a free-trial policy with a spend ceiling so trials cannot burn DataForSEO money, terms and privacy pages, a pricing page. Rework the onboarding chat copy from "personal SEO tool" to "agency workspace". + +- **Done when:** a stranger can sign up, add a client, run a first audit and receive a report without talking to us. Trial spend per organisation is capped and monitored. + +--- + +## 7. Phase 3: After launch + +Do not start these until paying agencies have asked. Each is a real project. Order by what pilots ask for. + +| ID | Item | Note | +|---|---|---| +| P3.1 | Custom domain and full white-label | Per-org hostname (Cloudflare for SaaS), branded login and emails. | +| P3.2 | Agency MCP and API | MCP and skills exist in hosted mode. Publish our own skill pack and per-org API keys. | +| P3.3 | Looker Studio connector or CSV export | Agencies keep their own dashboards. Today only export to Google Sheets exists. | +| P3.4 | Security hardening | 2FA, SSO for larger agencies, audit-log export, DPA template. | +| P3.5 | Local SEO depth | Local rank grids and Business Profile data, if agencies serve local businesses. | + +--- + +## 8. Quality bar for every task + +From `CLAUDE.md`. New work is reviewed against these. + +**Code** + +- Backend shape: TanStack server function → service → repository. +- Every schema change works on SQLite and Postgres; run `npm run db:generate` and keep the parity test green. +- Relations are real tables and foreign keys, not JSON. +- Validate untrusted input with Zod at the boundary. +- All DataForSEO calls go through the metered client. A raw `fetch*Raw` call in feature code is a billing bypass. + +**Tests and CI** + +- A test must protect real behaviour or a real edge case. No tests for their own sake. +- Test at the public entry point; do not mock ORM builder chains. +- Merge gate: `pnpm ci:check` (prettier, knip, tsc, oxlint) and `pnpm test`. +- Log small repo friction in `.agents/PAPERCUTS.md`. +- Changes to `.greptile/`, `AGENTS.md`, `CLAUDE.md` and `.github/` need maintainer review. + +--- + +## 9. Risks + +| Risk | Why it matters | Mitigation | +|---|---|---| +| DataForSEO cost overrun | We pay raw cost. Trials, rank tracking and large audits can burn money faster than revenue. | P2.4 caps, trial spend ceiling in P2.6, alert on account balance, weekly review of `pnpm billing:usage`. | +| Google verification delay | Blocks external GSC and GA4, which the report depends on. | Submit in week 1 (P0.4). Pilots can use their own Google Cloud client in the meantime. | +| Reseller terms | If DataForSEO disallows resale, the model needs a different arrangement. | Ask before building billing (P0.4). | +| Cross-tenant data leak | Fatal for trust, with agency clients' data. | P0.5 test suite, and a review of every new server function. | +| Blocked crawls | Site audit fails on many client sites. | P1.6 allowlist and identity, paid OnPage fallback. | +| Upstream drift | Upstream ships almost daily. Our rebrand touched 42 files. | Hard fork; brand constants in one file (P0.2); cherry-pick security fixes only. | +| Runtime not production-grade | Internal deployment runs a preview server under pm2. | P0.3 decision; do not put paying agencies on it. | + +--- + +## 10. Handover checklist + +**Accounts and access** + +- [ ] Gitea repository and CI +- [ ] Production and staging servers (SSH, pm2) +- [ ] Postgres credentials and backup location +- [ ] DataForSEO account and billing alerts +- [ ] Google Cloud project (OAuth client, consent screen) +- [ ] Cloudflare account and DNS for the product domain +- [ ] Stripe and Autumn accounts (create if not yet) +- [ ] Email provider, error tracking and PostHog project +- [ ] Shared support inbox + +**Day-one tasks for each engineer** + +- [ ] Run the app locally (`docs/LOCAL_DEVELOPMENT.md`) +- [ ] Read `CLAUDE.md`, `specs/0001`, `0002`, `0011` +- [ ] Run `pnpm test` and `pnpm ci:check` green +- [ ] Create a team-mode instance and click through every module +- [ ] Do P0.1 (telemetry off) as the first small change +- [ ] Skim `.agents/PAPERCUTS.md` for known friction