diff --git a/docs/site-audit-pm-research.md b/docs/site-audit-pm-research.md new file mode 100644 index 0000000..f4107c5 --- /dev/null +++ b/docs/site-audit-pm-research.md @@ -0,0 +1,142 @@ +# OpenSEO Site Audit: 80/20 Feature Set & Crawl Architecture Proposal (rev. 3, founder decision applied) + +**Author:** PM research synthesis | **Date:** 2026-06-09 | **Inputs:** SEOnaut + LibreCrawl code reviews, OpenSEO baseline review, competitor/market research, architecture research, adversarial panel review. Revision notes: panel critiques accepted are folded in silently; where we overrode the panel, an explicit **Panel pushback** note marks the disagreement. **Rev. 3:** founder decision — build on the in-house Workers crawler, do not adopt DataForSEO OnPage; "we were blocked" flagging accepted as the blocking posture. §4 now weighs that decision's tradeoffs instead of recommending an engine. + +--- + +## 1. How site audits actually work (primer) + +Every site audit tool is the same three-stage pipeline: + +1. **Crawl.** Start from a URL, fetch robots.txt and sitemaps, fetch pages, extract links, enqueue new same-site URLs (BFS), repeat until a page cap. Two engine choices matter: _plain HTTP fetch_ (fast, cheap, blind to JavaScript) vs _headless rendering_ (sees what Google sees, ~10x cost). JS rendering is table stakes in commercial tools; differentiation is fidelity and whether it's paywalled. +2. **Check.** Per-page rules (missing title, noindex, broken canonical) plus cross-crawl rules (duplicate titles, redirect chains, orphan pages). The cross-page checks carry most of the value, and they're set-queries over crawl data, not parsing. +3. **Report.** Severity grouping, plain-English explanations, drill-down to affected URLs, **export**, and diffs against previous crawls. + +**Desktop vs cloud.** Screaming Frog ($259/yr) and Sitebulb run on the user's laptop: residential IP, no bot-blocking, but no scheduling, RAM-limited crawls (the #1 SF complaint), and the machine must be on. Cloud tools (Ahrefs, Semrush, Moz) crawl datacenter IPs on a schedule — and **getting blocked is their #1 recurring complaint**: WAFs 403/challenge their bots, producing phantom 404s and false positives. + +**Why blocking happens.** WAFs score IP reputation (datacenter = suspicious) and fingerprint (no JS = bot). It hits every cloud auditor, including ours. The industry playbook: documented UAs, published allowlistable IPs, Cloudflare's Verified Bots program, and ownership verification to justify higher crawl rates (Ahrefs advertises dramatically higher rates for verified owners; the oft-quoted 30→30,000 URLs/min figure is unsourced in our fact base and Ahrefs' verification is its own, not GSC — treat as directional only). + +--- + +## 2. Where OpenSEO's audit stands today + +Honest summary: **we have a crawler and a Lighthouse integration, but not an audit product.** We extract per-page fields and show filterable tables, but no rule engine names issues; the only "issues" users see are Lighthouse failures. SEOnaut ships ~88 issue types; LibreCrawl ~26; Screaming Frog ~300. We effectively ship ~2 as table styling. + +Structural problems, in priority order: + +- **The link graph is thrown away.** Link lists are extracted but only counts persist to D1, so broken links, orphans, redirect chains, and inlink analysis are impossible without schema change. +- **Persistence is finalize-only and Workflow-shaped.** `runCrawlBatch` returns full page arrays (links, images) as durable step state (~1MiB/step cap — already borderline on link-dense sites), `allPages` accumulates everything in 128MB Worker memory, and `batchWriteResults` writes all rows inside the single finalize step (DB_BATCH_SIZE=100 → a 1M-row link graph would mean ~10,000 subrequests in one step, ~10x the limit). **Any link-graph feature requires restructuring persistence first.** This was deferred to P1 in draft 1; the panel correctly identified that as backwards — it is now the P0 prerequisite. +- **Step budget caps crawl size — ledger, not vibes.** Workflows allows ~1,024 steps/instance. Today: 3 steps per 25-page batch → ~341 batches ≈ **~8.5k pages** fits (minus discovery/finalize overhead), so the existing 10,000-page clamp slightly exceeds the budget. The sharper constraint is Lighthouse strategy `all`: 2 steps per 10-URL batch ≈ 1,000 steps at 5k pages — over budget on its own. Politeness delays must live _inside_ steps (wall-clock waits are fine), never as smaller batches. See §4 for the post-refactor ledger and caps. +- **No audit MCP tools** — keywords, SERP, backlinks, rank-tracker, GSC all have them; audit has zero (verified in `src/server/mcp/tools`). Ahrefs, meanwhile, **already ships site-audit MCP tools** (`site-audit-issues`, `site-audit-page-explorer`, etc.); Semrush has an MCP server too. We are behind here, not ahead — see §3.4 for what our actual wedge is. +- **SSRF gap:** `url-policy.ts` validates only the start URL; discovered links are fetched unvalidated. Must be fixed in P0 for as long as the in-house crawler runs. + +Condensed check inventory: status/redirects, title/meta, canonicals, indexability, headings, images = **partial** (extracted, unflagged; redirects followed silently; header variants like `X-Robots-Tag` unchecked). Duplicates (30–35% of sites, Semrush study¹), broken links (52% of domains¹), redirect chains (~12%¹), orphans, security headers, hreflang, JS rendering, scheduling/diffs, issue UX, MCP, block resilience = **missing**. Lighthouse/CWV = **strong**, better than both OSS tools. _(¹ Prevalence figures from market research §3; primary sources to be attached as a footnoted appendix before this circulates externally.)_ + +(Folklore correction stands: the 100k-page lifetime cap settles to actual crawled counts; only failed/stuck audits permanently pay their reservation.) + +--- + +## 3. The 80/20 feature set + +**Pricing posture (decides everything downstream).** Audit is the most commoditized feature in SEO — Ahrefs Webmaster Tools and Screaming Frog ≤500 URLs are free. Per our SEO strategy (goal = signups), **audit is acquisition, not monetization**: default 50-page audits are free per site; credits are charged for large crawls, scheduling, and history. The free posture also reinforces the engine decision (§4): with the in-house crawler, a free audit's marginal cost is D1 writes and Workers compute largely inside quotas we already pay for — no per-audit vendor bill, no metered cost on failed runs (DataForSEO charges even for failed tasks). + +The prevalence evidence says a credible audit needs ~6 check families, not 300. P0 below runs on the **in-house crawler (founder decision, §4)**; crawler-side hardening is therefore mandatory P0 work, not contingency. Everything except the crawler-side items remains **engine-independent by design** (schema, SQL checks, UX, MCP) — that portability is the cheap hedge that keeps a future vendor or local-agent engine swap on the table. + +### P0 — the audit becomes real (~6–8 weeks) + +**0. Incremental persistence (prerequisite, was P1).** Write `audit_pages`/`audit_links` rows to D1 _inside each crawl-batch step_ (one extra `db.batch` per batch keeps subrequests bounded), return only slim summaries (id, url, status) as step state, drop the `allPages` accumulator. Finalize becomes: multipage SQL step → completion. Bonus: failed audits keep partial results. Also fix the SSRF gap (run discovered links through url-policy). + +**1. Issue engine + schema.** `audit_links` edge table (source, target, anchor, rel, internal) and `audit_issues` table. **Reporter contract (decided now, per panel):** reporters consume a normalized page record — `(record: AuditPageRecord) => boolean` where the record includes response headers captured at crawl time — _not_ SEOnaut's `(page, $, headers)` cheerio signature. Cheerio runs once in `crawlPage`; reporters are DOM-free. This is what makes the engine portable across our crawler, a future local agent, or a vendor feed if the §4 revisit triggers ever fire. Cross-page checks are D1 SQL passes (duplicates = `GROUP BY hash HAVING count > 1`, per SEOnaut's multipage reporters). **Effort: 2–3 weeks including #0's restructure** (was "1 week" — panel was right). + +**2. High-prevalence checks** (post-#1; thresholds cribbed from SEOnaut/LibreCrawl): + +- _Broken internal links + redirect chains/loops_: switch to `redirect: "manual"`, record hops, enqueue targets. Decision: redirect hops do **not** count against maxPages or billing (they're cheap HEAD-sized fetches; metering/progress semantics budgeted at +1 day). Broken-link checks fire **only for targets we actually fetched or HEAD-checked** — never inferred from absence on partial crawls. ~4 days. +- _Duplicates_: body-hash group-by catches mirrored/parameterized duplicates only (any dynamic token defeats exact hashing); **duplicate titles/descriptions group-by is the workhorse** and carries the prevalence claim. Avoid LibreCrawl's O(n²) similarity scan. ~1–2 days. +- _Title/meta length, missing/multiple H1, heading order_ (SEOnaut thresholds). ~1–2 days. +- _Canonical + indexability with header fallbacks_ (`X-Robots-Tag`, `Link: rel=canonical` — SEOnaut's standout trick; headers are in the normalized record by design). ~2 days. +- _Orphan pages_: reorder seeding to **sitemap-last** (we currently seed sitemap URLs _first_, the opposite of the SEOnaut pattern we cite, so sitemap pages eat the page budget and link discovery never exhausts), and **emit orphan findings only when the crawl completed** (queue exhausted below maxPages) — on truncated crawls "no observed inlinks" is true of nearly everything and would mass-false-positive, the exact Semrush health-score noise we mock. ~3–4 days including reordering and gating. +- _Thin content + crawl depth_: explicit disposition — **build in P0** (word count already stored; BFS depth is one column). _Page size_: P1. ~1 day. + +**3. Issues UX.** Critical/Warning/Info grouping, one-paragraph plain-English explanation per type (adapt SEOnaut's YAML copy), affected-URL drill-down, LibreCrawl's ~100-glob default exclusions (wp-admin, checkout, feeds), and **CSV export** (previously unaccounted; it's one of the three core report capabilities from §1). ~4–5 days. + +**4. MCP audit tools — reframed.** Draft 1 claimed "a demo no competitor can match." **That was wrong: Ahrefs ships site-audit MCP tools today and Semrush has an MCP server.** Our wedge is not MCP existence; it's **credit-priced, zero-setup, agent-actionable audits**: audit any URL on demand for cents with no project setup or $129+/mo subscription, and return per-issue _fix instructions an agent can execute_ (the issue record carries remediation steps, not just labels). Tool shapes designed against Ahrefs' actual, inspectable MCP audit tools: `run_site_audit`, `get_audit_status`, `get_audit_issues(severity, type)`, `get_audit_pages(filter)`, each issue carrying `how_to_fix`. ~4 days. + +**5. Block detection — moved to weeks 1–2 (was week 5).** The panel is right that for our segment (indie founders on Vercel/Cloudflare/Netlify — bot protection there is a one-toggle setting and IP-reputation challenges hit datacenter egress even without it; the week-1 baseline will establish the real rate) blocking is a **first-run activation killer**, not an ops metric — and our own onboarding audit shows activation leaks are the existential problem. Classify 403s/challenge bodies/`cf-mitigated` headers as "we were blocked," say so honestly instead of recording error pages, and show owner-targeted allowlist instructions inline (GSC-verified flow, §4). This layer is response-classification, hence engine-independent, and instruments block-rate telemetry from day one. ~3 days. + +### P1 — weeks 8–14 + +- **Scheduled audits + diffing**: extend the existing cron; store per-run issue counts, show deltas ("12 new broken links since last week"). The retention feature and the reason cloud beats desktop. **Data retention policy (new):** full link graphs kept for the latest 2 runs only; older runs keep issue counts/deltas (a 10k-page graph is ~100–150MB against D1's 10GB cap — weekly large audits would eat gigabytes/year otherwise). Large-audit edge lists optionally archived as compressed R2 blobs. +- **AI-crawler accessibility audit — promoted from P2.** Which AI bots (GPTBot, ClaudeBot) can reach your content, robots/Cloudflare AI-block detection, is-key-content-in-static-HTML. The panel is right that our own words ("open greenfield, perfectly on-brand") contradicted P2 placement: it's mostly computable from data we already collect, and "is ChatGPT blocked from your site?" is the launch/marketing asset for the whole relaunch — nobody switches auditors for HSTS checks. (Skip llms.txt as a headline; no major LLM vendor consumes it.) +- **JS rendering — deferred (consequence of the no-vendor decision, §4).** Plain fetch can't render SPAs. P1 ships the honest mitigation instead: detect client-rendered shells (near-empty body text with a large script payload / root-div-only HTML) and flag "this site renders client-side; HTML-level checks are incomplete" rather than reporting false missing-title/thin-content issues. If demand materializes, the revisit path is Cloudflare Browser Rendering as an opt-in, credit-priced add-on for GSC-verified own-site audits (where the owner can allowlist, neutralizing the fingerprint problem) — not a vendor API. +- **Security/hygiene checks** (HTTPS, mixed content, HSTS/CSP, insecure forms) and **external broken links** (HEAD checks, per-crawl cache) — demoted below AI-crawler work. +- **Cloudflare Verified Bots**: reframed as a **lottery ticket, not a mitigation**. Eligibility requires established, meaningful crawl volume; a new low-volume crawler routinely doesn't qualify, and Web Bot Auth is beta on Cloudflare's side. File opportunistically for our own UA (`OpenSEOBot`); plan zero dependency on it. The owner-targeted allowlist flow is the only mitigation we count on. + +### P2 — later + +- **`npx openseo-audit` local agent** (see §4 option D) — gated on block-rate telemetry, which now exists from week 1. +- **Crawl snapshots in R2** ("what we saw" evidence; lightweight SEOnaut WACZ analog). +- **Hreflang validation, structured-data validation, page-size checks, link-placement classification.** + +**Skip entirely (explicit dispositions):** 300-check parity, crawl visualizations, E-E-A-T scores, multi-language UI. + +--- + +## 4. Architecture: where should the crawl run? + +| | **A. In-house Workers crawl + mitigations** | **B. DataForSEO OnPage offload** | **C. Desktop app** | **D. Hybrid: cloud + `npx` local agent** | +| -------------------- | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------- | ----------------------------------------------------------- | +| Blocking reliability | Low-medium: no stable IPs, no JS-challenge survival | Medium: still datacenter, but published IPs + named UA are _allowlistable_ | Highest (residential IP, real browser) | High when used; manual trigger only | +| Scheduled audits | Yes (≤ a few k pages — step budget) | Yes | **No** | Yes (cloud path); **no for the blocked sites D exists for** | +| JS rendering | No | Yes (priced) | Yes | Yes (local Playwright) | +| Eng effort | Ongoing hardening forever | Low (vendor/SDK/billing integrated) | **Months** + signing/notarization/3-platform QA | ~3–5 weeks (see below) | +| Cost per 1k pages | **Not $0**: D1 link writes ≈ $0.30/1k (with indexes) + Workers/Workflows | ~$0.125–0.50 base, ~$1.25 JS _(list prices from architecture research — re-verify if the §4 revisit triggers ever fire)_ | $0 marginal | $0 marginal + ingest | +| Max crawl size | ~8.5k today, ~12–25k post-refactor (Lighthouse `all` binds first; instance chaining removes the ceiling) | Vendor-side, no Workflow ceiling | RAM-limited | Local machine-limited | +| MCP / billing fit | Good / fine | Good / per-page cost maps to credits | **Bad** / awkward | Excellent for developers / fine | + +**Excluded options, explicitly (panel asked):** _Cloudflare Browser Rendering_ — solves JS but not blocking (egress is still Cloudflare datacenter IPs), adds per-browser-hour cost, and headless-Chrome fingerprints start their own anti-bot treadmill. With the vendor path now declined (see decision below), this becomes the only _cloud_ JS-rendering route (the P2 `npx` agent's local Playwright is the other) — scoped as a P2 opt-in for GSC-verified own-site audits where the owner can allowlist us first. _Residential/ISP proxy egress_ — rejected on principle and practice: it disguises crawler identity, which contradicts our documented-UA/allowlist posture and many proxy networks' sourcing is ethically murky; cost (~$1–8/GB) is also worst-in-class. + +**Decision: A — in-house Workers crawler (founder call: avoid DataForSEO vendor costs).** The original recommendation was B, contingent on a week-1 spike; the founder has decided against adding vendor spend, and "we were blocked" honest flagging is accepted as the blocking posture. That is a defensible position, but the proposal's job is to price it accurately. What the decision buys, and what it costs: + +**What going in-house buys:** + +- **No per-audit vendor bill on a free acquisition feature.** Every OnPage call is marginal cash out (~$0.006–0.025 per free 50-page audit, $1.25–5 per 10k-page crawl at list prices), and DataForSEO meters even failed tasks — a real footgun we've already hit on the SERP side. Crucially, the comparison must not double-count: D1 storage writes are common to **both** engines (vendor results would land in the same `audit_pages`/`audit_links` tables to run the same SQL checks), so the vendor fee is purely additive. In-house wins on marginal cost at every crawl size. +- **No new vendor dependency or pricing-change exposure** on the product's centerpiece free feature; full control of crawl behavior, UA identity, and politeness. +- **One less integration**: no task-queue/webhook polling model, no mapping a vendor's page schema into the normalized record. + +**What it costs — priced honestly:** + +1. **"Basically free" is true only until the free tier works.** D1 Paid includes 50M row-writes/month _shared with the entire product_. A 50-page free audit writes ~8–15k rows (pages + links + indexes), so **~3–5k free audits/month consumes the whole included allotment**; beyond it, $1/M rows — at 10k free audits/month, ~$50–100/month in-house vs ~$60–250/month vendor-side. A ~2x cost win, not zero, and it eats quota headroom from every other D1 feature. Required companion: a **TTL on free-tier audit data** (e.g., purge one-off free audits' link graphs after 30 days) — the P1 two-run retention policy only covers scheduled audits and doesn't protect the 10GB D1 cap from thousands of one-off free crawls. +2. **JS rendering is off the roadmap** until/unless we build it on Cloudflare Browser Rendering. SPA-heavy sites (a real slice of indie-founder stacks: Next.js CSR pages, Vite SPAs) get degraded audits. Mitigation: detect-and-disclose (§3 P1), never silently mis-report — and count CSR-shell prevalence in the week-1 telemetry so the JS-rendering revisit trigger has a denominator too. +3. **A step-budget ceiling — real, but it's Lighthouse that binds, not the crawl.** The ledger: today's 3 steps per 25-page batch supports ~8.5k pages in the 1,024-step budget; after the P0 #0 refactor (1–2 steps/batch), the crawl alone supports **~12–25k pages** — the 10k clamp can stand. The binding constraint is Lighthouse strategy `all` (2 steps per 10-URL batch ≈ 1,000 steps at 5k pages, over budget by itself): cap `all` at ~2k pages or switch to sampling, and document page caps _coupled to Lighthouse strategy_. Nor is the ceiling permanent: Workflow instances can chain (the P0 #0 refactor already puts frontier state in D1; a continuation instance resumes it) — roughly a week of cross-instance dedupe work, not an engine swap. The agency/large-site segment is deferred, not foreclosed. +4. **Weaker blocking mitigation — and the allowlist fix is unverified on the exact tiers our ICP uses.** Workers egress comes from shared Cloudflare IP ranges — we cannot publish a dedicated allowlistable IP, so the GSC-verified fix flow must be **User-Agent-based** (`OpenSEOBot/1.0` via a WAF custom rule; spoofable, tolerable when the owner allowlists their _own_ site). But: on **Cloudflare Free, Bot Fight Mode supports no exceptions** — a skip-by-UA rule does not bypass it; the owner's only fix there is turning BFM off entirely, a much heavier ask than "add this one rule." Vercel's challenge bypass is plan-gated and may be unavailable on Hobby. **Week 1 therefore includes a 1–2 day validation**: crawl test zones on Cloudflare Free (BFM on), Cloudflare Pro (Super BFM), and Vercel Hobby/Pro (Attack Challenge Mode), and write per-platform fix copy that is honest where the real answer is "disable bot protection." Verified Bots filing (P1) stays a lottery ticket. +5. **The ~1–2 weeks of crawler work inside P0 that the vendor path would have eliminated, plus permanent ownership.** Manual-redirect handling, per-host politeness inside steps, sitemap-last seeding, SSRF fix, step-budget restructuring, timeout/retry edges — already counted inside the 6–8-week P0 estimate (not additional to it), but under B most of it would have been deleted, and the hardening treadmill (new WAF behaviors, new edge cases) is ours indefinitely. +6. **No fallback engine for blocked sites.** Under B, a site blocking us might still be crawlable via the vendor's allowlisted infrastructure. In-house, a blocked site stays blocked until the owner acts. The honest-flagging posture makes this acceptable for owned sites: "we were blocked, here's the fix" instead of pretending. +7. **Third-party-URL audits have no remedy at all.** The MCP wedge includes "audit any URL on demand" (competitor audits, agent workflows) — but the allowlist flow only works for sites the user _owns_. For non-owned URLs, "we were blocked" is a terminal state, exactly where vendor infrastructure would have helped. Disposition: scope the claim — on-demand MCP audits are first-class for owned/verified sites, best-effort for everything else, and the tool response says which one it was. + +**Revisit triggers (set now, so the decision is cheap to reverse).** Define "audit blocked" first: start URL blocked, or ≥20% of fetched URLs classified as challenge/`cf-mitigated`. Then: (a) >15–20% of first-run audits blocked _and_ the fix funnel (block shown → instructions viewed → re-run unblocked, instrumented in week 1) shows owners not converting; (b) sustained demand for >10k-page crawls that chaining doesn't satisfy, or for JS rendering (CSR-shell prevalence from week-1 telemetry is the denominator); (c) DataForSEO/competitor pricing shifts. The reporter contract and schema are engine-agnostic by design, so adopting OnPage later — or the `npx` local agent (D) — is an adapter, not a rewrite. + +**Implementation notes for A (from §2's constraints):** politeness delays serialized _inside_ steps (wall-clock waits are fine; more steps are not), progress steps collapsed to stay under the 1,024-step budget, incremental per-batch D1 writes (P0 #0), block-rate baseline pulled from existing audit data (status 0/403 rates + PostHog) in week 1 so the revisit triggers have a denominator. + +One honest limitation, stated plainly: **for sites that block all datacenter crawlers, every recommended option loses scheduled audits** — D is a manual local run. The cloud-scheduling advantage evaporates for exactly that segment; our remedy there is the GSC-verified allowlist-fix flow, which restores cloud crawling rather than working around it. + +**Why not C (the desktop question, directly).** Desktop genuinely solves blocking — it's Screaming Frog's model. But scheduled audits die, server-side MCP can't wake a sleeping laptop, and it's a second product (signing, notarization, auto-update, months of work) whose own failure modes — RAM crashes, machine tie-up — are that model's loudest complaints. Draft 1's "~80% of the benefit for ~5% of the cost" for D was invented precision; the honest version: D recovers desktop's _crawl-origin_ benefit (residential IP, optional local Playwright) but not unattended operation, requires a **new** authenticated bulk-ingest endpoint feeding `audit_pages`/`audit_links` (no ingest path exists today; D depends on the P0 persistence refactor) plus a local crawler — the cheerio extraction logic is plain TS and ports, the Workflow orchestration doesn't. Estimate **~3–5 weeks**. **Panel pushback:** the panel suggested that if D balloons to 6–8 weeks the C-vs-D calculus changes. It doesn't: C strictly contains D's costs (local crawler, Playwright, sync) _plus_ installers/signing/3-platform QA, and still loses scheduling and MCP. Worst-case D remains cheaper than best-case C. Scope honesty: D serves the developer half of the ICP; for SEO freelancers (not terminal-native, and whose clients' sites are exactly the WAF'd ones) the realistic remedy is the GSC-verified allowlist flow — which is why that flow ranks above the CLI. + +**The GSC idea, honestly.** GSC verification **does not bypass any WAF** — Cloudflare neither knows nor cares about Google's ownership records. What it buys: (1) **policy cover** to raise crawl _rate_ for verified owners (industry-standard move; the Ahrefs precedent is directional, see §1) — but robots.txt respect stays the default with a per-project owner toggle, not auto-ignored: crawl traps and server load are real, and "verified" doesn't repeal politeness; (2) **the trustworthy block-fix loop** — on a detected block for a GSC-verified site, show the _owner_ the exact Cloudflare/Vercel allowlist rule for our UA (`OpenSEOBot` — UA-based since Workers egress IPs aren't publishable, see decision above). Highest-leverage mitigation available; OAuth already built; now surfaced in onboarding, not post-P0; (3) later, GSC-enriched audits (true orphans = GSC impressions, zero inlinks). + +--- + +## 5. Recommended roadmap + +**Week 1:** `audit_pages`/`audit_links`/`audit_issues` schema + engine-agnostic reporter input contract; ship block detection + honest "we were blocked" messaging + telemetry — audit-level blocked definition, fix-funnel events (block shown → instructions viewed → re-run unblocked), CSR-shell prevalence counting, baseline from existing audit data (status 0/403 + PostHog; note raw 403s overcount — classify challenges, don't just count status codes); set the documented UA (`OpenSEOBot/1.0` with a /bot info page); **1–2 day allowlist validation** on Cloudflare Free (BFM), Cloudflare Pro (SBFM), Vercel Hobby/Pro (Attack Challenge) → per-platform fix copy. + +**Weeks 2–4:** incremental persistence refactor + SSRF fix; crawler hardening (manual redirects, per-host politeness inside steps, sitemap-last seeding, step collapse; page caps coupled to Lighthouse strategy — `all` capped ~2k or sampled, crawl clamp stays 10k); issue engine + per-page reporters on the normalized record; multipage SQL checks (duplicates, broken links, chains); orphans (sitemap-last + completeness gating). + +**Weeks 4–6:** Issues UX (severity, copy, drill-down, exclusions, CSV export); MCP audit tools with `how_to_fix` payloads; GSC-verified allowlist-fix flow in onboarding (UA-based WAF rule instructions). + +**Weeks 6–8:** client-side-rendering detection + disclosure; pricing posture shipped (free 50-page audits, credits beyond) with the free-tier TTL/retention policy from §4 cost #1; buffer for the hardening tail that always exists when you own the crawler. + +**Weeks 8–14 (P1):** scheduled audits + deltas with the 2-run link-graph retention policy; **AI-crawler accessibility report as the relaunch marketing asset**; thin-content/depth checks if not landed; security + external-link checks; opportunistic Verified Bots filing for `OpenSEOBot`. + +**Later (P2):** `npx` local agent (new ingest endpoint; gate on block telemetry vs the §4 triggers); Cloudflare Browser Rendering JS audits for GSC-verified sites if demand shows; R2 crawl snapshots; hreflang/structured-data/page-size. + +**The one-sentence version:** keep audits in the cloud on our own crawler (founder call — no vendor spend), give the small ones away to drive signups, spend engineering on a portable issue engine plus agent-actionable MCP output, answer bot-blocking with day-one honest "we were blocked" flagging and owner-targeted allowlist fixes validated per-platform in week 1, accept no JS rendering for now and Lighthouse-coupled page caps — with pre-set, measurable telemetry triggers that tell us if the no-vendor bet needs revisiting, a local CLI later, and a desktop app never. diff --git a/drizzle-pg/0007_same_marvel_zombies.sql b/drizzle-pg/0007_same_marvel_zombies.sql new file mode 100644 index 0000000..c6ed9be --- /dev/null +++ b/drizzle-pg/0007_same_marvel_zombies.sql @@ -0,0 +1,36 @@ +CREATE TABLE "audit_issues" ( + "id" text PRIMARY KEY NOT NULL, + "audit_id" text NOT NULL, + "page_id" text, + "page_url" text NOT NULL, + "issue_type" text NOT NULL, + "severity" text DEFAULT 'info' NOT NULL, + "details_json" text +); +--> statement-breakpoint +CREATE TABLE "audit_links" ( + "id" text PRIMARY KEY NOT NULL, + "audit_id" text NOT NULL, + "source_page_id" text NOT NULL, + "source_url" text NOT NULL, + "target_url" text NOT NULL, + "anchor" text, + "is_internal" boolean DEFAULT true NOT NULL, + "is_nofollow" boolean DEFAULT false NOT NULL +); +--> statement-breakpoint +ALTER TABLE "audit_pages" ADD COLUMN "x_robots_tag" text;--> statement-breakpoint +ALTER TABLE "audit_pages" ADD COLUMN "header_canonical_url" text;--> statement-breakpoint +ALTER TABLE "audit_pages" ADD COLUMN "crawl_depth" integer;--> statement-breakpoint +ALTER TABLE "audit_pages" ADD COLUMN "in_sitemap" boolean DEFAULT false NOT NULL;--> statement-breakpoint +ALTER TABLE "audit_pages" ADD COLUMN "content_hash" text;--> statement-breakpoint +ALTER TABLE "audit_pages" ADD COLUMN "fetch_class" text DEFAULT 'ok' NOT NULL;--> statement-breakpoint +ALTER TABLE "audit_issues" ADD CONSTRAINT "audit_issues_audit_id_audits_id_fk" FOREIGN KEY ("audit_id") REFERENCES "public"."audits"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint +ALTER TABLE "audit_issues" ADD CONSTRAINT "audit_issues_page_id_audit_pages_id_fk" FOREIGN KEY ("page_id") REFERENCES "public"."audit_pages"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint +ALTER TABLE "audit_links" ADD CONSTRAINT "audit_links_audit_id_audits_id_fk" FOREIGN KEY ("audit_id") REFERENCES "public"."audits"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint +ALTER TABLE "audit_links" ADD CONSTRAINT "audit_links_source_page_id_audit_pages_id_fk" FOREIGN KEY ("source_page_id") REFERENCES "public"."audit_pages"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint +CREATE INDEX "audit_issues_audit_id_idx" ON "audit_issues" USING btree ("audit_id");--> statement-breakpoint +CREATE INDEX "audit_issues_audit_type_idx" ON "audit_issues" USING btree ("audit_id","issue_type");--> statement-breakpoint +CREATE INDEX "audit_links_audit_id_idx" ON "audit_links" USING btree ("audit_id");--> statement-breakpoint +CREATE INDEX "audit_links_audit_target_idx" ON "audit_links" USING btree ("audit_id","target_url");--> statement-breakpoint +CREATE INDEX "audit_pages_audit_url_idx" ON "audit_pages" USING btree ("audit_id","url"); \ No newline at end of file diff --git a/drizzle-pg/meta/0007_snapshot.json b/drizzle-pg/meta/0007_snapshot.json new file mode 100644 index 0000000..b608cf7 --- /dev/null +++ b/drizzle-pg/meta/0007_snapshot.json @@ -0,0 +1,3404 @@ +{ + "id": "7317f11e-f526-4ac5-be84-0b3c6fad6ab0", + "prevId": "d7a0c2f9-269d-44eb-bc3e-c13b4bc3cb47", + "version": "7", + "dialect": "postgresql", + "tables": { + "public.keyword_metrics": { + "name": "keyword_metrics", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "serial", + "primaryKey": true, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "keyword": { + "name": "keyword", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "location_code": { + "name": "location_code", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "language_code": { + "name": "language_code", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'en'" + }, + "search_volume": { + "name": "search_volume", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "cpc": { + "name": "cpc", + "type": "real", + "primaryKey": false, + "notNull": false + }, + "competition": { + "name": "competition", + "type": "real", + "primaryKey": false, + "notNull": false + }, + "keyword_difficulty": { + "name": "keyword_difficulty", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "intent": { + "name": "intent", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "monthly_searches": { + "name": "monthly_searches", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "fetched_at": { + "name": "fetched_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')" + } + }, + "indexes": { + "keyword_metrics_unique_project_keyword_location_language": { + "name": "keyword_metrics_unique_project_keyword_location_language", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "keyword", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "location_code", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "language_code", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "keyword_metrics_lookup_idx": { + "name": "keyword_metrics_lookup_idx", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "keyword", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "location_code", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "language_code", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "fetched_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "keyword_metrics_project_id_projects_id_fk": { + "name": "keyword_metrics_project_id_projects_id_fk", + "tableFrom": "keyword_metrics", + "tableTo": "projects", + "columnsFrom": [ + "project_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.projects": { + "name": "projects", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "domain": { + "name": "domain", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "location_code": { + "name": "location_code", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 2840 + }, + "language_code": { + "name": "language_code", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'en'" + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')" + }, + "archived_at": { + "name": "archived_at", + "type": "text", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "projects_one_default_per_organization_idx": { + "name": "projects_one_default_per_organization_idx", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"projects\".\"name\" = 'Default' AND \"projects\".\"domain\" IS NULL AND \"projects\".\"archived_at\" IS NULL", + "concurrently": false, + "method": "btree", + "with": {} + }, + "projects_organization_id_idx": { + "name": "projects_organization_id_idx", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "projects_organization_id_organization_id_fk": { + "name": "projects_organization_id_organization_id_fk", + "tableFrom": "projects", + "tableTo": "organization", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.rank_check_runs": { + "name": "rank_check_runs", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "config_id": { + "name": "config_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "keywords_total": { + "name": "keywords_total", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "keywords_checked": { + "name": "keywords_checked", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "is_subset_run": { + "name": "is_subset_run", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "error_message": { + "name": "error_message", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "started_at": { + "name": "started_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')" + }, + "completed_at": { + "name": "completed_at", + "type": "text", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "rank_check_runs_config_idx": { + "name": "rank_check_runs_config_idx", + "columns": [ + { + "expression": "config_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "started_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "rank_check_runs_project_idx": { + "name": "rank_check_runs_project_idx", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "started_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "rank_check_runs_one_active_per_config_idx": { + "name": "rank_check_runs_one_active_per_config_idx", + "columns": [ + { + "expression": "config_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"rank_check_runs\".\"status\" IN ('pending', 'running')", + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "rank_check_runs_config_id_rank_tracking_configs_id_fk": { + "name": "rank_check_runs_config_id_rank_tracking_configs_id_fk", + "tableFrom": "rank_check_runs", + "tableTo": "rank_tracking_configs", + "columnsFrom": [ + "config_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "rank_check_runs_project_id_projects_id_fk": { + "name": "rank_check_runs_project_id_projects_id_fk", + "tableFrom": "rank_check_runs", + "tableTo": "projects", + "columnsFrom": [ + "project_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.rank_snapshots": { + "name": "rank_snapshots", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "serial", + "primaryKey": true, + "notNull": true + }, + "run_id": { + "name": "run_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "tracking_keyword_id": { + "name": "tracking_keyword_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "keyword": { + "name": "keyword", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "device": { + "name": "device", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "position": { + "name": "position", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "url": { + "name": "url", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "serp_features": { + "name": "serp_features", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "checked_at": { + "name": "checked_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')" + } + }, + "indexes": { + "rank_snapshots_keyword_device_idx": { + "name": "rank_snapshots_keyword_device_idx", + "columns": [ + { + "expression": "tracking_keyword_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "device", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "checked_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "rank_snapshots_run_keyword_device_idx": { + "name": "rank_snapshots_run_keyword_device_idx", + "columns": [ + { + "expression": "run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "tracking_keyword_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "device", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "rank_snapshots_run_id_rank_check_runs_id_fk": { + "name": "rank_snapshots_run_id_rank_check_runs_id_fk", + "tableFrom": "rank_snapshots", + "tableTo": "rank_check_runs", + "columnsFrom": [ + "run_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.rank_tracking_configs": { + "name": "rank_tracking_configs", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "domain": { + "name": "domain", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "location_code": { + "name": "location_code", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 2840 + }, + "language_code": { + "name": "language_code", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'en'" + }, + "devices": { + "name": "devices", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'both'" + }, + "serp_depth": { + "name": "serp_depth", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "schedule_interval": { + "name": "schedule_interval", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'weekly'" + }, + "location_name": { + "name": "location_name", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "is_active": { + "name": "is_active", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": true + }, + "last_checked_at": { + "name": "last_checked_at", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "next_check_at": { + "name": "next_check_at", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "last_skip_reason": { + "name": "last_skip_reason", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')" + } + }, + "indexes": { + "rank_tracking_configs_national_idx": { + "name": "rank_tracking_configs_national_idx", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "domain", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "location_code", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"rank_tracking_configs\".\"location_name\" IS NULL", + "concurrently": false, + "method": "btree", + "with": {} + }, + "rank_tracking_configs_local_idx": { + "name": "rank_tracking_configs_local_idx", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "domain", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "location_code", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "location_name", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"rank_tracking_configs\".\"location_name\" IS NOT NULL", + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "rank_tracking_configs_project_id_projects_id_fk": { + "name": "rank_tracking_configs_project_id_projects_id_fk", + "tableFrom": "rank_tracking_configs", + "tableTo": "projects", + "columnsFrom": [ + "project_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.rank_tracking_keywords": { + "name": "rank_tracking_keywords", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "config_id": { + "name": "config_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "keyword": { + "name": "keyword", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "search_volume": { + "name": "search_volume", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "keyword_difficulty": { + "name": "keyword_difficulty", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "cpc": { + "name": "cpc", + "type": "real", + "primaryKey": false, + "notNull": false + }, + "metrics_fetched_at": { + "name": "metrics_fetched_at", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')" + } + }, + "indexes": { + "rank_tracking_keywords_config_keyword_idx": { + "name": "rank_tracking_keywords_config_keyword_idx", + "columns": [ + { + "expression": "config_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "keyword", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "rank_tracking_keywords_config_id_rank_tracking_configs_id_fk": { + "name": "rank_tracking_keywords_config_id_rank_tracking_configs_id_fk", + "tableFrom": "rank_tracking_keywords", + "tableTo": "rank_tracking_configs", + "columnsFrom": [ + "config_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.saved_keyword_tag_assignments": { + "name": "saved_keyword_tag_assignments", + "schema": "", + "columns": { + "saved_keyword_id": { + "name": "saved_keyword_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "tag_id": { + "name": "tag_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')" + } + }, + "indexes": { + "saved_keyword_tag_assignments_unique_idx": { + "name": "saved_keyword_tag_assignments_unique_idx", + "columns": [ + { + "expression": "saved_keyword_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "tag_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "saved_keyword_tag_assignments_tag_idx": { + "name": "saved_keyword_tag_assignments_tag_idx", + "columns": [ + { + "expression": "tag_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "saved_keyword_tag_assignments_saved_keyword_id_saved_keywords_id_fk": { + "name": "saved_keyword_tag_assignments_saved_keyword_id_saved_keywords_id_fk", + "tableFrom": "saved_keyword_tag_assignments", + "tableTo": "saved_keywords", + "columnsFrom": [ + "saved_keyword_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "saved_keyword_tag_assignments_tag_id_saved_keyword_tags_id_fk": { + "name": "saved_keyword_tag_assignments_tag_id_saved_keyword_tags_id_fk", + "tableFrom": "saved_keyword_tag_assignments", + "tableTo": "saved_keyword_tags", + "columnsFrom": [ + "tag_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.saved_keyword_tags": { + "name": "saved_keyword_tags", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "normalized_name": { + "name": "normalized_name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "color": { + "name": "color", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')" + } + }, + "indexes": { + "saved_keyword_tags_project_normalized_name_idx": { + "name": "saved_keyword_tags_project_normalized_name_idx", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "normalized_name", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "saved_keyword_tags_project_name_idx": { + "name": "saved_keyword_tags_project_name_idx", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "name", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "saved_keyword_tags_project_id_projects_id_fk": { + "name": "saved_keyword_tags_project_id_projects_id_fk", + "tableFrom": "saved_keyword_tags", + "tableTo": "projects", + "columnsFrom": [ + "project_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.saved_keywords": { + "name": "saved_keywords", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "keyword": { + "name": "keyword", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "location_code": { + "name": "location_code", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 2840 + }, + "language_code": { + "name": "language_code", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'en'" + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')" + } + }, + "indexes": { + "saved_keywords_unique_project_keyword_location_language": { + "name": "saved_keywords_unique_project_keyword_location_language", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "keyword", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "location_code", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "language_code", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "saved_keywords_project_created_idx": { + "name": "saved_keywords_project_created_idx", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "saved_keywords_project_id_projects_id_fk": { + "name": "saved_keywords_project_id_projects_id_fk", + "tableFrom": "saved_keywords", + "tableTo": "projects", + "columnsFrom": [ + "project_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.user_onboarding_answers": { + "name": "user_onboarding_answers", + "schema": "", + "columns": { + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "interested_features": { + "name": "interested_features", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'[]'" + }, + "work_for": { + "name": "work_for", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "client_website_count": { + "name": "client_website_count", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "found_via": { + "name": "found_via", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "mcp_setup_intent": { + "name": "mcp_setup_intent", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "completed_at": { + "name": "completed_at", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "gsc_nudge_dismissed_at": { + "name": "gsc_nudge_dismissed_at", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')" + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')" + } + }, + "indexes": { + "user_onboarding_answers_organization_idx": { + "name": "user_onboarding_answers_organization_idx", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "user_onboarding_answers_user_id_user_id_fk": { + "name": "user_onboarding_answers_user_id_user_id_fk", + "tableFrom": "user_onboarding_answers", + "tableTo": "user", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "user_onboarding_answers_organization_id_organization_id_fk": { + "name": "user_onboarding_answers_organization_id_organization_id_fk", + "tableFrom": "user_onboarding_answers", + "tableTo": "organization", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.audit_issues": { + "name": "audit_issues", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "audit_id": { + "name": "audit_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "page_id": { + "name": "page_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "page_url": { + "name": "page_url", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "issue_type": { + "name": "issue_type", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "severity": { + "name": "severity", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'info'" + }, + "details_json": { + "name": "details_json", + "type": "text", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "audit_issues_audit_id_idx": { + "name": "audit_issues_audit_id_idx", + "columns": [ + { + "expression": "audit_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "audit_issues_audit_type_idx": { + "name": "audit_issues_audit_type_idx", + "columns": [ + { + "expression": "audit_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "issue_type", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "audit_issues_audit_id_audits_id_fk": { + "name": "audit_issues_audit_id_audits_id_fk", + "tableFrom": "audit_issues", + "tableTo": "audits", + "columnsFrom": [ + "audit_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "audit_issues_page_id_audit_pages_id_fk": { + "name": "audit_issues_page_id_audit_pages_id_fk", + "tableFrom": "audit_issues", + "tableTo": "audit_pages", + "columnsFrom": [ + "page_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.audit_lighthouse_results": { + "name": "audit_lighthouse_results", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "audit_id": { + "name": "audit_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "page_id": { + "name": "page_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "strategy": { + "name": "strategy", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "performance_score": { + "name": "performance_score", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "accessibility_score": { + "name": "accessibility_score", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "best_practices_score": { + "name": "best_practices_score", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "seo_score": { + "name": "seo_score", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "lcp_ms": { + "name": "lcp_ms", + "type": "real", + "primaryKey": false, + "notNull": false + }, + "cls": { + "name": "cls", + "type": "real", + "primaryKey": false, + "notNull": false + }, + "inp_ms": { + "name": "inp_ms", + "type": "real", + "primaryKey": false, + "notNull": false + }, + "ttfb_ms": { + "name": "ttfb_ms", + "type": "real", + "primaryKey": false, + "notNull": false + }, + "error_message": { + "name": "error_message", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "r2_key": { + "name": "r2_key", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "payload_size_bytes": { + "name": "payload_size_bytes", + "type": "integer", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "audit_lighthouse_results_audit_id_idx": { + "name": "audit_lighthouse_results_audit_id_idx", + "columns": [ + { + "expression": "audit_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "audit_lighthouse_results_audit_id_audits_id_fk": { + "name": "audit_lighthouse_results_audit_id_audits_id_fk", + "tableFrom": "audit_lighthouse_results", + "tableTo": "audits", + "columnsFrom": [ + "audit_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "audit_lighthouse_results_page_id_audit_pages_id_fk": { + "name": "audit_lighthouse_results_page_id_audit_pages_id_fk", + "tableFrom": "audit_lighthouse_results", + "tableTo": "audit_pages", + "columnsFrom": [ + "page_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.audit_links": { + "name": "audit_links", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "audit_id": { + "name": "audit_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "source_page_id": { + "name": "source_page_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "source_url": { + "name": "source_url", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "target_url": { + "name": "target_url", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "anchor": { + "name": "anchor", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "is_internal": { + "name": "is_internal", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": true + }, + "is_nofollow": { + "name": "is_nofollow", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + } + }, + "indexes": { + "audit_links_audit_id_idx": { + "name": "audit_links_audit_id_idx", + "columns": [ + { + "expression": "audit_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "audit_links_audit_target_idx": { + "name": "audit_links_audit_target_idx", + "columns": [ + { + "expression": "audit_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "target_url", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "audit_links_audit_id_audits_id_fk": { + "name": "audit_links_audit_id_audits_id_fk", + "tableFrom": "audit_links", + "tableTo": "audits", + "columnsFrom": [ + "audit_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "audit_links_source_page_id_audit_pages_id_fk": { + "name": "audit_links_source_page_id_audit_pages_id_fk", + "tableFrom": "audit_links", + "tableTo": "audit_pages", + "columnsFrom": [ + "source_page_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.audit_pages": { + "name": "audit_pages", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "audit_id": { + "name": "audit_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "url": { + "name": "url", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status_code": { + "name": "status_code", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "redirect_url": { + "name": "redirect_url", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "meta_description": { + "name": "meta_description", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "canonical_url": { + "name": "canonical_url", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "robots_meta": { + "name": "robots_meta", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "og_title": { + "name": "og_title", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "og_description": { + "name": "og_description", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "og_image": { + "name": "og_image", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "h1_count": { + "name": "h1_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "h2_count": { + "name": "h2_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "h3_count": { + "name": "h3_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "h4_count": { + "name": "h4_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "h5_count": { + "name": "h5_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "h6_count": { + "name": "h6_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "heading_order_json": { + "name": "heading_order_json", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "word_count": { + "name": "word_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "images_total": { + "name": "images_total", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "images_missing_alt": { + "name": "images_missing_alt", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "images_json": { + "name": "images_json", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "internal_link_count": { + "name": "internal_link_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "external_link_count": { + "name": "external_link_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "has_structured_data": { + "name": "has_structured_data", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "hreflang_tags_json": { + "name": "hreflang_tags_json", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "is_indexable": { + "name": "is_indexable", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": true + }, + "x_robots_tag": { + "name": "x_robots_tag", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "header_canonical_url": { + "name": "header_canonical_url", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "crawl_depth": { + "name": "crawl_depth", + "type": "integer", + "primaryKey": false, + "notNull": false + }, + "in_sitemap": { + "name": "in_sitemap", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "content_hash": { + "name": "content_hash", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "fetch_class": { + "name": "fetch_class", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'ok'" + }, + "response_time_ms": { + "name": "response_time_ms", + "type": "integer", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "audit_pages_audit_id_idx": { + "name": "audit_pages_audit_id_idx", + "columns": [ + { + "expression": "audit_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "audit_pages_audit_url_idx": { + "name": "audit_pages_audit_url_idx", + "columns": [ + { + "expression": "audit_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "url", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "audit_pages_audit_id_audits_id_fk": { + "name": "audit_pages_audit_id_audits_id_fk", + "tableFrom": "audit_pages", + "tableTo": "audits", + "columnsFrom": [ + "audit_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.audits": { + "name": "audits", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "started_by_user_id": { + "name": "started_by_user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "start_url": { + "name": "start_url", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'running'" + }, + "workflow_instance_id": { + "name": "workflow_instance_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "config": { + "name": "config", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'{}'" + }, + "pages_crawled": { + "name": "pages_crawled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "pages_total": { + "name": "pages_total", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "lighthouse_total": { + "name": "lighthouse_total", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "lighthouse_completed": { + "name": "lighthouse_completed", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "lighthouse_failed": { + "name": "lighthouse_failed", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "current_phase": { + "name": "current_phase", + "type": "text", + "primaryKey": false, + "notNull": false, + "default": "'discovery'" + }, + "started_at": { + "name": "started_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')" + }, + "completed_at": { + "name": "completed_at", + "type": "text", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "audits_project_id_idx": { + "name": "audits_project_id_idx", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "audits_started_by_user_id_idx": { + "name": "audits_started_by_user_id_idx", + "columns": [ + { + "expression": "started_by_user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "audits_project_id_projects_id_fk": { + "name": "audits_project_id_projects_id_fk", + "tableFrom": "audits", + "tableTo": "projects", + "columnsFrom": [ + "project_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.sam_project_memory": { + "name": "sam_project_memory", + "schema": "", + "columns": { + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "content": { + "name": "content", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')" + } + }, + "indexes": {}, + "foreignKeys": { + "sam_project_memory_project_id_projects_id_fk": { + "name": "sam_project_memory_project_id_projects_id_fk", + "tableFrom": "sam_project_memory", + "tableTo": "projects", + "columnsFrom": [ + "project_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "sam_project_memory_project_id_label_pk": { + "name": "sam_project_memory_project_id_label_pk", + "columns": [ + "project_id", + "label" + ] + } + }, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.sam_sessions": { + "name": "sam_sessions", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'New chat'" + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')" + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')" + }, + "archived_at": { + "name": "archived_at", + "type": "text", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "sam_sessions_project_updated_idx": { + "name": "sam_sessions_project_updated_idx", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "updated_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "sam_sessions_project_id_projects_id_fk": { + "name": "sam_sessions_project_id_projects_id_fk", + "tableFrom": "sam_sessions", + "tableTo": "projects", + "columnsFrom": [ + "project_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "sam_sessions_user_id_user_id_fk": { + "name": "sam_sessions_user_id_user_id_fk", + "tableFrom": "sam_sessions", + "tableTo": "user", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.account": { + "name": "account", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "account_id": { + "name": "account_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "provider_id": { + "name": "provider_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "access_token": { + "name": "access_token", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "refresh_token": { + "name": "refresh_token", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "id_token": { + "name": "id_token", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "access_token_expires_at": { + "name": "access_token_expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "refresh_token_expires_at": { + "name": "refresh_token_expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "scope": { + "name": "scope", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "password": { + "name": "password", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + } + }, + "indexes": { + "account_userId_idx": { + "name": "account_userId_idx", + "columns": [ + { + "expression": "user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "account_accountId_providerId_idx": { + "name": "account_accountId_providerId_idx", + "columns": [ + { + "expression": "account_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "provider_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "account_user_id_user_id_fk": { + "name": "account_user_id_user_id_fk", + "tableFrom": "account", + "tableTo": "user", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.invitation": { + "name": "invitation", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "email": { + "name": "email", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "role": { + "name": "role", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "inviter_id": { + "name": "inviter_id", + "type": "text", + "primaryKey": false, + "notNull": true + } + }, + "indexes": { + "invitation_organizationId_idx": { + "name": "invitation_organizationId_idx", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "invitation_email_idx": { + "name": "invitation_email_idx", + "columns": [ + { + "expression": "email", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "invitation_organization_id_organization_id_fk": { + "name": "invitation_organization_id_organization_id_fk", + "tableFrom": "invitation", + "tableTo": "organization", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "invitation_inviter_id_user_id_fk": { + "name": "invitation_inviter_id_user_id_fk", + "tableFrom": "invitation", + "tableTo": "user", + "columnsFrom": [ + "inviter_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.member": { + "name": "member", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "role": { + "name": "role", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'member'" + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + } + }, + "indexes": { + "member_organizationId_idx": { + "name": "member_organizationId_idx", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "member_userId_idx": { + "name": "member_userId_idx", + "columns": [ + { + "expression": "user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "member_organization_id_organization_id_fk": { + "name": "member_organization_id_organization_id_fk", + "tableFrom": "member", + "tableTo": "organization", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "member_user_id_user_id_fk": { + "name": "member_user_id_user_id_fk", + "tableFrom": "member", + "tableTo": "user", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.organization": { + "name": "organization", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "slug": { + "name": "slug", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "logo": { + "name": "logo", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "metadata": { + "name": "metadata", + "type": "text", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "organization_slug_uidx": { + "name": "organization_slug_uidx", + "columns": [ + { + "expression": "slug", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "organization_slug_unique": { + "name": "organization_slug_unique", + "nullsNotDistinct": false, + "columns": [ + "slug" + ] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.session": { + "name": "session", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "token": { + "name": "token", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "ip_address": { + "name": "ip_address", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "user_agent": { + "name": "user_agent", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "active_organization_id": { + "name": "active_organization_id", + "type": "text", + "primaryKey": false, + "notNull": false + } + }, + "indexes": { + "session_userId_idx": { + "name": "session_userId_idx", + "columns": [ + { + "expression": "user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "session_user_id_user_id_fk": { + "name": "session_user_id_user_id_fk", + "tableFrom": "session", + "tableTo": "user", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "session_token_unique": { + "name": "session_token_unique", + "nullsNotDistinct": false, + "columns": [ + "token" + ] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.user": { + "name": "user", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "email": { + "name": "email", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "email_verified": { + "name": "email_verified", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "image": { + "name": "image", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "analytics_opted_out": { + "name": "analytics_opted_out", + "type": "boolean", + "primaryKey": false, + "notNull": false + } + }, + "indexes": {}, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": { + "user_email_unique": { + "name": "user_email_unique", + "nullsNotDistinct": false, + "columns": [ + "email" + ] + } + }, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.verification": { + "name": "verification", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "identifier": { + "name": "identifier", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "value": { + "name": "value", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "verification_identifier_idx": { + "name": "verification_identifier_idx", + "columns": [ + { + "expression": "identifier", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "verification_expiresAt_idx": { + "name": "verification_expiresAt_idx", + "columns": [ + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.billing_customer_status": { + "name": "billing_customer_status", + "schema": "", + "columns": { + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "is_paying": { + "name": "is_paying", + "type": "boolean", + "primaryKey": false, + "notNull": true, + "default": false + }, + "paid_plan_id": { + "name": "paid_plan_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "paid_plan_status": { + "name": "paid_plan_status", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "customer_json": { + "name": "customer_json", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "synced_at": { + "name": "synced_at", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')" + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')" + } + }, + "indexes": {}, + "foreignKeys": { + "billing_customer_status_organization_id_organization_id_fk": { + "name": "billing_customer_status_organization_id_organization_id_fk", + "tableFrom": "billing_customer_status", + "tableTo": "organization", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.gsc_connections": { + "name": "gsc_connections", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "site_url": { + "name": "site_url", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "connected_by_user_id": { + "name": "connected_by_user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "connected_account_email": { + "name": "connected_account_email", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')" + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')" + } + }, + "indexes": { + "gsc_connections_project_idx": { + "name": "gsc_connections_project_idx", + "columns": [ + { + "expression": "project_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "gsc_connections_organization_idx": { + "name": "gsc_connections_organization_idx", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "gsc_connections_project_id_projects_id_fk": { + "name": "gsc_connections_project_id_projects_id_fk", + "tableFrom": "gsc_connections", + "tableTo": "projects", + "columnsFrom": [ + "project_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "gsc_connections_organization_id_organization_id_fk": { + "name": "gsc_connections_organization_id_organization_id_fk", + "tableFrom": "gsc_connections", + "tableTo": "organization", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.reddit_attributions": { + "name": "reddit_attributions", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "click_id": { + "name": "click_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "uuid": { + "name": "uuid", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "landing_page": { + "name": "landing_page", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "referrer": { + "name": "referrer", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "utm_source": { + "name": "utm_source", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "utm_medium": { + "name": "utm_medium", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "utm_campaign": { + "name": "utm_campaign", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "utm_term": { + "name": "utm_term", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "utm_content": { + "name": "utm_content", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "signup_sent_at": { + "name": "signup_sent_at", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "purchase_sent_at": { + "name": "purchase_sent_at", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')" + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"')" + } + }, + "indexes": { + "reddit_attributions_user_idx": { + "name": "reddit_attributions_user_idx", + "columns": [ + { + "expression": "user_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "reddit_attributions_organization_idx": { + "name": "reddit_attributions_organization_idx", + "columns": [ + { + "expression": "organization_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "reddit_attributions_user_id_user_id_fk": { + "name": "reddit_attributions_user_id_user_id_fk", + "tableFrom": "reddit_attributions", + "tableTo": "user", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "reddit_attributions_organization_id_organization_id_fk": { + "name": "reddit_attributions_organization_id_organization_id_fk", + "tableFrom": "reddit_attributions", + "tableTo": "organization", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + } + }, + "enums": {}, + "schemas": {}, + "sequences": {}, + "roles": {}, + "policies": {}, + "views": {}, + "_meta": { + "columns": {}, + "schemas": {}, + "tables": {} + } +} \ No newline at end of file diff --git a/drizzle-pg/meta/_journal.json b/drizzle-pg/meta/_journal.json index d50969b..5c7196d 100644 --- a/drizzle-pg/meta/_journal.json +++ b/drizzle-pg/meta/_journal.json @@ -50,6 +50,13 @@ "when": 1783438455708, "tag": "0006_location_name", "breakpoints": true + }, + { + "idx": 7, + "version": "7", + "when": 1783306049167, + "tag": "0007_same_marvel_zombies", + "breakpoints": true } ] } \ No newline at end of file diff --git a/drizzle/0030_legal_reaper.sql b/drizzle/0030_legal_reaper.sql new file mode 100644 index 0000000..3bd9775 --- /dev/null +++ b/drizzle/0030_legal_reaper.sql @@ -0,0 +1,36 @@ +CREATE TABLE `audit_issues` ( + `id` text PRIMARY KEY NOT NULL, + `audit_id` text NOT NULL, + `page_id` text, + `page_url` text NOT NULL, + `issue_type` text NOT NULL, + `severity` text DEFAULT 'info' NOT NULL, + `details_json` text, + FOREIGN KEY (`audit_id`) REFERENCES `audits`(`id`) ON UPDATE no action ON DELETE cascade, + FOREIGN KEY (`page_id`) REFERENCES `audit_pages`(`id`) ON UPDATE no action ON DELETE cascade +); +--> statement-breakpoint +CREATE INDEX `audit_issues_audit_id_idx` ON `audit_issues` (`audit_id`);--> statement-breakpoint +CREATE INDEX `audit_issues_audit_type_idx` ON `audit_issues` (`audit_id`,`issue_type`);--> statement-breakpoint +CREATE TABLE `audit_links` ( + `id` text PRIMARY KEY NOT NULL, + `audit_id` text NOT NULL, + `source_page_id` text NOT NULL, + `source_url` text NOT NULL, + `target_url` text NOT NULL, + `anchor` text, + `is_internal` integer DEFAULT true NOT NULL, + `is_nofollow` integer DEFAULT false NOT NULL, + FOREIGN KEY (`audit_id`) REFERENCES `audits`(`id`) ON UPDATE no action ON DELETE cascade, + FOREIGN KEY (`source_page_id`) REFERENCES `audit_pages`(`id`) ON UPDATE no action ON DELETE cascade +); +--> statement-breakpoint +CREATE INDEX `audit_links_audit_id_idx` ON `audit_links` (`audit_id`);--> statement-breakpoint +CREATE INDEX `audit_links_audit_target_idx` ON `audit_links` (`audit_id`,`target_url`);--> statement-breakpoint +ALTER TABLE `audit_pages` ADD `x_robots_tag` text;--> statement-breakpoint +ALTER TABLE `audit_pages` ADD `header_canonical_url` text;--> statement-breakpoint +ALTER TABLE `audit_pages` ADD `crawl_depth` integer;--> statement-breakpoint +ALTER TABLE `audit_pages` ADD `in_sitemap` integer DEFAULT false NOT NULL;--> statement-breakpoint +ALTER TABLE `audit_pages` ADD `content_hash` text;--> statement-breakpoint +ALTER TABLE `audit_pages` ADD `fetch_class` text DEFAULT 'ok' NOT NULL;--> statement-breakpoint +CREATE INDEX `audit_pages_audit_url_idx` ON `audit_pages` (`audit_id`,`url`); \ No newline at end of file diff --git a/drizzle/meta/0030_snapshot.json b/drizzle/meta/0030_snapshot.json new file mode 100644 index 0000000..d5f4d23 --- /dev/null +++ b/drizzle/meta/0030_snapshot.json @@ -0,0 +1,3086 @@ +{ + "version": "6", + "dialect": "sqlite", + "id": "57a79baa-091a-4d26-9324-dd8623e530a9", + "prevId": "04f10078-feca-42c4-9158-f1ba8df6091f", + "tables": { + "keyword_metrics": { + "name": "keyword_metrics", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "keyword": { + "name": "keyword", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "location_code": { + "name": "location_code", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "language_code": { + "name": "language_code", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'en'" + }, + "search_volume": { + "name": "search_volume", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "cpc": { + "name": "cpc", + "type": "real", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "competition": { + "name": "competition", + "type": "real", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "keyword_difficulty": { + "name": "keyword_difficulty", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "intent": { + "name": "intent", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "monthly_searches": { + "name": "monthly_searches", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "fetched_at": { + "name": "fetched_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "keyword_metrics_unique_project_keyword_location_language": { + "name": "keyword_metrics_unique_project_keyword_location_language", + "columns": [ + "project_id", + "keyword", + "location_code", + "language_code" + ], + "isUnique": true + }, + "keyword_metrics_lookup_idx": { + "name": "keyword_metrics_lookup_idx", + "columns": [ + "project_id", + "keyword", + "location_code", + "language_code", + "fetched_at" + ], + "isUnique": false + } + }, + "foreignKeys": { + "keyword_metrics_project_id_projects_id_fk": { + "name": "keyword_metrics_project_id_projects_id_fk", + "tableFrom": "keyword_metrics", + "tableTo": "projects", + "columnsFrom": [ + "project_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "projects": { + "name": "projects", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "domain": { + "name": "domain", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "location_code": { + "name": "location_code", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 2840 + }, + "language_code": { + "name": "language_code", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'en'" + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "archived_at": { + "name": "archived_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "projects_one_default_per_organization_idx": { + "name": "projects_one_default_per_organization_idx", + "columns": [ + "organization_id" + ], + "isUnique": true, + "where": "\"projects\".\"name\" = 'Default' AND \"projects\".\"domain\" IS NULL AND \"projects\".\"archived_at\" IS NULL" + }, + "projects_organization_id_idx": { + "name": "projects_organization_id_idx", + "columns": [ + "organization_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "projects_organization_id_organization_id_fk": { + "name": "projects_organization_id_organization_id_fk", + "tableFrom": "projects", + "tableTo": "organization", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "rank_check_runs": { + "name": "rank_check_runs", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "config_id": { + "name": "config_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'pending'" + }, + "keywords_total": { + "name": "keywords_total", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "keywords_checked": { + "name": "keywords_checked", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "is_subset_run": { + "name": "is_subset_run", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "error_message": { + "name": "error_message", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "started_at": { + "name": "started_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "completed_at": { + "name": "completed_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "rank_check_runs_config_idx": { + "name": "rank_check_runs_config_idx", + "columns": [ + "config_id", + "started_at" + ], + "isUnique": false + }, + "rank_check_runs_project_idx": { + "name": "rank_check_runs_project_idx", + "columns": [ + "project_id", + "started_at" + ], + "isUnique": false + }, + "rank_check_runs_one_active_per_config_idx": { + "name": "rank_check_runs_one_active_per_config_idx", + "columns": [ + "config_id" + ], + "isUnique": true, + "where": "\"rank_check_runs\".\"status\" IN ('pending', 'running')" + } + }, + "foreignKeys": { + "rank_check_runs_config_id_rank_tracking_configs_id_fk": { + "name": "rank_check_runs_config_id_rank_tracking_configs_id_fk", + "tableFrom": "rank_check_runs", + "tableTo": "rank_tracking_configs", + "columnsFrom": [ + "config_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "rank_check_runs_project_id_projects_id_fk": { + "name": "rank_check_runs_project_id_projects_id_fk", + "tableFrom": "rank_check_runs", + "tableTo": "projects", + "columnsFrom": [ + "project_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "rank_snapshots": { + "name": "rank_snapshots", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "run_id": { + "name": "run_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "tracking_keyword_id": { + "name": "tracking_keyword_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "keyword": { + "name": "keyword", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "device": { + "name": "device", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "position": { + "name": "position", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "url": { + "name": "url", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "serp_features": { + "name": "serp_features", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "checked_at": { + "name": "checked_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "rank_snapshots_keyword_device_idx": { + "name": "rank_snapshots_keyword_device_idx", + "columns": [ + "tracking_keyword_id", + "device", + "checked_at" + ], + "isUnique": false + }, + "rank_snapshots_run_keyword_device_idx": { + "name": "rank_snapshots_run_keyword_device_idx", + "columns": [ + "run_id", + "tracking_keyword_id", + "device" + ], + "isUnique": true + } + }, + "foreignKeys": { + "rank_snapshots_run_id_rank_check_runs_id_fk": { + "name": "rank_snapshots_run_id_rank_check_runs_id_fk", + "tableFrom": "rank_snapshots", + "tableTo": "rank_check_runs", + "columnsFrom": [ + "run_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "rank_tracking_configs": { + "name": "rank_tracking_configs", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "domain": { + "name": "domain", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "location_code": { + "name": "location_code", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 2840 + }, + "language_code": { + "name": "language_code", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'en'" + }, + "devices": { + "name": "devices", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'both'" + }, + "serp_depth": { + "name": "serp_depth", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "schedule_interval": { + "name": "schedule_interval", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'weekly'" + }, + "location_name": { + "name": "location_name", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "is_active": { + "name": "is_active", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "last_checked_at": { + "name": "last_checked_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "next_check_at": { + "name": "next_check_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "last_skip_reason": { + "name": "last_skip_reason", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "rank_tracking_configs_national_idx": { + "name": "rank_tracking_configs_national_idx", + "columns": [ + "project_id", + "domain", + "location_code" + ], + "isUnique": true, + "where": "\"rank_tracking_configs\".\"location_name\" IS NULL" + }, + "rank_tracking_configs_local_idx": { + "name": "rank_tracking_configs_local_idx", + "columns": [ + "project_id", + "domain", + "location_code", + "location_name" + ], + "isUnique": true, + "where": "\"rank_tracking_configs\".\"location_name\" IS NOT NULL" + } + }, + "foreignKeys": { + "rank_tracking_configs_project_id_projects_id_fk": { + "name": "rank_tracking_configs_project_id_projects_id_fk", + "tableFrom": "rank_tracking_configs", + "tableTo": "projects", + "columnsFrom": [ + "project_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "rank_tracking_keywords": { + "name": "rank_tracking_keywords", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "config_id": { + "name": "config_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "keyword": { + "name": "keyword", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "search_volume": { + "name": "search_volume", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "keyword_difficulty": { + "name": "keyword_difficulty", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "cpc": { + "name": "cpc", + "type": "real", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "metrics_fetched_at": { + "name": "metrics_fetched_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "rank_tracking_keywords_config_keyword_idx": { + "name": "rank_tracking_keywords_config_keyword_idx", + "columns": [ + "config_id", + "keyword" + ], + "isUnique": true + } + }, + "foreignKeys": { + "rank_tracking_keywords_config_id_rank_tracking_configs_id_fk": { + "name": "rank_tracking_keywords_config_id_rank_tracking_configs_id_fk", + "tableFrom": "rank_tracking_keywords", + "tableTo": "rank_tracking_configs", + "columnsFrom": [ + "config_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "saved_keyword_tag_assignments": { + "name": "saved_keyword_tag_assignments", + "columns": { + "saved_keyword_id": { + "name": "saved_keyword_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "tag_id": { + "name": "tag_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "saved_keyword_tag_assignments_unique_idx": { + "name": "saved_keyword_tag_assignments_unique_idx", + "columns": [ + "saved_keyword_id", + "tag_id" + ], + "isUnique": true + }, + "saved_keyword_tag_assignments_tag_idx": { + "name": "saved_keyword_tag_assignments_tag_idx", + "columns": [ + "tag_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "saved_keyword_tag_assignments_saved_keyword_id_saved_keywords_id_fk": { + "name": "saved_keyword_tag_assignments_saved_keyword_id_saved_keywords_id_fk", + "tableFrom": "saved_keyword_tag_assignments", + "tableTo": "saved_keywords", + "columnsFrom": [ + "saved_keyword_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "saved_keyword_tag_assignments_tag_id_saved_keyword_tags_id_fk": { + "name": "saved_keyword_tag_assignments_tag_id_saved_keyword_tags_id_fk", + "tableFrom": "saved_keyword_tag_assignments", + "tableTo": "saved_keyword_tags", + "columnsFrom": [ + "tag_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "saved_keyword_tags": { + "name": "saved_keyword_tags", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "normalized_name": { + "name": "normalized_name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "color": { + "name": "color", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "saved_keyword_tags_project_normalized_name_idx": { + "name": "saved_keyword_tags_project_normalized_name_idx", + "columns": [ + "project_id", + "normalized_name" + ], + "isUnique": true + }, + "saved_keyword_tags_project_name_idx": { + "name": "saved_keyword_tags_project_name_idx", + "columns": [ + "project_id", + "name" + ], + "isUnique": false + } + }, + "foreignKeys": { + "saved_keyword_tags_project_id_projects_id_fk": { + "name": "saved_keyword_tags_project_id_projects_id_fk", + "tableFrom": "saved_keyword_tags", + "tableTo": "projects", + "columnsFrom": [ + "project_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "saved_keywords": { + "name": "saved_keywords", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "keyword": { + "name": "keyword", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "location_code": { + "name": "location_code", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 2840 + }, + "language_code": { + "name": "language_code", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'en'" + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "saved_keywords_unique_project_keyword_location_language": { + "name": "saved_keywords_unique_project_keyword_location_language", + "columns": [ + "project_id", + "keyword", + "location_code", + "language_code" + ], + "isUnique": true + }, + "saved_keywords_project_created_idx": { + "name": "saved_keywords_project_created_idx", + "columns": [ + "project_id", + "created_at" + ], + "isUnique": false + } + }, + "foreignKeys": { + "saved_keywords_project_id_projects_id_fk": { + "name": "saved_keywords_project_id_projects_id_fk", + "tableFrom": "saved_keywords", + "tableTo": "projects", + "columnsFrom": [ + "project_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "user_onboarding_answers": { + "name": "user_onboarding_answers", + "columns": { + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "interested_features": { + "name": "interested_features", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'[]'" + }, + "work_for": { + "name": "work_for", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "client_website_count": { + "name": "client_website_count", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "found_via": { + "name": "found_via", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "mcp_setup_intent": { + "name": "mcp_setup_intent", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "completed_at": { + "name": "completed_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "gsc_nudge_dismissed_at": { + "name": "gsc_nudge_dismissed_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "user_onboarding_answers_organization_idx": { + "name": "user_onboarding_answers_organization_idx", + "columns": [ + "organization_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "user_onboarding_answers_user_id_user_id_fk": { + "name": "user_onboarding_answers_user_id_user_id_fk", + "tableFrom": "user_onboarding_answers", + "tableTo": "user", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "user_onboarding_answers_organization_id_organization_id_fk": { + "name": "user_onboarding_answers_organization_id_organization_id_fk", + "tableFrom": "user_onboarding_answers", + "tableTo": "organization", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "audit_issues": { + "name": "audit_issues", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "audit_id": { + "name": "audit_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "page_id": { + "name": "page_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "page_url": { + "name": "page_url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "issue_type": { + "name": "issue_type", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "severity": { + "name": "severity", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'info'" + }, + "details_json": { + "name": "details_json", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "audit_issues_audit_id_idx": { + "name": "audit_issues_audit_id_idx", + "columns": [ + "audit_id" + ], + "isUnique": false + }, + "audit_issues_audit_type_idx": { + "name": "audit_issues_audit_type_idx", + "columns": [ + "audit_id", + "issue_type" + ], + "isUnique": false + } + }, + "foreignKeys": { + "audit_issues_audit_id_audits_id_fk": { + "name": "audit_issues_audit_id_audits_id_fk", + "tableFrom": "audit_issues", + "tableTo": "audits", + "columnsFrom": [ + "audit_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "audit_issues_page_id_audit_pages_id_fk": { + "name": "audit_issues_page_id_audit_pages_id_fk", + "tableFrom": "audit_issues", + "tableTo": "audit_pages", + "columnsFrom": [ + "page_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "audit_lighthouse_results": { + "name": "audit_lighthouse_results", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "audit_id": { + "name": "audit_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "page_id": { + "name": "page_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "strategy": { + "name": "strategy", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "performance_score": { + "name": "performance_score", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "accessibility_score": { + "name": "accessibility_score", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "best_practices_score": { + "name": "best_practices_score", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "seo_score": { + "name": "seo_score", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "lcp_ms": { + "name": "lcp_ms", + "type": "real", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "cls": { + "name": "cls", + "type": "real", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "inp_ms": { + "name": "inp_ms", + "type": "real", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "ttfb_ms": { + "name": "ttfb_ms", + "type": "real", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "error_message": { + "name": "error_message", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "r2_key": { + "name": "r2_key", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "payload_size_bytes": { + "name": "payload_size_bytes", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "audit_lighthouse_results_audit_id_idx": { + "name": "audit_lighthouse_results_audit_id_idx", + "columns": [ + "audit_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "audit_lighthouse_results_audit_id_audits_id_fk": { + "name": "audit_lighthouse_results_audit_id_audits_id_fk", + "tableFrom": "audit_lighthouse_results", + "tableTo": "audits", + "columnsFrom": [ + "audit_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "audit_lighthouse_results_page_id_audit_pages_id_fk": { + "name": "audit_lighthouse_results_page_id_audit_pages_id_fk", + "tableFrom": "audit_lighthouse_results", + "tableTo": "audit_pages", + "columnsFrom": [ + "page_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "audit_links": { + "name": "audit_links", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "audit_id": { + "name": "audit_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "source_page_id": { + "name": "source_page_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "source_url": { + "name": "source_url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "target_url": { + "name": "target_url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "anchor": { + "name": "anchor", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "is_internal": { + "name": "is_internal", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "is_nofollow": { + "name": "is_nofollow", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + } + }, + "indexes": { + "audit_links_audit_id_idx": { + "name": "audit_links_audit_id_idx", + "columns": [ + "audit_id" + ], + "isUnique": false + }, + "audit_links_audit_target_idx": { + "name": "audit_links_audit_target_idx", + "columns": [ + "audit_id", + "target_url" + ], + "isUnique": false + } + }, + "foreignKeys": { + "audit_links_audit_id_audits_id_fk": { + "name": "audit_links_audit_id_audits_id_fk", + "tableFrom": "audit_links", + "tableTo": "audits", + "columnsFrom": [ + "audit_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "audit_links_source_page_id_audit_pages_id_fk": { + "name": "audit_links_source_page_id_audit_pages_id_fk", + "tableFrom": "audit_links", + "tableTo": "audit_pages", + "columnsFrom": [ + "source_page_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "audit_pages": { + "name": "audit_pages", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "audit_id": { + "name": "audit_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "url": { + "name": "url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "status_code": { + "name": "status_code", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "redirect_url": { + "name": "redirect_url", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "meta_description": { + "name": "meta_description", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "canonical_url": { + "name": "canonical_url", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "robots_meta": { + "name": "robots_meta", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "og_title": { + "name": "og_title", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "og_description": { + "name": "og_description", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "og_image": { + "name": "og_image", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "h1_count": { + "name": "h1_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "h2_count": { + "name": "h2_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "h3_count": { + "name": "h3_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "h4_count": { + "name": "h4_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "h5_count": { + "name": "h5_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "h6_count": { + "name": "h6_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "heading_order_json": { + "name": "heading_order_json", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "word_count": { + "name": "word_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "images_total": { + "name": "images_total", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "images_missing_alt": { + "name": "images_missing_alt", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "images_json": { + "name": "images_json", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "internal_link_count": { + "name": "internal_link_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "external_link_count": { + "name": "external_link_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "has_structured_data": { + "name": "has_structured_data", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "hreflang_tags_json": { + "name": "hreflang_tags_json", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "is_indexable": { + "name": "is_indexable", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "x_robots_tag": { + "name": "x_robots_tag", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "header_canonical_url": { + "name": "header_canonical_url", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "crawl_depth": { + "name": "crawl_depth", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "in_sitemap": { + "name": "in_sitemap", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "content_hash": { + "name": "content_hash", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "fetch_class": { + "name": "fetch_class", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'ok'" + }, + "response_time_ms": { + "name": "response_time_ms", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "audit_pages_audit_id_idx": { + "name": "audit_pages_audit_id_idx", + "columns": [ + "audit_id" + ], + "isUnique": false + }, + "audit_pages_audit_url_idx": { + "name": "audit_pages_audit_url_idx", + "columns": [ + "audit_id", + "url" + ], + "isUnique": false + } + }, + "foreignKeys": { + "audit_pages_audit_id_audits_id_fk": { + "name": "audit_pages_audit_id_audits_id_fk", + "tableFrom": "audit_pages", + "tableTo": "audits", + "columnsFrom": [ + "audit_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "audits": { + "name": "audits", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "started_by_user_id": { + "name": "started_by_user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "start_url": { + "name": "start_url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'running'" + }, + "workflow_instance_id": { + "name": "workflow_instance_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "config": { + "name": "config", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'{}'" + }, + "pages_crawled": { + "name": "pages_crawled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "pages_total": { + "name": "pages_total", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "lighthouse_total": { + "name": "lighthouse_total", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "lighthouse_completed": { + "name": "lighthouse_completed", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "lighthouse_failed": { + "name": "lighthouse_failed", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "current_phase": { + "name": "current_phase", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "'discovery'" + }, + "started_at": { + "name": "started_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "completed_at": { + "name": "completed_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "audits_project_id_idx": { + "name": "audits_project_id_idx", + "columns": [ + "project_id" + ], + "isUnique": false + }, + "audits_started_by_user_id_idx": { + "name": "audits_started_by_user_id_idx", + "columns": [ + "started_by_user_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "audits_project_id_projects_id_fk": { + "name": "audits_project_id_projects_id_fk", + "tableFrom": "audits", + "tableTo": "projects", + "columnsFrom": [ + "project_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "sam_project_memory": { + "name": "sam_project_memory", + "columns": { + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "label": { + "name": "label", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "content": { + "name": "content", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": { + "sam_project_memory_project_id_projects_id_fk": { + "name": "sam_project_memory_project_id_projects_id_fk", + "tableFrom": "sam_project_memory", + "tableTo": "projects", + "columnsFrom": [ + "project_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "sam_project_memory_project_id_label_pk": { + "columns": [ + "project_id", + "label" + ], + "name": "sam_project_memory_project_id_label_pk" + } + }, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "sam_sessions": { + "name": "sam_sessions", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'New chat'" + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "archived_at": { + "name": "archived_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "sam_sessions_project_updated_idx": { + "name": "sam_sessions_project_updated_idx", + "columns": [ + "project_id", + "updated_at" + ], + "isUnique": false + } + }, + "foreignKeys": { + "sam_sessions_project_id_projects_id_fk": { + "name": "sam_sessions_project_id_projects_id_fk", + "tableFrom": "sam_sessions", + "tableTo": "projects", + "columnsFrom": [ + "project_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "sam_sessions_user_id_user_id_fk": { + "name": "sam_sessions_user_id_user_id_fk", + "tableFrom": "sam_sessions", + "tableTo": "user", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "account": { + "name": "account", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "account_id": { + "name": "account_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "provider_id": { + "name": "provider_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "access_token": { + "name": "access_token", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "refresh_token": { + "name": "refresh_token", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "id_token": { + "name": "id_token", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "access_token_expires_at": { + "name": "access_token_expires_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "refresh_token_expires_at": { + "name": "refresh_token_expires_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "scope": { + "name": "scope", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "password": { + "name": "password", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(cast(unixepoch('subsecond') * 1000 as integer))" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "account_userId_idx": { + "name": "account_userId_idx", + "columns": [ + "user_id" + ], + "isUnique": false + }, + "account_accountId_providerId_idx": { + "name": "account_accountId_providerId_idx", + "columns": [ + "account_id", + "provider_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "account_user_id_user_id_fk": { + "name": "account_user_id_user_id_fk", + "tableFrom": "account", + "tableTo": "user", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "invitation": { + "name": "invitation", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "email": { + "name": "email", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "role": { + "name": "role", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'pending'" + }, + "expires_at": { + "name": "expires_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(cast(unixepoch('subsecond') * 1000 as integer))" + }, + "inviter_id": { + "name": "inviter_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "invitation_organizationId_idx": { + "name": "invitation_organizationId_idx", + "columns": [ + "organization_id" + ], + "isUnique": false + }, + "invitation_email_idx": { + "name": "invitation_email_idx", + "columns": [ + "email" + ], + "isUnique": false + } + }, + "foreignKeys": { + "invitation_organization_id_organization_id_fk": { + "name": "invitation_organization_id_organization_id_fk", + "tableFrom": "invitation", + "tableTo": "organization", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "invitation_inviter_id_user_id_fk": { + "name": "invitation_inviter_id_user_id_fk", + "tableFrom": "invitation", + "tableTo": "user", + "columnsFrom": [ + "inviter_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "member": { + "name": "member", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "role": { + "name": "role", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'member'" + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + } + }, + "indexes": { + "member_organizationId_idx": { + "name": "member_organizationId_idx", + "columns": [ + "organization_id" + ], + "isUnique": false + }, + "member_userId_idx": { + "name": "member_userId_idx", + "columns": [ + "user_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "member_organization_id_organization_id_fk": { + "name": "member_organization_id_organization_id_fk", + "tableFrom": "member", + "tableTo": "organization", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "member_user_id_user_id_fk": { + "name": "member_user_id_user_id_fk", + "tableFrom": "member", + "tableTo": "user", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "organization": { + "name": "organization", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "slug": { + "name": "slug", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "logo": { + "name": "logo", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "metadata": { + "name": "metadata", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "organization_slug_unique": { + "name": "organization_slug_unique", + "columns": [ + "slug" + ], + "isUnique": true + }, + "organization_slug_uidx": { + "name": "organization_slug_uidx", + "columns": [ + "slug" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "session": { + "name": "session", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "expires_at": { + "name": "expires_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "token": { + "name": "token", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(cast(unixepoch('subsecond') * 1000 as integer))" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "ip_address": { + "name": "ip_address", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "user_agent": { + "name": "user_agent", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "active_organization_id": { + "name": "active_organization_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "session_token_unique": { + "name": "session_token_unique", + "columns": [ + "token" + ], + "isUnique": true + }, + "session_userId_idx": { + "name": "session_userId_idx", + "columns": [ + "user_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "session_user_id_user_id_fk": { + "name": "session_user_id_user_id_fk", + "tableFrom": "session", + "tableTo": "user", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "user": { + "name": "user", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "email": { + "name": "email", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "email_verified": { + "name": "email_verified", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "image": { + "name": "image", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(cast(unixepoch('subsecond') * 1000 as integer))" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(cast(unixepoch('subsecond') * 1000 as integer))" + }, + "analytics_opted_out": { + "name": "analytics_opted_out", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + } + }, + "indexes": { + "user_email_unique": { + "name": "user_email_unique", + "columns": [ + "email" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "verification": { + "name": "verification", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "identifier": { + "name": "identifier", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "value": { + "name": "value", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "expires_at": { + "name": "expires_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(cast(unixepoch('subsecond') * 1000 as integer))" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(cast(unixepoch('subsecond') * 1000 as integer))" + } + }, + "indexes": { + "verification_identifier_idx": { + "name": "verification_identifier_idx", + "columns": [ + "identifier" + ], + "isUnique": false + }, + "verification_expiresAt_idx": { + "name": "verification_expiresAt_idx", + "columns": [ + "expires_at" + ], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "billing_customer_status": { + "name": "billing_customer_status", + "columns": { + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "is_paying": { + "name": "is_paying", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "paid_plan_id": { + "name": "paid_plan_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "paid_plan_status": { + "name": "paid_plan_status", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "customer_json": { + "name": "customer_json", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "synced_at": { + "name": "synced_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": {}, + "foreignKeys": { + "billing_customer_status_organization_id_organization_id_fk": { + "name": "billing_customer_status_organization_id_organization_id_fk", + "tableFrom": "billing_customer_status", + "tableTo": "organization", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "gsc_connections": { + "name": "gsc_connections", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "project_id": { + "name": "project_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "site_url": { + "name": "site_url", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "connected_by_user_id": { + "name": "connected_by_user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "connected_account_email": { + "name": "connected_account_email", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "gsc_connections_project_idx": { + "name": "gsc_connections_project_idx", + "columns": [ + "project_id" + ], + "isUnique": true + }, + "gsc_connections_organization_idx": { + "name": "gsc_connections_organization_idx", + "columns": [ + "organization_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "gsc_connections_project_id_projects_id_fk": { + "name": "gsc_connections_project_id_projects_id_fk", + "tableFrom": "gsc_connections", + "tableTo": "projects", + "columnsFrom": [ + "project_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "gsc_connections_organization_id_organization_id_fk": { + "name": "gsc_connections_organization_id_organization_id_fk", + "tableFrom": "gsc_connections", + "tableTo": "organization", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "reddit_attributions": { + "name": "reddit_attributions", + "columns": { + "id": { + "name": "id", + "type": "text", + "primaryKey": true, + "notNull": true, + "autoincrement": false + }, + "user_id": { + "name": "user_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "organization_id": { + "name": "organization_id", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "click_id": { + "name": "click_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "uuid": { + "name": "uuid", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "landing_page": { + "name": "landing_page", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "referrer": { + "name": "referrer", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "utm_source": { + "name": "utm_source", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "utm_medium": { + "name": "utm_medium", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "utm_campaign": { + "name": "utm_campaign", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "utm_term": { + "name": "utm_term", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "utm_content": { + "name": "utm_content", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "signup_sent_at": { + "name": "signup_sent_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "purchase_sent_at": { + "name": "purchase_sent_at", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + }, + "updated_at": { + "name": "updated_at", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(current_timestamp)" + } + }, + "indexes": { + "reddit_attributions_user_idx": { + "name": "reddit_attributions_user_idx", + "columns": [ + "user_id" + ], + "isUnique": true + }, + "reddit_attributions_organization_idx": { + "name": "reddit_attributions_organization_idx", + "columns": [ + "organization_id" + ], + "isUnique": false + } + }, + "foreignKeys": { + "reddit_attributions_user_id_user_id_fk": { + "name": "reddit_attributions_user_id_user_id_fk", + "tableFrom": "reddit_attributions", + "tableTo": "user", + "columnsFrom": [ + "user_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "reddit_attributions_organization_id_organization_id_fk": { + "name": "reddit_attributions_organization_id_organization_id_fk", + "tableFrom": "reddit_attributions", + "tableTo": "organization", + "columnsFrom": [ + "organization_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + } + }, + "views": {}, + "enums": {}, + "_meta": { + "schemas": {}, + "tables": {}, + "columns": {} + }, + "internal": { + "indexes": {} + } +} \ No newline at end of file diff --git a/drizzle/meta/_journal.json b/drizzle/meta/_journal.json index 8c858fd..dd32744 100644 --- a/drizzle/meta/_journal.json +++ b/drizzle/meta/_journal.json @@ -211,6 +211,13 @@ "when": 1783438453855, "tag": "0029_location_name", "breakpoints": true + }, + { + "idx": 30, + "version": "6", + "when": 1783306047190, + "tag": "0030_legal_reaper", + "breakpoints": true } ] } \ No newline at end of file diff --git a/src/db/app.schema.ts b/src/db/app.schema.ts index 382dd65..a292982 100644 --- a/src/db/app.schema.ts +++ b/src/db/app.schema.ts @@ -345,123 +345,3 @@ export const rankSnapshots = sqliteTable( ), ], ); - -// ============================================================================ -// Site Audit tables -// ============================================================================ - -// One row per audit run -export const audits = sqliteTable( - "audits", - { - id: text("id").primaryKey(), - projectId: text("project_id") - .notNull() - .references(() => projects.id, { onDelete: "cascade" }), - startedByUserId: text("started_by_user_id").notNull(), - startUrl: text("start_url").notNull(), - status: text("status", { - enum: ["running", "completed", "failed"], - }) - .notNull() - .default("running"), - workflowInstanceId: text("workflow_instance_id"), - // JSON config: { maxPages, lighthouseStrategy } - config: text("config").notNull().default("{}"), - // Progress & summary - pagesCrawled: integer("pages_crawled").notNull().default(0), - pagesTotal: integer("pages_total").notNull().default(0), - lighthouseTotal: integer("lighthouse_total").notNull().default(0), - lighthouseCompleted: integer("lighthouse_completed").notNull().default(0), - lighthouseFailed: integer("lighthouse_failed").notNull().default(0), - currentPhase: text("current_phase").default("discovery"), - startedAt: text("started_at") - .notNull() - .default(sql`(current_timestamp)`), - completedAt: text("completed_at"), - }, - (table) => [ - index("audits_project_id_idx").on(table.projectId), - index("audits_started_by_user_id_idx").on(table.startedByUserId), - ], -); - -// One row per crawled page -export const auditPages = sqliteTable( - "audit_pages", - { - id: text("id").primaryKey(), - auditId: text("audit_id") - .notNull() - .references(() => audits.id, { onDelete: "cascade" }), - url: text("url").notNull(), - statusCode: integer("status_code"), - redirectUrl: text("redirect_url"), - // Metadata - title: text("title"), - metaDescription: text("meta_description"), - canonicalUrl: text("canonical_url"), - robotsMeta: text("robots_meta"), - // Open Graph - ogTitle: text("og_title"), - ogDescription: text("og_description"), - ogImage: text("og_image"), - // Headings - h1Count: integer("h1_count").notNull().default(0), - h2Count: integer("h2_count").notNull().default(0), - h3Count: integer("h3_count").notNull().default(0), - h4Count: integer("h4_count").notNull().default(0), - h5Count: integer("h5_count").notNull().default(0), - h6Count: integer("h6_count").notNull().default(0), - headingOrderJson: text("heading_order_json"), - // Content - wordCount: integer("word_count").notNull().default(0), - // Images - imagesTotal: integer("images_total").notNull().default(0), - imagesMissingAlt: integer("images_missing_alt").notNull().default(0), - imagesJson: text("images_json"), - // Links - internalLinkCount: integer("internal_link_count").notNull().default(0), - externalLinkCount: integer("external_link_count").notNull().default(0), - // Structured data - hasStructuredData: integer("has_structured_data", { mode: "boolean" }) - .notNull() - .default(false), - // Hreflang - hreflangTagsJson: text("hreflang_tags_json"), - // Indexability - isIndexable: integer("is_indexable", { mode: "boolean" }) - .notNull() - .default(true), - // Performance - responseTimeMs: integer("response_time_ms"), - }, - (table) => [index("audit_pages_audit_id_idx").on(table.auditId)], -); - -// One row per Lighthouse test (mobile + desktop per page). -export const auditLighthouseResults = sqliteTable( - "audit_lighthouse_results", - { - id: text("id").primaryKey(), - auditId: text("audit_id") - .notNull() - .references(() => audits.id, { onDelete: "cascade" }), - pageId: text("page_id") - .notNull() - .references(() => auditPages.id, { onDelete: "cascade" }), - strategy: text("strategy", { enum: ["mobile", "desktop"] }).notNull(), - performanceScore: integer("performance_score"), - accessibilityScore: integer("accessibility_score"), - bestPracticesScore: integer("best_practices_score"), - seoScore: integer("seo_score"), - lcpMs: real("lcp_ms"), - cls: real("cls"), - inpMs: real("inp_ms"), - ttfbMs: real("ttfb_ms"), - errorMessage: text("error_message"), - r2Key: text("r2_key"), - payloadSizeBytes: integer("payload_size_bytes"), - }, - (table) => [index("audit_lighthouse_results_audit_id_idx").on(table.auditId)], -); diff --git a/src/db/audit.schema.ts b/src/db/audit.schema.ts new file mode 100644 index 0000000..55c3fa6 --- /dev/null +++ b/src/db/audit.schema.ts @@ -0,0 +1,201 @@ +import { + sqliteTable, + text, + integer, + real, + index, +} from "drizzle-orm/sqlite-core"; +import { sql } from "drizzle-orm"; +import { projects } from "./app.schema"; + +// ============================================================================ +// Site Audit tables +// ============================================================================ + +// One row per audit run +export const audits = sqliteTable( + "audits", + { + id: text("id").primaryKey(), + projectId: text("project_id") + .notNull() + .references(() => projects.id, { onDelete: "cascade" }), + startedByUserId: text("started_by_user_id").notNull(), + startUrl: text("start_url").notNull(), + status: text("status", { + enum: ["running", "completed", "failed"], + }) + .notNull() + .default("running"), + workflowInstanceId: text("workflow_instance_id"), + // JSON config: { maxPages, lighthouseStrategy } + config: text("config").notNull().default("{}"), + // Progress & summary + pagesCrawled: integer("pages_crawled").notNull().default(0), + pagesTotal: integer("pages_total").notNull().default(0), + lighthouseTotal: integer("lighthouse_total").notNull().default(0), + lighthouseCompleted: integer("lighthouse_completed").notNull().default(0), + lighthouseFailed: integer("lighthouse_failed").notNull().default(0), + currentPhase: text("current_phase").default("discovery"), + startedAt: text("started_at") + .notNull() + .default(sql`(current_timestamp)`), + completedAt: text("completed_at"), + }, + (table) => [ + index("audits_project_id_idx").on(table.projectId), + index("audits_started_by_user_id_idx").on(table.startedByUserId), + ], +); + +// One row per crawled page +export const auditPages = sqliteTable( + "audit_pages", + { + id: text("id").primaryKey(), + auditId: text("audit_id") + .notNull() + .references(() => audits.id, { onDelete: "cascade" }), + url: text("url").notNull(), + statusCode: integer("status_code"), + redirectUrl: text("redirect_url"), + // Metadata + title: text("title"), + metaDescription: text("meta_description"), + canonicalUrl: text("canonical_url"), + robotsMeta: text("robots_meta"), + // Open Graph + ogTitle: text("og_title"), + ogDescription: text("og_description"), + ogImage: text("og_image"), + // Headings + h1Count: integer("h1_count").notNull().default(0), + h2Count: integer("h2_count").notNull().default(0), + h3Count: integer("h3_count").notNull().default(0), + h4Count: integer("h4_count").notNull().default(0), + h5Count: integer("h5_count").notNull().default(0), + h6Count: integer("h6_count").notNull().default(0), + headingOrderJson: text("heading_order_json"), + // Content + wordCount: integer("word_count").notNull().default(0), + // Images + imagesTotal: integer("images_total").notNull().default(0), + imagesMissingAlt: integer("images_missing_alt").notNull().default(0), + imagesJson: text("images_json"), + // Links + internalLinkCount: integer("internal_link_count").notNull().default(0), + externalLinkCount: integer("external_link_count").notNull().default(0), + // Structured data + hasStructuredData: integer("has_structured_data", { mode: "boolean" }) + .notNull() + .default(false), + // Hreflang + hreflangTagsJson: text("hreflang_tags_json"), + // Indexability + isIndexable: integer("is_indexable", { mode: "boolean" }) + .notNull() + .default(true), + // Indexability/canonical signals from response headers + xRobotsTag: text("x_robots_tag"), + headerCanonicalUrl: text("header_canonical_url"), + // Crawl metadata + // null depth = not reached via links (e.g. sitemap-seeded) + crawlDepth: integer("crawl_depth"), + inSitemap: integer("in_sitemap", { mode: "boolean" }) + .notNull() + .default(false), + // SHA-256 of the visible body text, for duplicate-content grouping + contentHash: text("content_hash"), + // How the fetch resolved: ok | blocked (WAF/bot challenge) | error + fetchClass: text("fetch_class", { enum: ["ok", "blocked", "error"] }) + .notNull() + .default("ok"), + // Performance + responseTimeMs: integer("response_time_ms"), + }, + (table) => [ + index("audit_pages_audit_id_idx").on(table.auditId), + index("audit_pages_audit_url_idx").on(table.auditId, table.url), + ], +); + +// One row per unique (source page, target URL) link edge. Currently only +// internal edges are stored (see AuditRepository); isInternal stays so +// external-link checks can start writing rows without a migration. +export const auditLinks = sqliteTable( + "audit_links", + { + id: text("id").primaryKey(), + auditId: text("audit_id") + .notNull() + .references(() => audits.id, { onDelete: "cascade" }), + sourcePageId: text("source_page_id") + .notNull() + .references(() => auditPages.id, { onDelete: "cascade" }), + sourceUrl: text("source_url").notNull(), + targetUrl: text("target_url").notNull(), + anchor: text("anchor"), + isInternal: integer("is_internal", { mode: "boolean" }) + .notNull() + .default(true), + isNofollow: integer("is_nofollow", { mode: "boolean" }) + .notNull() + .default(false), + }, + (table) => [ + index("audit_links_audit_id_idx").on(table.auditId), + index("audit_links_audit_target_idx").on(table.auditId, table.targetUrl), + ], +); + +// One row per (issue type, affected page) +export const auditIssues = sqliteTable( + "audit_issues", + { + id: text("id").primaryKey(), + auditId: text("audit_id") + .notNull() + .references(() => audits.id, { onDelete: "cascade" }), + pageId: text("page_id").references(() => auditPages.id, { + onDelete: "cascade", + }), + pageUrl: text("page_url").notNull(), + issueType: text("issue_type").notNull(), + severity: text("severity", { enum: ["critical", "warning", "info"] }) + .notNull() + .default("info"), + // JSON details specific to the issue type (e.g. broken link target) + detailsJson: text("details_json"), + }, + (table) => [ + index("audit_issues_audit_id_idx").on(table.auditId), + index("audit_issues_audit_type_idx").on(table.auditId, table.issueType), + ], +); + +// One row per Lighthouse test (mobile + desktop per page). +export const auditLighthouseResults = sqliteTable( + "audit_lighthouse_results", + { + id: text("id").primaryKey(), + auditId: text("audit_id") + .notNull() + .references(() => audits.id, { onDelete: "cascade" }), + pageId: text("page_id") + .notNull() + .references(() => auditPages.id, { onDelete: "cascade" }), + strategy: text("strategy", { enum: ["mobile", "desktop"] }).notNull(), + performanceScore: integer("performance_score"), + accessibilityScore: integer("accessibility_score"), + bestPracticesScore: integer("best_practices_score"), + seoScore: integer("seo_score"), + lcpMs: real("lcp_ms"), + cls: real("cls"), + inpMs: real("inp_ms"), + ttfbMs: real("ttfb_ms"), + errorMessage: text("error_message"), + r2Key: text("r2_key"), + payloadSizeBytes: integer("payload_size_bytes"), + }, + (table) => [index("audit_lighthouse_results_audit_id_idx").on(table.auditId)], +); diff --git a/src/db/d1/schema.ts b/src/db/d1/schema.ts index 74d0080..63ef60b 100644 --- a/src/db/d1/schema.ts +++ b/src/db/d1/schema.ts @@ -2,6 +2,7 @@ // which is the provider-aware barrel) so the D1 client always binds to the // SQLite tables regardless of DATABASE_PROVIDER. export * from "../app.schema"; +export * from "../audit.schema"; export * from "../sam.schema"; export * from "../better-auth-schema"; export * from "../billing.schema"; diff --git a/src/db/pg/app.schema.ts b/src/db/pg/app.schema.ts index ff427d4..b75fe5b 100644 --- a/src/db/pg/app.schema.ts +++ b/src/db/pg/app.schema.ts @@ -338,112 +338,3 @@ export const rankSnapshots = pgTable( ), ], ); - -// ============================================================================ -// Site Audit tables -// ============================================================================ - -// One row per audit run -export const audits = pgTable( - "audits", - { - id: text("id").primaryKey(), - projectId: text("project_id") - .notNull() - .references(() => projects.id, { onDelete: "cascade" }), - startedByUserId: text("started_by_user_id").notNull(), - startUrl: text("start_url").notNull(), - status: text("status", { - enum: ["running", "completed", "failed"], - }) - .notNull() - .default("running"), - workflowInstanceId: text("workflow_instance_id"), - // JSON config: { maxPages, lighthouseStrategy } - config: text("config").notNull().default("{}"), - // Progress & summary - pagesCrawled: integer("pages_crawled").notNull().default(0), - pagesTotal: integer("pages_total").notNull().default(0), - lighthouseTotal: integer("lighthouse_total").notNull().default(0), - lighthouseCompleted: integer("lighthouse_completed").notNull().default(0), - lighthouseFailed: integer("lighthouse_failed").notNull().default(0), - currentPhase: text("current_phase").default("discovery"), - startedAt: timestampColumn("started_at").notNull().default(isoNow), - completedAt: timestampColumn("completed_at"), - }, - (table) => [ - index("audits_project_id_idx").on(table.projectId), - index("audits_started_by_user_id_idx").on(table.startedByUserId), - ], -); - -// One row per crawled page -export const auditPages = pgTable( - "audit_pages", - { - id: text("id").primaryKey(), - auditId: text("audit_id") - .notNull() - .references(() => audits.id, { onDelete: "cascade" }), - url: text("url").notNull(), - statusCode: integer("status_code"), - redirectUrl: text("redirect_url"), - // Metadata - title: text("title"), - metaDescription: text("meta_description"), - canonicalUrl: text("canonical_url"), - robotsMeta: text("robots_meta"), - // Open Graph - ogTitle: text("og_title"), - ogDescription: text("og_description"), - ogImage: text("og_image"), - // Headings - h1Count: integer("h1_count").notNull().default(0), - h2Count: integer("h2_count").notNull().default(0), - h3Count: integer("h3_count").notNull().default(0), - h4Count: integer("h4_count").notNull().default(0), - h5Count: integer("h5_count").notNull().default(0), - h6Count: integer("h6_count").notNull().default(0), - headingOrderJson: text("heading_order_json"), - // Content - wordCount: integer("word_count").notNull().default(0), - // Images - imagesTotal: integer("images_total").notNull().default(0), - imagesMissingAlt: integer("images_missing_alt").notNull().default(0), - imagesJson: text("images_json"), - internalLinkCount: integer("internal_link_count").notNull().default(0), - externalLinkCount: integer("external_link_count").notNull().default(0), - hasStructuredData: boolean("has_structured_data").notNull().default(false), - hreflangTagsJson: text("hreflang_tags_json"), - isIndexable: boolean("is_indexable").notNull().default(true), - responseTimeMs: integer("response_time_ms"), - }, - (table) => [index("audit_pages_audit_id_idx").on(table.auditId)], -); - -// One row per Lighthouse test (mobile + desktop per page). -export const auditLighthouseResults = pgTable( - "audit_lighthouse_results", - { - id: text("id").primaryKey(), - auditId: text("audit_id") - .notNull() - .references(() => audits.id, { onDelete: "cascade" }), - pageId: text("page_id") - .notNull() - .references(() => auditPages.id, { onDelete: "cascade" }), - strategy: text("strategy", { enum: ["mobile", "desktop"] }).notNull(), - performanceScore: integer("performance_score"), - accessibilityScore: integer("accessibility_score"), - bestPracticesScore: integer("best_practices_score"), - seoScore: integer("seo_score"), - lcpMs: real("lcp_ms"), - cls: real("cls"), - inpMs: real("inp_ms"), - ttfbMs: real("ttfb_ms"), - errorMessage: text("error_message"), - r2Key: text("r2_key"), - payloadSizeBytes: integer("payload_size_bytes"), - }, - (table) => [index("audit_lighthouse_results_audit_id_idx").on(table.auditId)], -); diff --git a/src/db/pg/audit.schema.ts b/src/db/pg/audit.schema.ts new file mode 100644 index 0000000..5f57113 --- /dev/null +++ b/src/db/pg/audit.schema.ts @@ -0,0 +1,196 @@ +import { sql } from "drizzle-orm"; +import { + boolean, + index, + integer, + pgTable, + real, + text, +} from "drizzle-orm/pg-core"; +import { projects } from "./app.schema"; + +// Timestamps are stored as *text* (same column shape as the SQLite schema); see +// the note in pg/app.schema.ts. `isoNow` matches `new Date().toISOString()` so +// DB-defaulted and app-written values sort together lexicographically. +const isoNow = sql`to_char(now() AT TIME ZONE 'utc', 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"')`; +const timestampColumn = (name: string) => text(name); + +// ============================================================================ +// Site Audit tables +// ============================================================================ + +// One row per audit run +export const audits = pgTable( + "audits", + { + id: text("id").primaryKey(), + projectId: text("project_id") + .notNull() + .references(() => projects.id, { onDelete: "cascade" }), + startedByUserId: text("started_by_user_id").notNull(), + startUrl: text("start_url").notNull(), + status: text("status", { + enum: ["running", "completed", "failed"], + }) + .notNull() + .default("running"), + workflowInstanceId: text("workflow_instance_id"), + // JSON config: { maxPages, lighthouseStrategy } + config: text("config").notNull().default("{}"), + // Progress & summary + pagesCrawled: integer("pages_crawled").notNull().default(0), + pagesTotal: integer("pages_total").notNull().default(0), + lighthouseTotal: integer("lighthouse_total").notNull().default(0), + lighthouseCompleted: integer("lighthouse_completed").notNull().default(0), + lighthouseFailed: integer("lighthouse_failed").notNull().default(0), + currentPhase: text("current_phase").default("discovery"), + startedAt: timestampColumn("started_at").notNull().default(isoNow), + completedAt: timestampColumn("completed_at"), + }, + (table) => [ + index("audits_project_id_idx").on(table.projectId), + index("audits_started_by_user_id_idx").on(table.startedByUserId), + ], +); + +// One row per crawled page +export const auditPages = pgTable( + "audit_pages", + { + id: text("id").primaryKey(), + auditId: text("audit_id") + .notNull() + .references(() => audits.id, { onDelete: "cascade" }), + url: text("url").notNull(), + statusCode: integer("status_code"), + redirectUrl: text("redirect_url"), + // Metadata + title: text("title"), + metaDescription: text("meta_description"), + canonicalUrl: text("canonical_url"), + robotsMeta: text("robots_meta"), + // Open Graph + ogTitle: text("og_title"), + ogDescription: text("og_description"), + ogImage: text("og_image"), + // Headings + h1Count: integer("h1_count").notNull().default(0), + h2Count: integer("h2_count").notNull().default(0), + h3Count: integer("h3_count").notNull().default(0), + h4Count: integer("h4_count").notNull().default(0), + h5Count: integer("h5_count").notNull().default(0), + h6Count: integer("h6_count").notNull().default(0), + headingOrderJson: text("heading_order_json"), + // Content + wordCount: integer("word_count").notNull().default(0), + // Images + imagesTotal: integer("images_total").notNull().default(0), + imagesMissingAlt: integer("images_missing_alt").notNull().default(0), + imagesJson: text("images_json"), + // Links + internalLinkCount: integer("internal_link_count").notNull().default(0), + externalLinkCount: integer("external_link_count").notNull().default(0), + // Structured data + hasStructuredData: boolean("has_structured_data").notNull().default(false), + // Hreflang + hreflangTagsJson: text("hreflang_tags_json"), + // Indexability + isIndexable: boolean("is_indexable").notNull().default(true), + // Indexability/canonical signals from response headers + xRobotsTag: text("x_robots_tag"), + headerCanonicalUrl: text("header_canonical_url"), + // Crawl metadata + // null depth = not reached via links (e.g. sitemap-seeded) + crawlDepth: integer("crawl_depth"), + inSitemap: boolean("in_sitemap").notNull().default(false), + // SHA-256 of the visible body text, for duplicate-content grouping + contentHash: text("content_hash"), + // How the fetch resolved: ok | blocked (WAF/bot challenge) | error + fetchClass: text("fetch_class", { enum: ["ok", "blocked", "error"] }) + .notNull() + .default("ok"), + // Performance + responseTimeMs: integer("response_time_ms"), + }, + (table) => [ + index("audit_pages_audit_id_idx").on(table.auditId), + index("audit_pages_audit_url_idx").on(table.auditId, table.url), + ], +); + +// One row per unique (source page, target URL) link edge. Currently only +// internal edges are stored (see AuditRepository); isInternal stays so +// external-link checks can start writing rows without a migration. +export const auditLinks = pgTable( + "audit_links", + { + id: text("id").primaryKey(), + auditId: text("audit_id") + .notNull() + .references(() => audits.id, { onDelete: "cascade" }), + sourcePageId: text("source_page_id") + .notNull() + .references(() => auditPages.id, { onDelete: "cascade" }), + sourceUrl: text("source_url").notNull(), + targetUrl: text("target_url").notNull(), + anchor: text("anchor"), + isInternal: boolean("is_internal").notNull().default(true), + isNofollow: boolean("is_nofollow").notNull().default(false), + }, + (table) => [ + index("audit_links_audit_id_idx").on(table.auditId), + index("audit_links_audit_target_idx").on(table.auditId, table.targetUrl), + ], +); + +// One row per (issue type, affected page) +export const auditIssues = pgTable( + "audit_issues", + { + id: text("id").primaryKey(), + auditId: text("audit_id") + .notNull() + .references(() => audits.id, { onDelete: "cascade" }), + pageId: text("page_id").references(() => auditPages.id, { + onDelete: "cascade", + }), + pageUrl: text("page_url").notNull(), + issueType: text("issue_type").notNull(), + severity: text("severity", { enum: ["critical", "warning", "info"] }) + .notNull() + .default("info"), + // JSON details specific to the issue type (e.g. broken link target) + detailsJson: text("details_json"), + }, + (table) => [ + index("audit_issues_audit_id_idx").on(table.auditId), + index("audit_issues_audit_type_idx").on(table.auditId, table.issueType), + ], +); + +// One row per Lighthouse test (mobile + desktop per page). +export const auditLighthouseResults = pgTable( + "audit_lighthouse_results", + { + id: text("id").primaryKey(), + auditId: text("audit_id") + .notNull() + .references(() => audits.id, { onDelete: "cascade" }), + pageId: text("page_id") + .notNull() + .references(() => auditPages.id, { onDelete: "cascade" }), + strategy: text("strategy", { enum: ["mobile", "desktop"] }).notNull(), + performanceScore: integer("performance_score"), + accessibilityScore: integer("accessibility_score"), + bestPracticesScore: integer("best_practices_score"), + seoScore: integer("seo_score"), + lcpMs: real("lcp_ms"), + cls: real("cls"), + inpMs: real("inp_ms"), + ttfbMs: real("ttfb_ms"), + errorMessage: text("error_message"), + r2Key: text("r2_key"), + payloadSizeBytes: integer("payload_size_bytes"), + }, + (table) => [index("audit_lighthouse_results_audit_id_idx").on(table.auditId)], +); diff --git a/src/db/pg/schema.ts b/src/db/pg/schema.ts index 9d03f35..db15b8c 100644 --- a/src/db/pg/schema.ts +++ b/src/db/pg/schema.ts @@ -1,4 +1,5 @@ export * from "./app.schema"; +export * from "./audit.schema"; export * from "./sam.schema"; export * from "./better-auth-schema"; export * from "./billing.schema"; diff --git a/src/db/schema.ts b/src/db/schema.ts index 75f40d3..2e21d2d 100644 --- a/src/db/schema.ts +++ b/src/db/schema.ts @@ -1,11 +1,13 @@ import { getDatabaseProvider } from "./provider"; import * as sqliteApp from "./app.schema"; +import * as sqliteAudit from "./audit.schema"; import * as sqliteSam from "./sam.schema"; import * as sqliteAuth from "./better-auth-schema"; import * as sqliteBilling from "./billing.schema"; import * as sqliteGsc from "./gsc.schema"; import * as sqliteReddit from "./reddit-attribution.schema"; import * as pgApp from "./pg/app.schema"; +import * as pgAudit from "./pg/audit.schema"; import * as pgSam from "./pg/sam.schema"; import * as pgAuth from "./pg/better-auth-schema"; import * as pgBilling from "./pg/billing.schema"; @@ -23,6 +25,7 @@ import * as pgReddit from "./pg/reddit-attribution.schema"; // schema is the one structural artifact NOT regenerated by `db:generate`, so the // parity test is its drift guard. type AppSchema = typeof sqliteApp & + typeof sqliteAudit & typeof sqliteSam & typeof sqliteAuth & typeof sqliteBilling & @@ -31,9 +34,18 @@ type AppSchema = typeof sqliteApp & const runtimeSchema = getDatabaseProvider() === "postgres" - ? { ...pgApp, ...pgSam, ...pgAuth, ...pgBilling, ...pgGsc, ...pgReddit } + ? { + ...pgApp, + ...pgAudit, + ...pgSam, + ...pgAuth, + ...pgBilling, + ...pgGsc, + ...pgReddit, + } : { ...sqliteApp, + ...sqliteAudit, ...sqliteSam, ...sqliteAuth, ...sqliteBilling, @@ -57,6 +69,8 @@ export const { rankSnapshots, audits, auditPages, + auditLinks, + auditIssues, auditLighthouseResults, samSessions, samProjectMemory, diff --git a/src/server/features/audit/repositories/AuditRepository.ts b/src/server/features/audit/repositories/AuditRepository.ts index a01a075..9fa0ad0 100644 --- a/src/server/features/audit/repositories/AuditRepository.ts +++ b/src/server/features/audit/repositories/AuditRepository.ts @@ -1,17 +1,34 @@ /** * Data access layer for site audit tables. - * Provider-aware (D1 or Postgres) via the `@/db` handle. + * Provider-aware (D1 or Postgres) via the `@/db` handle. Covers audits, + * audit_pages, audit_links, audit_issues, and stored Lighthouse results. */ import { and, desc, eq } from "drizzle-orm"; import { db } from "@/db"; -import { audits, auditLighthouseResults, auditPages } from "@/db/schema"; +import { + audits, + auditIssues, + auditLighthouseResults, + auditLinks, + auditPages, +} from "@/db/schema"; import { executeInBatches } from "@/db/runBatch"; +import { AUDIT_ISSUE_TYPES } from "@/shared/audit-issues"; +import { deterministicAuditRowId } from "@/server/lib/audit/ids"; +import type { DetectedIssue } from "@/server/lib/audit/issues/page-reporters"; import type { AuditConfig, + CrawledPageResult, LighthouseResult, - StepPageResult, } from "@/server/lib/audit/types"; +// Only internal links are stored: both consumers (broken-internal-link and +// orphan checks) filter on isInternal, and per-page external counts already +// live on audit_pages. External rows come back when P1 adds external-link +// checks. Mega-menu/footer-heavy sites can carry 1000+ links per page; cap +// what we store so a 10k-page crawl can't write tens of millions of link rows. +const MAX_STORED_LINKS_PER_PAGE = 500; + async function createAudit(data: { id: string; projectId: string; @@ -84,6 +101,9 @@ async function completeAudit( } async function failAudit(auditId: string, workflowInstanceId: string) { + // Only a running audit can transition to failed: the getStatus reconciler + // races the workflow's own finalize, and without this guard it could flip + // a just-completed audit to failed. await db .update(audits) .set({ @@ -95,6 +115,7 @@ async function failAudit(auditId: string, workflowInstanceId: string) { and( eq(audits.id, auditId), eq(audits.workflowInstanceId, workflowInstanceId), + eq(audits.status, "running"), ), ); } @@ -111,25 +132,24 @@ async function getAuditForWorkflow( }); } -async function batchWriteResults( +/** + * Persist one crawl batch (pages + link edges + per-page issues). + * Called inside the crawl-batch Workflow step so results land in D1 + * incrementally instead of accumulating in memory until finalize. + * + * Idempotent on step retry: callers assign deterministic page ids + * (deterministicAuditRowId) and link/issue ids are derived from stable + * content. Page rows upsert (a retried fetch may legitimately differ — last + * attempt wins, matching what the step returns); links and issues are + * insert-or-ignore. + */ +async function insertCrawledBatch( auditId: string, - pages: StepPageResult[], - lighthouseResults: LighthouseResult[], + pages: CrawledPageResult[], + issues: DetectedIssue[], ) { - // The `finalize` workflow step can retry after a partial write (multi-chunk - // inserts aren't atomic, and steps after the insert can throw). Clear any - // rows from a prior attempt first so the re-run is idempotent — otherwise - // stable page ids collide on the PK and lighthouse rows silently duplicate. - // audit_lighthouse_results.page_id FKs audit_pages, so delete it first. - await db - .delete(auditLighthouseResults) - .where(eq(auditLighthouseResults.auditId, auditId)); - await db.delete(auditPages).where(eq(auditPages.auditId, auditId)); - - await executeInBatches(pages, (tx, page) => - tx.insert(auditPages).values({ - id: page.id, - auditId, + await executeInBatches(pages, (tx, page) => { + const dataColumns = { url: page.url, statusCode: page.statusCode, redirectUrl: page.redirectUrl, @@ -137,6 +157,8 @@ async function batchWriteResults( metaDescription: page.metaDescription, canonicalUrl: page.canonicalUrl, robotsMeta: page.robotsMeta, + xRobotsTag: page.xRobotsTag, + headerCanonicalUrl: page.headerCanonicalUrl, ogTitle: page.ogTitle, ogDescription: page.ogDescription, ogImage: page.ogImage, @@ -148,25 +170,83 @@ async function batchWriteResults( h6Count: page.h6Count, headingOrderJson: JSON.stringify(page.headingOrder), wordCount: page.wordCount, + contentHash: page.contentHash, imagesTotal: page.imagesTotal, imagesMissingAlt: page.imagesMissingAlt, imagesJson: JSON.stringify(page.images), - internalLinkCount: page.internalLinks.length, - externalLinkCount: page.externalLinks.length, + internalLinkCount: page.links.filter((l) => l.isInternal).length, + externalLinkCount: page.links.filter((l) => !l.isInternal).length, hasStructuredData: page.hasStructuredData, hreflangTagsJson: JSON.stringify(page.hreflangTags), isIndexable: page.isIndexable, + fetchClass: page.fetchClass, + crawlDepth: page.crawlDepth, + inSitemap: page.inSitemap, responseTimeMs: page.responseTimeMs, - }), + }; + return tx + .insert(auditPages) + .values({ id: page.id, auditId, ...dataColumns }) + .onConflictDoUpdate({ target: auditPages.id, set: dataColumns }); + }); + + const linkRows = await Promise.all( + pages.flatMap((page) => + page.links + .filter((link) => link.isInternal) + .slice(0, MAX_STORED_LINKS_PER_PAGE) + .map(async (link) => ({ + id: await deterministicAuditRowId(auditId, page.url, link.targetUrl), + auditId, + sourcePageId: page.id, + sourceUrl: page.url, + targetUrl: link.targetUrl, + anchor: link.anchor, + isInternal: link.isInternal, + isNofollow: link.isNofollow, + })), + ), + ); + await executeInBatches(linkRows, (tx, row) => + tx.insert(auditLinks).values(row).onConflictDoNothing(), ); - if (lighthouseResults.length === 0) { - return; - } + await insertIssues(auditId, issues); +} - await executeInBatches(lighthouseResults, (tx, result) => - tx.insert(auditLighthouseResults).values({ - id: crypto.randomUUID(), +async function insertIssues(auditId: string, issues: DetectedIssue[]) { + const issueRows = await Promise.all( + issues.map(async (issue) => ({ + id: await deterministicAuditRowId( + auditId, + issue.pageUrl, + issue.issueType, + issue.dedupeKey ?? "", + ), + auditId, + pageId: issue.pageId, + pageUrl: issue.pageUrl, + issueType: issue.issueType, + severity: AUDIT_ISSUE_TYPES[issue.issueType].severity, + detailsJson: issue.details ? JSON.stringify(issue.details) : null, + })), + ); + await executeInBatches(issueRows, (tx, row) => + tx.insert(auditIssues).values(row).onConflictDoNothing(), + ); +} + +async function insertLighthouseResults( + auditId: string, + lighthouseResults: LighthouseResult[], +) { + const rows = await Promise.all( + lighthouseResults.map(async (result) => ({ + id: await deterministicAuditRowId( + auditId, + result.pageId, + result.strategy, + ), auditId, pageId: result.pageId, strategy: result.strategy, @@ -181,8 +261,17 @@ async function batchWriteResults( errorMessage: result.errorMessage ?? null, r2Key: result.r2Key ?? null, payloadSizeBytes: result.payloadSizeBytes ?? null, - }), + })), ); + // Upsert: a step retry can charge a second DataForSEO call whose result + // must not be silently dropped in favor of a failed first attempt. + await executeInBatches(rows, (tx, row) => { + const { id: _id, auditId: _auditId, ...dataColumns } = row; + return tx.insert(auditLighthouseResults).values(row).onConflictDoUpdate({ + target: auditLighthouseResults.id, + set: dataColumns, + }); + }); } async function getAuditForProject(auditId: string, projectId: string) { @@ -191,6 +280,58 @@ async function getAuditForProject(auditId: string, projectId: string) { }); } +async function getLatestAuditForProject(projectId: string) { + return db.query.audits.findFirst({ + where: eq(audits.projectId, projectId), + orderBy: desc(audits.startedAt), + }); +} + +async function getIssuesForAudit( + auditId: string, + filters: { severity?: "critical" | "warning" | "info"; issueType?: string }, +) { + return db.query.auditIssues.findMany({ + where: and( + eq(auditIssues.auditId, auditId), + filters.severity ? eq(auditIssues.severity, filters.severity) : undefined, + filters.issueType + ? eq(auditIssues.issueType, filters.issueType) + : undefined, + ), + }); +} + +async function getPagesForAudit(auditId: string) { + return db + .select({ + id: auditPages.id, + url: auditPages.url, + statusCode: auditPages.statusCode, + fetchClass: auditPages.fetchClass, + redirectUrl: auditPages.redirectUrl, + title: auditPages.title, + metaDescription: auditPages.metaDescription, + wordCount: auditPages.wordCount, + isIndexable: auditPages.isIndexable, + crawlDepth: auditPages.crawlDepth, + inSitemap: auditPages.inSitemap, + internalLinkCount: auditPages.internalLinkCount, + responseTimeMs: auditPages.responseTimeMs, + }) + .from(auditPages) + .where(eq(auditPages.auditId, auditId)); +} + +async function hasPagesForAudit(auditId: string): Promise { + const rows = await db + .select({ id: auditPages.id }) + .from(auditPages) + .where(eq(auditPages.auditId, auditId)) + .limit(1); + return rows.length > 0; +} + async function getAuditsByProject(projectId: string) { const rows = await db .select({ audit: audits }) @@ -223,19 +364,22 @@ async function getAuditUsageForUser(userId: string) { async function getAuditResultsForProject(auditId: string, projectId: string) { const audit = await getAuditForProject(auditId, projectId); if (!audit) { - return { audit: null, pages: [], lighthouse: [] }; + return { audit: null, pages: [], lighthouse: [], issues: [] }; } - const [pages, lighthouse] = await Promise.all([ + const [pages, lighthouse, issues] = await Promise.all([ db.query.auditPages.findMany({ where: eq(auditPages.auditId, auditId), }), db.query.auditLighthouseResults.findMany({ where: eq(auditLighthouseResults.auditId, auditId), }), + db.query.auditIssues.findMany({ + where: eq(auditIssues.auditId, auditId), + }), ]); - return { audit, pages, lighthouse }; + return { audit, pages, lighthouse, issues }; } async function getLighthouseResultById(input: { @@ -285,8 +429,14 @@ export const AuditRepository = { completeAudit, failAudit, getAuditForWorkflow, - batchWriteResults, + insertCrawledBatch, + insertIssues, + insertLighthouseResults, getAuditForProject, + getLatestAuditForProject, + getIssuesForAudit, + getPagesForAudit, + hasPagesForAudit, getAuditsByProject, getAuditUsageForUser, getAuditResultsForProject, diff --git a/src/server/features/audit/services/AuditService.ts b/src/server/features/audit/services/AuditService.ts index d0ff340..f36146c 100644 --- a/src/server/features/audit/services/AuditService.ts +++ b/src/server/features/audit/services/AuditService.ts @@ -1,5 +1,9 @@ import { env } from "cloudflare:workers"; -import type { BillingCustomerContext } from "@/server/billing/subscription"; +import { + customerHasManagedAccess, + customerHasPaidPlan, + type BillingCustomerContext, +} from "@/server/billing/subscription"; import { AuditRepository } from "@/server/features/audit/repositories/AuditRepository"; import { AUDIT_LIMITS, @@ -15,6 +19,24 @@ import { type LighthouseStrategy, } from "@/server/lib/audit/types"; import { normalizeAndValidateStartUrl } from "@/server/lib/audit/url-policy"; +import { isHostedServerAuthMode } from "@/server/lib/runtime-env"; + +// Plan-tier limits are the abuse bound in hosted mode: free accounts get one +// small audit at a time, paid keeps the full limits, and customers with no +// Autumn product at all are turned away. Self-hosted isn't gated. +async function resolveAuditLimitTier( + organizationId: string, +): Promise { + if (!(await isHostedServerAuthMode())) return "paid"; + const [hasManagedAccess, hasPaidPlan] = await Promise.all([ + customerHasManagedAccess(organizationId), + customerHasPaidPlan(organizationId), + ]); + if (!hasManagedAccess) { + throw new AppError("PAYMENT_REQUIRED", "Subscribe to run site audits"); + } + return hasPaidPlan ? "paid" : "free"; +} async function startAudit(input: { actorUserId: string; @@ -98,8 +120,29 @@ async function startAudit(input: { } async function getStatus(auditId: string, projectId: string) { - const audit = await AuditRepository.getAuditForProject(auditId, projectId); - if (!audit) throw new AppError("NOT_FOUND"); + let audit = await AuditRepository.getAuditForProject(auditId, projectId); + if (!audit) + throw new AppError("NOT_FOUND", "Audit not found in this project."); + + // Self-heal audits whose workflow died without reaching the mark-failed + // step (instance terminated, mark-failed itself failed, deploys, ...). + // Without this they stay "running" forever and hold capacity. + if (audit.status === "running" && audit.workflowInstanceId) { + try { + const instance = await env.SITE_AUDIT_WORKFLOW.get( + audit.workflowInstanceId, + ); + const { status } = await instance.status(); + if (status === "errored" || status === "terminated") { + await AuditRepository.failAudit(audit.id, audit.workflowInstanceId); + audit = + (await AuditRepository.getAuditForProject(auditId, projectId)) ?? + audit; + } + } catch { + // Instance not found or status unavailable — leave the audit as-is. + } + } return { id: audit.id, @@ -117,7 +160,7 @@ async function getStatus(auditId: string, projectId: string) { } async function getResults(auditId: string, projectId: string) { - const { audit, pages, lighthouse } = + const { audit, pages, lighthouse, issues } = await AuditRepository.getAuditResultsForProject(auditId, projectId); if (!audit) throw new AppError("NOT_FOUND"); @@ -140,6 +183,7 @@ async function getResults(auditId: string, projectId: string) { }, pages, lighthouse, + issues, }; } @@ -216,6 +260,7 @@ async function remove(auditId: string, projectId: string) { } export const AuditService = { + resolveAuditLimitTier, startAudit, getStatus, getCrawlProgress, diff --git a/src/server/features/audit/services/audit-capacity.test.ts b/src/server/features/audit/services/audit-capacity.test.ts index adc6432..238991c 100644 --- a/src/server/features/audit/services/audit-capacity.test.ts +++ b/src/server/features/audit/services/audit-capacity.test.ts @@ -21,16 +21,6 @@ describe("audit capacity helpers", () => { lighthouseTotal: 0, total: 100, }); - expect( - getEstimatedAuditCapacity({ - maxPages: 100, - lighthouseStrategy: "manual", - }), - ).toEqual({ - pagesTotal: 100, - lighthouseTotal: 0, - total: 100, - }); expect( getEstimatedAuditCapacity({ maxPages: 100, lighthouseStrategy: "auto" }), ).toEqual({ diff --git a/src/server/lib/audit/discovery.ts b/src/server/lib/audit/discovery.ts index f37b0a4..68bd387 100644 --- a/src/server/lib/audit/discovery.ts +++ b/src/server/lib/audit/discovery.ts @@ -22,41 +22,41 @@ export interface RobotsResult { } /** - * Fetch and parse robots.txt for a given origin. - * Returns a helper to check if URLs are allowed + discovered sitemap URLs. + * Fetch the raw robots.txt body (null = missing/unreachable). Kept separate + * from parsing so Workflows can checkpoint the text as durable step state and + * re-derive the parsed result deterministically on replay. */ -export async function fetchRobotsTxt(origin: string): Promise { - const robotsUrl = `${origin}/robots.txt`; +async function fetchRobotsTxtText(origin: string): Promise { try { - const response = await fetch(robotsUrl, { + const response = await fetch(`${origin}/robots.txt`, { headers: { "User-Agent": "OpenSEO-Audit/1.0" }, signal: AbortSignal.timeout(10_000), }); - if (!response.ok) { - // No robots.txt = everything allowed - return { - isAllowed: () => true, - sitemapUrls: [], - }; - } - - const text = await response.text(); - const robots = robotsParser(robotsUrl, text); - - return { - isAllowed: (url: string) => robots.isAllowed(url) ?? true, - sitemapUrls: robots.getSitemaps(), - }; + if (!response.ok) return null; + return await response.text(); } catch (error) { console.warn("Failed to fetch robots.txt:", error); - return { - isAllowed: () => true, - sitemapUrls: [], - }; + return null; } } +/** Deterministic: same text in, same result out. Null = everything allowed. */ +export function parseRobotsTxt( + origin: string, + text: string | null, +): RobotsResult { + if (text === null) { + return { isAllowed: () => true, sitemapUrls: [] }; + } + + const robots = robotsParser(`${origin}/robots.txt`, text); + return { + isAllowed: (url: string) => robots.isAllowed(url) ?? true, + sitemapUrls: robots.getSitemaps(), + }; +} + /** * Fetch and parse a sitemap (supports sitemap index recursion). * Returns a flat list of page URLs found. @@ -184,8 +184,9 @@ async function fetchSitemapDocumentWithRetry(sitemapUrl: string): Promise<{ export async function discoverUrls( origin: string, maxPages = 50, -): Promise<{ urls: string[]; robots: RobotsResult; sitemapUrls: Set }> { - const robots = await fetchRobotsTxt(origin); +): Promise<{ urls: string[]; robotsText: string | null }> { + const robotsText = await fetchRobotsTxtText(origin); + const robots = parseRobotsTxt(origin, robotsText); // Collect sitemap URLs: from robots.txt + default location const sitemapSources = new Set(robots.sitemapUrls); @@ -262,9 +263,10 @@ export async function discoverUrls( ); } + // Cap at the crawl's page budget: these are seeds, the crawl can never use + // more — and an uncapped list can blow the ~1MiB Workflow step-state limit. return { - urls: Array.from(allUrls), - robots, - sitemapUrls: allUrls, + urls: Array.from(allUrls).slice(0, maxPages), + robotsText, }; } diff --git a/src/server/lib/audit/ids.ts b/src/server/lib/audit/ids.ts new file mode 100644 index 0000000..6dc30f2 --- /dev/null +++ b/src/server/lib/audit/ids.ts @@ -0,0 +1,24 @@ +/** + * Deterministic row ids for audit data. + * + * Crawl/lighthouse persistence happens inside Workflow steps, which retry on + * failure. Deriving ids from stable content (audit id + URL + ...) combined + * with `onConflictDoNothing` makes those writes idempotent across retries — + * a partially-written batch is simply completed on the next attempt instead + * of duplicated under fresh random ids. + */ +export async function sha256Hex(text: string): Promise { + const digest = await crypto.subtle.digest( + "SHA-256", + new TextEncoder().encode(text), + ); + return Array.from(new Uint8Array(digest)) + .map((byte) => byte.toString(16).padStart(2, "0")) + .join(""); +} + +export async function deterministicAuditRowId( + ...parts: string[] +): Promise { + return (await sha256Hex(parts.join("|"))).slice(0, 36); +} diff --git a/src/server/lib/audit/issues/multipage-checks.ts b/src/server/lib/audit/issues/multipage-checks.ts new file mode 100644 index 0000000..5858db5 --- /dev/null +++ b/src/server/lib/audit/issues/multipage-checks.ts @@ -0,0 +1,180 @@ +/** + * Pure cross-page checks (no database access): duplicate grouping and + * redirect chain/loop detection. The D1-backed checks (broken links, + * orphans) live in multipage.ts. + */ +import type { DetectedIssue } from "@/server/lib/audit/issues/page-reporters"; + +const DUPLICATE_GROUP_SAMPLE = 3; + +export interface SlimPage { + id: string; + url: string; + statusCode: number | null; + fetchClass: "ok" | "blocked" | "error"; + title: string | null; + metaDescription: string | null; + contentHash: string | null; + redirectUrl: string | null; + wordCount: number; + isIndexable: boolean; + canonicalUrl: string | null; + headerCanonicalUrl: string | null; +} + +function isOkHtmlPage(page: SlimPage): boolean { + return ( + page.fetchClass === "ok" && + page.statusCode !== null && + page.statusCode >= 200 && + page.statusCode < 300 + ); +} + +/** + * Pages the owner already de-duplicated (noindex, or canonicalized to + * another URL) don't belong in duplicate groups — flagging them tells the + * user to fix something they already fixed. + */ +function isDuplicateCandidate(page: SlimPage): boolean { + if (!isOkHtmlPage(page) || !page.isIndexable) return false; + const effectiveCanonical = page.canonicalUrl ?? page.headerCanonicalUrl; + return !effectiveCanonical || effectiveCanonical === page.url; +} + +export function findDuplicates(pages: SlimPage[]): DetectedIssue[] { + const okPages = pages.filter(isDuplicateCandidate); + + const groupBy = ( + keyOf: (page: SlimPage) => string | null, + ): Map => { + const groups = new Map(); + for (const page of okPages) { + const key = keyOf(page); + if (!key) continue; + const group = groups.get(key); + if (group) group.push(page); + else groups.set(key, [page]); + } + return groups; + }; + + const issues: DetectedIssue[] = []; + const emitGroups = ( + groups: Map, + issueType: DetectedIssue["issueType"], + ) => { + for (const group of groups.values()) { + if (group.length < 2) continue; + for (const page of group) { + issues.push({ + issueType, + pageId: page.id, + pageUrl: page.url, + details: { + groupSize: group.length, + otherUrls: group + .filter((other) => other.id !== page.id) + .slice(0, DUPLICATE_GROUP_SAMPLE) + .map((other) => other.url), + }, + }); + } + } + }; + + emitGroups( + groupBy((page) => page.title || null), + "duplicate-title", + ); + emitGroups( + groupBy((page) => page.metaDescription || null), + "duplicate-meta-description", + ); + emitGroups( + groupBy((page) => (page.wordCount > 0 ? page.contentHash : null)), + "duplicate-content", + ); + return issues; +} + +export function findRedirectChainsAndLoops(pages: SlimPage[]): DetectedIssue[] { + const redirects = new Map(); + for (const page of pages) { + const isRedirect = + page.statusCode !== null && + page.statusCode >= 300 && + page.statusCode < 400 && + page.redirectUrl; + if (isRedirect) redirects.set(page.url, page); + } + + const redirectTargets = new Set( + Array.from(redirects.values(), (page) => page.redirectUrl!), + ); + + const issues: DetectedIssue[] = []; + const walked = new Set(); + + // Walk from chain heads (redirects nothing else redirects to), so a 5-hop + // chain yields one issue, not five. + for (const [url, head] of redirects) { + if (redirectTargets.has(url)) continue; + + const hops: string[] = [url]; + const seen = new Set(hops); + walked.add(url); + let current = head.redirectUrl; + let isLoop = false; + while (current) { + if (seen.has(current)) { + isLoop = true; + hops.push(current); + break; + } + hops.push(current); + seen.add(current); + if (redirects.has(current)) walked.add(current); + current = redirects.get(current)?.redirectUrl ?? null; + } + + if (isLoop) { + issues.push({ + issueType: "redirect-loop", + pageId: head.id, + pageUrl: url, + details: { hops }, + }); + } else if (hops.length > 2) { + // url -> a -> b: two redirects before content = a chain + issues.push({ + issueType: "redirect-chain", + pageId: head.id, + pageUrl: url, + details: { hops, finalUrl: hops[hops.length - 1] }, + }); + } + } + + // Headless cycles (every member is also a target — e.g. a↔b, or a→a) are + // never reached from a head; emit one loop issue per cycle. + for (const [url, page] of redirects) { + if (walked.has(url)) continue; + + const cycle: string[] = []; + let current: string | null = url; + while (current && !walked.has(current)) { + walked.add(current); + cycle.push(current); + current = redirects.get(current)?.redirectUrl ?? null; + } + issues.push({ + issueType: "redirect-loop", + pageId: page.id, + pageUrl: url, + details: { hops: [...cycle, url] }, + }); + } + + return issues; +} diff --git a/src/server/lib/audit/issues/multipage.ts b/src/server/lib/audit/issues/multipage.ts new file mode 100644 index 0000000..b089e50 --- /dev/null +++ b/src/server/lib/audit/issues/multipage.ts @@ -0,0 +1,154 @@ +/** + * Cross-page (multipage) issue checks. + * + * These run once after the crawl, against the rows the crawl wrote to D1: + * duplicates, broken internal links, redirect chains/loops, orphan pages. + * Pure set-queries over crawl data — no fetching, no DOM. + */ +import { and, eq, gte, lt, ne, notExists, sql } from "drizzle-orm"; +import { alias } from "drizzle-orm/sqlite-core"; +import { db } from "@/db"; +import { auditLinks, auditPages } from "@/db/schema"; +import { normalizeUrl } from "@/server/lib/audit/url-utils"; +import { + findDuplicates, + findRedirectChainsAndLoops, + type SlimPage, +} from "@/server/lib/audit/issues/multipage-checks"; +import type { DetectedIssue } from "@/server/lib/audit/issues/page-reporters"; + +const BROKEN_LINK_ISSUE_CAP = 2_000; + +export async function runMultipageChecks(input: { + auditId: string; + startUrl: string; + /** Orphan detection only makes sense when the crawl wasn't truncated. */ + crawlCompleted: boolean; +}): Promise { + const pages: SlimPage[] = await db + .select({ + id: auditPages.id, + url: auditPages.url, + statusCode: auditPages.statusCode, + fetchClass: auditPages.fetchClass, + title: auditPages.title, + metaDescription: auditPages.metaDescription, + contentHash: auditPages.contentHash, + redirectUrl: auditPages.redirectUrl, + wordCount: auditPages.wordCount, + isIndexable: auditPages.isIndexable, + canonicalUrl: auditPages.canonicalUrl, + headerCanonicalUrl: auditPages.headerCanonicalUrl, + }) + .from(auditPages) + .where(eq(auditPages.auditId, input.auditId)); + + const issues: DetectedIssue[] = [ + ...findDuplicates(pages), + ...findRedirectChainsAndLoops(pages), + ...(await findBrokenInternalLinks(input.auditId)), + ]; + + if (input.crawlCompleted) { + // Page rows store normalized URLs; normalize the start URL the same way + // so the orphan exclusion matches. + const normalizedStart = normalizeUrl(input.startUrl) ?? input.startUrl; + issues.push(...(await findOrphanPages(input.auditId, normalizedStart))); + } + + return issues; +} + +async function findBrokenInternalLinks( + auditId: string, +): Promise { + // Only flag targets we actually crawled and saw fail — never inferred from + // absence. Blocked targets (WAF challenges) are excluded: a 403 from bot + // protection is not evidence of a broken link. + const rows = await db + .select({ + sourcePageId: auditLinks.sourcePageId, + sourceUrl: auditLinks.sourceUrl, + targetUrl: auditLinks.targetUrl, + targetStatus: auditPages.statusCode, + }) + .from(auditLinks) + .innerJoin( + auditPages, + and( + eq(auditPages.auditId, auditLinks.auditId), + eq(auditPages.url, auditLinks.targetUrl), + ), + ) + .where( + and( + eq(auditLinks.auditId, auditId), + eq(auditLinks.isInternal, true), + gte(auditPages.statusCode, 400), + eq(auditPages.fetchClass, "ok"), + ), + ) + .limit(BROKEN_LINK_ISSUE_CAP); + + return rows.map((row) => ({ + issueType: "broken-internal-link" as const, + pageId: row.sourcePageId, + pageUrl: row.sourceUrl, + dedupeKey: row.targetUrl, + details: { targetUrl: row.targetUrl, targetStatus: row.targetStatus }, + })); +} + +async function findOrphanPages( + auditId: string, + startUrl: string, +): Promise { + // A live 2xx page is an orphan when no OTHER crawled page links to it and + // nothing redirects to it. Only meaningful on a completed crawl: on a + // truncated one, "no observed inlinks" is true of nearly everything. + // Error/redirect rows aren't orphans — they already get their own issues. + const inlink = db + .select({ one: sql`1` }) + .from(auditLinks) + .where( + and( + eq(auditLinks.auditId, auditId), + eq(auditLinks.isInternal, true), + eq(auditLinks.targetUrl, auditPages.url), + // Self-links (breadcrumbs, anchors) don't make a page reachable. + ne(auditLinks.sourcePageId, auditPages.id), + ), + ); + + const redirectSourcePages = alias(auditPages, "redirect_sources"); + const redirectSources = db + .select({ one: sql`1` }) + .from(redirectSourcePages) + .where( + and( + eq(redirectSourcePages.auditId, auditId), + eq(redirectSourcePages.redirectUrl, auditPages.url), + ), + ); + + const rows = await db + .select({ id: auditPages.id, url: auditPages.url }) + .from(auditPages) + .where( + and( + eq(auditPages.auditId, auditId), + ne(auditPages.url, startUrl), + eq(auditPages.fetchClass, "ok"), + gte(auditPages.statusCode, 200), + lt(auditPages.statusCode, 300), + notExists(inlink), + notExists(redirectSources), + ), + ); + + return rows.map((row) => ({ + issueType: "orphan-page" as const, + pageId: row.id, + pageUrl: row.url, + })); +} diff --git a/src/server/lib/audit/issues/page-reporters.test.ts b/src/server/lib/audit/issues/page-reporters.test.ts new file mode 100644 index 0000000..576328f --- /dev/null +++ b/src/server/lib/audit/issues/page-reporters.test.ts @@ -0,0 +1,308 @@ +import { describe, expect, it } from "vitest"; +import { runPageReporters } from "@/server/lib/audit/issues/page-reporters"; +import { + findDuplicates, + findRedirectChainsAndLoops, + type SlimPage, +} from "@/server/lib/audit/issues/multipage-checks"; +import type { CrawledPageResult } from "@/server/lib/audit/types"; + +function makePage(overrides: Partial): CrawledPageResult { + return { + id: "page-1", + url: "https://example.com/a", + statusCode: 200, + fetchClass: "ok", + redirectUrl: null, + title: "A perfectly reasonable page title", + metaDescription: + "A reasonable meta description that says something useful about the page.", + canonicalUrl: null, + robotsMeta: null, + xRobotsTag: null, + headerCanonicalUrl: null, + ogTitle: null, + ogDescription: null, + ogImage: null, + h1Count: 1, + h2Count: 0, + h3Count: 0, + h4Count: 0, + h5Count: 0, + h6Count: 0, + headingOrder: [1, 2, 3], + wordCount: 500, + contentHash: "abc123", + isHtml: true, + imagesTotal: 0, + imagesMissingAlt: 0, + images: [], + links: [], + hasStructuredData: false, + hreflangTags: [], + isIndexable: true, + responseTimeMs: 200, + crawlDepth: 1, + inSitemap: true, + ...overrides, + }; +} + +function issueTypes(page: CrawledPageResult): string[] { + return runPageReporters(page).map((issue) => issue.issueType); +} + +describe("runPageReporters", () => { + it("reports nothing for a healthy page", () => { + expect(issueTypes(makePage({}))).toEqual([]); + }); + + it("reports only blocked-page for a blocked fetch", () => { + expect( + issueTypes(makePage({ fetchClass: "blocked", statusCode: 403 })), + ).toEqual(["blocked-page"]); + }); + + it("reports nothing for a fetch error", () => { + expect( + issueTypes(makePage({ fetchClass: "error", statusCode: 0 })), + ).toEqual([]); + }); + + it("classifies error statuses by range", () => { + expect(issueTypes(makePage({ statusCode: 500 }))).toEqual(["server-error"]); + expect(issueTypes(makePage({ statusCode: 404 }))).toEqual(["broken-page"]); + expect( + issueTypes( + makePage({ + statusCode: 301, + redirectUrl: "https://example.com/b", + }), + ), + ).toEqual([]); + }); + + it("checks titles and meta descriptions", () => { + expect(issueTypes(makePage({ title: "" }))).toContain("missing-title"); + expect(issueTypes(makePage({ title: "x".repeat(70) }))).toContain( + "title-too-long", + ); + expect(issueTypes(makePage({ title: "Tiny" }))).toContain( + "title-too-short", + ); + expect(issueTypes(makePage({ metaDescription: "" }))).toContain( + "missing-meta-description", + ); + expect( + issueTypes(makePage({ metaDescription: "x".repeat(200) })), + ).toContain("meta-description-too-long"); + }); + + it("checks headings", () => { + expect(issueTypes(makePage({ h1Count: 0 }))).toContain("missing-h1"); + expect(issueTypes(makePage({ h1Count: 3 }))).toContain("multiple-h1"); + expect(issueTypes(makePage({ headingOrder: [1, 2, 4] }))).toContain( + "heading-order-skip", + ); + }); + + it("skips content checks for non-HTML responses", () => { + const nonHtml = makePage({ + isHtml: false, + title: "", + metaDescription: "", + h1Count: 0, + headingOrder: [], + wordCount: 0, + contentHash: null, + }); + expect(issueTypes(nonHtml)).toEqual([]); + }); + + it("still checks empty-shell HTML pages", () => { + const shell = makePage({ + isHtml: true, + title: "", + metaDescription: "", + h1Count: 0, + headingOrder: [], + wordCount: 0, + contentHash: null, + }); + const types = issueTypes(shell); + expect(types).toContain("missing-title"); + expect(types).toContain("missing-h1"); + expect(types).toContain("thin-content"); + }); + + it("flags indexability and canonical signals", () => { + expect( + issueTypes(makePage({ isIndexable: false, robotsMeta: "noindex" })), + ).toContain("noindex-page"); + + const conflicted = issueTypes( + makePage({ + canonicalUrl: "https://example.com/canonical-a", + headerCanonicalUrl: "https://example.com/canonical-b", + }), + ); + expect(conflicted).toContain("canonical-conflict"); + expect(conflicted).toContain("canonicalized-page"); + + expect( + issueTypes(makePage({ canonicalUrl: "https://example.com/a" })), + ).not.toContain("canonicalized-page"); + }); + + it("flags thin content only on indexable pages", () => { + expect(issueTypes(makePage({ wordCount: 50 }))).toContain("thin-content"); + expect( + issueTypes( + makePage({ wordCount: 50, isIndexable: false, robotsMeta: "noindex" }), + ), + ).not.toContain("thin-content"); + }); + + it("flags slow responses and deep pages", () => { + expect(issueTypes(makePage({ responseTimeMs: 3000 }))).toContain( + "slow-response", + ); + expect(issueTypes(makePage({ crawlDepth: 6 }))).toContain("deep-page"); + expect(issueTypes(makePage({ crawlDepth: null }))).not.toContain( + "deep-page", + ); + }); +}); + +function makeSlimPage(overrides: Partial): SlimPage { + return { + id: overrides.url ?? "page", + url: "https://example.com/a", + statusCode: 200, + fetchClass: "ok", + title: null, + metaDescription: null, + contentHash: null, + redirectUrl: null, + wordCount: 100, + isIndexable: true, + canonicalUrl: null, + headerCanonicalUrl: null, + ...overrides, + }; +} + +describe("findDuplicates", () => { + it("flags duplicate titles across pages and includes the other URLs", () => { + const issues = findDuplicates([ + makeSlimPage({ url: "https://example.com/a", title: "Same" }), + makeSlimPage({ url: "https://example.com/b", title: "Same" }), + makeSlimPage({ url: "https://example.com/c", title: "Different" }), + ]); + const duplicateTitles = issues.filter( + (issue) => issue.issueType === "duplicate-title", + ); + expect(duplicateTitles).toHaveLength(2); + expect(duplicateTitles[0].details?.otherUrls).toEqual([ + "https://example.com/b", + ]); + }); + + it("excludes noindexed and canonicalized pages from duplicate groups", () => { + const issues = findDuplicates([ + makeSlimPage({ url: "https://example.com/a", title: "Same" }), + makeSlimPage({ + url: "https://example.com/b", + title: "Same", + canonicalUrl: "https://example.com/a", + }), + makeSlimPage({ + url: "https://example.com/c", + title: "Same", + isIndexable: false, + }), + ]); + expect(issues).toHaveLength(0); + }); + + it("ignores non-2xx and blocked pages", () => { + const issues = findDuplicates([ + makeSlimPage({ url: "https://example.com/a", title: "Same" }), + makeSlimPage({ + url: "https://example.com/b", + title: "Same", + fetchClass: "blocked", + statusCode: 403, + }), + ]); + expect(issues).toHaveLength(0); + }); + + it("groups duplicate content by hash only when there is text", () => { + const issues = findDuplicates([ + makeSlimPage({ url: "https://example.com/a", contentHash: "h1" }), + makeSlimPage({ url: "https://example.com/b", contentHash: "h1" }), + makeSlimPage({ + url: "https://example.com/empty-1", + contentHash: "h2", + wordCount: 0, + }), + makeSlimPage({ + url: "https://example.com/empty-2", + contentHash: "h2", + wordCount: 0, + }), + ]); + expect( + issues.filter((issue) => issue.issueType === "duplicate-content"), + ).toHaveLength(2); + }); +}); + +describe("findRedirectChainsAndLoops", () => { + const redirect = (url: string, target: string) => + makeSlimPage({ url, statusCode: 301, redirectUrl: target }); + + it("ignores single redirects", () => { + expect( + findRedirectChainsAndLoops([ + redirect("https://example.com/a", "https://example.com/b"), + makeSlimPage({ url: "https://example.com/b" }), + ]), + ).toHaveLength(0); + }); + + it("flags a chain once, on its head", () => { + const issues = findRedirectChainsAndLoops([ + redirect("https://example.com/a", "https://example.com/b"), + redirect("https://example.com/b", "https://example.com/c"), + makeSlimPage({ url: "https://example.com/c" }), + ]); + expect(issues).toHaveLength(1); + expect(issues[0].issueType).toBe("redirect-chain"); + expect(issues[0].pageUrl).toBe("https://example.com/a"); + expect(issues[0].details?.hops).toEqual([ + "https://example.com/a", + "https://example.com/b", + "https://example.com/c", + ]); + }); + + it("flags loops", () => { + const issues = findRedirectChainsAndLoops([ + redirect("https://example.com/a", "https://example.com/b"), + redirect("https://example.com/b", "https://example.com/a"), + ]); + expect( + issues.filter((issue) => issue.issueType === "redirect-loop").length, + ).toBeGreaterThan(0); + }); + + it("flags self-loops", () => { + const issues = findRedirectChainsAndLoops([ + redirect("https://example.com/a", "https://example.com/a"), + ]); + expect(issues).toHaveLength(1); + expect(issues[0].issueType).toBe("redirect-loop"); + }); +}); diff --git a/src/server/lib/audit/issues/page-reporters.ts b/src/server/lib/audit/issues/page-reporters.ts new file mode 100644 index 0000000..b988678 --- /dev/null +++ b/src/server/lib/audit/issues/page-reporters.ts @@ -0,0 +1,146 @@ +/** + * Per-page issue reporters. + * + * Each reporter is a pure function over a single crawled page record — + * DOM-free by design (cheerio runs once in crawlPage), so the engine works + * over any crawl source that can produce a CrawledPageResult. + * + * Cross-page checks (duplicates, broken links, orphans, redirect chains) + * live in multipage.ts and run over D1 after the crawl. + */ +import type { AuditIssueType } from "@/shared/audit-issues"; +import type { CrawledPageResult } from "@/server/lib/audit/types"; + +export interface DetectedIssue { + issueType: AuditIssueType; + pageId: string | null; + pageUrl: string; + details?: Record; + /** + * Distinguishes multiple issues of the same type on the same page + * (e.g. one broken-internal-link issue per target). Part of the + * deterministic row id, so step retries don't duplicate issues. + */ + dedupeKey?: string; +} + +const TITLE_MAX_CHARS = 60; +const TITLE_MIN_CHARS = 10; +const META_DESCRIPTION_MAX_CHARS = 160; +const THIN_CONTENT_WORDS = 150; +const SLOW_RESPONSE_MS = 1500; +const DEEP_PAGE_DEPTH = 5; + +function hasHeadingLevelSkip(headingOrder: number[]): boolean { + for (let i = 1; i < headingOrder.length; i++) { + if (headingOrder[i] > headingOrder[i - 1] + 1) return true; + } + return false; +} + +export function runPageReporters(page: CrawledPageResult): DetectedIssue[] { + const issues: DetectedIssue[] = []; + const report = ( + issueType: AuditIssueType, + details?: Record, + ) => issues.push({ issueType, pageId: page.id, pageUrl: page.url, details }); + + if (page.fetchClass === "blocked") { + report("blocked-page", { statusCode: page.statusCode }); + return issues; + } + if (page.fetchClass === "error") { + return issues; + } + + if (page.statusCode >= 500) { + report("server-error", { statusCode: page.statusCode }); + return issues; + } + if (page.statusCode >= 400) { + report("broken-page", { statusCode: page.statusCode }); + return issues; + } + // Redirects are normal on their own; chains/loops are flagged in multipage. + if (page.statusCode >= 300) { + return issues; + } + + if (page.responseTimeMs > SLOW_RESPONSE_MS) { + report("slow-response", { responseTimeMs: page.responseTimeMs }); + } + + // Content checks only make sense for analyzed HTML documents (a PDF has no + // title tag to miss; an empty-shell HTML page very much does). + if (!page.isHtml) { + return issues; + } + + // Titles + if (!page.title) { + report("missing-title"); + } else if (page.title.length > TITLE_MAX_CHARS) { + report("title-too-long", { length: page.title.length }); + } else if (page.title.length < TITLE_MIN_CHARS) { + report("title-too-short", { length: page.title.length }); + } + + // Meta description + if (!page.metaDescription) { + report("missing-meta-description"); + } else if (page.metaDescription.length > META_DESCRIPTION_MAX_CHARS) { + report("meta-description-too-long", { + length: page.metaDescription.length, + }); + } + + // Headings + if (page.h1Count === 0) { + report("missing-h1"); + } else if (page.h1Count > 1) { + report("multiple-h1", { h1Count: page.h1Count }); + } + if (hasHeadingLevelSkip(page.headingOrder)) { + report("heading-order-skip"); + } + + // Indexability + canonical signals + if (!page.isIndexable) { + report("noindex-page", { + robotsMeta: page.robotsMeta, + xRobotsTag: page.xRobotsTag, + }); + } + if ( + page.canonicalUrl && + page.headerCanonicalUrl && + page.canonicalUrl !== page.headerCanonicalUrl + ) { + report("canonical-conflict", { + htmlCanonical: page.canonicalUrl, + headerCanonical: page.headerCanonicalUrl, + }); + } + const effectiveCanonical = page.canonicalUrl ?? page.headerCanonicalUrl; + if (effectiveCanonical && effectiveCanonical !== page.url) { + report("canonicalized-page", { canonicalUrl: effectiveCanonical }); + } + + // Content quality + if (page.isIndexable && page.wordCount < THIN_CONTENT_WORDS) { + report("thin-content", { wordCount: page.wordCount }); + } + if (page.imagesMissingAlt > 0) { + report("images-missing-alt", { + imagesMissingAlt: page.imagesMissingAlt, + imagesTotal: page.imagesTotal, + }); + } + + // Structure + if (page.crawlDepth !== null && page.crawlDepth >= DEEP_PAGE_DEPTH) { + report("deep-page", { crawlDepth: page.crawlDepth }); + } + + return issues; +} diff --git a/src/server/lib/audit/lighthouse.ts b/src/server/lib/audit/lighthouse.ts index 5194ed0..50976f8 100644 --- a/src/server/lib/audit/lighthouse.ts +++ b/src/server/lib/audit/lighthouse.ts @@ -1,4 +1,4 @@ -import { detectUrlTemplate } from "./url-utils"; +import { detectUrlTemplate, normalizeUrl } from "./url-utils"; import type { BillingCustomerContext } from "@/server/billing/subscription"; import { createDataforseoClient } from "@/server/lib/dataforseo"; import type { LighthouseResult, LighthouseStrategy } from "./types"; @@ -127,16 +127,13 @@ export function selectLighthouseSample( (p) => p.statusCode >= 200 && p.statusCode < 300, ); - if (strategy === "manual") { - // manual = user picks after crawl; for now return empty - return []; - } - // strategy === "auto": homepage + 1 per URL pattern, capped at 10 const selected = new Set(); - // Always include the start URL / homepage - const startPage = validPages.find((p) => p.url === startUrl); + // Always include the start URL / homepage. Page URLs are normalized; + // normalize the start URL the same way or the comparison silently misses. + const normalizedStart = normalizeUrl(startUrl) ?? startUrl; + const startPage = validPages.find((p) => p.url === normalizedStart); if (startPage) selected.add(startPage.url); // Group by URL template pattern diff --git a/src/server/lib/audit/page-analyzer.ts b/src/server/lib/audit/page-analyzer.ts index e853ae0..8c424c8 100644 --- a/src/server/lib/audit/page-analyzer.ts +++ b/src/server/lib/audit/page-analyzer.ts @@ -7,7 +7,7 @@ */ import * as cheerio from "cheerio"; import { normalizeUrl, isSameOrigin } from "./url-utils"; -import type { PageAnalysis } from "./types"; +import type { PageAnalysis, PageLink } from "./types"; /** * Analyze an HTML string and extract all SEO-relevant data. @@ -71,9 +71,8 @@ export function analyzeHtml( }); }); - // --- Links --- - const internalLinks: string[] = []; - const externalLinks: string[] = []; + // --- Links (deduped by target URL; first anchor wins) --- + const linksByTarget = new Map(); $("a[href]").each((_, el) => { const href = $(el).attr("href"); @@ -84,13 +83,18 @@ export function analyzeHtml( const resolved = normalizeUrl(href, pageUrl); if (!resolved) return; + if (linksByTarget.has(resolved)) return; - if (isSameOrigin(resolved, pageUrl)) { - internalLinks.push(resolved); - } else { - externalLinks.push(resolved); - } + const anchor = $(el).text().replace(/\s+/g, " ").trim().slice(0, 200); + const rel = $(el).attr("rel")?.toLowerCase() ?? ""; + linksByTarget.set(resolved, { + targetUrl: resolved, + anchor: anchor || null, + isInternal: isSameOrigin(resolved, pageUrl), + isNofollow: rel.split(/\s+/).includes("nofollow"), + }); }); + const links = Array.from(linksByTarget.values()); // --- Structured data (JSON-LD) --- let hasStructuredData = false; @@ -119,9 +123,9 @@ export function analyzeHtml( h1s, headingOrder, wordCount, + bodyText, images, - internalLinks, - externalLinks, + links, hasStructuredData, hreflangTags, }; diff --git a/src/server/lib/audit/types.ts b/src/server/lib/audit/types.ts index bb2bf15..e7ca402 100644 --- a/src/server/lib/audit/types.ts +++ b/src/server/lib/audit/types.ts @@ -6,7 +6,7 @@ import { z } from "zod"; import { MIN_AUDIT_PAGES, PAID_MAX_AUDIT_PAGES } from "@/shared/audit-limits"; import { jsonCodec } from "@/shared/json"; -export type LighthouseStrategy = "auto" | "manual" | "none"; +export type LighthouseStrategy = "auto" | "none"; export interface AuditConfig { maxPages: number; @@ -14,11 +14,21 @@ export interface AuditConfig { } // Read-side only (writes stringify a typed AuditConfig). Stored rows may hold -// retired strategies (e.g. "all"); fall back to "auto" instead of failing the -// whole config parse and making the audit's results unviewable. +// retired strategies ("all", "manual") from older audits; map them onto the +// closest surviving strategy — and fall back to "auto" on anything unknown — +// instead of failing the whole config parse and making the audit's results +// unviewable. +const lighthouseStrategySchema = z + .enum(["auto", "all", "manual", "none"]) + .transform( + (value): LighthouseStrategy => + value === "all" ? "auto" : value === "manual" ? "none" : value, + ) + .catch("auto"); + const auditConfigSchema = z.object({ maxPages: z.number().int().min(MIN_AUDIT_PAGES).max(PAID_MAX_AUDIT_PAGES), - lighthouseStrategy: z.enum(["auto", "manual", "none"]).catch("auto"), + lighthouseStrategy: lighthouseStrategySchema, }); const auditConfigCodec = jsonCodec(auditConfigSchema); @@ -29,6 +39,17 @@ export function parseAuditConfig(configRaw: string | null): AuditConfig | null { return result.success ? result.data : null; } +/** How a page fetch resolved. "blocked" = WAF/bot challenge stood in the way. */ +export type PageFetchClass = "ok" | "blocked" | "error"; + +/** One outgoing link edge, deduped by target URL within a page. */ +export interface PageLink { + targetUrl: string; + anchor: string | null; + isInternal: boolean; + isNofollow: boolean; +} + /** Data extracted from a single page via cheerio. */ export interface PageAnalysis { url: string; @@ -51,13 +72,13 @@ export interface PageAnalysis { // Content wordCount: number; + bodyText: string; // Images images: Array<{ src: string | null; alt: string | null }>; - // Links (raw href values from the HTML) - internalLinks: string[]; - externalLinks: string[]; + // Links (normalized, deduped by target) + links: PageLink[]; // Structured data hasStructuredData: boolean; @@ -84,15 +105,22 @@ export interface LighthouseResult { payloadSizeBytes?: number | null; } -export interface StepPageResult { +/** + * Full result of crawling one page. Persisted to D1 inside the crawl-batch + * step; never accumulated in memory or returned as durable step state. + */ +export interface CrawledPageResult { id: string; url: string; statusCode: number; + fetchClass: PageFetchClass; redirectUrl: string | null; title: string; metaDescription: string; canonicalUrl: string | null; robotsMeta: string | null; + xRobotsTag: string | null; + headerCanonicalUrl: string | null; ogTitle: string | null; ogDescription: string | null; ogImage: string | null; @@ -104,13 +132,37 @@ export interface StepPageResult { h6Count: number; headingOrder: number[]; wordCount: number; + contentHash: string | null; + /** + * True when an HTML document was fetched and analyzed. Gates the content + * checks in page reporters (an empty-shell HTML page must still be + * checked; a PDF must not). Transient — not persisted. + */ + isHtml: boolean; imagesTotal: number; imagesMissingAlt: number; images: Array<{ src: string | null; alt: string | null }>; - internalLinks: string[]; - externalLinks: string[]; + links: PageLink[]; hasStructuredData: boolean; hreflangTags: string[]; isIndexable: boolean; responseTimeMs: number; + /** null = not reached via links (e.g. sitemap-seeded). */ + crawlDepth: number | null; + inSitemap: boolean; +} + +/** + * Slim per-page summary returned as durable step state from a crawl batch. + * Keep this small: full page data lives in D1, not in Workflow step state. + */ +export interface StepPageSummary { + id: string; + url: string; + statusCode: number; + fetchClass: PageFetchClass; + redirectUrl: string | null; + title: string; + /** Normalized same-origin link targets, for frontier expansion. */ + internalLinks: string[]; } diff --git a/src/server/lib/audit/url-policy.ts b/src/server/lib/audit/url-policy.ts index f5ddbb3..0c386fd 100644 --- a/src/server/lib/audit/url-policy.ts +++ b/src/server/lib/audit/url-policy.ts @@ -186,6 +186,26 @@ async function hostnameResolvesToBlockedAddress( } } +/** + * Synchronous SSRF check for URLs discovered mid-crawl (links, redirect + * targets, sitemap entries). Blocks non-http(s) schemes, private/loopback IP + * literals, and internal hostnames. DNS resolution is only performed for the + * start URL (see normalizeAndValidateStartUrl); per-link DoH lookups would be + * prohibitively slow. + */ +export function isCrawlableUrl(url: string): boolean { + let parsed: URL; + try { + parsed = new URL(url); + } catch { + return false; + } + if (parsed.protocol !== "http:" && parsed.protocol !== "https:") { + return false; + } + return !isBlockedHost(parsed.hostname); +} + export async function normalizeAndValidateStartUrl( input: string, ): Promise { diff --git a/src/server/mcp/server.ts b/src/server/mcp/server.ts index 22d224c..3d2d8d6 100644 --- a/src/server/mcp/server.ts +++ b/src/server/mcp/server.ts @@ -22,6 +22,12 @@ import { getSearchConsolePerformanceTool, inspectUrlsTool, } from "@/server/mcp/tools/search-console-tools"; +import { + getAuditIssuesTool, + getAuditPagesTool, + getAuditStatusTool, + runSiteAuditTool, +} from "@/server/mcp/tools/site-audit-tools"; import { whoamiTool } from "@/server/mcp/tools/whoami"; // Each handler is wrapped with instrumentMcpToolHandler so failures reach @@ -201,4 +207,24 @@ export function registerOpenSeoMcpTools(server: McpServer) { inspectUrlsTool.handler, ), ); + server.registerTool( + runSiteAuditTool.name, + runSiteAuditTool.config, + runSiteAuditTool.handler, + ); + server.registerTool( + getAuditStatusTool.name, + getAuditStatusTool.config, + getAuditStatusTool.handler, + ); + server.registerTool( + getAuditIssuesTool.name, + getAuditIssuesTool.config, + getAuditIssuesTool.handler, + ); + server.registerTool( + getAuditPagesTool.name, + getAuditPagesTool.config, + getAuditPagesTool.handler, + ); } diff --git a/src/server/mcp/tools/site-audit-tools.ts b/src/server/mcp/tools/site-audit-tools.ts new file mode 100644 index 0000000..488bc78 --- /dev/null +++ b/src/server/mcp/tools/site-audit-tools.ts @@ -0,0 +1,399 @@ +import { z } from "zod"; +import { AuditRepository } from "@/server/features/audit/repositories/AuditRepository"; +import { AuditService } from "@/server/features/audit/services/AuditService"; +import { AppError } from "@/server/lib/errors"; +import { captureServerEvent } from "@/server/lib/posthog"; +import { + AUDIT_ISSUE_TYPES, + getIssueDescriptor, + ISSUE_SEVERITY_ORDER, +} from "@/shared/audit-issues"; +import { mcpResponse } from "@/server/mcp/formatters"; +import { buildProjectMeta } from "@/server/mcp/context"; +import { + looseObjectOutputSchema, + optionalMetaOutputSchema, +} from "@/server/mcp/output-schemas"; +import { withMcpProjectAuth } from "@/server/mcp/project-auth"; +import { projectIdSchema } from "@/server/mcp/schemas"; + +const auditIdSchema = z + .string() + .optional() + .describe("Audit ID. If omitted, uses the project's most recent audit."); + +async function resolveAudit(projectId: string, auditId?: string) { + const audit = auditId + ? await AuditRepository.getAuditForProject(auditId, projectId) + : await AuditRepository.getLatestAuditForProject(projectId); + if (!audit) { + throw new AppError( + "NOT_FOUND", + auditId + ? `Audit ${auditId} not found in this project.` + : "No audits exist for this project yet. Start one with run_site_audit.", + ); + } + return audit; +} + +function auditPath(projectId: string, auditId: string) { + return `/p/${projectId}/audit?auditId=${auditId}`; +} + +// ─── run_site_audit ────────────────────────────────────────────────────────── + +const runInputSchema = { + projectId: projectIdSchema, + url: z.string().min(1).max(2048).describe("Start URL to crawl."), + maxPages: z + .number() + .int() + .min(10) + .max(10_000) + .optional() + .describe("Page budget for the crawl (default 50)."), + runLighthouse: z + .boolean() + .optional() + .describe( + "Run Lighthouse on a sample of up to 10 representative pages (default true).", + ), +} as const; + +type RunArgs = z.infer>; + +export const runSiteAuditTool = { + name: "run_site_audit", + config: { + title: "Run site audit", + description: + "Start a site audit: crawls the site (robots.txt-aware, same-origin), checks every page for SEO issues (broken links, duplicate/missing titles and descriptions, redirect chains, orphan pages, canonical problems, thin content, and more), and optionally runs Lighthouse on a sample of pages. Runs in the background — poll get_audit_status, then read get_audit_issues. If the site blocks our crawler, pages are honestly flagged as blocked rather than misreported.", + inputSchema: runInputSchema, + outputSchema: z + .object({ + auditId: z.string(), + ...optionalMetaOutputSchema, + }) + .passthrough(), + annotations: { + readOnlyHint: false, + openWorldHint: true, + destructiveHint: false, + }, + }, + handler: withMcpProjectAuth(async (args: RunArgs, context) => { + const lighthouseStrategy = (args.runLighthouse ?? true) ? "auto" : "none"; + const limitTier = await AuditService.resolveAuditLimitTier( + context.auth.organizationId, + ); + let auditId: string; + try { + ({ auditId } = await AuditService.startAudit({ + actorUserId: context.auth.userId, + billingCustomer: context.billing, + projectId: args.projectId, + startUrl: args.url, + maxPages: args.maxPages, + lighthouseStrategy, + limitTier, + })); + } catch (error) { + if ( + error instanceof AppError && + error.code === "AUDIT_CAPACITY_REACHED" + ) { + return mcpResponse({ + text: "Audit capacity reached for this account — delete old audits in the dashboard to free capacity, then try again.", + meta: buildProjectMeta( + context, + args.projectId, + `/p/${args.projectId}/audit`, + ), + }); + } + throw error; + } + + await captureServerEvent({ + distinctId: context.auth.userId, + event: "site_audit:start", + organizationId: context.auth.organizationId, + properties: { + project_id: args.projectId, + max_pages: args.maxPages ?? 50, + run_lighthouse: lighthouseStrategy !== "none", + source: "mcp", + }, + }); + + return mcpResponse({ + text: `Audit ${auditId} started for ${args.url}. Poll get_audit_status until it completes, then call get_audit_issues for the prioritized issue report.`, + meta: buildProjectMeta( + context, + args.projectId, + auditPath(args.projectId, auditId), + ), + structuredContent: { auditId }, + }); + }), +}; + +// ─── get_audit_status ──────────────────────────────────────────────────────── + +const statusInputSchema = { + projectId: projectIdSchema, + auditId: auditIdSchema, +} as const; + +type StatusArgs = z.infer>; + +export const getAuditStatusTool = { + name: "get_audit_status", + config: { + title: "Get site audit status", + description: + "Check the progress of a site audit (phase, pages crawled, Lighthouse progress). Free — reads OpenSEO state. Omit auditId for the most recent audit.", + inputSchema: statusInputSchema, + outputSchema: z + .object({ + status: looseObjectOutputSchema, + ...optionalMetaOutputSchema, + }) + .passthrough(), + annotations: { + readOnlyHint: true, + openWorldHint: false, + destructiveHint: false, + }, + }, + handler: withMcpProjectAuth(async (args: StatusArgs, context) => { + // getStatus fetches (and self-heals) the audit row itself; only hit the + // DB here when we need to default to the most recent audit. + const auditId = args.auditId ?? (await resolveAudit(args.projectId)).id; + const status = await AuditService.getStatus(auditId, args.projectId); + + const lighthouseNote = + status.lighthouseTotal > 0 + ? `, lighthouse ${status.lighthouseCompleted + status.lighthouseFailed}/${status.lighthouseTotal}` + : ""; + return mcpResponse({ + text: `Audit ${status.id} (${status.startUrl}): ${status.status} — phase ${status.currentPhase}, ${status.pagesCrawled}/${status.pagesTotal} pages${lighthouseNote}.${status.status === "completed" ? " Call get_audit_issues for the issue report." : ""}`, + meta: buildProjectMeta( + context, + args.projectId, + auditPath(args.projectId, status.id), + ), + structuredContent: { status }, + }); + }), +}; + +// ─── get_audit_issues ──────────────────────────────────────────────────────── + +const issuesInputSchema = { + projectId: projectIdSchema, + auditId: auditIdSchema, + severity: z + .enum(["critical", "warning", "info"]) + .optional() + .describe("Only return issues of this severity."), + issueType: z + .string() + .optional() + .describe( + `Only return issues of this type. One of: ${Object.keys(AUDIT_ISSUE_TYPES).join(", ")}`, + ), + limit: z + .number() + .int() + .min(1) + .max(1_000) + .optional() + .describe("Max issues to return (default 200)."), +} as const; + +type IssuesArgs = z.infer>; + +export const getAuditIssuesTool = { + name: "get_audit_issues", + config: { + title: "Get site audit issues", + description: + "Read the prioritized issue report from a completed site audit. Every issue carries a how_to_fix with concrete remediation steps an agent can act on. Free — reads OpenSEO state. Omit auditId for the most recent audit.", + inputSchema: issuesInputSchema, + outputSchema: z + .object({ + summary: z.array(looseObjectOutputSchema), + issues: z.array(looseObjectOutputSchema), + ...optionalMetaOutputSchema, + }) + .passthrough(), + annotations: { + readOnlyHint: true, + openWorldHint: false, + destructiveHint: false, + }, + }, + handler: withMcpProjectAuth(async (args: IssuesArgs, context) => { + const audit = await resolveAudit(args.projectId, args.auditId); + const unsorted = await AuditRepository.getIssuesForAudit(audit.id, { + severity: args.severity, + issueType: args.issueType, + }); + // Severity-first so truncation drops info rows, never critical ones. + const rows = unsorted.toSorted( + (a, b) => + ISSUE_SEVERITY_ORDER[a.severity] - ISSUE_SEVERITY_ORDER[b.severity] || + a.issueType.localeCompare(b.issueType), + ); + + const counts = new Map(); + for (const row of rows) { + counts.set(row.issueType, (counts.get(row.issueType) ?? 0) + 1); + } + const summary = Array.from(counts.entries()) + .map(([issueType, count]) => { + const descriptor = getIssueDescriptor(issueType); + return { + issueType, + title: descriptor?.title ?? issueType, + severity: descriptor?.severity ?? "info", + count, + }; + }) + .toSorted( + (a, b) => + ISSUE_SEVERITY_ORDER[a.severity] - ISSUE_SEVERITY_ORDER[b.severity] || + b.count - a.count, + ); + + const limit = args.limit ?? 200; + const issues = rows.slice(0, limit).map((row) => { + const descriptor = getIssueDescriptor(row.issueType); + return { + severity: row.severity, + issueType: row.issueType, + title: descriptor?.title ?? row.issueType, + url: row.pageUrl, + details: row.detailsJson + ? (JSON.parse(row.detailsJson) as unknown) + : null, + howToFix: descriptor?.howToFix ?? null, + }; + }); + + const text = + rows.length === 0 + ? args.severity || args.issueType + ? `No issues found for audit ${audit.id} matching the given filters.` + : `No issues recorded for audit ${audit.id}. Note: audits run before issue checks existed have no issue data — re-run the audit with run_site_audit to get a real report.` + : [ + `Audit ${audit.id} (${audit.startUrl}): ${rows.length} issues${rows.length > limit ? ` (showing ${limit})` : ""}.`, + "By type:", + ...summary.map( + (entry) => + `- [${entry.severity}] ${entry.title} (${entry.issueType}): ${entry.count}`, + ), + "Full issue rows with how_to_fix instructions are in structuredContent.issues.", + ].join("\n"); + + return mcpResponse({ + text, + meta: buildProjectMeta( + context, + args.projectId, + auditPath(args.projectId, audit.id), + ), + structuredContent: { summary, issues }, + }); + }), +}; + +// ─── get_audit_pages ───────────────────────────────────────────────────────── + +const pagesInputSchema = { + projectId: projectIdSchema, + auditId: auditIdSchema, + fetchClass: z + .enum(["ok", "blocked", "error"]) + .optional() + .describe( + 'Filter by fetch outcome ("blocked" = the site\'s bot protection challenged the crawler).', + ), + statusCode: z + .number() + .int() + .optional() + .describe("Filter by exact HTTP status code."), + urlContains: z + .string() + .optional() + .describe("Filter to URLs containing this substring."), + limit: z + .number() + .int() + .min(1) + .max(1_000) + .optional() + .describe("Max pages to return (default 100)."), +} as const; + +type PagesArgs = z.infer>; + +export const getAuditPagesTool = { + name: "get_audit_pages", + config: { + title: "Get site audit pages", + description: + "List crawled pages from a site audit with per-page SEO data (status, title, description, word count, indexability, crawl depth, link counts). Free — reads OpenSEO state. Omit auditId for the most recent audit.", + inputSchema: pagesInputSchema, + outputSchema: z + .object({ + pages: z.array(looseObjectOutputSchema), + total: z.number(), + ...optionalMetaOutputSchema, + }) + .passthrough(), + annotations: { + readOnlyHint: true, + openWorldHint: false, + destructiveHint: false, + }, + }, + handler: withMcpProjectAuth(async (args: PagesArgs, context) => { + const audit = await resolveAudit(args.projectId, args.auditId); + const allPages = await AuditRepository.getPagesForAudit(audit.id); + + const filtered = allPages.filter( + (page) => + (!args.fetchClass || page.fetchClass === args.fetchClass) && + (args.statusCode === undefined || + page.statusCode === args.statusCode) && + (!args.urlContains || page.url.includes(args.urlContains)), + ); + const limit = args.limit ?? 100; + const pages = filtered.slice(0, limit); + + const text = [ + `Audit ${audit.id}: ${filtered.length} pages${filtered.length > limit ? ` (showing ${limit})` : ""}.`, + ...pages + .slice(0, 25) + .map( + (page) => + `- ${page.statusCode} ${page.url}${page.fetchClass !== "ok" ? ` [${page.fetchClass}]` : ""} "${page.title ?? ""}"`, + ), + "Full rows are in structuredContent.pages.", + ].join("\n"); + + return mcpResponse({ + text, + meta: buildProjectMeta( + context, + args.projectId, + auditPath(args.projectId, audit.id), + ), + structuredContent: { pages, total: filtered.length }, + }); + }), +}; diff --git a/src/server/workflows/SiteAuditWorkflow.ts b/src/server/workflows/SiteAuditWorkflow.ts index 2ca3924..b6fe898 100644 --- a/src/server/workflows/SiteAuditWorkflow.ts +++ b/src/server/workflows/SiteAuditWorkflow.ts @@ -40,20 +40,25 @@ export class SiteAuditWorkflow extends WorkflowEntrypoint { const { auditId, billingCustomer, projectId, startUrl, config } = event.payload; - const audit = await AuditRepository.getAuditForWorkflow( - auditId, - event.instanceId, - ); - - if (!audit) { - throw new Error("Audit workflow context mismatch"); - } - - if (audit.projectId !== projectId) { - throw new Error("Audit workflow project mismatch"); - } - try { + // Inside a step so the D1 read is retried and replay-cached; a bare + // read here would re-execute on every replay and a transient failure + // would kill the instance before the catch below exists. + await pgStep(step, "validate-context", undefined, async () => { + const audit = await AuditRepository.getAuditForWorkflow( + auditId, + event.instanceId, + ); + + if (!audit) { + throw new Error("Audit workflow context mismatch"); + } + + if (audit.projectId !== projectId) { + throw new Error("Audit workflow project mismatch"); + } + }); + await runAuditPhases(step, { auditId, workflowInstanceId: event.instanceId, diff --git a/src/server/workflows/site-audit-workflow-helpers.ts b/src/server/workflows/site-audit-workflow-helpers.ts index 5b61602..aac8506 100644 --- a/src/server/workflows/site-audit-workflow-helpers.ts +++ b/src/server/workflows/site-audit-workflow-helpers.ts @@ -1,110 +1,258 @@ -import type { StepPageResult } from "@/server/lib/audit/types"; -import { isSameOrigin, normalizeUrl } from "@/server/lib/audit/url-utils"; +import type { + CrawledPageResult, + PageFetchClass, +} from "@/server/lib/audit/types"; +import { sha256Hex } from "@/server/lib/audit/ids"; +import { normalizeUrl } from "@/server/lib/audit/url-utils"; + +const CRAWL_USER_AGENT = "OpenSEO-Audit/1.0"; + +/** + * Markers of a bot-mitigation challenge page. We classify these honestly as + * "blocked" instead of recording the challenge HTML as if it were the page. + */ +const CHALLENGE_BODY_MARKERS = [ + "just a moment...", + "challenge-platform", + "cf-browser-verification", + "attention required! | cloudflare", + "verifying you are human", +]; + +function classifyFetch( + statusCode: number, + headers: Headers, + bodySnippet: string, +): PageFetchClass { + if (statusCode === 0) return "error"; + if (headers.get("cf-mitigated")) return "blocked"; + if (statusCode === 401 || statusCode === 403 || statusCode === 429) { + return "blocked"; + } + if (statusCode === 503) { + const snippet = bodySnippet.toLowerCase(); + if (CHALLENGE_BODY_MARKERS.some((marker) => snippet.includes(marker))) { + return "blocked"; + } + } + return "ok"; +} + +/** Resolve a Location header against its base without normalizing. */ +function resolveRawUrl(location: string, base: string): string | null { + try { + return new URL(location, base).toString(); + } catch { + return null; + } +} + +/** Parse `Link: ; rel="canonical"` response headers. */ +function parseLinkHeaderCanonical( + linkHeader: string | null, + pageUrl: string, +): string | null { + if (!linkHeader) return null; + for (const part of linkHeader.split(",")) { + const match = part.match(/<([^>]+)>\s*;([^]*)/); + if (!match) continue; + if (/rel\s*=\s*"?canonical"?/i.test(match[2])) { + return normalizeUrl(match[1].trim(), pageUrl); + } + } + return null; +} export async function crawlPage( url: string, - crawlOrigin: string, -): Promise { + crawlDepth: number | null, + inSitemap: boolean, +): Promise { const startTime = Date.now(); try { - const response = await fetch(url, { - headers: { - "User-Agent": "OpenSEO-Audit/1.0", - Accept: "text/html,application/xhtml+xml", - }, - redirect: "follow", - signal: AbortSignal.timeout(15_000), - }); + // Manual redirect handling: each hop is recorded as its own page row and + // the target is enqueued by the frontier, so chains/loops are detectable. + // Exception: redirects whose target normalizes to this same URL (e.g. + // /docs -> /docs/ on slash-canonical sites — our normalizer strips the + // slash) are followed inline; recording them would create self-redirect + // rows the frontier can never resolve. + let fetchUrl = url; + let response: Response; + let hops = 0; + for (;;) { + response = await fetch(fetchUrl, { + headers: { + "User-Agent": CRAWL_USER_AGENT, + Accept: "text/html,application/xhtml+xml", + }, + redirect: "manual", + signal: AbortSignal.timeout(15_000), + }); + + if (response.status < 300 || response.status >= 400) break; + + const location = response.headers.get("location"); + const rawTarget = location ? resolveRawUrl(location, fetchUrl) : null; + const normalizedTarget = location + ? normalizeUrl(location, fetchUrl) + : null; + const isSelfAfterNormalization = + normalizedTarget === url && + rawTarget !== null && + rawTarget !== fetchUrl; + if (!isSelfAfterNormalization || hops >= 3) break; + + fetchUrl = rawTarget; + hops += 1; + } const responseTimeMs = Date.now() - startTime; const statusCode = response.status; - const finalUrl = normalizeUrl(response.url) ?? response.url; - if (!isSameOrigin(finalUrl, crawlOrigin)) return null; + const xRobotsTag = response.headers.get("x-robots-tag"); + const headerCanonicalUrl = parseLinkHeaderCanonical( + response.headers.get("link"), + fetchUrl, + ); - const redirectUrl = - response.redirected && response.url !== url ? response.url : null; - const contentType = response.headers.get("content-type") ?? ""; - if (!contentType.includes("text/html")) { - return emptyPageResult(finalUrl, statusCode, redirectUrl, responseTimeMs); + if (statusCode >= 300 && statusCode < 400) { + const location = response.headers.get("location"); + const redirectUrl = location ? normalizeUrl(location, fetchUrl) : null; + return emptyPageResult({ + url, + statusCode, + fetchClass: "ok", + redirectUrl, + responseTimeMs, + xRobotsTag, + headerCanonicalUrl, + crawlDepth, + inSitemap, + }); } - const html = await response.text(); + const contentType = response.headers.get("content-type") ?? ""; + const isHtml = contentType.includes("text/html"); + const body = isHtml ? await response.text() : ""; + const fetchClass = classifyFetch( + statusCode, + response.headers, + body.slice(0, 4_000), + ); + + if (!isHtml || fetchClass !== "ok" || statusCode >= 400) { + return emptyPageResult({ + url, + statusCode, + fetchClass, + redirectUrl: null, + responseTimeMs, + xRobotsTag, + headerCanonicalUrl, + crawlDepth, + inSitemap, + }); + } + + // Resolve links/canonical against the URL that actually served the + // content (it may carry a trailing slash the recorded URL doesn't). // Dynamic import keeps cheerio (page-analyzer's HTML parser) out of the // worker's startup module graph: SiteAuditWorkflow is re-exported from // src/server.ts, so a static import would evaluate cheerio in every // isolate's baseline heap, not just when an audit actually crawls. const { analyzeHtml } = await import("@/server/lib/audit/page-analyzer"); - const analysis = analyzeHtml( - html, - finalUrl, - statusCode, - responseTimeMs, - redirectUrl, - ); - const isIndexable = !( - analysis.robotsMeta?.toLowerCase().includes("noindex") ?? false - ); - const h2Count = analysis.headingOrder.filter((h) => h === 2).length; - const h3Count = analysis.headingOrder.filter((h) => h === 3).length; - const h4Count = analysis.headingOrder.filter((h) => h === 4).length; - const h5Count = analysis.headingOrder.filter((h) => h === 5).length; - const h6Count = analysis.headingOrder.filter((h) => h === 6).length; + const analysis = analyzeHtml(body, fetchUrl, statusCode, responseTimeMs); + const robotsDirectives = [analysis.robotsMeta, xRobotsTag] + .filter(Boolean) + .join(",") + .toLowerCase(); + const isIndexable = !robotsDirectives.includes("noindex"); + const headingCount = (level: number) => + analysis.headingOrder.filter((h) => h === level).length; return { id: crypto.randomUUID(), - url: finalUrl, + url, statusCode, - redirectUrl, + fetchClass, + redirectUrl: null, title: analysis.title, metaDescription: analysis.metaDescription, - canonicalUrl: analysis.canonical, + canonicalUrl: analysis.canonical + ? (normalizeUrl(analysis.canonical, fetchUrl) ?? analysis.canonical) + : null, robotsMeta: analysis.robotsMeta, + xRobotsTag, + headerCanonicalUrl, ogTitle: analysis.ogTitle, ogDescription: analysis.ogDescription, ogImage: analysis.ogImage, h1Count: analysis.h1s.length, - h2Count, - h3Count, - h4Count, - h5Count, - h6Count, + h2Count: headingCount(2), + h3Count: headingCount(3), + h4Count: headingCount(4), + h5Count: headingCount(5), + h6Count: headingCount(6), headingOrder: analysis.headingOrder, wordCount: analysis.wordCount, + contentHash: analysis.bodyText + ? await sha256Hex(analysis.bodyText) + : null, + isHtml: true, imagesTotal: analysis.images.length, - imagesMissingAlt: analysis.images.filter( - (img) => !img.alt || img.alt === "", - ).length, + // Only a truly absent alt attribute counts: alt="" is the correct + // markup for decorative images. + imagesMissingAlt: analysis.images.filter((img) => img.alt === null) + .length, images: analysis.images, - internalLinks: analysis.internalLinks, - externalLinks: analysis.externalLinks, + links: analysis.links, hasStructuredData: analysis.hasStructuredData, hreflangTags: analysis.hreflangTags, isIndexable, responseTimeMs, + crawlDepth, + inSitemap, }; } catch (error) { const responseTimeMs = Date.now() - startTime; console.warn(`Failed to crawl ${url}:`, error); - return emptyPageResult(url, 0, null, responseTimeMs); + return emptyPageResult({ + url, + statusCode: 0, + fetchClass: "error", + redirectUrl: null, + responseTimeMs, + xRobotsTag: null, + headerCanonicalUrl: null, + crawlDepth, + inSitemap, + }); } } -function emptyPageResult( - url: string, - statusCode: number, - redirectUrl: string | null, - responseTimeMs: number, -): StepPageResult { +function emptyPageResult(input: { + url: string; + statusCode: number; + fetchClass: PageFetchClass; + redirectUrl: string | null; + responseTimeMs: number; + xRobotsTag: string | null; + headerCanonicalUrl: string | null; + crawlDepth: number | null; + inSitemap: boolean; +}): CrawledPageResult { return { id: crypto.randomUUID(), - url, - statusCode, - redirectUrl, + url: input.url, + statusCode: input.statusCode, + fetchClass: input.fetchClass, + redirectUrl: input.redirectUrl, title: "", metaDescription: "", canonicalUrl: null, robotsMeta: null, + xRobotsTag: input.xRobotsTag, + headerCanonicalUrl: input.headerCanonicalUrl, ogTitle: null, ogDescription: null, ogImage: null, @@ -116,14 +264,17 @@ function emptyPageResult( h6Count: 0, headingOrder: [], wordCount: 0, + contentHash: null, + isHtml: false, imagesTotal: 0, imagesMissingAlt: 0, images: [], - internalLinks: [], - externalLinks: [], + links: [], hasStructuredData: false, hreflangTags: [], isIndexable: false, - responseTimeMs, + responseTimeMs: input.responseTimeMs, + crawlDepth: input.crawlDepth, + inSitemap: input.inSitemap, }; } diff --git a/src/server/workflows/siteAuditWorkflowCrawl.ts b/src/server/workflows/siteAuditWorkflowCrawl.ts index a915e34..08b9871 100644 --- a/src/server/workflows/siteAuditWorkflowCrawl.ts +++ b/src/server/workflows/siteAuditWorkflowCrawl.ts @@ -1,13 +1,21 @@ import type { WorkflowStep } from "cloudflare:workers"; import type { RobotsResult } from "@/server/lib/audit/discovery"; -import type { StepPageResult } from "@/server/lib/audit/types"; +import type { StepPageSummary } from "@/server/lib/audit/types"; import { isSameOrigin, normalizeUrl } from "@/server/lib/audit/url-utils"; +import { isCrawlableUrl } from "@/server/lib/audit/url-policy"; +import { deterministicAuditRowId } from "@/server/lib/audit/ids"; +import { runPageReporters } from "@/server/lib/audit/issues/page-reporters"; import { AuditRepository } from "@/server/features/audit/repositories/AuditRepository"; import { AuditProgressKV } from "@/server/lib/audit/progress-kv"; import { crawlPage } from "@/server/workflows/site-audit-workflow-helpers"; import { pgStep } from "@/server/workflows/pgStep"; const CRAWL_CONCURRENCY = 25; +// Keep durable step state under the ~1MiB limit: full link lists live in D1; +// the step return only carries new-to-the-frontier targets, deduped across +// the batch and capped. +const MAX_FRONTIER_LINKS_PER_BATCH = 2_000; +const MAX_SUMMARY_TITLE_CHARS = 300; function shouldQueueCrawlLink( link: string, @@ -18,12 +26,19 @@ function shouldQueueCrawlLink( ): boolean { return ( isSameOrigin(link, origin) && + isCrawlableUrl(link) && robots.isAllowed(link) && !visited.has(link) && !queued.has(link) ); } +interface QueueEntry { + url: string; + /** Clicks from the start URL; null when only reachable via sitemap. */ + depth: number | null; +} + type CrawlPhaseParams = { auditId: string; workflowInstanceId: string; @@ -34,10 +49,19 @@ type CrawlPhaseParams = { sitemapUrls: string[]; }; +/** What later phases need per page — no link lists (those stay in D1). */ +export type CrawledPageSummary = Omit; + +export type CrawlPhaseResult = { + pages: CrawledPageSummary[]; + /** True when the frontier was exhausted before hitting maxPages. */ + completed: boolean; +}; + export async function runCrawlPhase( step: WorkflowStep, params: CrawlPhaseParams, -): Promise { +): Promise { const { auditId, workflowInstanceId, @@ -48,43 +72,69 @@ export async function runCrawlPhase( sitemapUrls, } = params; const visited = new Set(); - const queue: string[] = []; const queued = new Set(); - const allPages: StepPageResult[] = []; + // Link-discovered URLs crawl first (BFS from the start URL); sitemap-only + // URLs drain last so link discovery isn't starved of page budget and + // orphan detection stays meaningful. + const linkQueue: QueueEntry[] = []; + const sitemapQueue: QueueEntry[] = []; + const sitemapSet = new Set(); + const summaries: CrawledPageSummary[] = []; - seedCrawlQueue({ - startUrl, - origin, - robots, - sitemapUrls, - visited, - queued, - queue, - }); + const normalizedStart = normalizeUrl(startUrl) ?? startUrl; + if ( + robots.isAllowed(normalizedStart) && + isSameOrigin(normalizedStart, origin) + ) { + linkQueue.push({ url: normalizedStart, depth: 0 }); + queued.add(normalizedStart); + } + + for (const sitemapUrl of sitemapUrls) { + const normalized = normalizeUrl(sitemapUrl); + if (!normalized) continue; + sitemapSet.add(normalized); + if (!shouldQueueCrawlLink(normalized, origin, robots, visited, queued)) { + continue; + } + sitemapQueue.push({ url: normalized, depth: null }); + queued.add(normalized); + } let crawlBatchIndex = 0; - while (queue.length > 0 && allPages.length < maxPages) { - const urlsToCrawl = selectNextCrawlBatch( - queue, + while ( + (linkQueue.length > 0 || sitemapQueue.length > 0) && + summaries.length < maxPages + ) { + const batchEntries = selectNextCrawlBatch({ + linkQueue, + sitemapQueue, queued, visited, robots, - maxPages - allPages.length, - ); - if (urlsToCrawl.length === 0) continue; + remaining: maxPages - summaries.length, + }); + if (batchEntries.length === 0) continue; crawlBatchIndex += 1; - const crawledBatch = await runCrawlBatch( - step, + const crawledBatch = await runCrawlBatch(step, { crawlBatchIndex, - urlsToCrawl, - origin, + auditId, + batchEntries, + sitemapSet, + visited, + queued, + }); + // Keep only the slim summary in memory: at 10k pages, retaining link + // lists for the whole crawl would not fit in the 128MB Worker heap. + summaries.push( + ...crawledBatch.map(({ internalLinks: _links, ...summary }) => summary), ); - allPages.push(...crawledBatch); enqueueDiscoveredLinks({ crawledBatch, - queue, + batchEntries, + linkQueue, queued, visited, origin, @@ -96,110 +146,151 @@ export async function runCrawlPhase( auditId, workflowInstanceId, crawledBatch, - pagesCrawled: allPages.length, + pagesCrawled: summaries.length, visitedCount: visited.size, - queueLength: queue.length, + queueLength: linkQueue.length + sitemapQueue.length, maxPages, }); } - return allPages; + return { + pages: summaries, + completed: linkQueue.length === 0 && sitemapQueue.length === 0, + }; } -function seedCrawlQueue({ - startUrl, - origin, - robots, - sitemapUrls, - visited, - queued, - queue, -}: { - startUrl: string; - origin: string; - robots: RobotsResult; - sitemapUrls: string[]; - visited: Set; +function selectNextCrawlBatch(params: { + linkQueue: QueueEntry[]; + sitemapQueue: QueueEntry[]; queued: Set; - queue: string[]; + visited: Set; + robots: RobotsResult; + remaining: number; }) { - const normalizedStart = normalizeUrl(startUrl) ?? startUrl; - if ( - robots.isAllowed(normalizedStart) && - isSameOrigin(normalizedStart, origin) - ) { - queue.push(normalizedStart); - queued.add(normalizedStart); - } - - for (const sitemapUrl of sitemapUrls) { - const normalized = normalizeUrl(sitemapUrl); - if (!normalized) continue; - if (!shouldQueueCrawlLink(normalized, origin, robots, visited, queued)) { - continue; - } - queue.push(normalized); - queued.add(normalized); - } -} - -function selectNextCrawlBatch( - queue: string[], - queued: Set, - visited: Set, - robots: RobotsResult, - remaining: number, -) { + const { linkQueue, sitemapQueue, queued, visited, robots, remaining } = + params; const batchSize = Math.min(CRAWL_CONCURRENCY, remaining); - const urlsToCrawl: string[] = []; + const batchEntries: QueueEntry[] = []; - while (queue.length > 0 && urlsToCrawl.length < batchSize) { - const url = queue.shift()!; - queued.delete(url); - if (visited.has(url)) continue; - if (!robots.isAllowed(url)) continue; - visited.add(url); - urlsToCrawl.push(url); + while ( + (linkQueue.length > 0 || sitemapQueue.length > 0) && + batchEntries.length < batchSize + ) { + const entry = (linkQueue.length > 0 ? linkQueue : sitemapQueue).shift()!; + queued.delete(entry.url); + if (visited.has(entry.url)) continue; + if (!robots.isAllowed(entry.url)) continue; + visited.add(entry.url); + batchEntries.push(entry); } - return urlsToCrawl; + return batchEntries; } async function runCrawlBatch( step: WorkflowStep, - crawlBatchIndex: number, - urlsToCrawl: string[], - origin: string, -): Promise { - return step.do(`crawl-batch-${crawlBatchIndex}`, async () => { - const settled = await Promise.allSettled( - urlsToCrawl.map((url) => crawlPage(url, origin)), + input: { + crawlBatchIndex: number; + auditId: string; + batchEntries: QueueEntry[]; + sitemapSet: Set; + visited: Set; + queued: Set; + }, +): Promise { + const { + crawlBatchIndex, + auditId, + batchEntries, + sitemapSet, + visited, + queued, + } = input; + return pgStep(step, `crawl-batch-${crawlBatchIndex}`, undefined, async () => { + const pages = await Promise.all( + batchEntries.map((entry) => + crawlPage(entry.url, entry.depth, sitemapSet.has(entry.url)), + ), ); - return settled.flatMap((result) => { - if (result.status === "fulfilled" && result.value) { - return [result.value]; + + // Deterministic ids keep the D1 writes idempotent across step retries. + for (const page of pages) { + page.id = await deterministicAuditRowId(auditId, page.url); + } + + const issues = pages.flatMap((page) => runPageReporters(page)); + await AuditRepository.insertCrawledBatch(auditId, pages, issues); + + // Frontier candidates only: drop targets already visited/queued and + // dedupe across the batch, so the step return stays far under the + // ~1MiB durable-state limit even on mega-menu sites. + const seenTargets = new Set(); + return pages.map((page) => { + const internalLinks: string[] = []; + for (const link of page.links) { + if (!link.isInternal) continue; + if (seenTargets.size >= MAX_FRONTIER_LINKS_PER_BATCH) break; + if (visited.has(link.targetUrl) || queued.has(link.targetUrl)) continue; + if (seenTargets.has(link.targetUrl)) continue; + seenTargets.add(link.targetUrl); + internalLinks.push(link.targetUrl); } - return []; + return { + id: page.id, + url: page.url, + statusCode: page.statusCode, + fetchClass: page.fetchClass, + redirectUrl: page.redirectUrl, + title: page.title.slice(0, MAX_SUMMARY_TITLE_CHARS), + internalLinks, + }; }); }); } function enqueueDiscoveredLinks(params: { - crawledBatch: StepPageResult[]; - queue: string[]; + crawledBatch: StepPageSummary[]; + batchEntries: QueueEntry[]; + linkQueue: QueueEntry[]; queued: Set; visited: Set; origin: string; robots: RobotsResult; }) { - const { crawledBatch, queue, queued, visited, origin, robots } = params; + const { + crawledBatch, + batchEntries, + linkQueue, + queued, + visited, + origin, + robots, + } = params; + const depthByUrl = new Map( + batchEntries.map((entry) => [entry.url, entry.depth]), + ); + for (const pageResult of crawledBatch) { - for (const link of pageResult.internalLinks.filter((candidate) => - shouldQueueCrawlLink(candidate, origin, robots, visited, queued), - )) { - queue.push(link); + const pageDepth = depthByUrl.get(pageResult.url) ?? null; + const childDepth = pageDepth === null ? null : pageDepth + 1; + + for (const link of pageResult.internalLinks) { + if (!shouldQueueCrawlLink(link, origin, robots, visited, queued)) { + continue; + } + linkQueue.push({ url: link, depth: childDepth }); queued.add(link); } + + // Redirect targets continue the same navigation path: same depth. + const redirectTarget = pageResult.redirectUrl; + if ( + redirectTarget && + shouldQueueCrawlLink(redirectTarget, origin, robots, visited, queued) + ) { + linkQueue.push({ url: redirectTarget, depth: pageDepth }); + queued.add(redirectTarget); + } } } @@ -208,7 +299,7 @@ async function persistCrawlProgress(params: { crawlBatchIndex: number; auditId: string; workflowInstanceId: string; - crawledBatch: StepPageResult[]; + crawledBatch: StepPageSummary[]; pagesCrawled: number; visitedCount: number; queueLength: number; @@ -225,23 +316,24 @@ async function persistCrawlProgress(params: { queueLength, maxPages, } = params; - await step.do(`kv-progress-batch-${crawlBatchIndex}`, async () => { - await AuditProgressKV.pushCrawledUrls( - auditId, - crawledBatch.map((pageResult) => ({ - url: pageResult.url, - statusCode: pageResult.statusCode, - title: pageResult.title, - crawledAt: Date.now(), - })), - ); - }); - + // KV push + D1 progress in one step — merging them halves the per-batch + // step count against the ~1k step budget. The D1 update is idempotent; the + // KV push can duplicate entries on a partial retry, which is acceptable for + // an ephemeral progress feed (capped list, short TTL). await pgStep( step, `progress-batch-${crawlBatchIndex}`, undefined, async () => { + await AuditProgressKV.pushCrawledUrls( + auditId, + crawledBatch.map((pageResult) => ({ + url: pageResult.url, + statusCode: pageResult.statusCode, + title: pageResult.title, + crawledAt: Date.now(), + })), + ); await AuditRepository.updateAuditProgress(auditId, workflowInstanceId, { pagesCrawled, pagesTotal: Math.min(visitedCount + queueLength, maxPages), diff --git a/src/server/workflows/siteAuditWorkflowPhases.ts b/src/server/workflows/siteAuditWorkflowPhases.ts index 034a481..1946802 100644 --- a/src/server/workflows/siteAuditWorkflowPhases.ts +++ b/src/server/workflows/siteAuditWorkflowPhases.ts @@ -1,6 +1,6 @@ import type { WorkflowStep } from "cloudflare:workers"; import type { BillingCustomerContext } from "@/server/billing/subscription"; -import { discoverUrls, fetchRobotsTxt } from "@/server/lib/audit/discovery"; +import { discoverUrls, parseRobotsTxt } from "@/server/lib/audit/discovery"; import { fetchAndStoreLighthouseResult, selectLighthouseSample, @@ -8,13 +8,14 @@ import { import { getOrigin } from "@/server/lib/audit/url-utils"; import { AuditRepository } from "@/server/features/audit/repositories/AuditRepository"; import { AuditProgressKV } from "@/server/lib/audit/progress-kv"; -import type { - AuditConfig, - LighthouseResult, - StepPageResult, -} from "@/server/lib/audit/types"; +import { runMultipageChecks } from "@/server/lib/audit/issues/multipage"; +import type { AuditConfig } from "@/server/lib/audit/types"; import { captureServerEvent } from "@/server/lib/posthog"; -import { runCrawlPhase } from "@/server/workflows/siteAuditWorkflowCrawl"; +import { + runCrawlPhase, + type CrawledPageSummary, + type CrawlPhaseResult, +} from "@/server/workflows/siteAuditWorkflowCrawl"; import { pgStep } from "@/server/workflows/pgStep"; const LIGHTHOUSE_URL_BATCH_SIZE = 10; @@ -35,22 +36,6 @@ function capSitemapSeeds(urls: string[], maxPages: number): string[] { return seeds; } -function countLighthouseBatchResults(results: LighthouseResult[]): { - completed: number; - failed: number; -} { - let completed = 0; - let failed = 0; - for (const result of results) { - if (result.errorMessage) { - failed += 1; - continue; - } - completed += 1; - } - return { completed, failed }; -} - type AuditPhasesParams = { auditId: string; workflowInstanceId: string; @@ -82,8 +67,11 @@ export async function runAuditPhases( origin, maxPages, ); - const robots = await fetchRobotsTxt(origin); - const allPages = await runCrawlPhase(step, { + // Parsed outside the step from checkpointed text, so replays see the exact + // robots rules the original run used (a live re-fetch could differ and + // desync the frontier from already-persisted crawl batches). + const robots = parseRobotsTxt(origin, discovery.robotsText); + const crawl = await runCrawlPhase(step, { auditId, workflowInstanceId, origin, @@ -92,14 +80,14 @@ export async function runAuditPhases( robots, sitemapUrls: discovery.sitemapUrls, }); - const lighthouseResults = await runLighthousePhase(step, { + await runLighthousePhase(step, { auditId, workflowInstanceId, billingCustomer, projectId, startUrl, config, - allPages, + pages: crawl.pages, }); await finalizeAudit({ step, @@ -107,9 +95,9 @@ export async function runAuditPhases( workflowInstanceId, billingCustomer, projectId, + startUrl, config, - allPages, - lighthouseResults, + crawl, }); } @@ -126,7 +114,10 @@ async function runDiscoveryPhase( pagesTotal: Math.min(result.urls.length + 1, maxPages), currentPhase: "crawling", }); - return { sitemapUrls: capSitemapSeeds(result.urls, maxPages) }; + return { + sitemapUrls: capSitemapSeeds(result.urls, maxPages), + robotsText: result.robotsText, + }; }); } @@ -137,13 +128,13 @@ type LighthousePhaseParams = { projectId: string; startUrl: string; config: AuditConfig; - allPages: StepPageResult[]; + pages: CrawledPageSummary[]; }; async function runLighthousePhase( step: WorkflowStep, params: LighthousePhaseParams, -): Promise { +) { const { auditId, workflowInstanceId, @@ -151,20 +142,19 @@ async function runLighthousePhase( projectId, startUrl, config, - allPages, + pages, } = params; - if (config.lighthouseStrategy === "none") return []; + if (config.lighthouseStrategy === "none") return; const lighthouseWork = await selectLighthousePages({ step, auditId, workflowInstanceId, - allPages, + pages, startUrl, strategy: config.lighthouseStrategy, }); - const lighthouseResults: LighthouseResult[] = []; let completedChecks = 0; let failedChecks = 0; let lighthouseBatchIndex = 0; @@ -172,47 +162,69 @@ async function runLighthousePhase( for (let i = 0; i < lighthouseWork.length; i += LIGHTHOUSE_URL_BATCH_SIZE) { const batch = lighthouseWork.slice(i, i + LIGHTHOUSE_URL_BATCH_SIZE); lighthouseBatchIndex += 1; - const lighthouseBatchResults = await runLighthouseBatch({ - step, - lighthouseBatchIndex, - batch, - billingCustomer, - projectId, - auditId, - }); + const priorCompleted = completedChecks; + const priorFailed = failedChecks; - lighthouseResults.push(...lighthouseBatchResults); - const counts = countLighthouseBatchResults(lighthouseBatchResults); - failedChecks += counts.failed; - completedChecks += counts.completed; - await pgStep( + // Fetch, store (R2 + D1) and update progress inside one step. The step + // returns only counts; full results live in D1. + const counts = await pgStep( step, - `lighthouse-progress-batch-${lighthouseBatchIndex}`, + `lighthouse-batch-${lighthouseBatchIndex}`, undefined, async () => { + const perUrlResults = await Promise.all( + batch.map(async ({ url, pageId }) => { + const [mobileResult, desktopResult] = await Promise.all([ + fetchAndStoreLighthouseResult({ + url, + pageId, + strategy: "mobile", + billingCustomer, + projectId, + auditId, + }), + fetchAndStoreLighthouseResult({ + url, + pageId, + strategy: "desktop", + billingCustomer, + projectId, + auditId, + }), + ]); + return [mobileResult, desktopResult]; + }), + ); + const results = perUrlResults.flat(); + await AuditRepository.insertLighthouseResults(auditId, results); + + const failed = results.filter((result) => result.errorMessage).length; + const completed = results.length - failed; await AuditRepository.updateAuditProgress(auditId, workflowInstanceId, { - lighthouseCompleted: completedChecks, - lighthouseFailed: failedChecks, + lighthouseCompleted: priorCompleted + completed, + lighthouseFailed: priorFailed + failed, }); + return { completed, failed }; }, ); - } - return lighthouseResults; + completedChecks += counts.completed; + failedChecks += counts.failed; + } } async function selectLighthousePages(params: { step: WorkflowStep; auditId: string; workflowInstanceId: string; - allPages: StepPageResult[]; + pages: CrawledPageSummary[]; startUrl: string; strategy: AuditConfig["lighthouseStrategy"]; }) { - const { step, auditId, workflowInstanceId, allPages, startUrl, strategy } = + const { step, auditId, workflowInstanceId, pages, startUrl, strategy } = params; return pgStep(step, "select-lighthouse-sample", undefined, async () => { - const sample = selectLighthouseSample(allPages, startUrl, strategy); + const sample = selectLighthouseSample(pages, startUrl, strategy); const selectedUrls = new Set(sample); await AuditRepository.updateAuditProgress(auditId, workflowInstanceId, { @@ -221,66 +233,21 @@ async function selectLighthousePages(params: { lighthouseCompleted: 0, lighthouseFailed: 0, }); - return allPages.flatMap((page) => + return pages.flatMap((page) => selectedUrls.has(page.url) ? [{ url: page.url, pageId: page.id }] : [], ); }); } -async function runLighthouseBatch(params: { - step: WorkflowStep; - lighthouseBatchIndex: number; - batch: Array<{ url: string; pageId: string }>; - billingCustomer: BillingCustomerContext; - projectId: string; - auditId: string; -}) { - const { - step, - lighthouseBatchIndex, - batch, - billingCustomer, - projectId, - auditId, - } = params; - return step.do(`lighthouse-batch-${lighthouseBatchIndex}`, async () => { - const perUrlResults = await Promise.all( - batch.map(async ({ url, pageId }) => { - const [mobileResult, desktopResult] = await Promise.all([ - fetchAndStoreLighthouseResult({ - url, - pageId, - strategy: "mobile", - billingCustomer, - projectId, - auditId, - }), - fetchAndStoreLighthouseResult({ - url, - pageId, - strategy: "desktop", - billingCustomer, - projectId, - auditId, - }), - ]); - return [mobileResult, desktopResult]; - }), - ); - - return perUrlResults.flat(); - }); -} - async function finalizeAudit(args: { step: WorkflowStep; auditId: string; workflowInstanceId: string; billingCustomer: BillingCustomerContext; projectId: string; + startUrl: string; config: AuditConfig; - allPages: StepPageResult[]; - lighthouseResults: LighthouseResult[]; + crawl: CrawlPhaseResult; }) { const { step, @@ -288,23 +255,42 @@ async function finalizeAudit(args: { workflowInstanceId, billingCustomer, projectId, + startUrl, config, - allPages, - lighthouseResults, + crawl, } = args; - await pgStep(step, "finalize", undefined, async () => { + await pgStep(step, "multipage-checks", undefined, async () => { await AuditRepository.updateAuditProgress(auditId, workflowInstanceId, { currentPhase: "finalizing", }); - await AuditRepository.batchWriteResults( + + // Integrity guard: pages are persisted inside crawl-batch steps. If the + // crawl claims pages but D1 has none (e.g. an instance started under the + // pre-incremental-persistence code was replayed under this code), fail + // loudly instead of completing with an empty audit. + if ( + crawl.pages.length > 0 && + !(await AuditRepository.hasPagesForAudit(auditId)) + ) { + throw new Error( + `Audit ${auditId}: crawl reported ${crawl.pages.length} pages but none were persisted`, + ); + } + + const issues = await runMultipageChecks({ auditId, - allPages, - lighthouseResults, - ); + startUrl, + crawlCompleted: crawl.completed, + }); + await AuditRepository.insertIssues(auditId, issues); + return { issueCount: issues.length }; + }); + + await pgStep(step, "finalize", undefined, async () => { await AuditRepository.completeAudit(auditId, workflowInstanceId, { - pagesCrawled: allPages.length, - pagesTotal: allPages.length, + pagesCrawled: crawl.pages.length, + pagesTotal: crawl.pages.length, }); await captureServerEvent({ distinctId: billingCustomer.userId, @@ -313,8 +299,12 @@ async function finalizeAudit(args: { properties: { project_id: projectId, status: "completed", - pages_crawled: allPages.length, - pages_total: allPages.length, + pages_crawled: crawl.pages.length, + pages_total: crawl.pages.length, + crawl_completed: crawl.completed, + pages_blocked: crawl.pages.filter( + (page) => page.fetchClass === "blocked", + ).length, run_lighthouse: config.lighthouseStrategy !== "none", }, }); diff --git a/src/serverFunctions/audit.ts b/src/serverFunctions/audit.ts index 050573e..eb56fef 100644 --- a/src/serverFunctions/audit.ts +++ b/src/serverFunctions/audit.ts @@ -1,14 +1,7 @@ import { createServerFn } from "@tanstack/react-start"; import { waitUntil } from "cloudflare:workers"; import { AuditService } from "@/server/features/audit/services/AuditService"; -import type { AuditLimitTier } from "@/server/features/audit/services/audit-capacity"; -import { - customerHasManagedAccess, - customerHasPaidPlan, -} from "@/server/billing/subscription"; -import { AppError } from "@/server/lib/errors"; import { captureServerEvent } from "@/server/lib/posthog"; -import { isHostedServerAuthMode } from "@/server/lib/runtime-env"; import { requireProjectContext } from "@/serverFunctions/middleware"; import { deleteAuditSchema, @@ -23,21 +16,9 @@ export const startAudit = createServerFn({ method: "POST" }) .middleware(requireProjectContext) .validator(startAuditSchema) .handler(async ({ data, context }) => { - // The crawler runs on our Workers compute and isn't credit-metered, so - // plan-tier limits are the abuse bound in hosted mode: free accounts get - // one small audit at a time, paid keeps the full limits, and customers - // with no Autumn product at all are turned away. Self-hosted isn't gated. - let limitTier: AuditLimitTier = "paid"; - if (await isHostedServerAuthMode()) { - const [hasManagedAccess, hasPaidPlan] = await Promise.all([ - customerHasManagedAccess(context.organizationId), - customerHasPaidPlan(context.organizationId), - ]); - if (!hasManagedAccess) { - throw new AppError("PAYMENT_REQUIRED", "Subscribe to run site audits"); - } - limitTier = hasPaidPlan ? "paid" : "free"; - } + const limitTier = await AuditService.resolveAuditLimitTier( + context.organizationId, + ); const result = await AuditService.startAudit({ actorUserId: context.userId, diff --git a/src/shared/audit-issues.ts b/src/shared/audit-issues.ts new file mode 100644 index 0000000..c3f13ba --- /dev/null +++ b/src/shared/audit-issues.ts @@ -0,0 +1,235 @@ +/** + * Registry of site-audit issue types. + * + * Shared between the server (issue engine, MCP tools) and the client + * (issues UI, CSV export). Each issue row in `audit_issues` references one + * of these types by id. + */ + +type IssueSeverity = "critical" | "warning" | "info"; + +interface AuditIssueDescriptor { + severity: IssueSeverity; + title: string; + explanation: string; + howToFix: string; +} + +export const AUDIT_ISSUE_TYPES = { + "blocked-page": { + severity: "critical", + title: "Crawler was blocked", + explanation: + "The site returned a bot challenge or access denial (e.g. a Cloudflare challenge, 403, or 429) instead of the page. We report this honestly rather than pretending the page is broken — but it means this page could not be audited, and other crawlers like search engines may face similar friction.", + howToFix: + 'If you own this site, allowlist the "OpenSEO-Audit" user agent in your WAF/bot-protection settings (on Cloudflare: a WAF custom rule that skips bot protection when the user agent contains "OpenSEO-Audit"; on some free tiers you may need to relax bot protection). Then re-run the audit.', + }, + "server-error": { + severity: "critical", + title: "Server error (5xx)", + explanation: + "The page returned a 5xx server error. Search engines that repeatedly see server errors will crawl the site less and may drop the page from the index.", + howToFix: + "Check the server logs for this URL and fix the underlying error. If the page is gone, return a 404/410 or redirect it to a relevant page instead of erroring.", + }, + "broken-internal-link": { + severity: "critical", + title: "Broken internal link", + explanation: + "This page links to an internal URL that returns an error status (4xx/5xx). Broken links waste crawl budget, leak link equity, and frustrate users — they are among the most common and most damaging technical SEO issues.", + howToFix: + "Update the link to point at the correct live URL, or remove it. If the target was moved, prefer linking directly to the new URL rather than relying on a redirect.", + }, + "missing-title": { + severity: "critical", + title: "Missing title tag", + explanation: + "The page has no . The title is the strongest on-page relevance signal and the headline shown in search results; without it search engines generate one themselves, usually badly.", + howToFix: + "Add a unique, descriptive <title> of roughly 50–60 characters that includes the page's primary topic.", + }, + "broken-page": { + severity: "warning", + title: "Page returns an error (4xx)", + explanation: + "This crawled URL returned a client error (e.g. 404). If it is referenced from your sitemap or other pages, crawlers keep wasting requests on it.", + howToFix: + "If the page should exist, restore it. If it is intentionally gone, remove it from the sitemap and internal links, and consider a 301 redirect to the closest live page.", + }, + "duplicate-title": { + severity: "warning", + title: "Duplicate title", + explanation: + "Multiple pages share the same title tag. Search engines use titles to differentiate pages; duplicates make pages compete with each other and depress click-through rates.", + howToFix: + "Write a unique title for each page describing its specific content. For templated pages, include the distinguishing attribute (name, category, location) in the template.", + }, + "duplicate-meta-description": { + severity: "warning", + title: "Duplicate meta description", + explanation: + "Multiple pages share the same meta description, so search results show identical snippets and users cannot tell the pages apart.", + howToFix: + "Write a unique meta description per page, or remove the duplicated one entirely — search engines will generate a snippet from page content, which beats a wrong duplicate.", + }, + "duplicate-content": { + severity: "warning", + title: "Duplicate page content", + explanation: + "Two or more URLs serve byte-identical visible text. Search engines pick one version to index and ignore the rest, and ranking signals get split across the duplicates.", + howToFix: + "Consolidate duplicates: pick the canonical URL, add rel=canonical from the others, and 301-redirect duplicate URLs where possible (common causes: trailing-slash variants, URL parameters, http/https or www variants).", + }, + "missing-meta-description": { + severity: "warning", + title: "Missing meta description", + explanation: + "The page has no meta description. Search engines will assemble a snippet from page text, which is often less compelling and hurts click-through rate.", + howToFix: + "Add a meta description of roughly 70–160 characters that summarizes the page and gives a reason to click.", + }, + "missing-h1": { + severity: "warning", + title: "Missing H1 heading", + explanation: + "The page has no H1. The H1 tells users and search engines what the page is about; pages without one tend to have weaker topical clarity.", + howToFix: + "Add a single H1 that states the page's main topic, consistent with the title tag.", + }, + "multiple-h1": { + severity: "warning", + title: "Multiple H1 headings", + explanation: + "The page has more than one H1, which dilutes the main-topic signal and usually indicates a templating mistake (e.g. a logo and a headline both marked up as H1).", + howToFix: + "Keep one H1 for the page's main heading and demote the others to H2/H3 (or unstyled elements for non-headings like logos).", + }, + "redirect-chain": { + severity: "warning", + title: "Redirect chain", + explanation: + "Reaching the final page requires two or more consecutive redirects. Each hop adds latency, leaks link equity, and burns crawl budget; long chains may not be followed at all.", + howToFix: + "Point the first URL (and any internal links) directly at the final destination so there is at most one redirect.", + }, + "redirect-loop": { + severity: "warning", + title: "Redirect loop", + explanation: + "This redirect eventually points back to itself, so the URL never resolves. Browsers and crawlers give up with an error.", + howToFix: + "Trace the redirect rules for this URL and break the cycle so the chain terminates at a real 200 page.", + }, + "canonical-conflict": { + severity: "warning", + title: "Conflicting canonical signals", + explanation: + "The page declares different canonical URLs in its HTML <link rel=canonical> and its HTTP Link header. When signals conflict, search engines ignore both and choose their own canonical.", + howToFix: + "Pick one canonical URL and declare it in exactly one place (HTML head is the most common); remove or align the other declaration.", + }, + "thin-content": { + severity: "warning", + title: "Thin content", + explanation: + "The page has very little visible text. Thin pages rarely rank, can drag down sitewide quality assessments, and (if the site renders client-side) may indicate content invisible to plain-HTML crawlers.", + howToFix: + "Either expand the page with genuinely useful content, noindex it, or consolidate it into a stronger page. If the content exists but is rendered by JavaScript, ensure it is server-rendered or pre-rendered.", + }, + "images-missing-alt": { + severity: "warning", + title: "Images missing alt text", + explanation: + "One or more images on the page lack alt attributes. Alt text is an accessibility requirement and the main way search engines understand images.", + howToFix: + 'Add descriptive alt text to meaningful images; use an empty alt (alt="") only for purely decorative ones.', + }, + "orphan-page": { + severity: "warning", + title: "Orphan page", + explanation: + "No crawled page links to this URL — it was only discoverable via the sitemap. Pages without internal links receive little crawl attention and no internal link equity, and users can't find them by browsing.", + howToFix: + "Link to this page from relevant pages (navigation, related content, hub pages), or remove it from the sitemap if it shouldn't be indexed.", + }, + "title-too-long": { + severity: "info", + title: "Title too long", + explanation: + "The title exceeds ~60 characters, so search results will truncate it and the ending may be cut off mid-phrase.", + howToFix: + "Shorten the title to roughly 50–60 characters, front-loading the most important words.", + }, + "title-too-short": { + severity: "info", + title: "Title too short", + explanation: + "The title is under ~10 characters, which is usually too generic to describe the page or attract clicks.", + howToFix: + "Expand the title into a descriptive phrase (roughly 30–60 characters) that states what the page offers.", + }, + "meta-description-too-long": { + severity: "info", + title: "Meta description too long", + explanation: + "The meta description exceeds ~160 characters, so search engines will truncate the snippet.", + howToFix: + "Trim the description to roughly 70–160 characters while keeping the core message and call to action.", + }, + "heading-order-skip": { + severity: "info", + title: "Heading levels skip", + explanation: + "The heading hierarchy skips levels (e.g. an H4 directly after an H2). This weakens document structure for accessibility tools and content parsing.", + howToFix: + "Adjust heading levels so they descend one step at a time (H1 → H2 → H3) without skipping.", + }, + "slow-response": { + severity: "info", + title: "Slow server response", + explanation: + "The HTML response took over 1.5 seconds. Slow time-to-first-byte drags down every downstream performance metric and reduces crawl rate on large sites.", + howToFix: + "Investigate server/database time and caching for this route; serving cached or statically generated HTML usually fixes it.", + }, + "noindex-page": { + severity: "info", + title: "Page is noindex", + explanation: + "The page asks search engines not to index it (via robots meta tag or X-Robots-Tag header). That's often intentional — this is a heads-up, not an error.", + howToFix: + "If this page should rank, remove the noindex directive. If it's intentional (admin, thank-you, filter pages), no action is needed.", + }, + "canonicalized-page": { + severity: "info", + title: "Canonicalized to another URL", + explanation: + "The page declares a different URL as its canonical, telling search engines to index that URL instead. Fine when intentional (parameter pages, syndication) — a problem if this page was meant to rank.", + howToFix: + "If this page should rank on its own, set its canonical to itself. Otherwise no action is needed.", + }, + "deep-page": { + severity: "info", + title: "Page is deep in the site structure", + explanation: + "The page is 5+ clicks from the homepage. Deep pages get crawled less often and receive less link equity.", + howToFix: + "Add links from higher-level pages (hubs, category pages, navigation) to flatten the path to this page.", + }, +} as const satisfies Record<string, AuditIssueDescriptor>; + +export type AuditIssueType = keyof typeof AUDIT_ISSUE_TYPES; + +export const ISSUE_SEVERITY_ORDER: Record<IssueSeverity, number> = { + critical: 0, + warning: 1, + info: 2, +}; + +const issueRegistry: Record<string, AuditIssueDescriptor> = AUDIT_ISSUE_TYPES; + +export function getIssueDescriptor( + issueType: string, +): AuditIssueDescriptor | null { + return issueRegistry[issueType] ?? null; +} diff --git a/src/types/schemas/audit.ts b/src/types/schemas/audit.ts index b12c783..864f3f4 100644 --- a/src/types/schemas/audit.ts +++ b/src/types/schemas/audit.ts @@ -17,10 +17,7 @@ export const startAuditSchema = z.object({ .max(PAID_MAX_AUDIT_PAGES) .optional() .default(DEFAULT_AUDIT_PAGES), - lighthouseStrategy: z - .enum(["auto", "manual", "none"]) - .optional() - .default("auto"), + lighthouseStrategy: z.enum(["auto", "none"]).optional().default("auto"), }); export const getAuditStatusSchema = z.object({