1c74fded7b
* Site audit P0 (1/3): issue engine, incremental persistence, block detection
Server-side foundation of the P0 feature set from docs/site-audit-pm-research.md:
- Issue engine: shared registry of issue types (severity, explanation,
how-to-fix). Per-page reporters run inside crawl steps; cross-page checks
(duplicate titles/descriptions/content, broken internal links, redirect
chains/loops, orphan pages) run at finalize as SQL over the persisted crawl.
- New audit_links + audit_issues tables, audit_pages columns (depth, content
hash, header signals, fetch class, sitemap flag); audit tables moved to
src/db/{,pg/}audit.schema.ts; migrations 0029 (D1) / 0006 (PG).
- Incremental persistence: pages/links/issues written inside each crawl-batch
step with deterministic row ids + upserts (retry idempotent); slim step
state; robots.txt checkpointed as step state; merged progress steps keep a
10k-page crawl within the Workflows step budget.
- Crawler: manual redirect handling with inline follow of normalization-
equivalent redirects, response header capture (X-Robots-Tag, Link
rel=canonical), BFS depth, sitemap-last seeding, SSRF check on discovered
links, honest 'we were blocked' classification (403/429/cf-mitigated/
challenge).
- MCP: run_site_audit, get_audit_status, get_audit_issues, get_audit_pages;
limitTier resolved via shared AuditService.resolveAuditLimitTier.
- Lighthouse strategies reduced to auto/none (legacy all/manual map on read).
- Self-healing: getStatus reconciles audits whose workflow instance errored/
terminated without reaching mark-failed.
The Issues UI and the badseo.dev e2e fixture site stack on top of this PR.
Deploy notes: run db:migrate:prod (additive); terminate running audits before
deploying — the workflow step structure changed and in-flight instances cannot
replay under the new code (a finalize guard fails them loudly instead of
completing empty).
* Store only internal link edges in audit_links
Both consumers (broken-internal-link and orphan checks) filter on
isInternal; per-page external counts already live on audit_pages.
Dropping external rows cuts stored edges on outbound-heavy sites.
Column stays so P1 external-link checks can re-add rows without a
migration.
* Review fixes: failAudit CAS guard, dedupe hash helpers, cheaper checks
- failAudit only transitions running audits, so the getStatus reconciler
can't flip a just-completed audit to failed when it races finalize
- collapse the duplicate SHA-256 helper into audit/ids.ts
- finalize integrity guard uses a limit-1 existence probe instead of
fetching every page row
- get_audit_status MCP tool no longer reads the audit row twice when an
explicit auditId is given
88 lines
2.5 KiB
TypeScript
88 lines
2.5 KiB
TypeScript
import { getDatabaseProvider } from "./provider";
|
|
import * as sqliteApp from "./app.schema";
|
|
import * as sqliteAudit from "./audit.schema";
|
|
import * as sqliteSam from "./sam.schema";
|
|
import * as sqliteAuth from "./better-auth-schema";
|
|
import * as sqliteBilling from "./billing.schema";
|
|
import * as sqliteGsc from "./gsc.schema";
|
|
import * as sqliteReddit from "./reddit-attribution.schema";
|
|
import * as pgApp from "./pg/app.schema";
|
|
import * as pgAudit from "./pg/audit.schema";
|
|
import * as pgSam from "./pg/sam.schema";
|
|
import * as pgAuth from "./pg/better-auth-schema";
|
|
import * as pgBilling from "./pg/billing.schema";
|
|
import * as pgGsc from "./pg/gsc.schema";
|
|
import * as pgReddit from "./pg/reddit-attribution.schema";
|
|
|
|
// Canonical schema barrel. Repositories import their tables from here and the
|
|
// provider-aware `db` from "@/db", so each repository is written ONCE for both
|
|
// backends.
|
|
//
|
|
// The TYPE identity is the SQLite definitions; the runtime VALUES are whichever
|
|
// provider is active. `schema-parity.test.ts` asserts the two dialect schemas
|
|
// are structurally interchangeable (same tables/columns/nullability/PKs/unique
|
|
// indexes), which is what makes the single cast below sound. The Postgres
|
|
// schema is the one structural artifact NOT regenerated by `db:generate`, so the
|
|
// parity test is its drift guard.
|
|
type AppSchema = typeof sqliteApp &
|
|
typeof sqliteAudit &
|
|
typeof sqliteSam &
|
|
typeof sqliteAuth &
|
|
typeof sqliteBilling &
|
|
typeof sqliteGsc &
|
|
typeof sqliteReddit;
|
|
|
|
const runtimeSchema =
|
|
getDatabaseProvider() === "postgres"
|
|
? {
|
|
...pgApp,
|
|
...pgAudit,
|
|
...pgSam,
|
|
...pgAuth,
|
|
...pgBilling,
|
|
...pgGsc,
|
|
...pgReddit,
|
|
}
|
|
: {
|
|
...sqliteApp,
|
|
...sqliteAudit,
|
|
...sqliteSam,
|
|
...sqliteAuth,
|
|
...sqliteBilling,
|
|
...sqliteGsc,
|
|
...sqliteReddit,
|
|
};
|
|
|
|
// oxlint-disable-next-line typescript/no-unsafe-type-assertion -- guarded by schema-parity.test.ts
|
|
const schema = runtimeSchema as unknown as AppSchema;
|
|
|
|
export const {
|
|
userOnboardingAnswers,
|
|
projects,
|
|
savedKeywords,
|
|
savedKeywordTags,
|
|
savedKeywordTagAssignments,
|
|
keywordMetrics,
|
|
rankTrackingConfigs,
|
|
rankTrackingKeywords,
|
|
rankCheckRuns,
|
|
rankSnapshots,
|
|
audits,
|
|
auditPages,
|
|
auditLinks,
|
|
auditIssues,
|
|
auditLighthouseResults,
|
|
samSessions,
|
|
samProjectMemory,
|
|
user,
|
|
session,
|
|
account,
|
|
verification,
|
|
organization,
|
|
member,
|
|
invitation,
|
|
billingCustomerStatus,
|
|
gscConnections,
|
|
redditAttributions,
|
|
} = schema;
|