1c74fded7b
* Site audit P0 (1/3): issue engine, incremental persistence, block detection
Server-side foundation of the P0 feature set from docs/site-audit-pm-research.md:
- Issue engine: shared registry of issue types (severity, explanation,
how-to-fix). Per-page reporters run inside crawl steps; cross-page checks
(duplicate titles/descriptions/content, broken internal links, redirect
chains/loops, orphan pages) run at finalize as SQL over the persisted crawl.
- New audit_links + audit_issues tables, audit_pages columns (depth, content
hash, header signals, fetch class, sitemap flag); audit tables moved to
src/db/{,pg/}audit.schema.ts; migrations 0029 (D1) / 0006 (PG).
- Incremental persistence: pages/links/issues written inside each crawl-batch
step with deterministic row ids + upserts (retry idempotent); slim step
state; robots.txt checkpointed as step state; merged progress steps keep a
10k-page crawl within the Workflows step budget.
- Crawler: manual redirect handling with inline follow of normalization-
equivalent redirects, response header capture (X-Robots-Tag, Link
rel=canonical), BFS depth, sitemap-last seeding, SSRF check on discovered
links, honest 'we were blocked' classification (403/429/cf-mitigated/
challenge).
- MCP: run_site_audit, get_audit_status, get_audit_issues, get_audit_pages;
limitTier resolved via shared AuditService.resolveAuditLimitTier.
- Lighthouse strategies reduced to auto/none (legacy all/manual map on read).
- Self-healing: getStatus reconciles audits whose workflow instance errored/
terminated without reaching mark-failed.
The Issues UI and the badseo.dev e2e fixture site stack on top of this PR.
Deploy notes: run db:migrate:prod (additive); terminate running audits before
deploying — the workflow step structure changed and in-flight instances cannot
replay under the new code (a finalize guard fails them loudly instead of
completing empty).
* Store only internal link edges in audit_links
Both consumers (broken-internal-link and orphan checks) filter on
isInternal; per-page external counts already live on audit_pages.
Dropping external rows cuts stored edges on outbound-heavy sites.
Column stays so P1 external-link checks can re-add rows without a
migration.
* Review fixes: failAudit CAS guard, dedupe hash helpers, cheaper checks
- failAudit only transitions running audits, so the getStatus reconciler
can't flip a just-completed audit to failed when it races finalize
- collapse the duplicate SHA-256 helper into audit/ids.ts
- finalize integrity guard uses a limit-1 existence probe instead of
fetching every page row
- get_audit_status MCP tool no longer reads the audit row twice when an
explicit auditId is given
223 lines
4.4 KiB
JSON
223 lines
4.4 KiB
JSON
{
|
|
"version": "7",
|
|
"dialect": "sqlite",
|
|
"entries": [
|
|
{
|
|
"idx": 0,
|
|
"version": "6",
|
|
"when": 1772065863070,
|
|
"tag": "0000_fantastic_vanisher",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 1,
|
|
"version": "6",
|
|
"when": 1772127913577,
|
|
"tag": "0001_round_unus",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 2,
|
|
"version": "6",
|
|
"when": 1773261363719,
|
|
"tag": "0002_fair_toad_men",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 3,
|
|
"version": "6",
|
|
"when": 1773853209809,
|
|
"tag": "0003_light_sage",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 4,
|
|
"version": "6",
|
|
"when": 1773935379368,
|
|
"tag": "0004_faithful_sunset_bain",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 5,
|
|
"version": "6",
|
|
"when": 1773965298920,
|
|
"tag": "0005_low_red_hulk",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 6,
|
|
"version": "6",
|
|
"when": 1774320825595,
|
|
"tag": "0006_magical_alex_wilder",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 7,
|
|
"version": "6",
|
|
"when": 1776208279781,
|
|
"tag": "0007_sour_risque",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 8,
|
|
"version": "6",
|
|
"when": 1776288697036,
|
|
"tag": "0008_luxuriant_colossus",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 9,
|
|
"version": "6",
|
|
"when": 1776347182067,
|
|
"tag": "0009_smart_kitty_pryde",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 10,
|
|
"version": "6",
|
|
"when": 1777422740449,
|
|
"tag": "0010_high_liz_osborn",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 11,
|
|
"version": "6",
|
|
"when": 1778031161783,
|
|
"tag": "0011_colorful_dark_beast",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 12,
|
|
"version": "6",
|
|
"when": 1778113978173,
|
|
"tag": "0012_closed_impossible_man",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 13,
|
|
"version": "6",
|
|
"when": 1778532548655,
|
|
"tag": "0013_fat_network",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 14,
|
|
"version": "6",
|
|
"when": 1778750000000,
|
|
"tag": "0014_tag_color",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 15,
|
|
"version": "6",
|
|
"when": 1778750000001,
|
|
"tag": "0015_huge_celestials",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 16,
|
|
"version": "6",
|
|
"when": 1779138211145,
|
|
"tag": "0016_magical_the_fallen",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 17,
|
|
"version": "6",
|
|
"when": 1779331224828,
|
|
"tag": "0017_omniscient_dagger",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 18,
|
|
"version": "6",
|
|
"when": 1779849000462,
|
|
"tag": "0018_minor_mad_thinker",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 19,
|
|
"version": "6",
|
|
"when": 1780519331717,
|
|
"tag": "0019_true_absorbing_man",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 20,
|
|
"version": "6",
|
|
"when": 1780599087400,
|
|
"tag": "0020_drop_delegated_users",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 21,
|
|
"version": "6",
|
|
"when": 1780625404317,
|
|
"tag": "0021_autumn_billing",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 22,
|
|
"version": "6",
|
|
"when": 1781107930529,
|
|
"tag": "0022_purple_hitman",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 23,
|
|
"version": "6",
|
|
"when": 1781467831570,
|
|
"tag": "0023_sad_cobalt_man",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 24,
|
|
"version": "6",
|
|
"when": 1781809245835,
|
|
"tag": "0024_clumsy_barracuda",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 25,
|
|
"version": "6",
|
|
"when": 1781892941214,
|
|
"tag": "0025_loving_mojo",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 26,
|
|
"version": "6",
|
|
"when": 1782833723883,
|
|
"tag": "0026_wide_red_skull",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 27,
|
|
"version": "6",
|
|
"when": 1782851682143,
|
|
"tag": "0027_reflective_molten_man",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 28,
|
|
"version": "6",
|
|
"when": 1783117904905,
|
|
"tag": "0028_empty_beyonder",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 29,
|
|
"version": "6",
|
|
"when": 1783438453855,
|
|
"tag": "0029_location_name",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 30,
|
|
"version": "6",
|
|
"when": 1783306047190,
|
|
"tag": "0030_legal_reaper",
|
|
"breakpoints": true
|
|
}
|
|
]
|
|
} |