# Site audit crawl architecture ## Status Accepted ## Context The site audit runs as a Cloudflare Workflow. The original design kept all crawl state inside the workflow run: the URL frontier and page summaries lived in workflow memory and step returns, pages were parsed with a full DOM parser (cheerio), and link edges were bulk-inserted into the app database during the crawl. On large or slow sites this collided with platform limits: - **Isolate memory.** DOM parsing expands a page several times over, multiplied by concurrent parses, plus a whole-crawl summaries array that was rebuilt on every workflow replay. - **Step-output cap (~1 MiB).** Sitemap seed lists flowed through step returns and could exceed it outright. - **Step timeout × retry.** The default 10-minute step timeout with default retries meant a deterministically failing step burned about an hour before the audit died. - **Batch head-of-line blocking.** Fixed-size crawl batches waited on their slowest fetch; one tarpit URL stalled a whole batch. - **Invisible failure.** Workflow-level kills (OOM, CPU) skip the error handler, so audit rows could stay `running` forever with no recorded reason, and the UI refused to show the pages that had already been crawled and persisted. ## Decision Split the audit into a control plane and a data plane. ### Orchestration stays in Workflows; crawl state moves to a Durable Object `SiteAuditWorkflow` remains the orchestrator (phase ordering, durable steps, retries). All transient crawl state lives in `AuditScratchpad`, a SQLite-backed Durable Object, one instance per audit (`idFromName(auditId)`): - **frontier** — URL queue and dedup set (URL primary key), with `pending / leased / crawled` states keyed by chunk number; - **links** — internal link edges, primary-keyed for idempotent re-insert; - **page_mirror** — the few columns the finalize link checks need. The app database (D1/Postgres) keeps only what the product reads: `audits` (plus failure columns), `audit_pages`, `audit_issues`, `audit_lighthouse_results`. Link edges are never persisted to the app DB; the cross-page link checks (broken internal links, orphan pages) run as SQL inside the DO at finalize. Rejected alternatives: Cloudflare Queues (no dedup, no counters, no completion signal — a frontier is mostly those things), a Postgres frontier table (chatty hot-path writes, dual-dialect surface), R2 spill files (not queryable). ### Chunked rolling crawl The crawl phase is a loop of `crawl-chunk-N` steps. Each chunk leases up to ~200 URLs from the DO and crawls them with a rolling concurrency window: the moment a fetch settles, the next URL launches. The window self-adjusts between 5 and 40 based on recent fetch health (errors, blocked fetches, slow responses, oversized bodies shrink it; clean fast batches grow it). A soft deadline ends the chunk early and releases unfetched leases. Persistence is pipelined with fetching but serialized with itself: page rows and per-page issues go to the app DB, link edges and frontier updates go to the DO, and progress counters update per sub-batch. Step returns carry only counters, so no step output scales with site size. Everything is idempotent under step retries: chunk-keyed leases (a retried step re-receives exactly the URLs its failed attempt held, and refuses a fresh claim for an already-crawled chunk number), deterministic page-row ids, and insert-or-ignore/replace writes on stable keys. ### Streaming HTML parsing Pages are parsed with htmlparser2's streaming tokenizer — no DOM is built, so per-page memory is constant. cheerio (which uses the same tokenizer internally) remains only as a test reference: the parser test suite asserts extraction parity against it, and the badseo fixture harness asserts identical issue output end to end. ### Failure handling and graceful degradation - `audits` carries `error_code` (a closed vocabulary mapped from real platform error strings — `step_timeout`, `oom`, `cpu_limit`, `db_error`, `step_output_too_large`, `workflow_internal`, `instance_lost`, `unknown`), `error_detail`, and `failed_phase`. - A watchdog on the existing `*/15` cron reconciles audit rows stuck in `running` against the Workflows API; the audit status read path does the same lazily. Only confirmed instance-not-found errors (after a grace period) count as a lost instance, so transient API failures never fail a live audit. - A failed audit shows everything crawled before the failure ("stopped early after N pages") instead of hiding results. - The start URL follows redirects (with per-hop SSRF revalidation) before the crawl anchors its origin, so apex→www and cross-TLD redirects don't dead-end the crawl after one page. ### DO lifecycle and platform-limit guards - `destroy()` (delete alarm, then all storage) runs on audit success and on audit deletion. Every DO construction schedules a 7-day self-cleanup alarm, so any instantiation — including a write racing in after destroy, or a workflow that dies before seeding — is eventually wiped. Failed audits keep their scratchpad for those 7 days as the resume/debug artifact. - Guards where the platform has hard edges: link writes stop at a storage budget (below the free-plan per-object SQLite cap; orphan detection is skipped when the link graph was truncated), discovered-URL batches are capped below the serialized-RPC limit, and sitemap documents are read up to a byte cap and skipped whole beyond it. ## Consequences - Workflow heap and step state are O(one chunk) regardless of site size; the OOM/step-output failure classes are gone by construction. - Hostile or slow sites degrade to a slower window and finish (or fail with a classified reason and visible partials) instead of burning an hour of retries. - Every failure is aggregable by `error_code` in plain SQL, and zombie `running` rows self-heal. - Crawl-state code is provider-independent (DO SQLite exists in workerd), so self-host does not add a dialect surface for the frontier or links. - The scratchpad is opaque from outside the DO; the 7-day retention of failed audits' state is the debugging window. - Resume-from-frontier ("retry finishes the last N pages instead of recrawling") is enabled by the retained frontier but intentionally not built yet.