- DaisyUI themes openseo/openseo-dark -> crawlerx/crawlerx-dark on the
MetatronCube palette (primary #3779b9, secondary #0f1d34, accent #4faede)
- Load Inter as the app font; add --brand-gradient / .brand-gradient
- Cube mark + "CrawlerX" wordmark in the sidebar and mobile top bar
- Add public/crawlerx-mark.png; point transparent-logo.png at the cube mark
- <title>, site.webmanifest, and all user-facing "OpenSEO" copy across
src/client and src/routes -> "CrawlerX"
Deferred: favicons (need resizing), agent-facing identity (MCP server
name, SAM/onboarding prompts, fact sheet), web/ marketing site.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace leftover 'personal, noncommercial' boilerplate with a license
covering internal business use and client-services work, permit using
generated Outputs in client deliverables, and carve Outputs out of the
Section 2.2 commercial-exploitation restriction.
* Audit crawl: retry chunks conservatively and stop re-learning the window per chunk
Production audits on heavy-page sites (500-700KB/page) died with
exceededMemory in the first crawl chunk, and the single step retry re-ran
the exact same profile and died again.
- claimChunk now reports isRetry (leftover leases from a dead attempt);
a retried chunk crawls under RETRY_CRAWL_WINDOW (start 3, max 5,
halved byte budget) instead of restarting at window 10.
- The adapted window carries across chunks via durable step results, so
every ~200 pages no longer re-spikes to the initial window.
- First persist sub-batch shrinks to 5 pages so the byte bound sees the
site's page weight before a full 25-page batch is in flight.
- In-flight HTML budget halved to 8 MiB (16 MiB never constrained the
observed ~650KB pages: bound was 25, above the 20 max); growth now
requires a full 25-page sample.
- Parse-time caps: 1,000 extracted links/images per page so mega-menu and
crawler-trap pages can't bloat retained persist batches.
* Guard endWindow for instances replaying pre-deploy step results
* ci: un-export internal-only interfaces (knip)
* Blocked-audit alert: point to desktop crawlers instead of support email
* Soften blocked-audit alert opening
* Partial-block banner: recommend desktop crawlers, drop allowlist tip
