6d40767d95
Explicit, user-triggered button ("Get report from DataForSEO") shown when
the native crawler is blocked — never an automatic retry. DataForSEO's
OnPage API crawls from its own infrastructure with JS rendering, which
clears blocks a plain fetch() from our server can't.
- audits.crawlSource ("native" | "dataforseo") + dataforseoTaskId columns.
- dataforseo/onpage.ts: task_post (JS rendering + store_raw_html) / summary
polling / pages listing / raw_html retrieval. Only task_post is billed
(~$0.00125/page); the rest are free reads of already-billed results, per
DataForSEO's pricing docs.
- DataForSeoAuditService: hard quota of 5 runs per project per calendar
month, enforced server-side via the activity log (audit.dataforseo_report)
before any DataForSEO spend — a rejected 6th run never reaches the API.
- Reuses the native pipeline instead of duplicating it: extracted
buildAnalyzedPageResult() out of crawlPage() so both sources feed the same
analyzeHtml -> runPageReporters -> runMultipageChecks -> auditPages/
auditIssues path. No Cloudflare Workflow backs these audits; the existing
getAuditStatus poll (already running every 3s while "running") drives an
advance step each call instead.
- Known gap: broken-internal-link and orphan-page checks are native-only
(they read the crawl's link graph from the AuditScratchpad Durable Object,
which only the native crawl populates).
- UI: page-limit picker (25-500) + live quota display on the existing
"blocked" screen.
Migration: drizzle/0046_*, drizzle-pg/0024_*.
tsc / oxlint / knip clean. New onpage.test.ts (7) + DataForSeoAuditService
.test.ts (5, including the quota-rejection path); full suite otherwise
unchanged (1195 pass, pre-existing samSkills Windows-CRLF failure only).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2 lines
161 B
SQL
2 lines
161 B
SQL
ALTER TABLE "audits" ADD COLUMN "crawl_source" text DEFAULT 'native' NOT NULL;--> statement-breakpoint
|
|
ALTER TABLE "audits" ADD COLUMN "dataforseo_task_id" text; |