6d40767d95
Explicit, user-triggered button ("Get report from DataForSEO") shown when
the native crawler is blocked — never an automatic retry. DataForSEO's
OnPage API crawls from its own infrastructure with JS rendering, which
clears blocks a plain fetch() from our server can't.
- audits.crawlSource ("native" | "dataforseo") + dataforseoTaskId columns.
- dataforseo/onpage.ts: task_post (JS rendering + store_raw_html) / summary
polling / pages listing / raw_html retrieval. Only task_post is billed
(~$0.00125/page); the rest are free reads of already-billed results, per
DataForSEO's pricing docs.
- DataForSeoAuditService: hard quota of 5 runs per project per calendar
month, enforced server-side via the activity log (audit.dataforseo_report)
before any DataForSEO spend — a rejected 6th run never reaches the API.
- Reuses the native pipeline instead of duplicating it: extracted
buildAnalyzedPageResult() out of crawlPage() so both sources feed the same
analyzeHtml -> runPageReporters -> runMultipageChecks -> auditPages/
auditIssues path. No Cloudflare Workflow backs these audits; the existing
getAuditStatus poll (already running every 3s while "running") drives an
advance step each call instead.
- Known gap: broken-internal-link and orphan-page checks are native-only
(they read the crawl's link graph from the AuditScratchpad Durable Object,
which only the native crawl populates).
- UI: page-limit picker (25-500) + live quota display on the existing
"blocked" screen.
Migration: drizzle/0046_*, drizzle-pg/0024_*.
tsc / oxlint / knip clean. New onpage.test.ts (7) + DataForSeoAuditService
.test.ts (5, including the quota-rejection path); full suite otherwise
unchanged (1195 pass, pre-existing samSkills Windows-CRLF failure only).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
181 lines
3.6 KiB
JSON
181 lines
3.6 KiB
JSON
{
|
|
"version": "7",
|
|
"dialect": "postgresql",
|
|
"entries": [
|
|
{
|
|
"idx": 0,
|
|
"version": "7",
|
|
"when": 1780611624042,
|
|
"tag": "0000_fixed_nico_minoru",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 1,
|
|
"version": "7",
|
|
"when": 1781809009342,
|
|
"tag": "0001_striped_bulldozer",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 2,
|
|
"version": "7",
|
|
"when": 1781902417326,
|
|
"tag": "0002_clean_moira_mactaggert",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 3,
|
|
"version": "7",
|
|
"when": 1782833725049,
|
|
"tag": "0003_sturdy_may_parker",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 4,
|
|
"version": "7",
|
|
"when": 1782851683039,
|
|
"tag": "0004_dashing_betty_ross",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 5,
|
|
"version": "7",
|
|
"when": 1783117905771,
|
|
"tag": "0005_talented_wild_pack",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 6,
|
|
"version": "7",
|
|
"when": 1783438455708,
|
|
"tag": "0006_location_name",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 7,
|
|
"version": "7",
|
|
"when": 1783306049167,
|
|
"tag": "0007_same_marvel_zombies",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 8,
|
|
"version": "7",
|
|
"when": 1783630716543,
|
|
"tag": "0008_yummy_annihilus",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 9,
|
|
"version": "7",
|
|
"when": 1783699015488,
|
|
"tag": "0009_supreme_captain_stacy",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 10,
|
|
"version": "7",
|
|
"when": 1784070000325,
|
|
"tag": "0010_overrated_amazoness",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 11,
|
|
"version": "7",
|
|
"when": 1784410131656,
|
|
"tag": "0011_friendly_morlun",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 12,
|
|
"version": "7",
|
|
"when": 1784423181488,
|
|
"tag": "0012_dashboard",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 13,
|
|
"version": "7",
|
|
"when": 1784426969390,
|
|
"tag": "0013_sleepy_black_tarantula",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 14,
|
|
"version": "7",
|
|
"when": 1784588248793,
|
|
"tag": "0014_solid_centennial",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 15,
|
|
"version": "7",
|
|
"when": 1784610564142,
|
|
"tag": "0015_sticky_dagger",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 16,
|
|
"version": "7",
|
|
"when": 1785608388841,
|
|
"tag": "0016_panoramic_blob",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 17,
|
|
"version": "7",
|
|
"when": 1786066279773,
|
|
"tag": "0017_ga4_connections",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 18,
|
|
"version": "7",
|
|
"when": 1786208325786,
|
|
"tag": "0018_drop_reddit_attributions",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 19,
|
|
"version": "7",
|
|
"when": 1786209409297,
|
|
"tag": "0019_clammy_selene",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 20,
|
|
"version": "7",
|
|
"when": 1787099999115,
|
|
"tag": "0020_project_memory",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 21,
|
|
"version": "7",
|
|
"when": 1787594239250,
|
|
"tag": "0021_user_last_active_organization",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 22,
|
|
"version": "7",
|
|
"when": 1787773500453,
|
|
"tag": "0022_third_supernaut",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 23,
|
|
"version": "7",
|
|
"when": 1788983569510,
|
|
"tag": "0023_clever_prowler",
|
|
"breakpoints": true
|
|
},
|
|
{
|
|
"idx": 24,
|
|
"version": "7",
|
|
"when": 1789226958095,
|
|
"tag": "0024_heavy_arachne",
|
|
"breakpoints": true
|
|
}
|
|
]
|
|
} |