4 Commits

Author SHA1 Message Date
Ben Senescu
eec998a762
feat(rank-tracking): raise scheduler throughput 5x with rate-limit-derived sizing (#465)
* feat(rank-tracking): raise scheduler budget to 2000 units/tick

The 200-unit budget used ~2-6% of DataForSEO's 2,000 req/min account
cap and would take days to drain the post-#462 backlog. Scheduled checks
run through the task queue (1 task_post per 100 units + free task_get
polls), so a full 2,000-unit tick peaks around 1,200 req/min — still
leaving headroom for the other DataForSEO products on the account.

Also raise the due-config fetch limit to 500 so skip-heavy stretches
(free orgs, keywordless configs) drain more than 200 rows per tick.

* fix(rank-tracking): retune budget to 1000 with accurate sizing and a tick deadline

Review corrections to the 10x bump: task_get polling is one call per unit
per round and rounds wake synchronized per tick, with up to three ~15-min
poll windows overlapping the */5 cron — 2000 units/tick could saturate
DataForSEO's 2000 req/min cap, silently aging throttled polls into the
~3x-cost live fallback billed to customers. 1000/tick keeps real headroom
and is still ~45x steady-state demand.

Add a 3-minute wall-clock deadline to the per-config loop (stoppedByDeadline
in the tick summary): a skip-heavy 500-candidate tick pays serial Autumn
round-trips per distinct org and could otherwise run into the 15-minute cron
kill. Name the fetch limit (DUE_CONFIGS_PER_TICK) and correct its comment.

Test fixtures now derive from MAX_KEYWORDS_PER_CONFIG instead of asserting
an unreachable 1500-keyword config.

* fix(cron): run the audit watchdog before the rank loop

reconcileStaleAudits ran after runScheduledRankChecks in the same
invocation, so a slow rank tick would delay the watchdog and a wall-clock
kill would skip it entirely.

* fix(cron): preserve watchdog failure signal; codex review polish

Rethrow a caught reconcileStaleAudits error after the rank loop so the
invocation still reports failed (matching pre-reorder semantics), use an
inclusive deadline comparison, and note overlapping-tick poll residue in
the sizing comment.
2026-08-07 16:44:16 -04:00
Ben Senescu
f048dc3bd1
fix(rank-tracking): drain due configs deterministically (clean-room alternative to #456) (#462) 2026-08-07 13:53:30 -04:00
Ben Senescu
4f17fe4942 Triage production log errors: audit crash, Autumn webhook FK, PostHog capture, auth rate-limit IP, log noise (#327) 2026-07-02 18:15:46 -04:00
Ben Senescu
c5cbe84ce6 Unify dual-backend DB layer (D1 default + Postgres opt-in) (#238) 2026-07-02 18:15:46 -04:00