* Audit crawl: retry chunks conservatively and stop re-learning the window per chunk
Production audits on heavy-page sites (500-700KB/page) died with
exceededMemory in the first crawl chunk, and the single step retry re-ran
the exact same profile and died again.
- claimChunk now reports isRetry (leftover leases from a dead attempt);
a retried chunk crawls under RETRY_CRAWL_WINDOW (start 3, max 5,
halved byte budget) instead of restarting at window 10.
- The adapted window carries across chunks via durable step results, so
every ~200 pages no longer re-spikes to the initial window.
- First persist sub-batch shrinks to 5 pages so the byte bound sees the
site's page weight before a full 25-page batch is in flight.
- In-flight HTML budget halved to 8 MiB (16 MiB never constrained the
observed ~650KB pages: bound was 25, above the 20 max); growth now
requires a full 25-page sample.
- Parse-time caps: 1,000 extracted links/images per page so mega-menu and
crawler-trap pages can't bloat retained persist batches.
* Guard endWindow for instances replaying pre-deploy step results
* ci: un-export internal-only interfaces (knip)
* Blocked-audit alert: point to desktop crawlers instead of support email
* Soften blocked-audit alert opening
* Partial-block banner: recommend desktop crawlers, drop allowlist tip
* Add Google user data disclosures and Limited Use statement to privacy policy
* Show GA4 connect surfaces to the Google OAuth verification reviewer account
* List OpenAI as a potential AI model provider in the privacy policy
* fix(sam): stream canned refusals without a provider call
Out-of-credits and session-gone refusals ran a real LLM turn with a
200-token cap; MiniMax M3 could spend the whole budget on reasoning
tokens, leaving the user a raw truncated chain-of-thought (which also
named the backing model) and no reply. Refusal turns now swap in a
static LanguageModelV3 that streams the refusal text through Think's
normal pipeline — rendered and persisted like any assistant message,
with no provider request at all.
Fixesevery-app/open-seo#161
* fix(chat): stop pinning the transcript to the bottom while the user scrolls up
Both chat surfaces (SAM and onboarding) forced scrollTop to the bottom
on every streamed chunk, so scrolling up mid-reply was undone within
milliseconds. A shared stick-to-bottom hook now tracks pinned-ness from
real scroll events: scrolling away releases the pin, returning to the
bottom (or sending a message) re-arms it.
Fixesevery-app/open-seo#160
* fix(rank-tracking): allow any SERP language for any country
The Add Domain modal restricted the language picker to the Labs
per-country subset and disabled it when only one option existed, so
e.g. tracking English searches in Czechia was impossible — even though
rank tracking runs against the SERP API, which serves every supported
language in every country. The picker now offers the full SERP language
list; create/update schemas validate codes against the master list so
unknown codes still fail before DataForSEO charges for them; and
keyword-metrics refreshes resolve a Labs-served language so an
unserved pair never reaches a charged Labs call.
Fixesevery-app/open-seo#183
* fix(chat): surface silent turn failures in Workers logs
A provider stream dying mid-turn (chat:request:failed) and a DO restart
whose recovery gives up (chat:recovery:exhausted) leave the user a
replayed "Something went wrong" banner and a half-streamed message, but
never reach the onChatError hook — their only signal is the agents:chat
diagnostics channel, which was unsubscribed, so the chat agents' most
common failure modes produced zero log lines. A module-level
subscription now logs both for every chat DO.
SamChatAgent.onChatError also returns the error now: Think uses the
return value as the stored chat-terminal body that reconnecting clients
replay, and returning void stored the literal string "undefined".
* feat(rank-tracking): raise scheduler budget to 2000 units/tick
The 200-unit budget used ~2-6% of DataForSEO's 2,000 req/min account
cap and would take days to drain the post-#462 backlog. Scheduled checks
run through the task queue (1 task_post per 100 units + free task_get
polls), so a full 2,000-unit tick peaks around 1,200 req/min — still
leaving headroom for the other DataForSEO products on the account.
Also raise the due-config fetch limit to 500 so skip-heavy stretches
(free orgs, keywordless configs) drain more than 200 rows per tick.
* fix(rank-tracking): retune budget to 1000 with accurate sizing and a tick deadline
Review corrections to the 10x bump: task_get polling is one call per unit
per round and rounds wake synchronized per tick, with up to three ~15-min
poll windows overlapping the */5 cron — 2000 units/tick could saturate
DataForSEO's 2000 req/min cap, silently aging throttled polls into the
~3x-cost live fallback billed to customers. 1000/tick keeps real headroom
and is still ~45x steady-state demand.
Add a 3-minute wall-clock deadline to the per-config loop (stoppedByDeadline
in the tick summary): a skip-heavy 500-candidate tick pays serial Autumn
round-trips per distinct org and could otherwise run into the 15-minute cron
kill. Name the fetch limit (DUE_CONFIGS_PER_TICK) and correct its comment.
Test fixtures now derive from MAX_KEYWORDS_PER_CONFIG instead of asserting
an unreachable 1500-keyword config.
* fix(cron): run the audit watchdog before the rank loop
reconcileStaleAudits ran after runScheduledRankChecks in the same
invocation, so a slow rank tick would delay the watchdog and a wall-clock
kill would skip it entirely.
* fix(cron): preserve watchdog failure signal; codex review polish
Rethrow a caught reconcileStaleAudits error after the rank loop so the
invocation still reports failed (matching pre-reorder semantics), use an
inclusive deadline comparison, and note overlapping-tick poll residue in
the sizing comment.
