* Audit crawl: retry chunks conservatively and stop re-learning the window per chunk
Production audits on heavy-page sites (500-700KB/page) died with
exceededMemory in the first crawl chunk, and the single step retry re-ran
the exact same profile and died again.
- claimChunk now reports isRetry (leftover leases from a dead attempt);
a retried chunk crawls under RETRY_CRAWL_WINDOW (start 3, max 5,
halved byte budget) instead of restarting at window 10.
- The adapted window carries across chunks via durable step results, so
every ~200 pages no longer re-spikes to the initial window.
- First persist sub-batch shrinks to 5 pages so the byte bound sees the
site's page weight before a full 25-page batch is in flight.
- In-flight HTML budget halved to 8 MiB (16 MiB never constrained the
observed ~650KB pages: bound was 25, above the 20 max); growth now
requires a full 25-page sample.
- Parse-time caps: 1,000 extracted links/images per page so mega-menu and
crawler-trap pages can't bloat retained persist batches.
* Guard endWindow for instances replaying pre-deploy step results
* ci: un-export internal-only interfaces (knip)
* fix(sam): stream canned refusals without a provider call
Out-of-credits and session-gone refusals ran a real LLM turn with a
200-token cap; MiniMax M3 could spend the whole budget on reasoning
tokens, leaving the user a raw truncated chain-of-thought (which also
named the backing model) and no reply. Refusal turns now swap in a
static LanguageModelV3 that streams the refusal text through Think's
normal pipeline — rendered and persisted like any assistant message,
with no provider request at all.
Fixesevery-app/open-seo#161
* fix(chat): stop pinning the transcript to the bottom while the user scrolls up
Both chat surfaces (SAM and onboarding) forced scrollTop to the bottom
on every streamed chunk, so scrolling up mid-reply was undone within
milliseconds. A shared stick-to-bottom hook now tracks pinned-ness from
real scroll events: scrolling away releases the pin, returning to the
bottom (or sending a message) re-arms it.
Fixesevery-app/open-seo#160
* fix(rank-tracking): allow any SERP language for any country
The Add Domain modal restricted the language picker to the Labs
per-country subset and disabled it when only one option existed, so
e.g. tracking English searches in Czechia was impossible — even though
rank tracking runs against the SERP API, which serves every supported
language in every country. The picker now offers the full SERP language
list; create/update schemas validate codes against the master list so
unknown codes still fail before DataForSEO charges for them; and
keyword-metrics refreshes resolve a Labs-served language so an
unserved pair never reaches a charged Labs call.
Fixesevery-app/open-seo#183
* fix(chat): surface silent turn failures in Workers logs
A provider stream dying mid-turn (chat:request:failed) and a DO restart
whose recovery gives up (chat:recovery:exhausted) leave the user a
replayed "Something went wrong" banner and a half-streamed message, but
never reach the onChatError hook — their only signal is the agents:chat
diagnostics channel, which was unsubscribed, so the chat agents' most
common failure modes produced zero log lines. A module-level
subscription now logs both for every chat DO.
SamChatAgent.onChatError also returns the error now: Think uses the
return value as the stored chat-terminal body that reconnecting clients
replay, and returning void stored the literal string "undefined".
* feat(rank-tracking): raise scheduler budget to 2000 units/tick
The 200-unit budget used ~2-6% of DataForSEO's 2,000 req/min account
cap and would take days to drain the post-#462 backlog. Scheduled checks
run through the task queue (1 task_post per 100 units + free task_get
polls), so a full 2,000-unit tick peaks around 1,200 req/min — still
leaving headroom for the other DataForSEO products on the account.
Also raise the due-config fetch limit to 500 so skip-heavy stretches
(free orgs, keywordless configs) drain more than 200 rows per tick.
* fix(rank-tracking): retune budget to 1000 with accurate sizing and a tick deadline
Review corrections to the 10x bump: task_get polling is one call per unit
per round and rounds wake synchronized per tick, with up to three ~15-min
poll windows overlapping the */5 cron — 2000 units/tick could saturate
DataForSEO's 2000 req/min cap, silently aging throttled polls into the
~3x-cost live fallback billed to customers. 1000/tick keeps real headroom
and is still ~45x steady-state demand.
Add a 3-minute wall-clock deadline to the per-config loop (stoppedByDeadline
in the tick summary): a skip-heavy 500-candidate tick pays serial Autumn
round-trips per distinct org and could otherwise run into the 15-minute cron
kill. Name the fetch limit (DUE_CONFIGS_PER_TICK) and correct its comment.
Test fixtures now derive from MAX_KEYWORDS_PER_CONFIG instead of asserting
an unreachable 1500-keyword config.
* fix(cron): run the audit watchdog before the rank loop
reconcileStaleAudits ran after runScheduledRankChecks in the same
invocation, so a slow rank tick would delay the watchdog and a wall-clock
kill would skip it entirely.
* fix(cron): preserve watchdog failure signal; codex review polish
Rethrow a caught reconcileStaleAudits error after the rank loop so the
invocation still reports failed (matching pre-reorder semantics), use an
inclusive deadline comparison, and note overlapping-tick poll residue in
the sizing comment.
* fix(mcp): stop breaking public OAuth clients at token refresh
The DCR shim force-upgraded every public client (token_endpoint_auth_method
"none" or omitted) to client_secret_post so Perplexity would accept the
registration response. That made the stored client confidential, so the
token endpoint demanded client authentication on every grant — and MCP
clients that discard the secret (Codex) lost their session at first token
expiry with "invalid_client: missing client_secret".
Register those clients as true public clients instead (PKCE + the
provider's grant-to-client binding secure that flow), and satisfy
Perplexity by decorating only the registration response with a placeholder
client_secret and client_secret_post. The provider skips secret validation
for public clients, so clients that send the placeholder and clients that
never store it both keep working, including on refresh.
* refactor(mcp): only rebuild DCR requests that actually change
* fix(mcp): satisfy type-aware lint in DCR shims
oxlint --type-aware rejected the Record<string, unknown> assertions used to
read untrusted DCR payloads. Parse both with loose Zod schemas instead, per
the repo's trust-boundary convention, which also replaces the hand-rolled
object guards.
A user's #1 organic result showed as position 3-4 because rank_absolute
counts SERP features (local pack, PAA, AI overviews) as positions. Rank
tracker, keyword-research SERP list, and onboarding market snapshot now
prefer rank_group; MCP get-serp-results keeps rank_absolute since it
returns all SERP item types.
Cap crawled HTML bodies at 2 MiB before Cheerio parsing, bulk-insert
audit links on Postgres instead of per-row Hyperdrive round trips, and
raise the Worker CPU ceiling to the 5-minute paid max for link-heavy
crawl steps.
Remove the RankMovementCard (setup pitch + rank movement stats) from the
dashboard grid until the rank tracking experience gets more polish. The
overview endpoint still returns the rank summary, so restoring the card
is a small revert of this commit.
void setCached() leaves the R2 put unregistered, and workerd cancels
unregistered pending I/O once the response is sent — the write never
lands, so the caches behind domain overview, domain keyword/page pages,
and SERP research are silently re-fetched (and re-charged) on every
call. Register the writes with waitUntil, matching the pattern already
used in instrumentation.ts and brandLookup.ts.
