BIKEGEAR_PROXY_URL now applies to the browser-mode fetcher too.
The VPS IP is permanently blocked at Cloudflare network level — a residential
proxy is the only remaining solution. Set BIKEGEAR_PROXY_URL=http://user:pass@host:port.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previous approach (newContext per run) was still detected via navigator.plugins=0
and window.chrome=undefined. Two fixes:
1. launchPersistentContext writes a real browser profile to disk — cookies,
cache and fingerprint survive restarts so Cloudflare trusts the same "user"
2. addInitScript patches plugins, chrome runtime, webdriver and languages on
every page, hiding remaining headless Chromium signals
recycleBrowserContext now just closes stale pages (profile stays on disk).
closeBrowser just closes the browser process — profile already persisted.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Running Playwright Chromium for 2+ hours with one shared context accumulates
memory until PM2/OOM kills the process with no error. Fix: save cookies and
close/reopen the context every 20 brands so Chromium memory is freed mid-run.
Cookie persistence means the new context resumes with the cf_clearance session.
Override interval with BIKEGEAR_RECYCLE_EVERY env var.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Cloudflare 429s when running 2+ concurrent browser pages against bikegear.in.
Browser mode now runs 1 page at a time with a 1.5s polite delay after each load.
HTTP mode keeps its original concurrency (3 listing / 2 detail).
Override with BIKEGEAR_LISTING_CONCURRENCY / BIKEGEAR_DETAIL_CONCURRENCY / BIKEGEAR_REQUEST_DELAY.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
HTTP requests from datacenter IPs get 403 from bikegear.in Cloudflare WAF.
Adds BIKEGEAR_FETCH_MODE=browser (default) which runs a headless Chromium via
Playwright — already installed — and navigates pages like a real user, bypassing
the JS challenge without needing a paid proxy.
Set BIKEGEAR_FETCH_MODE=http to keep the old axios path (for local/residential IPs).
Browser instance is shared across the full scrape run and closed cleanly on finish.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Datacenter IPs are blocked by bikegear.in Cloudflare WAF.
Switches fetchHtml from native fetch to axios with BIKEGEAR_PROXY_URL support.
Set BIKEGEAR_PROXY_URL=http://user:pass@host:port in server .env to enable.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1. parseProductJsonLd: remove sku requirement — out-of-stock products have
Product JSON-LD but no sku field, causing false "No product JSON-LD found" errors.
Now picks block with offers.price or description instead.
2. extractProductImages: add fallback for non-standard image paths — products
with images at /catalog/KTM/... or other custom paths were falling through
to JSON-LD fallback (1 image). Now also scans the product-img anchor HTML.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
