BIKEGEAR_PROXY_URL now applies to the browser-mode fetcher too.
The VPS IP is permanently blocked at Cloudflare network level — a residential
proxy is the only remaining solution. Set BIKEGEAR_PROXY_URL=http://user:pass@host:port.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previous approach (newContext per run) was still detected via navigator.plugins=0
and window.chrome=undefined. Two fixes:
1. launchPersistentContext writes a real browser profile to disk — cookies,
cache and fingerprint survive restarts so Cloudflare trusts the same "user"
2. addInitScript patches plugins, chrome runtime, webdriver and languages on
every page, hiding remaining headless Chromium signals
recycleBrowserContext now just closes stale pages (profile stays on disk).
closeBrowser just closes the browser process — profile already persisted.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Running Playwright Chromium for 2+ hours with one shared context accumulates
memory until PM2/OOM kills the process with no error. Fix: save cookies and
close/reopen the context every 20 brands so Chromium memory is freed mid-run.
Cookie persistence means the new context resumes with the cf_clearance session.
Override interval with BIKEGEAR_RECYCLE_EVERY env var.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Cloudflare 429s when running 2+ concurrent browser pages against bikegear.in.
Browser mode now runs 1 page at a time with a 1.5s polite delay after each load.
HTTP mode keeps its original concurrency (3 listing / 2 detail).
Override with BIKEGEAR_LISTING_CONCURRENCY / BIKEGEAR_DETAIL_CONCURRENCY / BIKEGEAR_REQUEST_DELAY.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
HTTP requests from datacenter IPs get 403 from bikegear.in Cloudflare WAF.
Adds BIKEGEAR_FETCH_MODE=browser (default) which runs a headless Chromium via
Playwright — already installed — and navigates pages like a real user, bypassing
the JS challenge without needing a paid proxy.
Set BIKEGEAR_FETCH_MODE=http to keep the old axios path (for local/residential IPs).
Browser instance is shared across the full scrape run and closed cleanly on finish.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Datacenter IPs are blocked by bikegear.in Cloudflare WAF.
Switches fetchHtml from native fetch to axios with BIKEGEAR_PROXY_URL support.
Set BIKEGEAR_PROXY_URL=http://user:pass@host:port in server .env to enable.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1. parseProductJsonLd: remove sku requirement — out-of-stock products have
Product JSON-LD but no sku field, causing false "No product JSON-LD found" errors.
Now picks block with offers.price or description instead.
2. extractProductImages: add fallback for non-standard image paths — products
with images at /catalog/KTM/... or other custom paths were falling through
to JSON-LD fallback (1 image). Now also scans the product-img anchor HTML.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Each source module now exports a brands[] array:
- motousher: 44 brands (full list from brands.js)
- dirtstreet: 5 brands (SC Project, Evotech, etc.)
- retrorides: 10 bike brands (BMW, Ducati, Aprilia, etc.)
- kyt: KYT
- brocks-performance: Brock's Performance
- yuasa: Yuasa
listSources() passes brands[] through in the API response so the
frontend can render the per-source brand list without a separate call.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Added 32 new brands from motousher.com/pages/partner-brands:
ACE Bikes, Baja Designs, Barkbusters, Bumot, Cliff Top - Chargers,
Domino, Doubletake Mirrors, Flo Tool, Giant Loop Moto, HAD Originals,
Hepco Becker, HexEzCan, IntAct Batteries, Key Smart, Motone,
MotoSolutions, Nebo Tools, NGK Plugs, Nite Ize, NoNoise,
Own Your Adventure, Posi-Products, Quad Lock, Quixx, Rok Straps,
Rubba Tech, S100 and A1, SBV Tools, sPOD, True Utility, Vertex, Victor
Also corrected Maxima Racing Oils collection URL:
/maxima-racing-oils -> /maxima-oils (actual slug used on site)
Each entry commented with "already added" or "added 2026-07-07".
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Cache is now skipped if every product in the cached file has scrapeError
(prevents a 403-failed run from poisoning the next 24h of imports)
- If all scrapes fail, throw an error so the pipeline stops at step 1
instead of silently creating 13 empty Shopify products with no images/price
- Only successfully scraped batteries are included in allProducts passed
to the converter/download/upload/upsert stages
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Production server IP was being rejected by yuasabatteries.com as a bot.
Added full Chrome Sec-Fetch-* and Sec-CH-UA header suite (same fix applied
to retrorides earlier) so the server appears as a real browser navigation.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- scraper.js: fetches product detail pages from yuasabatteries.com for a configurable battery ID list; auto-discovers and downloads the current MSRP xlsx from the Yuasa MSRP page to build a battery-type → price map using the xlsx package; extracts specs, images, and vehicle fitment from HTML tables
- converter.js: converts scraped battery data to Shopify-ready format (title, SKU, price from MSRP, description from specs, images)
- index.js: standard pipeline integration with 24h cache (YUASA_CACHE_HOURS=0 to bypass); battery ID list configurable via YUASA_BATTERY_IDS env var (comma-separated)
- sources/index.js: registers yuasa as the 6th import source
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Production server IP triggered mod_security 409 even on sitemap fetch.
- Added Sec-Fetch-Dest/Mode/Site/User and Sec-CH-UA headers that
mod_security requires to recognise a real browser request
- Separate SITEMAP_HEADERS set for XML sitemap request (Accept: application/xml)
- fetchText now accepts optional headers override
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
HTTP 409 Conflict from the site WAF when requests run in parallel.
- 409 now retried with backoff (same as 429), up to 5 attempts at 5s increments
- Default concurrency changed from 3 to 1 (sequential fetches)
- RETRORIDES_CONCURRENCY env var to override if needed
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New source: retrorides (retrorides.co.in, 1168 products)
- Scraper reads all product URLs from WordPress sitemap in one call
- Extracts title/SKU/price/stock from JSON-LD, gallery images from data-large_image
- Bike brand/model from breadcrumb (case-insensitive BIKE_MAKES normalisation)
- Converts Jetpack CDN image URLs to direct retrorides.co.in URLs
- 24h cache with RETRORIDES_CACHE_HOURS env var, concurrency=3 default
- Converter follows same record.scraped pattern as motousher/dirtstreet
- Registered in sources/index.js alongside kyt, brocks-performance, motousher, dirtstreet
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pipelineJobs.js:
- cancelJob(jobId): marks job as cancelled=true, status=cancelling
- isJobCancelled(jobId): checked by the pipeline between stages
runSourcePipeline.js:
- PipelineCancelledError class
- checkCancelled() called before each of the 6 pipeline stages
- Accepts options.isCancelled() callback from the job runner
runKytPipelineJob.js:
- Passes isCancelled: () => isJobCancelled(job.id) into pipeline
- Catches PipelineCancelledError separately, sets status=cancelled
routes/pipeline.js:
- POST /pipeline/cancel/:jobId — marks job for cancellation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both motousher and dirtstreet converters were reading product fields
(title, sku, price, images) directly from the aggregated record, but
those fields live inside record.scraped after fetchWebsiteData wraps
them. Results were: Untitled Product, missing images, SKU=variant-1.
Also fixed brand priority: per-product brand (e.g. Evans Coolant,
SC Project) now takes precedence over the global SHOPIFY_BRAND env
var (KYT), which was incorrectly overriding all products from the
new sources.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both sources are now registered in sources/index.js and fully wired
into the 6-stage pipeline (fetch → download → watermark → upload →
convert → upsert). The frontend will automatically show them as tabs
via GET /pipeline/sources without any frontend changes needed.
motousher/ (Shopify JSON API — 12 brands, ~2,446 products):
- scraper.js: fetches /collections/{slug}/products.json + /products/{handle}.json
- converter.js: maps scraped products to standard pipeline format
- index.js: fetchWebsiteData() loops all brands, normalises to
productSummary.img format for shared download/upload utilities
- Supports MOTOUSHER_BRANDS env var to filter brands on a run
dirtstreet/ (WooCommerce HTML + JSON-LD — 5 brands, ~1,087 products):
- scraper.js: pure fetch, paginates /brand/{slug}/page/N/,
extracts price from offers.priceSpecification[0].price,
stock from JSON-LD availability field
- converter.js: maps scraped products to standard pipeline format,
builds descriptionHtml from body + short desc + attributes table
- index.js: fetchWebsiteData() loops all brands, normalises to
productSummary.img format
- Supports DIRTSTREET_BRANDS env var to filter brands on a run
sources/index.js: registered all 4 sources (kyt, brocks-performance,
motousher, dirtstreet). GET /pipeline/sources now returns all 4.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds two new experimental product scrapers under test_source/, isolated
from the active pipeline until verified and ready to promote.
motousher/ (Shopify store — Shopify JSON API):
- Scrapes 12 brands: All Balls Racing, DID Chains, EBC Brakes, Esjot
Sprockets, Evans Coolant, Grip Puppies, HiFlo Filters, JT Sprockets,
Maxima Racing Oils, Putoline, Ram Mount, Wunderlich
- 2,446 products total scraped and verified
- Uses /collections/{slug}/products.json + /products/{handle}.json
- Parallel fetch (concurrency 3), paginated collection listing
dirtstreet/ (WooCommerce store — HTML + JSON-LD):
- Scrapes 5 brands: SC Project, Evotech Performance, DNA Air Filters,
WRS, Zero Gravity Racing
- 1,087 products total scraped and verified
- Pure fetch with JSON-LD schema.org extraction (no browser)
- Handles paginated /brand/{slug}/page/N/ archives
- Price extracted from offers.priceSpecification[0].price
- Stock status derived from JSON-LD availability field
Both scrapers are standalone (node index.js), support --brand and
--limit flags, save per-brand JSON files and a combined.json.
Scraped data lives in data/sources/test_source/ (gitignored).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
