Each source module now exports a brands[] array:
- motousher: 44 brands (full list from brands.js)
- dirtstreet: 5 brands (SC Project, Evotech, etc.)
- retrorides: 10 bike brands (BMW, Ducati, Aprilia, etc.)
- kyt: KYT
- brocks-performance: Brock's Performance
- yuasa: Yuasa
listSources() passes brands[] through in the API response so the
frontend can render the per-source brand list without a separate call.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Added 32 new brands from motousher.com/pages/partner-brands:
ACE Bikes, Baja Designs, Barkbusters, Bumot, Cliff Top - Chargers,
Domino, Doubletake Mirrors, Flo Tool, Giant Loop Moto, HAD Originals,
Hepco Becker, HexEzCan, IntAct Batteries, Key Smart, Motone,
MotoSolutions, Nebo Tools, NGK Plugs, Nite Ize, NoNoise,
Own Your Adventure, Posi-Products, Quad Lock, Quixx, Rok Straps,
Rubba Tech, S100 and A1, SBV Tools, sPOD, True Utility, Vertex, Victor
Also corrected Maxima Racing Oils collection URL:
/maxima-racing-oils -> /maxima-oils (actual slug used on site)
Each entry commented with "already added" or "added 2026-07-07".
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Cache is now skipped if every product in the cached file has scrapeError
(prevents a 403-failed run from poisoning the next 24h of imports)
- If all scrapes fail, throw an error so the pipeline stops at step 1
instead of silently creating 13 empty Shopify products with no images/price
- Only successfully scraped batteries are included in allProducts passed
to the converter/download/upload/upsert stages
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Production server IP was being rejected by yuasabatteries.com as a bot.
Added full Chrome Sec-Fetch-* and Sec-CH-UA header suite (same fix applied
to retrorides earlier) so the server appears as a real browser navigation.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- scraper.js: fetches product detail pages from yuasabatteries.com for a configurable battery ID list; auto-discovers and downloads the current MSRP xlsx from the Yuasa MSRP page to build a battery-type → price map using the xlsx package; extracts specs, images, and vehicle fitment from HTML tables
- converter.js: converts scraped battery data to Shopify-ready format (title, SKU, price from MSRP, description from specs, images)
- index.js: standard pipeline integration with 24h cache (YUASA_CACHE_HOURS=0 to bypass); battery ID list configurable via YUASA_BATTERY_IDS env var (comma-separated)
- sources/index.js: registers yuasa as the 6th import source
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Production server IP triggered mod_security 409 even on sitemap fetch.
- Added Sec-Fetch-Dest/Mode/Site/User and Sec-CH-UA headers that
mod_security requires to recognise a real browser request
- Separate SITEMAP_HEADERS set for XML sitemap request (Accept: application/xml)
- fetchText now accepts optional headers override
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
HTTP 409 Conflict from the site WAF when requests run in parallel.
- 409 now retried with backoff (same as 429), up to 5 attempts at 5s increments
- Default concurrency changed from 3 to 1 (sequential fetches)
- RETRORIDES_CONCURRENCY env var to override if needed
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New source: retrorides (retrorides.co.in, 1168 products)
- Scraper reads all product URLs from WordPress sitemap in one call
- Extracts title/SKU/price/stock from JSON-LD, gallery images from data-large_image
- Bike brand/model from breadcrumb (case-insensitive BIKE_MAKES normalisation)
- Converts Jetpack CDN image URLs to direct retrorides.co.in URLs
- 24h cache with RETRORIDES_CACHE_HOURS env var, concurrency=3 default
- Converter follows same record.scraped pattern as motousher/dirtstreet
- Registered in sources/index.js alongside kyt, brocks-performance, motousher, dirtstreet
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pipelineJobs.js:
- cancelJob(jobId): marks job as cancelled=true, status=cancelling
- isJobCancelled(jobId): checked by the pipeline between stages
runSourcePipeline.js:
- PipelineCancelledError class
- checkCancelled() called before each of the 6 pipeline stages
- Accepts options.isCancelled() callback from the job runner
runKytPipelineJob.js:
- Passes isCancelled: () => isJobCancelled(job.id) into pipeline
- Catches PipelineCancelledError separately, sets status=cancelled
routes/pipeline.js:
- POST /pipeline/cancel/:jobId — marks job for cancellation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both motousher and dirtstreet converters were reading product fields
(title, sku, price, images) directly from the aggregated record, but
those fields live inside record.scraped after fetchWebsiteData wraps
them. Results were: Untitled Product, missing images, SKU=variant-1.
Also fixed brand priority: per-product brand (e.g. Evans Coolant,
SC Project) now takes precedence over the global SHOPIFY_BRAND env
var (KYT), which was incorrectly overriding all products from the
new sources.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both sources are now registered in sources/index.js and fully wired
into the 6-stage pipeline (fetch → download → watermark → upload →
convert → upsert). The frontend will automatically show them as tabs
via GET /pipeline/sources without any frontend changes needed.
motousher/ (Shopify JSON API — 12 brands, ~2,446 products):
- scraper.js: fetches /collections/{slug}/products.json + /products/{handle}.json
- converter.js: maps scraped products to standard pipeline format
- index.js: fetchWebsiteData() loops all brands, normalises to
productSummary.img format for shared download/upload utilities
- Supports MOTOUSHER_BRANDS env var to filter brands on a run
dirtstreet/ (WooCommerce HTML + JSON-LD — 5 brands, ~1,087 products):
- scraper.js: pure fetch, paginates /brand/{slug}/page/N/,
extracts price from offers.priceSpecification[0].price,
stock from JSON-LD availability field
- converter.js: maps scraped products to standard pipeline format,
builds descriptionHtml from body + short desc + attributes table
- index.js: fetchWebsiteData() loops all brands, normalises to
productSummary.img format
- Supports DIRTSTREET_BRANDS env var to filter brands on a run
sources/index.js: registered all 4 sources (kyt, brocks-performance,
motousher, dirtstreet). GET /pipeline/sources now returns all 4.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds two new experimental product scrapers under test_source/, isolated
from the active pipeline until verified and ready to promote.
motousher/ (Shopify store — Shopify JSON API):
- Scrapes 12 brands: All Balls Racing, DID Chains, EBC Brakes, Esjot
Sprockets, Evans Coolant, Grip Puppies, HiFlo Filters, JT Sprockets,
Maxima Racing Oils, Putoline, Ram Mount, Wunderlich
- 2,446 products total scraped and verified
- Uses /collections/{slug}/products.json + /products/{handle}.json
- Parallel fetch (concurrency 3), paginated collection listing
dirtstreet/ (WooCommerce store — HTML + JSON-LD):
- Scrapes 5 brands: SC Project, Evotech Performance, DNA Air Filters,
WRS, Zero Gravity Racing
- 1,087 products total scraped and verified
- Pure fetch with JSON-LD schema.org extraction (no browser)
- Handles paginated /brand/{slug}/page/N/ archives
- Price extracted from offers.priceSpecification[0].price
- Stock status derived from JSON-LD availability field
Both scrapers are standalone (node index.js), support --brand and
--limit flags, save per-brand JSON files and a combined.json.
Scraped data lives in data/sources/test_source/ (gitignored).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
