MOHAN 4e536f08b3 Add 24h cache to motousher and dirtstreet fetchWebsiteData ...

Subsequent pipeline runs within 24 hours reuse the existing
01_products_aggregated.json instead of re-scraping all brands,
eliminating redundant HTTP requests and 429 rate-limit retries.

Cache lifetime controlled per source:
  MOTOUSHER_CACHE_HOURS=0  → always re-scrape
  DIRTSTREET_CACHE_HOURS=0 → always re-scrape
  (default: 24h)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-06-04 16:44:38 +05:30

..

shared

Add multi-source import pipeline

2026-05-14 23:57:27 +05:30

sources

Add 24h cache to motousher and dirtstreet fetchWebsiteData

2026-06-04 16:44:38 +05:30

test_source

Add test_source scrapers for motousher.com and dirtstreet.in

2026-06-04 12:17:23 +05:30

runSourcePipeline.js

Add multi-source import pipeline

2026-05-14 23:57:27 +05:30