MOHAN 4e536f08b3 Add 24h cache to motousher and dirtstreet fetchWebsiteData ...

Subsequent pipeline runs within 24 hours reuse the existing
01_products_aggregated.json instead of re-scraping all brands,
eliminating redundant HTTP requests and 429 rate-limit retries.

Cache lifetime controlled per source:
  MOTOUSHER_CACHE_HOURS=0  → always re-scrape
  DIRTSTREET_CACHE_HOURS=0 → always re-scrape
  (default: 24h)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-06-04 16:44:38 +05:30

routes

Add multi-source import pipeline

2026-05-14 23:57:27 +05:30

src

Add 24h cache to motousher and dirtstreet fetchWebsiteData

2026-06-04 16:44:38 +05:30

.gitignore

Add multi-source import pipeline

2026-05-14 23:57:27 +05:30

auth.js

first commit

2026-04-13 17:31:26 +05:30

fulfillmentService.js

first commit

2026-04-13 17:31:26 +05:30

logger.js

first commit

2026-04-13 17:31:26 +05:30

package-lock.json

Add multi-source import pipeline

2026-05-14 23:57:27 +05:30

package.json

Install Playwright browser for backend

2026-05-15 00:25:40 +05:30

server.js

Fix typo in health check service name

2026-04-14 14:23:00 +05:30

tokenStore.js

first commit

2026-04-13 17:31:26 +05:30

Description

No description provided

220 KiB

Languages

JavaScript 100%