Hi everyone,
I’ve been building a training set for a region-sensitive classification task (retail product images across 4 markets), and since no public dataset covers the regional variants I need, I ended up collecting one myself. Wanted to share where I landed and get feedback from people who’ve done dataset construction at scale.
Setup: scraping product image URLs from regional e-commerce listings, collecting each market’s data from in-country IP vantage points (residential proxies with country targeting — otherwise the sites serve the US-catalog variant regardless of the URL), downloading images, then a preprocessing pass: resize, EXIF strip, dedup.
Current pipeline:
- Collect listing pages per market (~2k pages/market/week)
- Extract image URLs + metadata with BeautifulSoup
- Download with a rate-limited asyncio pool
- Perceptual hashing (imagehash, pHash) for near-duplicate removal
- Manual audit of a 5% sample for label noise
Where I’d appreciate feedback:
- Dedup threshold: pHash hamming distance ≤ 8 catches exact/near dupes, but I worry about over-merging legitimately similar variants (same product, different angle). Anyone found a good balance, or a better approach than pHash for this?
- Diversity vs. volume: markets have wildly different catalog sizes, so the dataset is imbalanced (US 40%, others 15% each). Re-weighting in the sampler, or collect more from smaller markets until balanced?
- Train/test leakage: same product can appear in multiple markets’ data. I dedupe by product ID across splits, but visually-similar-but-different products still leak style-level information. Has anyone quantified how much this matters in practice?
What surprised me so far: the collection infrastructure mattered more than I expected — early experiments with a single-region vantage point produced noticeably worse transfer to other markets, which I only caught because I’d kept the per-market provenance in the metadata. Would recommend anyone doing multi-region datasets record collection origin per sample, even if you think you won’t need it.
Not here to promote anything — genuinely looking for feedback from the dataset-construction trenches. Thanks in advance.