I’m working on a long-running pipeline that collects public image-text and video-text records before they are normalized for training.
The model code is fairly straightforward, but the data loader needs to handle intermittent failures, duplicate records, retries, and different response sizes. I’m considering a design with:
- a durable queue for collection jobs
- bounded retries with failure reasons
- a raw-response store separate from processed samples
- deterministic dataset manifests
- region-specific workers when content varies by location
For PyTorch projects, what patterns have you found useful for making this kind of input pipeline observable and restartable? I’m particularly interested in how people keep a failed batch from blocking the entire training or evaluation run.