I am working on a long-running pipeline that collects public image-text and video-text records before they are normalized for training.
The model code is straightforward, but the data pipeline needs to handle intermittent failures, duplicate records, retries, different response sizes, and occasional changes in the source format.
The current design I am considering has:
- a durable queue for collection jobs
- bounded retries with explicit failure reasons
- a raw-response store separate from processed samples
- deterministic dataset manifests
- region-specific workers when content varies by location
- a validation stage before records are exposed to the training loader
For PyTorch projects, what patterns have you found useful for making this kind of pipeline observable and restartable?
In particular:
- How do you prevent one failed batch from blocking the entire training or evaluation run?
- Do you keep failed records in the same dataset manifest or in a separate retry queue?
- How do you expose data freshness and source drift to the training job?
- Are there PyTorch DataLoader or TorchData patterns that work well for this setup?
I am mainly looking for production lessons around checkpointing, backpressure, and reproducibility rather than a specific vendor or scraping tool.