How Do You Handle Incremental Video Dataset Updates in PyTorch?

vision
1

I am designing a video training pipeline where new samples arrive periodically instead of replacing the entire dataset.

The update process may include:

  • New video records
  • Updated metadata or captions
  • Removed or unavailable samples
  • Dataset and schema versions
  • Reproducible train/validation/test splits

For PyTorch users, how do you usually handle incremental dataset updates?

  • Do you create a new immutable manifest for every training run?
  • How do you prevent removed samples from remaining in cached indexes?
  • Do you rebuild the Dataset/DataLoader after each release?
  • How do you keep sample IDs stable across versions?
  • Do you store dataset diffs separately from the full manifest?

I am mainly interested in practical patterns for keeping training runs reproducible while avoiding a full reprocessing job for every small update.

2

I’d keep every release as an immutable manifest keyed by stable sample IDs, and treat deletes as tombstones rather than rewriting the previous file. Rebuild Dataset/DataLoader indexes from the new manifest only — never mutate a cached index in place — so a removed clip can’t linger under an old offset. Store a small diff (added/updated/removed IDs) next to the full manifest so you can audit a run without reprocessing the whole tree. (I’m with aicost.tools; mostly looking at this from the data-prep cost side.) How are you keeping train/val/test membership stable when a sample’s caption changes but its ID stays the same?