I’d keep the segments in a separate manifest (one row per segment, keyed by video ID, start/end in seconds as floats) and build an interval index per video at Dataset init, so each sampled window is just an overlap query. For overlaps I keep every segment that intersects the window, clip its times to the window, and store them relative to window start; dropping or merging them in preprocessing tends to bite you later. Tokenize text offline once, but do the window-to-segment matching in __getitem__, since that’s where your sampling strategy changes most. For batching, fixed-length clips plus a padded token tensor with a segment-id/time tensor and masks in collate_fn keeps things simple. (I’m with aicost.tools, just sharing what has worked for me.)
This is a very helpful pattern. Keeping one row per segment in a separate manifest makes the temporal annotations much easier to inspect and version.
I also like keeping the window-to-segment matching in __getitem__. That preserves flexibility when changing the sampling strategy, while offline tokenization avoids repeating text processing for every clip.
For segments that cross the sampled window boundary, clipping the segment times to the window seems safer than merging them, especially when the same transcript segment may be relevant to multiple windows.
How do you handle windows with no matching segments? Do you drop them, return an empty text mask, or use them as negative samples?
I keep those windows instead of dropping them. A clip with no overlapping segment is a valid negative: return an empty token row and a text mask of all zeros so the caption or contrastive loss can ignore it, or treat it as no-speech if your labels include silence. Dropping them biases the loader toward dense dialogue and hides how the model behaves on quiet footage. If the empty rate gets very high, filter only at sampling time with a minimum-overlap threshold you can turn off for eval, rather than deleting rows from the manifest.