Active-speaker-driven video cropping: what the inference actually cost

vision
1

Sharing a local video pipeline and, more usefully, the two inference lessons
that cost me the most time.

The task: crop a 16:9 stream to 9:16 following whoever is speaking. YOLOv8 for
people, OpenCV for the crop path, TalkNet-ASD to pick the speaker from the
tracked faces.

  1. The model was never the expensive part. Feeding it was. On an RTX 3060,
    for a 60-second clip:

    tracking at 8fps                          62.3s
TalkNet forward pass, two people           1.2s
frame reads + crops reusing 8fps boxes     8.4s   (+15% overall)
re-detecting with YOLO at 25fps          280.2s   (+452% overall)

    Same output either way. Reusing boxes the tracker already produced, rather
    than re-running detection at the ASD’s frame rate, is the entire difference.

  2. Ultralytics inference is not thread-safe on a single model instance. The
    renders run in parallel and this surfaced as corrupted detections rather
    than an exception, which made it slow to find. The shared model now sits
    behind a lock. OpenCV’s detectMultiScale has the same problem on a shared
    cascade, so those are per-thread.

Also worth knowing if you do global video passes: making OpenCV walk every
frame to sample 1fps costs about ten minutes on a 30-minute video, because
H.264 frames depend on previous ones. Handing the decode to FFmpeg and having
it emit only the sampled frames, already downscaled and grayscale, straight
into numpy is roughly 20x faster for the same signals.

The clip-selection side is a plug-in point: signals are arrays over the
video’s timeline, percentile-ranked, and peaks become candidates. If you have
a detector for content this was not tuned for, it slots in there.