I’m sketching out a pipeline to flag near-duplicate drafts pulled from an external browser-based image generation source (in my case, Nano Banana 2 Lite, a third-party independent tool site unrelated to Google or DeepMind). When exploring ad concepts or storyboard variations there, I end up with dozens of drafts per session, many visually very similar, and I’d like an automated way to cluster or deduplicate them before manual review.
Model choice
Has anyone compared torchvision’s pretrained ResNet50 penultimate-layer features against something like a CLIP visual encoder for this kind of perceptual similarity task? I suspect CLIP embeddings might be more robust to compositional variation, but I’m unsure how much preprocessing consistency matters (resize strategy, center-crop vs letterbox, normalization stats).
Similarity thresholds and batching
For cosine similarity, is there a reasonably principled way to pick a threshold rather than eyeballing it, especially when drafts differ mainly in lighting, color grading, or minor object placement rather than structural content? Also curious about efficient batching strategies when embeddings need to be computed incrementally as new drafts arrive, rather than all at once.
Storing embeddings
For persistence, is it reasonable to just store L2-normalized embedding tensors alongside image hashes in a simple flat file or SQLite table, or does this quickly become a case for an approximate nearest neighbor index once the dataset grows past a few thousand images?
I haven’t built anything production-ready yet, just experimenting with feature extraction and similarity scoring locally. Any pointers on preprocessing pitfalls or embedding storage patterns would be appreciated.