Generate a stream of random pairs for training


(Wei Chen) #1

Hi, I am trying to implement a model of which the loss function depends on a pair of training data. Specifically, given dataset D={x_i}, the loss function will be E(f(x_i), f(x_j)), where f is the neural network model. I wonder if there is an efficient way to generate a stream of random pairs {x_i, x_j} for training?

Currently I have two following approaches in mind:

  1. preprocess D to get the complete pair set containing N(N-1)/2 pairs (N is number of data in D) and then feed them into the model. The problem is the memory usage (even at preprocessing stage) scales with N^2 which is not tractable.

  2. sample {i,j} index at runtime for each pair, and manually pass the index to D (without using DataLoader) to get data for each batch. But it introduces significant overhead and can be inefficient.

I am wondering if there exists some sort of “generator” that could handle this sampling problem efficiently. Any suggestion will be appreciated. Thanks!