Hi, I am trying to implement a model of which the loss function depends on a pair of training data. Specifically, given dataset D={x_i}, the loss function will be E(f(x_i), f(x_j)), where f is the neural network model. I wonder if there is an efficient way to generate a stream of random pairs {x_i, x_j} for training?
Currently I have two following approaches in mind:
-
preprocess D to get the complete pair set containing N(N-1)/2 pairs (N is number of data in D) and then feed them into the model. The problem is the memory usage (even at preprocessing stage) scales with N^2 which is not tractable.
-
sample {i,j} index at runtime for each pair, and manually pass the index to D (without using
DataLoader) to get data for each batch. But it introduces significant overhead and can be inefficient.
I am wondering if there exists some sort of “generator” that could handle this sampling problem efficiently. Any suggestion will be appreciated. Thanks!