I was wondering if you put an embeddings Neural network in your dataloader without freezing weights, will back-propagation modify the weights of that embeddings network?
As long as these embeddings are part of the computation graph on which
backward was called, they will get their gradient assigned (check
embeddings.weight.grad after the
backward call). To modify these weights, either you have to manually update them or you can pass them to the optimizer that will do the update for you (
embeddings.weight is an instance of
model.parameters() returns them as well)
Note though that the Dataloader uses multiprocessing for loading when you use >0 workers. And autograd won’t track across these.
So in general, this might be tricky to get working if you use multiple workers for loading.