Hi folks,
I’m trying to write a model where each image is made up of pixels and each pixel associated with a latent vector. This quickly explodes the memory usage of the embedding matrix since it grows like (number of images x number of pixels x latent dimensionality) which quickly exceeds GPU capacities. However, updates are sparse: only the vectors for images in a given minibatch are updated. Is there a drop-in solution like a “file-backed” embedding matrix that is cached only in memory, instead of being entirely in memory? Or is there a another recommended technique here?
Thanks!
chris moody