Hi, I’m building a generator
g, that receives a latent-code (vector of shape 100) and outputs an image. Specifically, I have 1000 MNIST images, and I want the network to learn a latent code
z_i for each image
x_i, such that
g(z_i)=x_i (this approach is known as Generative Latent Optimization). So I’ve used nn.Embedding(1000,embedding_dim=100) and standard generator architecture, that receives the code from embedding and outputs an image. As for loss, I combine reconstruction loss with regularization on the embedding-vector weights.
My probelm is: I’d like to add noise to the latent-code vector before it is inserted to the generator (in order to make the latent-code compact). However I’m a beginner, and I don’t know whether I should call detach() when adding the noise or not. I’m not sure of my approach entirely. I don’t want to learn the scale of the noise or anything.
Here’s my attempt:
class net(nn.Module): def __init__(): self.embed = nn.Embedding(1000,embedding_dim=100) self.generator = nn.sequential( nn.Linear(100, 84), .... ) def forward(batch_indices): batch_codes = self.embed(batch_indices) noise = torch.randn_like(batch_codes) * sigma noisy_batch_codes = batch_codes + noise # SHOULD THIS BE batch_codes.detach() + noise ?? return self.generator(noisy_batch_codes) g = net() optim = SGD(g.parameters(), lr=0.01) for epoch in range(num_epochs): for orig_images, orig_images_idx in trainloader: optim.zero_grad() output = g(orig_images_idx) reconstruction_loss = nn.MSELoss()(output, orig_images) embed_vector_weights = g.embed.weight[orig_images_idx] reg_loss = torch.norm(embed_vector_weights) * reg_coeff loss = reconstruction_loss + reg_loss loss.backward() optim.step()