Ok, I give up.
I am trying to get a 2D convolutional autoencoder working with non image based data. The data is of size[-1,1,264,132].
The loss function get stuck at around 430, which means that some of the data is encoded, however some is not. I have tried playing with the hyper parameters on ADAM, and found that if i drop the LR to below 5e-4 the it gets a loss function Nan. I have tried different loss functions and nothing works (using the standard KLD value the error still gets stuck just at different non 0 values). The one i am currently using is:
def log_normal_pdf(sample,mean,logvar):
log2pi=torch.tensor(np.log(2.*np.pi))
return torch.sum(-0.5*((sample-mean)**2.*torch.exp(-logvar)+logvar+log2pi),1)
def loss_fn(Gen_model, data,lab):
mean, logvar = Gen_model.encode(data,lab)
z2=Gen_model.reparm(mean, logvar)
out=Gen_model.decode(z2,lab)
criterion = torch.nn.BCEWithLogitsLoss(size_average=True,reduce=False, reduction='mean')
BCE=criterion(out,data)
logpx_z=-torch.sum(BCE,(1,2,3),keepdim=False)
logpz=log_normal_pdf(z2,torch.tensor(0.),torch.tensor(1.))
logqz_x=log_normal_pdf(z2, mean, logvar)
return -torch.mean(logpx_z+logpz-logqz_x),out
I have included the reparamatization
def encode(self,data,lab):
data=self.Encoder(data,lab)
mean, logvar=torch.chunk(data,2,dim=1)
return mean, logvar
def reparm(self, mean, logvar):
epsilon = torch.randn(mean.size()).to(device)
z = epsilon*torch.exp(logvar.mul(0.5))+mean
#print("z=",z.shape)
return z
I now i have got something wrong (obviously), but i cannot see it. I read somewhere that the the input data should be normalised with a mean of zero and an std of 1, however this cannot happen because the BCE function needs data between [0,1].
The raw data (before normalising) ranges from +1000 to -1000, normalising this data sets the pre normalised 0 to 0.5, the -1000 to 0 and +1000 to 1. the increments can therefore be very small, am i right in thinking that this maybe causing the issue??? if so, would it be better to create an input data set of [-1,3,264,132]?