Same output for all input value

1

Hi there

I have tained an 1D Autoencoder NN, which you can find the its details below:

Autoencoder(
(encoder): Sequential(
(0): Conv1d(1, 5, kernel_size=(5,), stride=(2,))
(1): MaxPool1d(kernel_size=3, stride=1, padding=0, dilation=1, ceil_mode=False)
(2): ReLU(inplace=True)
(3): Conv1d(5, 10, kernel_size=(5,), stride=(2,))
(4): MaxPool1d(kernel_size=3, stride=1, padding=0, dilation=1, ceil_mode=False)
(5): ReLU(inplace=True)
(6): Conv1d(10, 15, kernel_size=(5,), stride=(2,))
(7): MaxPool1d(kernel_size=3, stride=1, padding=0, dilation=1, ceil_mode=False)
(8): ReLU(inplace=True)
(9): Conv1d(15, 20, kernel_size=(4,), stride=(1,))
(10): ReLU(inplace=True)
)
(decoder): Sequential(
(0): ConvTranspose1d(20, 15, kernel_size=(1,), stride=(4,))
(1): ReLU(inplace=True)
(2): ConvTranspose1d(15, 10, kernel_size=(2,), stride=(4,))
(3): ReLU(inplace=True)
(4): ConvTranspose1d(10, 5, kernel_size=(9,), stride=(2,))
(5): ReLU(inplace=True)
(6): ConvTranspose1d(5, 1, kernel_size=(10,), stride=(2,))
(7): ReLU(inplace=True)
)
)

my loss decrease like 1/f function however, when I feed my NN with some data, I get almost same output for different input. Has any one had any similar experience before?
I should mention that my training sample was 65536 signal each has length of 94 points.
When I said feeding my data I meant I did this below:

for i in range(65536) :
out_put[i] = model(data_pixel2[i].unsqueeze(0))

which here data_pixel2 is my input.
Appreciate any comments.

2

Could you try to remove the last ReLU from your decoder and rerun the code again?
I’ve seen similar issues, where the last non-linearity was messing with the training procedure.

1 Like
3

thank you Patrick however it became worse than what it was before. Don’t you think if there would be a problem with the way that I normalized my input data? below you can see how I did that:

new_data = np.empty((256,256,94))
for i in range (256) :
for j in range (256) :
for k in range (94) :

  new_data[i,j,k] = ((data[i,j,k])-(mean))/(max-min)

data is a tensor that I stored my signal there and new data was used as input of my autoencoder

Regards

4

How did you calculate mean, min, and max?
Usually you would normalize the input signals using the channel stats, but your use case might be different.

5

thank you for your reply.
I did it in the below way:

mean = np.mean(data)

max = np.amax(data)

min = np.amin(data)
new_data = np.empty((256,256,94))
for i in range (256):
for j in range (256):
for k in range (94):
new_data[i,j,k] = (data[i,j,k]-mean)/(max-min)

where data is my input that has the shape of (256,256,94), its a tiff file.

6

Since you are calculating the global statistics, you could simply use:

data = (data - mean) / (max - min)

Did you play around with some hyperparameters, e.g. the learning rate and checked, if this would change the behavior?
If that doesn’t help, I would recommend to try to overfit your model on a small dataset samples, e.g. just 10 samples, and make sure your model predicts these samples perfectly.
If that doesn’t work, there might be another error in the code I haven’t found yet.

1 Like
7

Yes I did, things didn’t changed significantly even at some sets of hyper parameter it became worse.
Sure, I’m going to try to overfit my model with just a few example.
would u please take a look at my training section to see if I am feeding my NN correctly?

learning_rate = 1e-5

num_epochs = 360

model = Autoencoder().cuda()

criterion = nn.MSELoss()

optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate,

weight_decay=1e-5)

#training loop
loss = 0
epoch_num = 0
error = []
for epoch in range(num_epochs):
for data in train_loader:
img = data
img = Variable(img).cuda()

output = model (img)
loss = criterion (output,img)

optimizer.zero_grad()
loss.backward()
optimizer.step()
error.append(loss.item())


if epoch%10 == 9:
  epoch_num += epoch_num
  plt.plot(error)


  print ('\r Train Epoch : {}/{} \tLoss : {:.4f}'.format (epoch+1,num_epochs,loss/32))

model_save_name = ‘Autoencoder 09 Jan 2020 2nd with new normalzation 2nd trial.pt’
path = F"/content/drive/My Drive/{model_save_name}"
torch.save(model.state_dict(), path)
plt.plot(error)

I have to recall you thay my input size is a tensor of (65536,1,94) shape, which 65536 is number of signals 1 is number of channel and 94 is length of each signal.

Kind Regards,

8

I cannot see any obvious mistakes, besides the usage of the deprecated Variable, so you can just remove it.

Did you overfit the small sample successfully?

1 Like
9

Thank you for your time
Actually I tried that one as well, however got similar results. same output for all different input values,

:frowning:

10

I generated some artificial data to see if anything changes,
although generated data was quite different from my actual data, surprisingly I got the same out out, quite similar to what I have had with actual data :open_mouth:

1 Like
11

Did you get a resolution?