Small gradient problems

[CJS (Conor Sheehan)]

Solved: Using softmax with a large number of layers was leading to overflow. Added a batch normalisation layer before the softmax.

Problem using softmax

here is the Variable before softmax [image] [image] after softmax, the biggest one of them became 1 and others became 0 [image] my forward function def forward(self, x): x = F.relu(self.conv1(x)) x = F.relu(self.conv2(x)) x = F.relu(self.conv3(x)) x = F.relu(self.conv4(x)) x = F.relu(self.conv5(x)) x = F.relu(self.conv6(x)) x = F.relu(self.conv7(x)) x = x.view(x.size(0), -1) x = F.softmax(x) return x is there something wrong in my usage of softmax…