Test accuracy remains the same even though loss is changing

vision
1

Hello as you can see below, my model always has the same test accuracy, any kind of help would be nice.

Epoch: 1 | train_loss: 233082867353.8990 | train_acc: 0.1761 | test_loss: 1.7912 | test_acc: 0.1790
Epoch: 2 | train_loss: 1.7914 | train_acc: 0.1781 | test_loss: 1.7912 | test_acc: 0.1790
Epoch: 3 | train_loss: 1.7917 | train_acc: 0.1763 | test_loss: 1.7909 | test_acc: 0.1790
Epoch: 4 | train_loss: 1.7915 | train_acc: 0.1760 | test_loss: 1.7913 | test_acc: 0.1618
Epoch: 5 | train_loss: 1.7912 | train_acc: 0.1756 | test_loss: 1.7912 | test_acc: 0.1790
Epoch: 6 | train_loss: 1.7915 | train_acc: 0.1752 | test_loss: 1.7910 | test_acc: 0.1790
Epoch: 7 | train_loss: 1.7915 | train_acc: 0.1756 | test_loss: 1.7910 | test_acc: 0.1790
Epoch: 8 | train_loss: 1.7915 | train_acc: 0.1769 | test_loss: 1.7911 | test_acc: 0.1790
Epoch: 9 | train_loss: 1.7916 | train_acc: 0.1747 | test_loss: 1.7909 | test_acc: 0.1790
Epoch: 10 | train_loss: 1.7914 | train_acc: 0.1748 | test_loss: 1.7909 | test_acc: 0.1790

class KAMYOnet(nn.Module):
    def __init__(self, input_shape: int, output_shape: int):
        super().__init__()
        self.hidden_units = 128
        num_blocks = 2
        self.blocks = nn.ModuleList()
        
        self.blocks.append(nn.Sequential(
            nn.Conv2d(in_channels=input_shape,
                      out_channels=self.hidden_units,
                      kernel_size=3,
                      stride=1,
                      padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2)
        ))
                
        for j in range(num_blocks):
            self.hidden_units += 128
            self.block1()
            self.hidden_units += 128
            self.block1()
            self.blocks.append(nn.Sequential(nn.MaxPool2d(kernel_size=2, stride=2)))
            

        self.blocks.append(nn.Sequential(
            nn.Conv2d(in_channels=self.hidden_units,
                      out_channels=output_shape,
                      kernel_size=3,
                      stride=1,
                      padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2)
        ))
                

        # Dynamically calculate the input size for the linear layer
        with torch.no_grad():
            x = torch.randn(1, input_shape, 64, 64)
            self.weightedAvgNum = 0
            
            for block_idx, block in enumerate(self.blocks):
                for layer_idx, layer in enumerate(block):
                        x = layer(x)
                    #print(f"Block {block_idx}, Layer {layer_idx}: Output shape: {x.shape}")
                
        # Flatten the output tensor before passing it to the linear layer
        x = torch.flatten(x, start_dim=1)
        
        self.classifier = nn.Linear(x.size(1), output_shape)
    
    def block1(self):
        self.blocks.append(nn.Sequential(
            nn.Conv2d(in_channels=(self.hidden_units-128),
                        out_channels=self.hidden_units,
                        kernel_size=3,
                        stride=1,
                        padding=1),
            nn.ReLU(),
            nn.Conv2d(in_channels=(self.hidden_units),
                        out_channels=self.hidden_units,
                        kernel_size=3,
                        stride=1,
                        padding=1),
            nn.ReLU(),
        ))

    def forward(self, x: torch.Tensor):
        for block in self.blocks:
            for layer in block:
                    x = layer(x)
                    
        x = self.classifier(x.view(x.size(0), -1))  # Flatten before passing to linear layer
        return x

model = KAMYOnet(input_shape=3, output_shape=6).to(device)

criterion = nn.CrossEntropyLoss()

optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

loss_fn = nn.CrossEntropyLoss()
2

The initial loss seems to be quite large. Did you check the first output(s) and target(s) to see why the loss seems to be to large?

1 Like
3

This is what I got. I am still not sure why it is too large

Outputs: tensor([[-0.1163, -0.0291, -0.0039,  0.0696, -0.0227,  0.0322]],
       device='cuda:0', grad_fn=<AddmmBackward0>)
Targets: tensor([5], device='cuda:0')
Epoch: 1 | train_loss: 1004291905552786522112.0000 | train_acc: 0.1732 | test_loss: 1.7910 | test_acc: 0.1790
5

I cannot reproduce the large loss from this batch as the outputs also contain values in the expected range:

output = torch.tensor([[-0.1163, -0.0291, -0.0039,  0.0696, -0.0227,  0.0322]])
targets = torch.tensor([5])

criterion = nn.CrossEntropyLoss()
loss = criterion(output, targets)
print(loss)
# tensor(1.7495)

Are you using the latest PyTorch version? If not, could you update and rerun your code?
In an older release some error checks were broken (E.g. 1.13 as described here) and your loss values might indeed be caused by accessing invalid values and thus uninitialized memory.