CNN Image Classification

I am trying to build image classification with CNN. It did first with Restnet18 which works. But when i try to make my own CNN model the outputs are all same.

This is my modal:

class Network(nn.Module):

def __init__(self):

    super(Network,self).__init__()

    self.cn1=nn.Conv2d(3,16,3,1,1)

    self.cn2=nn.Conv2d(16,32,3,1,1)

    self.cn3=nn.Conv2d(32,64,3,1,1)

    self.cn4=nn.Conv2d(64,128,3,1,1)

    self.pool=nn.AvgPool2d(2,2)

    self.fc1=nn.Linear(128*14*14,1024)

    self.fc2=nn.Linear(1024,512)

    self.fc3=nn.Linear(512,256)

    self.fc4=nn.Linear(256,2)

def forward(self,x):

    x= F.relu(self.pool(self.cn1(x)))

    x = F.relu(self.pool(self.cn2(x)))

    x = F.relu(self.pool(self.cn3(x)))

    x = F.relu(self.pool(self.cn4(x)))

    

    x=x.view(-1,128*14*14)

    

    x=self.fc1(x)

    x=self.fc2(x)

    x=self.fc3(x)

    x=self.fc4(x)

    x=F.log_softmax(x,dim=1)

    return x

Batch size is 20. So then i checked the it is giving same output for all the images in the batch.

I would recommend to add non-linearities between the linear layers and try to overfit a small data sample (e.g. just 10 samples) by playing around with the hyper parameters, such as the learning rate.
Once your model can successfully overfit the small data sample, you could try to scale up the problem again.