Very Strange output with Conv2D and ReLU

Hi, I’m trying to build a model for monocular depth prediction using a DenseNet inspired architecture. DenseNet inspired meaning that there are sections of the network that are comprised of dense blocks of 2d convolutions, specifically 1x1 conv->ReLU->3x3 conv ->ReLU.

def BNBlock(inputSize,stride):
			return nn.Sequential(
					nn.Conv2d(inputSize,inputSize*stride,kernel_size = 1,stride = stride, padding = 0, bias = False),
					nn.ReLU(inplace=False),
					nn.Conv2d(inputSize,inputSize,kernel_size = 3,stride = 1, padding = 1, bias = False),
					nn.ReLU(inplace=False)
					)

def DenseBlock(numLayers, inputSize, stride):
			layers = []
			for i in range(numLayers):
				layers.append(BNBlock(inputSize,stride))
			return(nn.Sequential(*layers))

The problem I’m having is that when I run data through these dense blocks, no matter how dense it is, the output comes out either nearly entirely zeros or nearly entirely a singular value. Does anybody know why this could be?

Important things to note:

  1. The input to the network is normalized between 0-1
  2. I’ve tried replacing the ReLU layers with LeakyReLU, thinking that it could have fixed the “dying ReLU” problem, but that didn’t work either.

Any insight as to what’s happening here would be greatly appreciated!

I get an output containing approx. 50% zeros using randn inputs, which seems to be expected:

model = DenseBlock(5, 3, 1)
x = torch.randn(8, 3, 24, 24)
out = model(x)
print((out == 0.).float().sum() / out.nelement())
> tensor(0.4957)