I have sequence of series connection of “conv(3x3)-norm-leakyrelu”-“dialated_conv(3x3)-norm” with short skip connection(like Resnet) followed by last layer of “conv(1x1)-tanh”. I train model for only 2 images which gives below histogram of weight and gradients. One loss out of 2 is stuck after some epochs. All the layers before “layer_10” has histogram like “layer_10” which i think these layers are not learning. I use the xavier initialisation with initial gain 0.15. Also note the values of weight and gradients on x-axis. I think i am missing something after looking in histogram.
