Can't overfit after small change in data


I’m using the following repo:GitHub - OshriHalimi/shape_completion for partial shape reconstruction. Their architecture is Encoder-Decoder, where the encoder is 3 conv1 layers with relu between them and linear as final layer, the decoder is 5 conv1 layers.
When I make an overfit test, I give it as input always the same partial shape with the desired reconstruction, it works as expected, and after about 30 epochs there are nice results. when I make simple change of deleting about 20% of the mesh vertices(and complete those missing with coping random vertices until I get full number of vertices again), the network does not learn anything, although the completion is still rather pretty simple.
The loss function is MSE between the output mesh and desired one, with ADAM optimizer.

I will appreciate any help to understand the reason for this failure.