So it turns out that when using torch.functional.dropout(layer, p=1), the default value for training=False, which did not use dropout each time hence it makes sense the network would learn.
I’ve now set it to take torch.functional.dropout(layer, p=1, training=self.training) and now the output of this call is a fully zeroed out tensor, and the network no longer performs.
Sorry for the misunderstanding! I assumed it worked like the torch.nn.dropout which uses self.training by default.