Autoencoder + classifier - chance level learning

I am implementing a neural net model that has the following architecture:

        # Encoder
        self.encoder = nn.Sequential(encoder_config)

        # Decoder
        self.decoder = nn.Sequential(decoder_config)

        # Classifier
        self.classifier = nn.Sequential(classifier_config)

The idea is to first train the autoencoder part of the model then train the classifier on the encoded representations. I have tried many different layer configurations and it turns out the autoencoder portion never learns anything. Before, training the autoencoder, I freeze the classifier as follows:

    def freeze(self, network):

        for p in network.parameters():
            p.requires_grad = False

This is done by calling model.freeze(model.classifier).

Here is an example of the latest layer configuration I used:

#Neural net layer config
encoder_config = OrderedDict({'lin1_encoder':nn.Linear(256, 216), 'norm1_encoder':nn.BatchNorm1d(216),
                              'sig1_encoder':nn.Sigmoid(), 'lin2_encoder':nn.Linear(216, 196),
                              'norm2_encoder':nn.BatchNorm1d(196), 'sig2_encoder':nn.Sigmoid()})
decoder_config = OrderedDict({'lin1_decoder':nn.Linear(196, 216),'norm1_decoder':nn.BatchNorm1d(216),
                              'sig2_decoder':nn.Sigmoid(), 'lin2_decoder':nn.Linear(216, 256),
                              'norm2_decoder':nn.BatchNorm1d(256), 'sig2_decoder':nn.Sigmoid()})
classifier_config = OrderedDict({'lin1_classifier':nn.Linear(196,128), 'norm1_classifier':nn.BatchNorm1d(128),
                                 'sig1_classifier':nn.Sigmoid(), 'lin2_classifier':nn.Linear(128, 32),
                                 'norm2_classifier':nn.BatchNorm1d(32), 'sig2_classifier':nn.Sigmoid(),
                                 'lin3_classifier':nn.Linear(32,2), 'norm3_classifier':nn.BatchNorm1d(2), 

I then train the autoencoder as follows (using data that has added Gaussian noise but even with 0 noise the model does not learn):

    def train_autoencoder(self, num_epochs, stimuli, batch_size, noise_factor, optimizer, criterion, scheduler, conv=False, verbose=False):

        running_loss = []
        test_loss = []
        device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

        # Load test and training datasets
        testloader = load_dataset_AE(stimuli=stimuli, batch_size=batch_size, noise_factor=0.0, conv=conv)
        trainloader = load_dataset_AE(stimuli=stimuli, batch_size=batch_size, noise_factor=noise_factor, conv=conv)

        for epoch in range(num_epochs):
            cur_running_loss = 0.0
            for i, (stimuli, target) in enumerate(trainloader):
                stimuli =
                target =
                # Train mode
                # Forward pass
                x = self.forward(stimuli, decode=True)
                loss = criterion(x, target)
                # Backward pass
                # Add loss of current pass
                cur_running_loss += loss.item()

            cur_running_loss /= len(trainloader)

            # Eval loss
            eval_loss = self.evaluate_AE(testloader, criterion)

            if verbose:
                print('Autoencoder --> Running Loss: {} \t Eval loss: {}'.format(cur_running_loss, eval_loss))

            # Scheduler

        return running_loss, test_loss

However I configure the encoder portion, it learns [0.500, 0.500, 0.500, … , 0.500] and decoder seems to learn to output the approximately the zero vector. The training data are binary vectors. I use torch.MSELoss() and torch.optim.Adam() as loss and optimizer. Weirdly enough, for vectors of length 256, I get a very low loss when training the autoencoder (~0.002). Given that the encoder learns nothing, the classifier categorizes at chance. Lastly, an SVM can learn to classify this data with near perfect accuracy so I doubt this is a problem with my data. Thanks for your help!