Extracting latent space after training

1

I built a DCGAN to generate images of alphanumeric characters from different font styles.
After training, I would like to work with the latent space (i.e. the input of the generator).
How do I do this with Pytorch?
Here is the architecture:

class Generator(nn.Module):
    def __init__(self, ngpu):
        super(Generator, self).__init__()
        self.ngpu = ngpu
        self.main = nn.Sequential(
            # input is Z, going into a convolution
            nn.ConvTranspose2d( nz, ngf * 8, 4, 1, 0, bias=False),
            nn.BatchNorm2d(ngf * 8),
            nn.ReLU(True),
            # state size. (ngf*8) x 4 x 4
            nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 4),
            nn.ReLU(True),
            # state size. (ngf*4) x 8 x 8
            nn.ConvTranspose2d( ngf * 4, ngf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 2),
            nn.ReLU(True),
            # state size. (ngf*2) x 16 x 16
            nn.ConvTranspose2d( ngf * 2, ngf, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf),
            nn.ReLU(True),
            # state size. (ngf) x 32 x 32
            nn.ConvTranspose2d( ngf, nc, 4, 2, 1, bias=False),
            nn.Tanh()
            # state size. (nc) x 64 x 64
        )

    def forward(self, input):
        return self.main(input)
2

What exactly would you like to do?
You could simply sample nz from a (gaussian) distribution.
Or would you like to interpolate etc.?

3

Thanks for the prompt response @ptrblck
I want to use it for visualization purposes. Namely, apply t-SNE projection of the latent space representations from my dataset to visualize clusters.

4

Which latent space representations would you like to visualize?
In the DCGAN example you are sampling the generator input from a gaussian distribution.

Or would you like to use t-SNE to visualize the feature layer of the discriminator?

5

Sorry, I am probably getting concepts confused. I know that in the realm of autoencoders, there lies a bottleneck between the decoder and encoder; the encoder, when is fed with an input image, brings the data into a compressed representation of the image, i.e. the latent space representation.
Isn’t there such a bottleneck in GAN’s as well? This is what I want to extract.

6

You are right about autoencoders and the architecture with the latent variables in the bottleneck.
However, in the DCGAN setup you have basically a generator (G) and a discriminator (D).
The input to G might vary based on your current setup. One option is to feed in random noise vectors and let G generate fake images using transposed convolutions. You could also feed in other images and learn to transform these images, but let’s stick to the DCGAN example here.
D on the other side is often a model which might be used in a classification setup, i.e. it gets images as input and tries to classify fake and real images.

Back to your original question: you might try to visualize the input tensor to G in relation to G’s output, if I understand your use case correctly. Even though you are randomly sampling the input, you’ll see that the output is smoothly generated, if you use a reasonable interpolation method, which means that certain “areas” of your multidomensional gaussian create certain features in the images. I’m not sure, if t-SNE is the best method to do this, so let’s wait for other people to give some feedback on this. :wink:
I guess you’ll

7

hi @ptrblck! I actually want to do something sort of similar - using a trained DCGAN, I would like to give the encoder a new image and visualize what the decoder’s output would look like if I changed one latent variable and kept the rest constant.

Using the code from the official [DCGAN (Pytorch)], (https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html) I’m wondering how to interpolate the latent space. Any help appreciated!

8

In the tutorial you are dealing with 1000-dimensional latent vectors as far as I know, so an interpolation technique like slerp should work quite well.
You could simply clone the latent variabe, change one feature to a new value of the copy, and interpolate between the two vectors.