CVAE: Kernel size can't be greater than actual input size

I am trying to run a Convolutional VAE using the following model. My input images are single channel images of size 32x32. The input tensor size is [32, 1, 32, 32] since the batch size is 32. Why am I getting this error, the example code is supposed to run on 3-channel 32x32 images [32, 3, 32, 32]. I have a total of 499 images. How do I fix this?

The model’s specification:
DCVAE(
(encoder): Sequential(
(0): Conv2d(1, 32, kernel_size=(4, 4), stride=(2, 2))
(1): LeakyReLU(negative_slope=0.2)
(2): Conv2d(32, 64, kernel_size=(4, 4), stride=(2, 2))
(3): LeakyReLU(negative_slope=0.2)
(4): Conv2d(64, 128, kernel_size=(4, 4), stride=(2, 2))
(5): LeakyReLU(negative_slope=0.2)
(6): Conv2d(128, 256, kernel_size=(4, 4), stride=(2, 2))
(7): LeakyReLU(negative_slope=0.2)
(8): Flatten()
)
(encoder_mean): Linear(in_features=1024, out_features=32, bias=True)
(encoder_logvar): Linear(in_features=1024, out_features=32, bias=True)
(fc): Linear(in_features=32, out_features=1024, bias=True)
(decoder): Sequential(
(0): UnFlatten()
(1): ConvTranspose2d(1024, 128, kernel_size=(5, 5), stride=(2, 2))
(2): ReLU()
(3): ConvTranspose2d(128, 64, kernel_size=(5, 5), stride=(2, 2))
(4): ReLU()
(5): ConvTranspose2d(64, 32, kernel_size=(6, 6), stride=(2, 2))
(6): ReLU()
(7): ConvTranspose2d(32, 1, kernel_size=(6, 6), stride=(2, 2))
(8): Sigmoid()
)
)

The error message:
RuntimeError: Calculated padded input size per channel: (2 x 2). Kernel size: (4 x 4). Kernel size can’t be greater than actual input size

As the error message claims, your output activation is too small for the last conv layer.
Inline are the output activation shapes:

(encoder): Sequential(
(0): Conv2d(1, 32, kernel_size=(4, 4), stride=(2, 2)) # [batch_size, 32, 15, 15]
(1): LeakyReLU(negative_slope=0.2)
(2): Conv2d(32, 64, kernel_size=(4, 4), stride=(2, 2)) # [batch_size, 64, 6, 6]
(3): LeakyReLU(negative_slope=0.2)
(4): Conv2d(64, 128, kernel_size=(4, 4), stride=(2, 2)) # [batch_size, 128, 2, 2]
(5): LeakyReLU(negative_slope=0.2)
(6): Conv2d(128, 256, kernel_size=(4, 4), stride=(2, 2)) # ERROR: 2x2 is too small for a 4x4 kernel
(7): LeakyReLU(negative_slope=0.2)
(8): Flatten()
)

You could either increase the input shape or decrease the kernel shapes.
Alternatively you could also add padding to the preceding layers, so that the spatial size won’t be decreased so much.