Include torchvision in singularity image

I have been trying to set up a singularity container containing PyTorch via this image and it seems that it works out fine. I have, however, some trouble including torchvision in my setup. I tried the following lines without much success

Bootstrap: docker
From: pytorch/pytorch:latest

%runscript
    exec echo "Building singularity image..."

%post
apt update && apt install -y python3-pip
python3 -m pip install --upgrade pip
python3 -m pip install torchvision
pip --no-cache-dir install --upgrade torchvision
pip install torchvision
python3 -m pip install numpy
python3 -m pip install matplotlib
python3 -m pip install h5py

but when I try to queue in a job with slurm I get the error that torchvision cannot be found. Since there is no separate image for torchvision what is the most suitable way to include torchvision as a dependency?

noone here working with singularity images?