I want to use the pretrained models from pytorch and evaluate them on imagenet val data. That should be fairly straightforward, but I am getting stuck on the dataloader.
I downloaded the imagenet and the folder structure that I have is like this:
I changed the path_to_imagenet_to /media/SSD2/ILSVRC/ like this
torchvision.datasets.ImageNet('/media/SSD2/ILSVRC/',split='val',download=False)
but I get this error:
RuntimeError: The archive ILSVRC2012_devkit_t12.tar.gz is not present in the root directory or is corrupted. You need to download it externally and place it in /media/SSD2/ILSVRC/.
Then I tried using datasets.ImageFolder in the following way:
I don’t know how and where you’ve downloaded the ImageNet dataset, but on my system each class uses a separate subfolder, which is needed in order to find all classes.
E.g. the val folder contains:
I think @seyeeet probably downloaded the Imagenet dataset from kaggle: ImageNet Object Localization Challenge | Kaggle. It’s no longer possible to download it from the original place on the imagenet website, instead the website also only refers to the dataset that is hosted on kaggle. I also downloaded it there and I get the same error when loading it via the torchvision.datasets.ImageNet class.
Are we just using it wrong or has the layout of the ImageNet dataset changed? The error message mentions that the dataloder cannot find ILSVRC2012_devkit_t12.tar.gz - kaggle’s verison of the dataset actually does not contain such an archive.
I guess Kaggle might have changed the data layout and if so I would assume there would be PyTorch scripts to load this new dataset type.
Based on the previous output it seems as if the images are just stored in the test/train/val folders without any subfolders. In that case ImageFolder wouldn’t be compatible, since the class indices won’t be created. If so, I would also assume that a target file is provided additionally.
Thanks, you are right, there’s actually an example script in an official pytorch repo that shows how ImageNet data can be accessed by using ImageFolder, which works for the version of the dataset that’s available on kaggle: examples/imagenet at master · pytorch/examples · GitHub
That said, I haven’t used the script directly, I simply tried to reuse the code part that sets up the dataloaders which starts here examples/main.py at master · pytorch/examples · GitHub and that works for me.
Just make sure that you follow the setup instructions that are listed here examples/imagenet at master · pytorch/examples · GitHub. In particular, you have to execute the shell script, that’s mentioned in the last step, inside the val folder (ILSVRC/Data/CLS-LOC/val/). This makes sure that the val folder is structured in a way that can be understood by the ImageFolder class.
Meet the same problem and solve it with https://github.com/fh295/semanticCNN
Specifically, to make the val dataset have the same structure as train set, you should run the following commands. It will move all validation pictures to corresponding class subfolders.
cd val/
wget -qO- https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh | bash
I figured out that you can download the ILSVRC2012 dataset by logging and requesting for permission in ImageNet website. By that you will have the .tar file for further inner processing of torchvision ImageNet dataset.
Guys, I dont get it. I also got the data from Kaggle and loading it via torchvision.dataset.ImageNet.
Even if we structure val according to train with subdir as class label. The file “ILSVRC2012_devkit_t12.tar.gz” still does not exist. This was the reason for the OP (and mine) error
Ok I guess the folks were just using ImageFolder dataset class
Here is a small wrapper around ImageFolder to have some utility functions to mimic the original ImageNet class:
class ImageNet(ImageFolder):
def __init__(self, root, transform = None, target_transform = None, split = "train"):
root = str(Path(root / split))
super().__init__(root, transform, target_transform)
self.wnid_to_idx = self.build_wnid_to_idx()
self.idx_to_wnid = {index: wnid for wnid, index in self.wnid_to_idx.items()}
self.synset_to_class = self.build_synset_to_classes()
self.class_to_synset = {cl: syn for syn, cl in self.synset_to_class.items()}
self.classes = [(cl.strip(), syn) for syn, cl in self.synset_to_class.items()]
self.targets = self.build_targets()
def build_targets(self):
# Builds the targets for each sample, assigning each image a class index
sorted_subdirs = sorted([d.name for d in os.scandir(self.root) if d.is_dir()]) # Ensures only directories are considered
targets = [
self.wnid_to_idx[wnid]
for wnid in sorted_subdirs
for f in os.listdir(os.path.join(self.root, wnid))
if os.path.isfile(os.path.join(self.root, wnid, f))
]
return targets
def build_wnid_to_idx(self):
# Builds a WNID to Index mapping based on the assumption that class indices in imagenet are from alphabetical sorting of wn synsets
sorted_subdirs = sorted([d.name for d in os.scandir(self.root) if d.is_dir()]) # Ensures only directories are considered
return {wnid: index for index, wnid in enumerate(sorted_subdirs)}
def build_synset_to_classes(self):
mapping_path = Path(self.root).parent.parent.parent.parent / "LOC_synset_mapping.txt"
synset_mapping = {}
with open(mapping_path, "r") as f:
for l in f.readlines():
synset, word = l.split(" ", 1)
synset_mapping[synset] = word
return synset_mapping
def idx_to_class(self, i):
wnid = self.idx_to_wnid[i]
return self.synset_to_class[wnid]
