ImageNet DataLoader "too many open files" error

1

I create an instance of a torchvision ImageNet validation-set DataLoader called imagenet_val.

I then try to make an iterator for it: iter(imagenet_val).

And get this trace:

File "/u/slerman/u1/conda/envs/Sam/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 441, in __iter__
    return self._get_iterator()
           ^^^^^^^^^^^^^^^^^^^^
  File "/u/slerman/u1/conda/envs/Sam/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 388, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/u/slerman/u1/conda/envs/Sam/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 1042, in __init__
    w.start()
  File "/u/slerman/u1/conda/envs/Sam/lib/python3.11/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/u/slerman/u1/conda/envs/Sam/lib/python3.11/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/u/slerman/u1/conda/envs/Sam/lib/python3.11/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/u/slerman/u1/conda/envs/Sam/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/u/slerman/u1/conda/envs/Sam/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/u/slerman/u1/conda/envs/Sam/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 58, in _launch
    self.pid = util.spawnv_passfds(spawn.get_executable(),
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/u/slerman/u1/conda/envs/Sam/lib/python3.11/multiprocessing/util.py", line 451, in spawnv_passfds
    errpipe_read, errpipe_write = os.pipe()
                                  ^^^^^^^^^
OSError: [Errno 24] Too many open files
2

Check how many file handles can be used via ulimit -n and increase this value if needed.

4

Do you know if the imagenet validation set is expected to have images of shape [3, 500, 375] and others [3, 375, 500]?

5

ImageNet samples have a variable spatial shape, if Iā€™m not mistaken, and are thus resized to e.g. 224x224 in common architectures.

1 Like