# Imagenet Training taking too long **URL:** \ **Category:** Uncategorized\ **Created:** [May 26, 2021, 1:31pm UTC](https://discuss.pytorch.org/t/imagenet-training-taking-too-long/122430 "2021-05-26T13:31:58Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![lucastononrodrigues](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lucastononrodrigues/32/23856_2.png) [@lucastononrodrigues](https://discuss.pytorch.org/u/lucastononrodrigues)\ **Post date:** [May 26, 2021, 1:31pm UTC](https://discuss.pytorch.org/t/imagenet-training-taking-too-long/122430/1 "2021-05-26T13:31:58Z")
I am trying to train a ViT model modification on the ImageNet dataset from scratch. I am using 8 Teslas V100 GPUs and it is taking enormously too long. While inspecting the gpus with nvidia-smi I get: ![WhatsApp Image 2021-05-26 at 9.59.37 AM](https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d4663447ccebe81c70d09c0f5246d0c1d87682f.jpeg) I am using nn.DataParallel to train it. In my dataloader I am using num\_workers = 8 and pin\_memory=True of course. I tried to increase the number of workers up to 16 as adviced in [Guidelines for assigning num\_workers to DataLoader - #4 by YossiB](https://discuss.pytorch.org/t/guidelines-for-assigning-num-workers-to-dataloader/813/4) but it only froze the machine. Is that normal? The estimated time per epoch is around 9 hours, I think that’s too long, specially because I intend to train it for 300 epochs ---
**Author:** ![lucastononrodrigues](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lucastononrodrigues/32/23856_2.png) [@lucastononrodrigues](https://discuss.pytorch.org/u/lucastononrodrigues)\ **Post date:** [May 26, 2021, 7:26pm UTC](https://discuss.pytorch.org/t/imagenet-training-taking-too-long/122430/2 "2021-05-26T19:26:22Z")
Obs: while increasing the number of workers from 0 to 8 the training time per epoch reduced from 16h to 6h, but that’s still too long. I’ve seen some other imagenet training in 29 hours. ResNet-50 takes 29 hours using 8 Tesla P100 GPU. Would it be a Torch problem? ---
**Author:** ![eqy](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eqy/32/46233_2.png) [@eqy](https://discuss.pytorch.org/u/eqy)\ **Post date:** [May 26, 2021, 8:49pm UTC](https://discuss.pytorch.org/t/imagenet-training-taking-too-long/122430/3 "2021-05-26T20:49:11Z")
Even with 16 workers there might be a large imbalance between data loading time and GPU time. Do you have some statistics on the proportion of time spent on data in each batch (e.g., like in the classic ImageNet example [here](https://github.com/pytorch/examples/blob/01539f9eada34aef67ae7b3d674f30634c35f468/imagenet/main.py#L267))? ---
**Author:** ![lucastononrodrigues](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lucastononrodrigues/32/23856_2.png) [@lucastononrodrigues](https://discuss.pytorch.org/u/lucastononrodrigues)\ **Post date:** [May 26, 2021, 8:57pm UTC](https://discuss.pytorch.org/t/imagenet-training-taking-too-long/122430/4 "2021-05-26T20:57:21Z")
So you are saying that the bottleneck is mostly loading the data (is what I suspected since for CIFAR100 it works fine). I don’t have the statistics but I can implement them. ---
**Author:** ![eqy](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eqy/32/46233_2.png) [@eqy](https://discuss.pytorch.org/u/eqy)\ **Post date:** [May 26, 2021, 9:00pm UTC](https://discuss.pytorch.org/t/imagenet-training-taking-too-long/122430/5 "2021-05-26T21:00:51Z")
Right, I think a few simple `time.time()` checks won’t affect performance and are a good sanity check for obvious bottlenecks. ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [May 27, 2021, 6:06am UTC](https://discuss.pytorch.org/t/imagenet-training-taking-too-long/122430/7 "2021-05-27T06:06:04Z")
We also recommend to use `DistributedDataParallel` with one process per GPU for the best performance instead of using `nn.DataParallel`. ---
**Author:** ![lucastononrodrigues](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lucastononrodrigues/32/23856_2.png) [@lucastononrodrigues](https://discuss.pytorch.org/u/lucastononrodrigues)\ **Post date:** [May 27, 2021, 3:54pm UTC](https://discuss.pytorch.org/t/imagenet-training-taking-too-long/122430/8 "2021-05-27T15:54:00Z")
Thanks for the reply! I wonder why is Distributed Data Parallel any better? I am using 8 gpus in the same machine Anyway, any tips/ideas on how I could load imagenet faster with pytorch? Hopefully by changing the HDD to a SSD will help. The loading time is 5.2 sec average and the batch time is 0.8 sec =/ @eqy ---
**Author:** ![eqy](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eqy/32/46233_2.png) [@eqy](https://discuss.pytorch.org/u/eqy)\ **Post date:** [May 27, 2021, 4:28pm UTC](https://discuss.pytorch.org/t/imagenet-training-taking-too-long/122430/9 "2021-05-27T16:28:43Z")
You can see the short note in the docs [here](https://pytorch.org/docs/stable/notes/cuda.html#cuda-nn-ddp-instead) for why Distributed Data Parallel can be faster for multi-GPU training. Yes, switching HDD to SSD can make a large difference, especially for “random” file I/O as loading many ImageNet images can look like random reads to storage. If you have sufficient memory, you might consider increasing the `prefetch_factor` in your `DataLoader` as well to see if increased buffering might help. However, if your HDD cannot keep up with the data loading speed, it is tricky fully utilize the GPUs.