This issue is happening during inference.
- I have set num_worker=3 in dataloader.
- Using distributed data parallel.
- My batch size is 256.
- Each image is 256x256
- Using A100, 8 gpus,
- each gpu has 40gb memory.
- total dataset size is=20gb
- System: Ubuntu
Still I am getting this error:
Exception has occurred: UnpicklingError (note: full exception trace is shown but execution is paused at: )
pickle data was truncated
File “”, line 1, in (Current frame)
_pickle.UnpicklingError: pickle data was truncated
Sometimes it looks like this:
self = reduction.pickle.load(from_parent)
_pickle.UnpicklingError: pickle data was truncated