Hi, The following problem occurs when I train the model without Docker, but I don’t know how to fix it:
Epoch 1 | Iter 1 | Average Loss 50.771 | Current Loss 50.771217 | 3165.4 ms/batch
Traceback (most recent call last):
File "/home/marktsai/Conv-TasNet-master_2/egs/wsj0/../../src/train.py", line 147, in <module>
main(args)
File "/home/marktsai/Conv-TasNet-master_2/egs/wsj0/../../src/train.py", line 141, in main
solver.train()
File "/home/marktsai/Conv-TasNet-master_2/src/solver.py", line 76, in train
tr_avg_loss = self._run_one_epoch(epoch)
File "/home/marktsai/Conv-TasNet-master_2/src/solver.py", line 173, in _run_one_epoch
for i, (data) in enumerate(data_loader):
File "/home/marktsai/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 322, in __next__
return self._process_next_batch(batch)
File "/home/marktsai/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 357, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
AssertionError: Traceback (most recent call last):
File "/home/marktsai/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 106, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/marktsai/Conv-TasNet-master_2/src/data.py", line 142, in _collate_fn
mixtures, sources = load_mixtures_and_sources(batch[0])
File "/home/marktsai/Conv-TasNet-master_2/src/data.py", line 252, in load_mixtures_and_sources
assert mix_info[1] == s1_info[1] and s1_info[1] == s2_info[1]
AssertionError
Exception ignored in: <bound method _DataLoaderIter.__del__ of <torch.utils.data.dataloader._DataLoaderIter object at 0x7fbc87086f28>>
Traceback (most recent call last):
File "/home/marktsai/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 399, in __del__
self._shutdown_workers()
File "/home/marktsai/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 378, in _shutdown_workers
self.worker_result_queue.get()
File "/home/marktsai/anaconda3/envs/pytorch/lib/python3.6/multiprocessing/queues.py", line 337, in get
return _ForkingPickler.loads(res)
File "/home/marktsai/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 151, in rebuild_storage_fd
fd = df.detach()
File "/home/marktsai/anaconda3/envs/pytorch/lib/python3.6/multiprocessing/resource_sharer.py", line 57, in detach
with _resource_sharer.get_connection(self._id) as conn:
File "/home/marktsai/anaconda3/envs/pytorch/lib/python3.6/multiprocessing/resource_sharer.py", line 87, in get_connection
c = Client(address, authkey=process.current_process().authkey)
File "/home/marktsai/anaconda3/envs/pytorch/lib/python3.6/multiprocessing/connection.py", line 487, in Client
c = SocketClient(address)
File "/home/marktsai/anaconda3/envs/pytorch/lib/python3.6/multiprocessing/connection.py", line 614, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
I ran on Ubuntu 20.04, one Geforce GTX 1660, RAM size is 32GB. The dataset is around 100 hours of audio. It works when I use a smaller dataset, but it not works when the dataset is expanded. I read some suggestions on the forum and set num_worker to 0 and set batch size to 1, but it’s still not working.