I am trying to load my shanghai dataset for training . When I used the normal default Data Loader , i kept getting an empty dataset , even if my path was correct .
So , i circumvented it by using torch.utils.data.subset , and the same dataset was read .
My Problem is , when I check the length of the subset , it is giving me correctly , the length of the dataset I have, but when I run my code for training , I keep getting a None Type Error .
Another issue is , if I give subset size as just 2 , then my training occurs , only for those 2 samples .
My Code :
shanghai_dataset = SHANGHAITECH(path_ )
#data_loader2 = torch.utils.data.DataLoader(shanghai_dataset) #This is loading an empty dataset
data_loader2 = torch.utils.data.Subset(shanghai_dataset , indices = [i for i in range(0 , 300)] )
num_epochs = 800
for epoch in range(num_epochs):
for batch_idx, batch in enumerate(data_loader2):
video_frames = batch[batch_idx]
print(batch_idx)
print(video_frames)
visual_features = combined_model(video_frames)
visual_features = visual_features.view(-1)
#anomaly_scores = anomaly_detection_head(video_frames)
labels=torch.tensor([0.0 , 0.0 , 0.0 , 0.0])
loss = criterion(visual_features, labels )
optimizer.zero_grad()
loss.backward()
optimizer.step()
print(f'Epoch [{epoch + 1}/{num_epochs}], Loss: {loss.item()}')
Output , if subset length is 300
<First 2 video array values are printed>
2
None
Traceback (most recent call last):
File “/content/gdrive/MyDrive/Final/video_swin_transformer/train.py”, line 289, in
visual_features = combined_model(video_frames)
File “/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File “/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1527, in _call_impl
return forward_call(*args, **kwargs)
File “/usr/local/lib/python3.10/site-packages/torch/nn/modules/container.py”, line 215, in forward
input = module(input)
File “/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File “/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1527, in _call_impl
return forward_call(*args, **kwargs)
File “/content/gdrive/MyDrive/Final/video_swin_transformer/mmaction/models/backbones/swin_transformer.py”, line 652, in forward
x = self.patch_embed(x)
File “/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File “/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1527, in _call_impl
return forward_call(*args, **kwargs)
File “/content/gdrive/MyDrive/Final/video_swin_transformer/mmaction/models/backbones/swin_transformer.py”, line 441, in forward
_, _, D, H, W = x.size()
AttributeError: ‘NoneType’ object has no attribute ‘size’
Output if length of the subset is 2
Normal epoch vs loss gets printed without any issues