@ptrblck as per your suggestions I tried with DataLoader
but the result was the same.
The code snippet below creates a custom dataset object which takes in images and targets of type torch cuda float .
class CustomDataset(torch.utils.data.Dataset):
def __init__(self,xtr,ytr):
self.xtr = xtr
self.ytr = ytr
def __getitem__(self,idx):
img = self.xtr[idx]
tar = self.ytr[idx]
return img,tar
def __len__(self):
return len(self.xtr)
def collate_fn(batch):
return list(zip(*batch))
The dataset sample is pretty small. I was trying with a single image. I am facing some problems with num_workers
as it is throwing me Cuda Runtime error so decided to skip it. The rest of the code is given below.
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=False,num_classes=4)
model.to(device)
optimizer = optim.Adam(model.parameters(),lr=0.000001)
## trying with one image only
x_train = images[0:1]
y_train = labels[0:1]
## Converts numpy to TorchCudaFloat
x_train,y_train = from_numpy_to_tensor(x_train,y_train)
## DataLoader object
dataset = CustomDataset(x_train,y_train)
dataloader = DataLoader(dataset,batch_size=1,collate_fn=collate_fn)
## Iterations
for i in range(20):
print("Iter No:",i)
for xtr,ytr in dataloader:
#print("Iter No:",i)
#optimizer.zero_grad()
ytr = list(ytr)
#print("In train before:")
print(ytr)
print(xtr)
output = model(xtr,ytr)
At the start of the iteration my targets are something like this:-
[{'boxes': tensor([[ 311.9933, 1013.7640, 719.6339, 1142.7417],
[ 308.1646, 928.4176, 739.4443, 961.6580],
[ 308.7562, 830.7968, 740.0359, 864.0373],
[ 305.8657, 680.8315, 763.0424, 708.1243],
[ 300.3259, 439.0691, 790.4506, 523.2395],
[ 306.6932, 248.2031, 741.7596, 458.5648]], device='cuda:0'), 'labels': tensor([4, 3, 3, 3, 2, 1], device='cuda:0')}]
And at the end of the 20th iteration the targets have changed to:-
[{'boxes': tensor([[117.7318, 383.4572, 271.5563, 432.2433],
[116.2870, 351.1749, 279.0319, 363.7481],
[116.5102, 314.2498, 279.2552, 326.8230],
[115.4195, 257.5252, 287.9367, 267.8488],
[113.3290, 166.0784, 298.2794, 197.9159],
[115.7318, 93.8831, 279.9056, 173.4527]], device='cuda:0'), 'labels': tensor([4, 3, 3, 3, 2, 1], device='cuda:0')}]
As you can see the value of the targets are getting altered by a huge marging just in 20 iterations in just a simple forward pass.
@ptrblck Is this a bug? Do I have to open an issue in that case?