Divide validation data into n parts

After loading the data into val_loader, I need to divide the validation data into n (n=2 or 5 etc.) parts.
Here is what I tried, but it throws error, ‘DataLoader’ object is not subscriptable.

total=len(val_loader.dataset)
print(total) 
val1=val_loader[0:41629] # error, object is not subscriptable
val2=val_loader[41629:]
print(len(val1.dataset), len(val2.dataset))

Any suggestion on how to divide the val_loader in n parts.

You could use a SubsetRandomSampler to create different validation DataLoader, which would use the specified sample indices. Slicing the DataLoader is not supported.

Thank you @ptrblck for your suggestion, this is how I tried to split the validation data into two parts, and it worked, I hope it is ok.

indices = list(range(83258)) # 83258 is the total validation samples
np.random.shuffle(indices)
    
val1=indices[:41629]
val2=indices[41629:]
sampler1=SubsetRandomSampler(indices=val1)
sampler2=SubsetRandomSampler(indices=val2)

val_loader1 = torch.utils.data.DataLoader(
        datasets.ImageFolder(valdir, transform=transform),
        batch_size=32, shuffle=False,
        num_workers=args.workers, pin_memory=True, sampler=sampler1)
    
val_loader2 = torch.utils.data.DataLoader(
        datasets.ImageFolder(valdir, transform=transform),
        batch_size=32, shuffle=False,
        num_workers=args.workers, pin_memory=True, sampler=sampler2)

Yes, you approach looks perfectly fine! :slight_smile:

1 Like