Hi,
I am extracting features and storing them in a dataset. Currently, I use a batch size of 1 where I append each batch into a hdf5 dataset. However, I’m stuck on how to extract each item from a batch, when the batch size is greater than 1. I basically want to store each item from a batch as its own row/instance in the dataset.
Any idea on what I can do?
Cheers,
for i, (inputs40x, inputs20x, paths40x, paths20x, labels) in enumerate(dataloaders_dict):
print(f'Batch ID: {batch_idx}')
inputs40x = inputs40x.to(device)
inputs20x = inputs20x.to(device)
labels = labels.to(device)
paths = paths40x
# delete the last fc layer.
modules = list(resnet50.children())[:-1]
resnet = nn.Sequential(*modules)
x40 = resnet(inputs40x)
x20 = resnet(inputs20x)
x_all = torch.cat([x40, x20], dim=1)
# torch.Size([1, 2048, 1, 1]) batch, feats, 1l, 1l
array_40[batch_idx, ...] = x40.cpu()
array_20[batch_idx, ...] = x20.cpu()
array_all[batch_idx, ...] = x_all.cpu()
array_labels[batch_idx, ...] = labels[:].cpu()
array_paths[batch_idx, ...] = paths
array_batch_idx[:,...] = batch_idx
batch_idx +=1