Append items from batch to dataset


I am extracting features and storing them in a dataset. Currently, I use a batch size of 1 where I append each batch into a hdf5 dataset. However, I’m stuck on how to extract each item from a batch, when the batch size is greater than 1. I basically want to store each item from a batch as its own row/instance in the dataset.

Any idea on what I can do?


for i, (inputs40x, inputs20x, paths40x, paths20x, labels) in enumerate(dataloaders_dict):

                print(f'Batch ID: {batch_idx}')

                inputs40x =
                inputs20x =

                labels =
                paths = paths40x

                # delete the last fc layer.
                modules = list(resnet50.children())[:-1]
                resnet = nn.Sequential(*modules)

                x40 = resnet(inputs40x)
                x20 = resnet(inputs20x)
                x_all =[x40, x20], dim=1)

                # torch.Size([1, 2048, 1, 1]) batch, feats, 1l, 1l
                array_40[batch_idx, ...] = x40.cpu()
                array_20[batch_idx, ...] = x20.cpu()
                array_all[batch_idx, ...] = x_all.cpu()
                array_labels[batch_idx, ...] = labels[:].cpu()
                array_paths[batch_idx, ...] = paths
                array_batch_idx[:,...] = batch_idx

                batch_idx +=1