Torch Dataloader gives torch.Tensor as ouput

I think the main problem is:

as stacked tensors are expected.
You should be able to directly return the pd.DataFrames, but then your model would most likely not accept it (or you would transform the input to proper tensors in your forward which also sounds quite late).

Yes, I would recommend transforming the data into a tensor inside the __getitem__ so that the default collate function could then create batches from these samples.
Would this work or does it not fit your use case?