Data reading from text files

Hi, I am new to Pytorch. I have images stored in many text files and labels (which are vectors) stored in separate text files. I want to extract from these images some physical properties, which are buried in them, that is why the labels are vectors. I want to make a dataset to be able to use dataloaders on these files. What is the best way to load these data? Should I write a custom Dataset class or there is a standard way to read these data. One way, I imagine is to convert the text files into some image format and then use ImageLoader. What is your opinion ? Many thanks.

Based on your description I think writing a custom Dataset sounds like a valid approach.
You can take a look at this tutorial for an example.

Thanks. I transformed the txt files to grayscale images using matplotlib’s imsave( When I use a ToTensor() transform on such an image opened with PIL’s open method I get a tensor shape of [4, 256, 256]. Since the images are grayscale, shouldn’t I get [1, 256, 256]? Many thanks.

ToTensor shouldn’t change the number of channels. Could you check the shape of the loaded PIL.Image (or its underlying numpy array)? I would guess that PIL is already loading it in this shape.

The PIL image is of 256x256 size, but as I printed out the image object it said that it has an image mode RGBA. I guess this could be the problem. I still don’t understand why imsave() does not save it as grayscale with a single channel. I also tried PIL’s fromarray() function to save to image format, but my arrays contain float values (some of the very small values, therefore a direct conversion to int values results in zeros for the whole array) and fromarray expects uint8. Is there any way to save as an image a float valued numpy array ? Many thanks. You are very kind to answer all these questions.

You could try to specify the desired image format in imsave, but would need to make sure that PIL is not transforming the array under the hood and could thus quantize the data (if I’m not mistaken you don’t want that).

You could store the numpy array directly, in case you want to save the exact values. I don’t know if this would fit your use case and what kind of image format your current data would represent.