I build a Dataset class and followed the tutorials to use pandas DataFrame read my data. The size of my data are 2GB. When I call it in my main file, I found it costs too much cpu resources and memory(12GB). It’s not like image that people can load one image per time, so I wonder how could I process this more efficient.
Any idea would help. Thanks.