Operating systems assumes Locality about file reading and caches some content.
But in deep learning, we usually have large datasets that can’t fit in the memory, so we have to load image by image into the memory. However, here we never need the locality. We assume the input is i.i.d. So the operating system file cache strategy always fails.
When I train deep learning algorithms, a large proportion of buffer/cache is wasted for this.
Can I disable file cache for a specific process?