When part of a datapipe is cached on disk with the OnDiskCacheHolder datapipe, how to manage cache invalidation?
More specifically, say I have multiple files stored in a remote object storage. These files are fed into a datapipe using FSSpecFileLister, and cached locally using .on_disk_cache. I want to invalidate the cache and re-compute the datapipe when one or more remote files are changed, probably based on their hash.
Is there a way to achieve something similar?
While OnDiskCacheHolder accepts an extra_check_fn to carry out extra validation given the file paths, I don’t think we have automatic cache invalidation at the moment. It may requires changes to our existing cache holder implementation or a separate DataPipe. Feel free to open an issue on GitHub or a PR with what you end up implementing. We are happy to discuss further.