I created one dataset for each file, and if there’s only 3000 files then it isn’t that much to hold it inside an array (object that has a reference to).
If you wrote your DataSet with linecache, then it won’t read each file into memory.
At least this is my observation after reading more files than my computer’s memory can support.