How to safely load training data from memory

1

Hi

My training data has the shape torch.Size([330000,7225]) and is currently saved and retrived by using torch.save() and torch.load() before being split into batches using TensorDataset and DataLoader . However when loading i get the error:

RuntimeError: [enforce fail at ..\c10\core\CPUAllocator.cpp:47] 
((ptrdiff_t)nbytes) >= 0. alloc_cpu() seems to have been called with negative number: 18446744066066682432

I suspect this is because the file is too large to be loaded all at once (it is 8.88 GB). Is there an alternative way in which I don’t have to load all of the training data first or would you recommend I save my data using an alternative approach?

2

I tried to reproduce this error my storing and loading a tensor with the posted shape, but cannot reproduce this issue.
Could you post some information about your system, i.e. OS, PyTorch version, etc.?

3

Here is some information about my system:

PyTorch version: 1.4.0

Hardware and system information:

======================================== System Information ========================================
System: Windows
Node Name: LAPTOP-LFIC0N5G
Release: 10
Version: 10.0.18362
Machine: AMD64
Processor: Intel64 Family 6 Model 158 Stepping 13, GenuineIntel
======================================== Boot Time ========================================
Boot Time: 2020/3/2 12:30:58
======================================== CPU Info ========================================
Physical cores: 6
Total cores: 12
Max Frequency: 2592.00Mhz
Min Frequency: 0.00Mhz
Current Frequency: 2592.00Mhz
CPU Usage Per Core:
Core 0: 12.5%
Core 1: 3.4%
Core 2: 12.5%
Core 3: 5.9%
Core 4: 14.3%
Core 5: 2.1%
Core 6: 5.6%
Core 7: 1.4%
Core 8: 3.2%
Core 9: 2.2%
Core 10: 3.7%
Core 11: 2.4%
Total CPU Usage: 5.8%
======================================== Memory Information ========================================
Total: 15.84GB
Available: 4.18GB
Used: 11.66GB
Percentage: 73.6%
==================== SWAP ====================
Total: 52.62GB
Free: 37.13GB
Used: 15.49GB
Percentage: 29.4%
======================================== Disk Information ========================================
Partitions and Usage:
=== Device: C:\ ===
  Mountpoint: C:\
  File system type: NTFS
  Total Size: 476.15GB
  Used: 250.62GB
  Free: 225.53GB
  Percentage: 52.6%
Total read: 451.03GB
Total write: 450.79GB
4

Could you try to free more system memory, as you only have ~4GB + swap?