Hi all, I am new to PyTorch and deep learning in general and I am developing a classification model based on a study I found.
The data is initially in signals form but the study I am following converts these signals into images (of size 3x150x150) first and feeds them into the model, which is why I did the same. The model is supposed to take 3 image inputs: blood pressure (bp), heart rate (hr), and ecg and emg data in one image (ecg_emg).
I have 24-hour data, as the BP and HR were collected only once every 20-30 minutes it was easy to convert the entire signal into one 3x150x150 image. However, the ecg_emg sampling rate is 1000 Hz and so I cannot fit the entire signal onto a 150x150x3 image.
So for ecg_emg data I have set it to be as shape (num_subjects, graphs_per_subject, 3, 150, 150) whereas the other BP and HR data are (num_subjects, 3, 150, 150).
This leads to memory problems as the graphs_per_subject are more than 1000, and even using batch sizes of 4 causes the CUDA out of memory: error:
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 10.60 GiB. GPU 0 has a total capacity of 79.15 GiB of which 8.32 GiB is free. Including non-PyTorch memory, this process has 70.81 GiB memory in use. Of the allocated memory 70.22 GiB is allocated by PyTorch, and114.36 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (CUDA semantics — PyTorch 2.5 documentation)
In the scenario above I had 988 images per subject so in batches I was passing tensors of (4, 988, 3, 150, 150).
Is this a common problem and how do people usually aproach it? I am ofcourse open to other ideas as well as I have been stuck on this problem for a few weeks now and I am stuck between downsampling the ecg_emg heavily or using 1-10 ecg_emg images per subject, which causes significant loss in information, or using lots of images per subject but crashing due to memory.
I also am implementing the DataLoader in my code but I am still getting the same error:
data_combined = torch.cat((streaming_data_bp_flat, streaming_data_hr_flat, streaming_data_ecg_emg_flat, structured_data.view(structured_data.size(0), -1)), dim=1)
# 'labels' is already the correct shape [81]
dataset = TensorDataset(data_combined, labels)
# Split dataset into train, validation, and test sets
train_size = int(0.6 * len(dataset))
val_size = int(0.2 * len(dataset))
test_size = len(dataset) - train_size - val_size
train_dataset, val_dataset, test_dataset = random_split(dataset, [train_size, val_size, test_size])
# Create DataLoaders
batch_size = 4
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True, num_workers=4)
val_loader = DataLoader(val_dataset, batch_size=batch_size, num_workers=4)
test_loader = DataLoader(test_dataset, batch_size=batch_size, num_workers=4)
Training the model:
for epoch in range(n_epochs):
model.train()
train_loss = 0.0
for combined_data, labels in train_loader:
# Dynamically split the flattened data into BP, HR, and ECG/EMG
# Size of each modality after flattening
bp_size = 67500 # BP: 150 * 150 * 3
hr_size = 67500 # HR: 150 * 150 * 3
ecg_emg_size = 67500*num_ecg_emg_images # ECG/EMG size after flattening
# Extract BP, HR, ECG/EMG, and structured data
bp = combined_data[:, :bp_size]
hr = combined_data[:, bp_size:bp_size + hr_size]
ecg_emg = combined_data[:, bp_size + hr_size:bp_size + hr_size + ecg_emg_size]
structured_data = combined_data[:, -70:] # Extract structured data (last 70 columns)
# Reshape BP, HR, and ECG/EMG to [batch_size, 3, 150, 150]
bp = bp.view(bp.size(0), 3, 150, 150)
hr = hr.view(hr.size(0), 3, 150, 150)
ecg_emg = ecg_emg.view(ecg_emg.size(0), 3, num_ecg_emg_images, 150, 150) # Shape is now identical to BP and HR
print("Checking data loader: Shape of ecg_emg: ", ecg_emg.shape)
# Reshape structured data to [batch_size, 1, 70]
structured_data = structured_data.view(structured_data.size(0), 1, 70)
# Move data to the appropriate device
bp, hr, ecg_emg, structured_data, labels = (
bp.to(device),
hr.to(device),
ecg_emg.to(device),
structured_data.to(device),
labels.to(device),
)
optimizer.zero_grad()
# Pass the data through the FeatureFusionModel
outputs = model(bp, hr, ecg_emg, structured_data)
loss = criterion(outputs.squeeze(), labels)
loss.backward()
optimizer.step()
train_loss += loss.item()
gc.collect() # Triggers garbage collection to clean up unused memory
del bp, hr, ecg_emg, structured_data # Delete tensors after use
torch.cuda.empty_cache() # Ensure memory is freed
Thank you!