Hi I am new to Pytorch (and ML and NN). I need some general guidance.
I have a basic question that I could not find a straight answer for anywhere.
In an ideal case, should CPU RAM usage be increasing with each mini-batch? To give numbers: train_data size is ~6 million. min-batch size=128.
My ram started at like 5% when epoch started and i am at 23000th training step, and it has gone up to 60%. The increase seems linear in number of mini-batches. (about 2.5% increase with each 1000 mini batches)
However once epoch ends, RAM usage comes back down again. But why should it increase by each min-batch? I dont see it.
For context i am running training on a gpu machine and the flow is something like this:
model=mymodel( parameters) #initialize nn model
model.to("cuda:0")
train_dataloader=CustomDataloader(train_features)
for epoch in range(n_epochs):
steps=0
for X,y in train_dataloader():
steps+=1
if steps%1000==0:
gc.collect() # i added this line to see if it makes a difference, it didnt.
X.todevice("cuda:0")
#X is actually a custom object which has both tensor data
# but also some meta data. ".todevice" is a method that send tensors to cuda but doesnt
#touch meta data. Can the problem lie here? it still does not explain why RAM usage
#should increase with each minibatch
y.to("cuda:0" )
yhat=model(X)
criterion = torch.nn.BCEWithLogitsLoss(weight=weight).to("cuda:0")
loss=criterion(logit,target.to(torch.float32))
train_losses.append(loss)
optimizer.zero_grad()
loss.backward()
optimizer.step()
optimizer.zero_grad()
Please not the comment after X.todevice(“cuda:0”).
The dataloader is custom as well
class CustomDataLoader():
def __init__(self,ds,bs):
self.ds,self.bs=ds,bs
def __iter__(self):
for i in range(0,len(self.ds),self.bs):
indices=self.ds[key].data[i:i+self.bs].clone()
batch,y=self.ds.generate_batch(indices)
yield batch,y
train_features is not a dataset. It is a custom object (a UserDict) with a method “generate_batch” which, given indices, generates a “subset” of the same object. By subset I mean an instance of the same class with smaller data but same meta data.
You may wonder why i am doing it this way. Well, if i try to make train_features a dataset with a getitem that produces single data objects (again, with same meta-data as train_features) and go the usual route, i have to write a collate function that takes these single data objects and “stacks” them up. This seem an unnecessary process in which i first extract single data objects, and then out it back together. i can just cut the middle man and directly produce batches from the object itself.
Does Pytorch’s factory Dataloder has something that i will miss doing it this way?
Thank you for taking the time to read my question and I really appreciate your guidance. Let me know if you need further details.
Update: I just iterated over train_dataloader (and sent things to cuda) without any operations on them, and the RAM use didnt increase. So it is not the dataloader but model probably.
