Hello! I have a NN that is trained to predict the output of an equation. It is a very basic fully-connected NN. I used it for weeks perfectly fine, but for some reason today I started to get the CUDA error: out of memory error. I reduced the batch size to 1 (initially it was working just fine with a batch size of 1024) and I am still getting this error. I am running this on a shared linux machine (however I am the only user most of the time). I logged in and out several times and I am still getting that error. I tried nvidia-smi and I noticed that the GPU usage was 99-100%, however, later, that value dropped to 0% and I am still getting the error. I am really new to AI and I don’t know anything about GPU’s. Does anyone have any idea what could the problem be? Thank you!
Update: Here is a part of the code:
for filename in os.listdir(pathdir):
print(filename)
for i in range(1):
n_variables = np.loadtxt(pathdir+"/%s" %filename, dtype='str').shape[1]-1
variables = np.loadtxt(pathdir+"/%s" %filename, usecols=(0,))
if n_variables==0:
print("Solved! ", variables[0])
continue
elif n_variables==1:
variables = np.reshape(variables,(len(variables),1))
else:
for j in range(1,n_variables):
v = np.loadtxt(pathdir+"/%s" %filename, usecols=(j,))
variables = np.column_stack((variables,v))
f_dependent = np.loadtxt(pathdir+"/%s" %filename, usecols=(n_variables,))
f_dependent = np.reshape(f_dependent,(len(f_dependent),1))
factors = torch.from_numpy(variables[0:100000])
print(len(factors))
factors = factors.cuda()
It seems that the error appears in the last line, at factors = factors.cuda(). Factors is a torch tensor of size 4 x 100000. I made it size 4 x 1 and still running out of CUDA memory.