Hi!
So I have a question, I have this recursive algorithm to train multiple Neural Networks in different domains, but same architecture , every time you go deeper into the algorithm the less training data you have, meaning it should be faster training the deeper you go.
I tested my algorithm with a small(2000) training set to be sure that it worked. And it does work! But now i want to try it with more real world training set, with size 262144(512x512). Since I knew I was going to work with the GPU so I tried to carefully move variables to the GPU only if I needed it.
When I ran it, I saw that the first network I trained(with the biggest training set of 512x512 samples) trains quite fast it takes 130s. However, next one that is going to be trained using only 60% of the data to train it, takes significantly way more(I have taken the time to see how much it actually takes because it so long). But if I test the network using that 60% of the data to train it, it takes approximately 50s.
Now that you have the context my question is the following. Is there any way that the first network I trained slows the second one?
I will show you the NN and the my training functions:
class MLPflat(nn.Module):
def __init__(self,in_dim: int, out_dim: int, N, H):
super().__init__()
assert(N > 0)
assert(H > 0)
net = [nn.Linear(in_dim, H),nn.BatchNorm1d(H), nn.LeakyReLU()]
for _ in range(N-1): # make N layers
net += [nn.Linear(H, H),nn.BatchNorm1d(H), nn.LeakyReLU()]
net += [nn.Linear(H,out_dim,bias=False)]
self.model = nn.Sequential(*net)
def forward(self, x):
x = self.model(x)
output = x
return output
I use N(hidden layers) = 5 and H = 64.
def train(phi,train_loader,epochs,criterion,optimizer):
fit_start_time = time.time()
for epoch in range(epochs):
batch = 0
for x_batch, y_batch in train_loader:
optimizer.zero_grad()
print(x_batch.shape)
y_pred = phi(x_batch.to(device))
loss = criterion(y_pred.squeeze(), y_batch.to(device).squeeze())
loss.backward(retain_graph=True)# I retain the graph because I get an error if I don't do it.
optimizer.step()
batch+=1
fit_end_time = time.time()
print("Total time = %f" % (fit_end_time - fit_start_time))
I tried to do is to get rid any tensor inside of the GPU I don’t need. However it didn’t seem to improve.
I think that this problem is not really apparent in with really small training sets, but only comes when I train it with big amounts of training samples.
I would love to hear if this problem sounds like anything you have encounter before. If this is not enough let me know and I could elaborate more on my algorithm.
