Non scalar backward and self mini batch implementation

[meipuru344]

My program is using non scalar backward

and self mini batch implementation like bellow(because each data has huge size)

for i in range(len(batch_list)):

     output = network(batch_list[i])   #non scalar output
     grad = grad_fn(output, teacher[i])  #calc non scalar grad
     torch.autograd.backward([output],[grad])  #non scalar backwarding

optimizer.step()
optimizer.zero_grad() 

It seems my Network isn’t training correctly.

I think the section I cropped has a problem.

(Note that, for the simplicity the code isn’t original.

but it has a essence of my flowchart.)

Are there some weird point ?

In self implementation, how minibatch work?

I think i should divide gradient.

[meipuru344]

Does PyTorch average or sum gradients over a minibatch?

For example when you retrieve the gradients like so: loss = F.nll_loss(output, target) loss.backward() for key, value in model.named_parameters(): mygrad = value.grad is mygrad the sum of gradients over the minibatch or the average?

My question relates this one.

#Quote Mr SimonW
All autograd does is just to calculate the gradient, it has no notion of batching, and I don’t see how it can have different behavior with different batching mechanism.

[InnovArul (Arul)]

@meipuru344

grad = grad_fn(output, teacher[i]) #calc non scalar grad

You can divide the grad with the length of batch. It is needed especially when the batches are of different lengths.

If the batches are of same length, dividing by length of batch size can be thought to be absorbed by learning rate anyway, in which case its not needed.

[meipuru344]

I appreciate your reply

In this case

torch.autograd.backward([output],[grad])

Is grad of each data in minibatch accumulated like bellow?

How to implement accumulated gradient in pytorch (i.e. iter_size in caffe prototxt)

how to can i accumulate gradient during gradient descent in pytorch (i.e. iter_size in caffe prototxt). Currently, my code is: for iter, (images, labels, indices) in enumerate(train_loader, 0): optimizer.zero_grad() outputs = net(Variable(images.cuda())) loss = criterion(outputs, Variable(labels.cuda())) loss.backward() optimizer.step() Do i do this? for iter in range(N): optimizer.zero_grad() …

Thanks

[InnovArul (Arul)]

Yes. It is accumulated as mentioned in the post.
This code looks fine:

How to implement accumulated gradient in pytorch (i.e. iter_size in caffe prototxt)

Here is the corrected code for i in range(num_iters): optimizer.zero_grad() batch_loss_value = 0 for m in range(M): (images, labels, indices) = train_loader.next(): outputs = net(Variable(images.cuda())) loss = criterion(outputs, Variable(labels.cuda())) loss.backward() batch_loss_value += loss.cpu().numpy()[0] optimizer.step() …

[meipuru344]

Thank you