# Why applying "sum" twice when calculating the regularisation term manually?

[Context]
Book: Deep Learning with PyTorch by Eli Stevens, Luca Antiga, and Thomas Viehmann
Notebook: Chapter 8 Convolutional Neural Network

[Question]
Why sum is used twice when calculating the regularisation terms manually as shown in the code below?

``````def training_loop_l2reg(n_epochs, optimizer, model, loss_fn,
for epoch in range(1, n_epochs + 1):
loss_train = 0.0
imgs = imgs.to(device=device)
labels = labels.to(device=device)
outputs = model(imgs)
loss = loss_fn(outputs, labels)

l2_lambda = 0.001
l2_norm = sum(p.pow(2.0).sum()
for p in model.parameters())  # <1>
loss = loss + l2_lambda * l2_norm

loss.backward()
optimizer.step()

loss_train += loss.item()
if epoch == 1 or epoch % 10 == 0:
print('{} Epoch {}, Training loss {}'.format(
datetime.datetime.now(), epoch,
``````

Hi Oat!

The first (scanning left to right, rather than in order of operation) `sum`
is a python sum where you sum over the elements generated by the
“generator” (the expression inside of `sum`'s parentheses that looks like
a python “list comprehension” without the list). These elements happen
to be pytorch tensors that contain single values.

The second sum, `p.pow(2.0).sum()`, is a pytorch sum that sums over
the elements of a pytorch tensor (and returns a tensor that contains a
single value).

Just for fun, you can turn the nested sum inside out, first performing a
python sum over the generator (that generates python tensors containing
multiple values), and then performing the pytorch sum that sums over
elements of the resulting tensor (to get a single value):

``````l2_norm = sum (p.pow(2.0)  for p in model.parameters()).sum()
``````

Best.

K. Frank