Why applying "sum" twice when calculating the regularisation term manually?

Book: Deep Learning with PyTorch by Eli Stevens, Luca Antiga, and Thomas Viehmann
Notebook: Chapter 8 Convolutional Neural Network

Why sum is used twice when calculating the regularisation terms manually as shown in the code below?

def training_loop_l2reg(n_epochs, optimizer, model, loss_fn,
    for epoch in range(1, n_epochs + 1):
        loss_train = 0.0
        for imgs, labels in train_loader:
            imgs = imgs.to(device=device)
            labels = labels.to(device=device)
            outputs = model(imgs)
            loss = loss_fn(outputs, labels)

            l2_lambda = 0.001
            l2_norm = sum(p.pow(2.0).sum()
                          for p in model.parameters())  # <1>
            loss = loss + l2_lambda * l2_norm

            loss_train += loss.item()
        if epoch == 1 or epoch % 10 == 0:
            print('{} Epoch {}, Training loss {}'.format(
                datetime.datetime.now(), epoch,
                loss_train / len(train_loader)))

Hi Oat!

The first (scanning left to right, rather than in order of operation) sum
is a python sum where you sum over the elements generated by the
“generator” (the expression inside of sum's parentheses that looks like
a python “list comprehension” without the list). These elements happen
to be pytorch tensors that contain single values.

The second sum, p.pow(2.0).sum(), is a pytorch sum that sums over
the elements of a pytorch tensor (and returns a tensor that contains a
single value).

Just for fun, you can turn the nested sum inside out, first performing a
python sum over the generator (that generates python tensors containing
multiple values), and then performing the pytorch sum that sums over
elements of the resulting tensor (to get a single value):

l2_norm = sum (p.pow(2.0)  for p in model.parameters()).sum()


K. Frank