if you notice the examples, DataParallel is not applied to the entire network + loss. It is only applied to part of the network.
before adding DataParallel:
network = features (conv layers) -> classifier (linear layers)
error = loss_function(network(input), target)
error.backward()
After adding DataParallel:
network = DataParallel(features (conv layers)) -> classifier (linear layers)
error = loss_function(network(input), target)
error.backward()