Why is this not being send to GPU?

1

I’m sending the net to GPU, still complains about not being on GPU.

This is a piece of it:

class DynamicEncoder(nn.Module):
    """Encoder.

    Args:
        increments: list of input_channel-output_channel tuples.
        config: parameters for layers configuration.

    """

    def __init__(self, increments: list[tuple[int, int]], config: DAEConfig) -> None:
        super().__init__()
        self.config = config
        self.pool = (
            nn.MaxPool2d(
                kernel_size=config.get("p_kernel"),
                stride=config.get("p_stride"),
                return_indices=True,
            )
            if self.config.get("use_pool")
            else None
        )
        c, b = create_layers(
            channels_list=increments,
            is_transpose=False,
            config=config,
        )
        self.convs = c
        self.batch_norms = b
        self.drop = nn.Dropout(0.3)
        self.dense = None

    def forward(
        self,
        x: Tensor,
    ) -> tuple[Tensor, list[Tensor], list[tuple[str, torch.Size]]]:
        """Forward pass for Encoding.

        Returns:
            x: Tensor
            indices: From the convolution
            shapes: before each pooling, and before flattening.

        """
        pool_indices = []
        shapes = []
        for c, b in zip(self.convs, self.batch_norms, strict=False):
            shapes.append(("conv", x.size()))
            x = self.drop(F.leaky_relu(b(c(x))))
            if self.pool is not None:
                shapes.append(("pool", x.size()))
                x, index = self.pool(x)
                pool_indices.append(index)
        shapes.append(("none", x.size()))  # unflattened size.
        x = x.view(x.size(0), -1)
        dense_i, dense_o = sum(x.shape[1:]), 128
        if self.dense is None:
            self.dense = nn.Linear(dense_i, dense_o)
        x = self.drop(nn.functional.leaky_relu(self.dense(x)))
        shapes.append(("dense", (dense_i, dense_o)))

        return x, pool_indices, shapes

All I get is:

—> 81 x = self.drop(nn.functional.leaky_relu(self.dense(x)))

i don’t know what to do

2

How do you send the model to gpu?
All this code does is defining the model, but not the device.

3

i use this:

def to_device_available(net: nn.Module) -> str:
    """Send to device (GPU/MPS) if available."""
    device = "cpu"
    if torch.cuda.is_available():
        device = "cuda"
        if torch.cuda.device_count() > 1:
            net = nn.DataParallel(net)
            return device
    elif torch.mps.is_available():
        device = "mps"
    net.to(device)
    return net, device

i also print params and says “cuda” :sob:

4

net.to(device) is not in-place.
You should do net=net.to(device)

5

Ive tried that as well; apparently layers initialised within forward aren’t created in cuda (the Decoder part in this case).

Running a dummy input beforehand seems to fix it? (then sending the whole net to cuda.)

7

Oh I didn’t read the forward. forward is not the right place to initiallize the modules.
Pytorch’s nn is a tree structure and pretty much all the functions that call model.paremeters() and iterating over this tree.
Defining layers within forward is preventing all the methods before being aware that these layers exist and will create tons of silent issues. I would encourage you not to do this.

The most obvious is:
if you defined the optimizer before these layers are intialized by doing
optimizer=some_opt(model.paremeters())
The optimizer will not upgrade the layers with the gradients for those intialized in forward, which will cause your network not to perform well and will be hard to catch.

1 Like
8

Hi there, thanks for the comment. Let me assert how I understand this, then you can correct me.

I’ve an autoencoder:

class AE:
     self.encoder = Encoder()
     self.decoder = None

decoder is None because it takes results self.encoder(..) for instantiation.

…:thinking: …

PS: as I was writing this I realised it could be modified to:

```python
class AE:
     self.encoder = Encoder(...)
     self.decoder = Decoder(...)

But,

  • Aren’t there some cases where what I did would make sense, given a decoder that needs parameters returned from the encoder forward method to instantiate it @JuanFMontesinos ?

This seems a valid case to me.

9

I don’t recall a clear case where you cannot define the architecture without encoder parameters.
Could you provide an example?
In addition, there are many “lazy” layers where you don’t know number of channels of the input to a module and it is initialized at runtime:
https://pytorch.org/docs/stable/generated/torch.nn.LazyConv2d.html
Most operations in the latests pytorch have an analogous lazy layer (conv2d, conv1d, batchnorm, etcetera…)

1 Like
10

I just assign my model and the batches .to ‘cuda’ without problem.
It can also be helpful to import accelerate which helps assign dataloaders, losses, etc to the GPU

1 Like
11

you may be right there, i’ve got no example in my mind either

12

fantastic package, ty! ig ill need to do some more searches; yet it’s ok to implement things once

taking a look here now as well… Ecosystem | PyTorch