Really basic question on declaring variables in pytorch classes

I’m trying to learn the implementation of Transformers, using this tutorial. I’ve noticed that when declaring a variable inside a class using __init__, everything is set as self such as

class EncoderDecoder(nn.Module):
    A standard Encoder-Decoder architecture. Base for this and many 
    other models.
    def __init__(self, encoder, decoder, src_embed, tgt_embed, generator):
        super(EncoderDecoder, self).__init__()
        self.encoder = encoder
        self.decoder = decoder
        self.src_embed = src_embed
        self.tgt_embed = tgt_embed
        self.generator = generator
    def forward(self, src, tgt, src_mask, tgt_mask):
        "Take in and process masked src and target sequences."
        return self.decode(self.encode(src, src_mask), src_mask,
                            tgt, tgt_mask)
    def encode(self, src, src_mask):
        return self.encoder(self.src_embed(src), src_mask)
    def decode(self, memory, src_mask, tgt, tgt_mask):
        return self.decoder(self.tgt_embed(tgt), memory, src_mask, tgt_mask)

But sometimes not everything is set as self, such as

class Encoder(nn.Module):
    "Core encoder is a stack of N layers"
    def __init__(self, layer, N):
        super(Encoder, self).__init__()
        self.layers = clones(layer, N)
        self.norm = LayerNorm(layer.size)
    def forward(self, x, mask):
        "Pass the input (and mask) through each layer in turn."
        for layer in self.layers:
            x = layer(x, mask)
        return self.norm(x)

Where layer and N are not set as self

Previously when I implemented models I have been setting everything as self, but it’s really got me wondering if there is a difference, and if setting everything as self is wrong?

I’m sorry but that code is very ambiguous. As a python object you can set whatever as “self”. Self just indicates it’s an attribute of the object, this is not related to pytorch itself but python.

The pytorch way is to set nn.Module instances as attributes in init and call them during forward however this does not necessary mean that everything set as an attribute were a nn.Module.

I know self is not exclusive to pytorch, but I am curious why some of the declared variables in the object are set as self but others aren’t (like layer and N in the Encoder)

Sorry could you explain what layer is and what does clones do?

The layer is the self-attention + feed forward layer for transformer networks, and the clone just makes N copies of the layer so in the encoder the data passes through N copies of the layer

So in that case I’m pretty sure that (if it’s been properly done) clones will generate a torch.nn. ModuleList, right? If it’s the case, those modules are properly registered and you can work with that attribute as a python list