What is the linear embedding layer in Swin Transformer?

jasonrichdarmawan · June 21, 2024, 10:02am

my goal is to understand what is Linear Embedding layer actually is in Swin Transformer

The problem is the Swin Transformer’s paper do not explain what is the linear embedding layer.

arXiv:

arxiv org/pdf/2103.14030

A linear embedding layer is applied on this raw-valued feature to project it to an arbitrary dimension (denoted as C).

This is the only explanation given for linear embedding in the paper.

PyTorch documentation:

Linear pytorch org/docs/stable/generated/torch.nn.Linear.html
Embedding pytorch org/docs/stable/generated/torch.nn.Embedding.html

YouTube:

Embedding youtube com/watch?v=aWFllV6WsAs

StackOverflow:

The difference between Linear and Embedding in PyTorch stackoverflow com/questions/65445174/what-is-the-difference-between-an-embedding-layer-with-a-bias-immediately-afterw

Honestly, I do not know where to start

soulitzer · June 21, 2024, 6:09pm

From the PyTorch implementation:

github.com

microsoft/Swin-Transformer/blob/main/models/swin_transformer.py#L437


      
          
              def flops(self):
                  flops = 0
                  for blk in self.blocks:
                      flops += blk.flops()
                  if self.downsample is not None:
                      flops += self.downsample.flops()
                  return flops
          
          
          class PatchEmbed(nn.Module):
              r""" Image to Patch Embedding
          
              Args:
                  img_size (int): Image size.  Default: 224.
                  patch_size (int): Patch token size. Default: 4.
                  in_chans (int): Number of input image channels. Default: 3.
                  embed_dim (int): Number of linear projection output channels. Default: 96.
                  norm_layer (nn.Module, optional): Normalization layer. Default: None
              """

The Conv2D should be responsible for both the patch splitting and then the embedding