Tensor dimensionality and speed impact

1

Newbie question here. In my research I found this thread on word embedding dimensions stating:

You can pick a power of 32 (64, 128, 256) to speed up modeling training depending on your software framework and hardware.

Is that also a property in PyTorch and is using a power of 32 increasing speed?

2

Generally speaking, multiples of 8 or 32 will perform better on most GPUs. Depends on your specific setup, but it’s not something I’d prioritize too much unless you are just optimizing.

1 Like