# Transformer architecture in PyTorch using Mac M1 GPU via MPS

**URL:** <https://discuss.pytorch.org/t/transformer-architecture-in-pytorch-using-mac-m1-gpu-via-mps/188405>\
**Category:** mps\
**Created:** [September 15, 2023, 4:45pm UTC](https://discuss.pytorch.org/t/transformer-architecture-in-pytorch-using-mac-m1-gpu-via-mps/188405 "2023-09-15T16:45:05Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![radicalprotns](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/radicalprotns/32/61958_2.png) [@radicalprotns](https://discuss.pytorch.org/u/radicalprotns)\
**Post date:** [September 15, 2023, 4:45pm UTC](https://discuss.pytorch.org/t/transformer-architecture-in-pytorch-using-mac-m1-gpu-via-mps/188405/1 "2023-09-15T16:45:05Z")

</div>

I have a Mac M1 GPU (macOS 13.5.1, PyTorch 2.0.1) and I’m new to using the M1 GPU for deep learning. I’ve been playing around with the [Informer](https://github.com/zhouhaoyi/Informer2020) architecture which is the transformer architecture applied to time series forecasting. However, upon changing the device from ‘cuda’ to ‘mps’ in the code, I cannot replicate the example provided by the authors in this [google colab notebook](https://colab.research.google.com/drive/1_X7O2BkFLvqyCdZzDZvV2MB0aAvYALLC). I obtain the same results running with CPU but not with MPS.

As the informer code is more involved, I looked for a simpler example that I’d have problems reconciling with the M1 GPU. The image classification tutorial on PyTorch did not pose an issue. However, [this example on sequence to sequence network and attention](https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html) fails on my M1 GPU. In particular, I get the following result while training the transformer model:

 ![Screenshot 2023-09-16 at 12.27.46 AM](https://discuss.pytorch.org/uploads/default/original/3X/9/3/93fb285d4619731f1ee84e7f04736f7994a1a5bd.png)

I get nan’s for the loss while the loss gets down to 0.03 in the notebook. (The speed is another issue, unfortunately). The notebook shows a run time of around 6 minutes (CUDA) while it takes 40 minutes on my M1 GPU).

To those who have Mac M1 GPU’s, can you replicate the results in the PyTorch tutorial above? Does anyone know if there is an element of the transformer architecture which is not supported by MPS yet? It doesn’t throw off any errors but I suspect that adjustments have to be made?

Help appreciated!

---

<div class="post-metadata">

**Author:** ![radicalprotns](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/radicalprotns/32/61958_2.png) [@radicalprotns](https://discuss.pytorch.org/u/radicalprotns)\
**Post date:** [September 17, 2023, 6:23am UTC](https://discuss.pytorch.org/t/transformer-architecture-in-pytorch-using-mac-m1-gpu-via-mps/188405/2 "2023-09-17T06:23:47Z")

</div>

To anyone wondering, apparently, there are problems with GRU on mps. See [this link](https://github.com/pytorch/pytorch/issues/94691)