MPS Back End Out of Memory on GitHub Action

mps
1

When running a pytest action on GitHub Actions Mac OS, I inconsistently get an error message:

RuntimeError: MPS backend out of memory (MPS allocated: 0 bytes, other allocations: 0 bytes, max allowed: 1.70 GB). Tried to allocate 0 bytes on private pool. Use PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system failure).

from a line running a_tensor.to('mps')

If I keep re-running the test it eventually succeeds, but sometimes it takes >5 attempts. I don’t understand why the error is occurring or how to prevent it. Setting the environment variable PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 as suggested in the error message does not resolve the problem. It occurs, at least occasionally, with Python versions 3.9, 3.10, and 3.11.

The issue does not occur when I run pytest locally on Mac OS with Apple Silicon.

The github action lists the following operating system information:

macOS

12.6.9

21G726

The installed pytorch is torch (2.0.0)

1 Like
2

Hi,

Did you find any solution to this? I have the exact same issue with GitHub workflows, but all tests pass on my local macOS.

Thanks

3

Having the same issue (see here and here), but only since 2.1.0.

4

I don’t have a solution

5

even GH’s new macos-14 runners don’t have access to MPS hardware unfortunately, see pytorch/pytorch#111449 (comment).

6

I used to be able to run. Since ~May, the memory seems to be more limited (or a having other issues)

RuntimeError: MPS backend out of memory (MPS allocated: 0 bytes, other allocations: 0 bytes, max allowed: 7.93 GB). Tried to allocate 44.71 MB on private pool. Use PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system failure).
7

Switching runs-on from macos-latest (which is running on M1) to macos-13 fixed this issue for me. Hopefully they’ll get the issue with the M1 architecture fixed, but this is working for us for now.

1 Like