When running a pytest action on GitHub Actions Mac OS, I inconsistently get an error message:
RuntimeError: MPS backend out of memory (MPS allocated: 0 bytes, other allocations: 0 bytes, max allowed: 1.70 GB). Tried to allocate 0 bytes on private pool. Use PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system failure).
from a line running a_tensor.to('mps')
If I keep re-running the test it eventually succeeds, but sometimes it takes >5 attempts. I don’t understand why the error is occurring or how to prevent it. Setting the environment variable PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 as suggested in the error message does not resolve the problem. It occurs, at least occasionally, with Python versions 3.9, 3.10, and 3.11.
The issue does not occur when I run pytest locally on Mac OS with Apple Silicon.
The github action lists the following operating system information:
macOS
12.6.9
21G726
The installed pytorch is torch (2.0.0)