Hello. I have a Gunicorn application running in a sync mode. Firstly, I load models into memory via share_module().
Then Gunicorn forks the main process, and I additionally fork Gunicorn workers to achieve parallelism on the request level.
Everything works locally and in Docker. But once I enable docker read-only mode (give a RW access to /var/tmp) (this is a security requirement), the state of models (state_dict) can sometimes be corrupted during os.fork(). By corrupted I mean the state_dict coefficients are changed randomly which leads to incorrect model output.
I experience this behavior with both file_descriptor and file_system sharing strategies.
How can I use these Docker flags and fork processes with PyTorch safely?
I increased shm_size parameter in Docker so this is not a bottleneck.
I use only CPU calculations.