TorchServe: MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding

venkat.r · November 25, 2021, 8:25pm

While serving the model on Docker, i get the following error when it tried to load the saved model.

Strangely it cant also not find torch although it is install.

2021-11-25 20:18:12,416 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager -  Loading snapshot serializer plugin...
2021-11-25 20:18:12,438 [INFO ] main org.pytorch.serve.ModelServer - Loading initial models: ponzi.mar
2021-11-25 20:18:12,455 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 0.1 for model ponzi
2021-11-25 20:18:12,456 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 0.1 for model ponzi
2021-11-25 20:18:12,456 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model ponzi loaded.
2021-11-25 20:18:12,456 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: ponzi, count: 12
2021-11-25 20:18:12,514 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel.
2021-11-25 20:18:12,683 [WARN ] W-9002-ponzi_0.1-stderr MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding
2021-11-25 20:18:12,684 [INFO ] W-9005-ponzi_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9005-ponzi_0.1-stdout
2021-11-25 20:18:12,683 [WARN ] W-9009-ponzi_0.1-stderr MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding
2021-11-25 20:18:12,687 [INFO ] W-9003-ponzi_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9003-ponzi_0.1-stdout
2021-11-25 20:18:12,683 [WARN ] W-9010-ponzi_0.1-stderr MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding
2021-11-25 20:18:12,683 [WARN ] W-9003-ponzi_0.1-stderr MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding
2021-11-25 20:18:12,689 [WARN ] W-9010-ponzi_0.1-stderr MODEL_LOG - Traceback (most recent call last):
2021-11-25 20:18:12,689 [WARN ] W-9005-ponzi_0.1 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9005-ponzi_0.1-stderr
2021-11-25 20:18:12,683 [WARN ] W-9007-ponzi_0.1-stderr MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding
2021-11-25 20:18:12,689 [WARN ] W-9005-ponzi_0.1 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9005-ponzi_0.1-stdout
2021-11-25 20:18:12,690 [WARN ] W-9007-ponzi_0.1-stderr MODEL_LOG - Traceback (most recent call last):
2021-11-25 20:18:12,690 [WARN ] W-9007-ponzi_0.1-stderr MODEL_LOG -   File "/home/model-server/tmp/models/d272938c278d47eb9c045c2a929e667b/encodings.py", line 1, in <module>
2021-11-25 20:18:12,683 [WARN ] W-9004-ponzi_0.1-stderr MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding
2021-11-25 20:18:12,683 [WARN ] W-9001-ponzi_0.1-stderr MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding
2021-11-25 20:18:12,690 [WARN ] W-9007-ponzi_0.1-stderr MODEL_LOG - ModuleNotFoundError: No module named 'torch'
2021-11-25 20:18:12,689 [WARN ] W-9003-ponzi_0.1-stderr MODEL_LOG - Traceback (most recent call last):
2021-11-25 20:18:12,689 [WARN ] W-9002-ponzi_0.1-stderr MODEL_LOG - Traceback (most recent call last):
2021-11-25 20:18:12,688 [WARN ] W-9005-ponzi_0.1-stderr MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding
2021-11-25 20:18:12,690 [WARN ] W-9003-ponzi_0.1-stderr MODEL_LOG -   File "/home/model-server/tmp/models/d272938c278d47eb9c045c2a929e667b/encodings.py", line 1, in <module>
2021-11-25 20:18:12,690 [WARN ] W-9004-ponzi_0.1-stderr MODEL_LOG - Traceback (most recent call last):
2021-11-25 20:18:12,690 [WARN ] W-9003-ponzi_0.1-stderr MODEL_LOG - ModuleNotFoundError: No module named 'torch'
2021-11-25 20:18:12,684 [INFO ] W-9006-ponzi_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9006-ponzi_0.1-stdout
2021-11-25 20:18:12,683 [WARN ] W-9000-ponzi_0.1-stderr MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding
2021-11-25 20:18:12,683 [WARN ] W-9006-ponzi_0.1-stderr MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding
2021-11-25 20:18:12,690 [WARN ] W-9002-ponzi_0.1-stderr MODEL_LOG -   File "/home/model-server/tmp/models/d272938c278d47eb9c045c2a929e667b/encodings.py", line 1, in <module>
2021-11-25 20:18:12,690 [WARN ] W-9004-ponzi_0.1-stderr MODEL_LOG -   File "/home/model-server/tmp/models/d272938c278d47eb9c045c2a929e667b/encodings.py", line 1, in <module>
2021-11-25 20:18:12,690 [INFO ] W-9005-ponzi_0.1-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9005-ponzi_0.1-stderr
2021-11-25 20:18:12,690 [WARN ] W-9001-ponzi_0.1-stderr MODEL_LOG - Traceback (most recent call last):
2021-11-25 20:18:12,691 [WARN ] W-9004-ponzi_0.1-stderr MODEL_LOG - ModuleNotFoundError: No module named 'torch'
2021-11-25 20:18:12,691 [WARN ] W-9001-ponzi_0.1-stderr MODEL_LOG -   File "/home/model-server/tmp/models/d272938c278d47eb9c045c2a929e667b/encodings.py", line 1, in <module>
2021-11-25 20:18:12,691 [WARN ] W-9001-ponzi_0.1-stderr MODEL_LOG - ModuleNotFoundError: No module named 'torch'
2021-11-25 20:18:12,689 [WARN ] W-9009-ponzi_0.1-stderr MODEL_LOG - Traceback (most recent call last):
2021-11-25 20:18:12,692 [WARN ] W-9009-ponzi_0.1-stderr MODEL_LOG -   File "/home/model-server/tmp/models/d272938c278d47eb9c045c2a929e667b/encodings.py", line 1, in <module>
2021-11-25 20:18:12,692 [WARN ] W-9009-ponzi_0.1-stderr MODEL_LOG - ModuleNotFoundError: No module named 'torch'
2021-11-25 20:18:12,689 [WARN ] W-9010-ponzi_0.1-stderr MODEL_LOG -   File "/home/model-server/tmp/models/d272938c278d47eb9c045c2a929e667b/encodings.py", line 1, in <module>
2021-11-25 20:18:12,683 [WARN ] W-9008-ponzi_0.1-stderr MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding
2021-11-25 20:18:12,684 [WARN ] W-9011-ponzi_0.1-stderr MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding
2021-11-25 20:18:12,692 [WARN ] W-9010-ponzi_0.1-stderr MODEL_LOG - ModuleNotFoundError: No module named 'torch'
2021-11-25 20:18:12,691 [WARN ] W-9002-ponzi_0.1-stderr MODEL_LOG - ModuleNotFoundError: No module named 'torch'
2021-11-25 20:18:12,691 [WARN ] W-9003-ponzi_0.1 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9003-ponzi_0.1-stderr
2021-11-25 20:18:12,691 [WARN ] W-9006-ponzi_0.1-stderr MODEL_LOG - Traceback (most recent call last):
2021-11-25 20:18:12,692 [WARN ] W-9006-ponzi_0.1-stderr MODEL_LOG -   File "/home/model-server/tmp/models/d272938c278d47eb9c045c2a929e667b/encodings.py", line 1, in <module>
2021-11-25 20:18:12,693 [WARN ] W-9006-ponzi_0.1-stderr MODEL_LOG - ModuleNotFoundError: No module named 'torch'

The Docker file setup is like this.

FROM pytorch/torchserve:latest

USER root
RUN printf "\nservice_envelope=json" >> /home/model-server/config.properties \
    && pip install pytorch-lightning torchmetrics google-cloud-storage scikit-learn
USER model-server


COPY src /home/model-server/src
COPY load_model.py /home/model-server/src
COPY configs /home/model-server/configs
COPY /models/state_dict.pt /home/model-server/src/
RUN mkdir "model_store"

RUN torch-model-archiver \
    --model-name ponzi \
    --version 0.1 \
    --model-file /home/model-server/src/model.py \
    --serialized-file /home/model-server/src/state_dict.pt \
    --handler /home/model-server/src/handler.py \
    --export-path /home/model-server/model_store \
    --extra-files /home/model-server/configs/features.yml,/home/model-server/configs/model_configs.yml,/home/model-server/src/utils.py,/home/model-server/src/encodings.py,/home/model-server/src/dataset.py,/home/model-server/src/__init__.py

CMD ["torchserve", \
     "--start", \
     "--ncs", \
     "--model-store model_store", \
     "--ts-config=/home/model-server/config.properties", \
     "--models ponzi=ponzi.mar"]

Can someone help here? Thanks.

ptrblck · November 29, 2021, 3:33am

Based on these error messages:

Fatal Python error: Py_Initialize: Unable to get the locale encoding
[...]
ModuleNotFoundError: No module named 'torch'

I would guess that your current setup might be mixing different Python versions (or environments) and thus breaks in Py_Initialize and cannot find torch.
Look through the build logs of your container and check if some pre-installed packages are uninstalled etc.

venkat.r · November 30, 2021, 5:00pm

My local python is 3.8.10 vs Docker which is using 3.6. That could be the problem. What are the possible work around here.

Do i have to use a multi-stage docker build which pulls python3.8 and then does the torch serve bit ?

ptrblck · November 30, 2021, 10:04pm

I don’t have any custom docker containers for torchserve, but you could use the official ones.