Hello
I would like to ask for help, advice and hints about issue I am facing when deploying NN model for time-series predictions with custom handler.
I finally get working the model and inference for time-series prediction. Everything is working fine when i set prediction/output to 1.
But what i need is to get it working for predicting more then 1 values. My model and handler provide these output but I’m getting some errors from Torchserve.
Response i get from curl is
{
“code”: 503,
“type”: “InternalServerException”,
“message”: “number of batch response mismatched”
}
In LOG from Trochserve i see following ( i put there some print for debug):
Postprocess out: [0.6994340419769287, 0.9235875606536865, 0.7007522583007812, 0.8973164558410645, 0.7195687294006348]
[INFO ] W-9003-RNN_HR_model_1.0-stdout MODEL_LOG - model: RNN_HR_model, number of batch response mismatched, expect: 1, got: 5.
So i get 5 values in Inference.
According to documentation for handlers i see there:
def inference(self, model_input):
:return: list of inference output in NDArray
def postprocess(self, inference_output):
:return: list of predict results
When i check my output i see that
MODEL_LOG - The type of model output is <class ‘list’>
But even when i transform the inference output into NDArray and output for postprocess to list as provided in documentation I am still getting the same responses/errors
Any Idea what am i doing wrong? I can’t and dont want to use it in batch mode, since i have only 1 input and output should be 5 forecasted values.
Thank you