I’ve been wondering about the proper way of handling and stopping inference if a client cancels their request. My use case is that I have a custom TorchServe handler, which implements the
postprocess methods. The process takes a while - maybe around 50-60 seconds, so I thought it would be a good idea to handle any cancelations by interrupting the inference progress so that further requests can continue faster.
Is there any method, which I can implement to give me details about the underlying connection, such as when it was forcefully closed by the client?