Inference of transformer model on mac os x laptop

deployment
1

Hi All

I have been working on an email assistant. I fine tuned one of Huggingface’s transformers on my emails to classify them into different categories. It works really well on my linux ML workstation

But now I’m trying to implement the model to actually work on my emails in Apple Mail and I’m quite lost on how to do this. I managed to convert the model to ONNX.

I know I can run Applescripts on incoming email. So my idea would be to call an AppleScript on every incoming email, that calls a python script, which runs the model and returns probabilities for the trained categories.
But that all seems quite cumbersome and will probably be slow with the model loading on every incoming email.

Does any of you know a simpler way? Or an easy tutorial or example about how to do this?

2

I’d suggest you use an inferencing framework like https://github.com/pytorch/serve

So you’d setup your model behind a REST API which you would then call from applescript