Serving Model Trained with PyTorch

1

Tensorflow has Tensorflow Serving. I know pytorch is a framework in its early stages, but how do people serve models trained with pytorch. Must it be from Python? I’m specifically looking to serve from C++.

4 Likes
2

We don’t have a way to serve models from C++ right now, and it’s not a priority for us at this stage. There are many things like distributed training and double backward that we’ll be implementing first. Sorry!

2 Likes
3

Would you say that pytorch was built with serving in mind, e.g. for an API, or more for research purposes?

4

We’re more research oriented. We’re rather thinking of creating tools to export models to frameworks that are more focused on production usage like Caffe2 and TensorFlow.

3 Likes
5

Also you mentioned double backward. This is the first I’ve heard of it. I found a paper by Yann LeCun on double backpropagation, but was wondering whether it’s common to use such a method.

6

Hi, I’m playing with a possible solution for serving from C based on TH and THNN. It’ll be limited to statically compilable graphs of course. I should have something to share in the not so distant future.

5 Likes
7

@lantiga Awesome! Let us know if you need any help! I can answer any questions about the structure of our graphs and how can you export them. We still consider these things internal and they will have to change in the near future to support multiple backward and lazy execution.

4 Likes
8

Thank you @apaszke! I’m aware of the fact that the graph structure is going to change considerably in the future, but delving into it now while things are simpler sounds like a good idea to me.

My plan is to focus solely on inference and implement a first graph2c “transpiler”, which will generate C code directly, without exporting to an intermediate format. It may sound hacky but it could actually be enough for us for the moment and it would avoid having to struggle with polymorphic C.
Eventually, this could become a basis for a more refined solution in which we export the graph and have a C runtime execute it.

This is driven by our need of slim deploys and our determination to use pytorch in production :slight_smile:

5 Likes
9

Sure that sounds cool. It doesn’t seem hacky, it’s just a graph compiler. It’s a very good start, and will likely be capable of producing small binaries. Let us know when there’s going to be any progress or in case you have any trouble. We’ll definitely showcase your solution somewhere.

10

let us know, also interested.
For now we will create a python script to export to a Torch7 model, and then use: https://github.com/mvitez/thnets in production code

1 Like
11

Making progress. As soon as I get the first MNIST example to compile I’ll share what I have.

1 Like
12

We need to deploy pytorch models to e.g. Android, so we need a method to export a model. This is my starting point. Can you please tell me if I am on the right way or if I am doing something totally stupid?

import sys
import torch
from torch import nn
from torchvision import models
from torch.utils.serialization import load_lua

def dump(f):
	s = str(f.__class__)
	sys.stdout.write(s[s.rfind('.')+1:-2]+'(')
	for fa in f.previous_functions:
		if isinstance(fa[0], torch.autograd.Function):
			dump(fa[0])
			sys.stdout.write(',')
		if isinstance(fa[0], torch.nn.parameter.Parameter):
			sys.stdout.write('param,')
		elif isinstance(fa[0], torch.autograd.Variable):
			sys.stdout.write('input,')
	sys.stdout.write(')')


class MyNet(nn.Module):
    def __init__(self):
        super(MyNet, self).__init__()
        self.conv1 = nn.Conv2d(3, 16, kernel_size=1, bias=False)
        self.bn1 = nn.BatchNorm2d(16)
        self.conv2 = nn.Conv2d(3, 16, kernel_size=1, bias=True)

    def forward(self, x):
        return self.bn1(self.conv1(x))+self.conv2(x)
        

#net = models.alexnet()
#net=load_lua('model.net') #Legacy networks won't work (no support for Variables)
net = MyNet()
input=torch.autograd.Variable(torch.zeros(1,3,128,128))
output=net.forward(input)
dump(output.creator)
print('')

The output for the simple MyNet will be

Add(BatchNorm(ConvNd(input,param,),param,param,),ConvNd(input,param,param,),)

Thanks

13

This will work for now, but may break in the future. We’re still actively working on autograd internals, and there are two possible ways we can take now, but we’re still thinking which one is the best. The only caveat right now is that instead of BatchNorm you may find BatchNormBackward in the graph. Are you on slack? I can keep you posted about the currently used data structures if you want.

1 Like
14

Yes, please. I have just sent an invitation request for slack to soumith.

15

So, if you’re interested this is what I have so far: https://github.com/lantiga/pytorch2c. (I think) I’m close, I’m working at serializing THStorage right now and probably there’s a number of other issues, but you can start to take a peek.

I’m not sure how profoundly things will have to be reworked with the upcoming changes in autograd, but it’s fun anyway.

3 Likes
16

Quick update: as of commit 9d0fd21, both the feedforward and MNIST tests pass (they verify that the output of the compiled code matches the output from PyTorch for the same input). I also added a few scripts to get up and running quickly, so things are kind of starting to shape up. /cc @apaszke @Eugenio_Culurciello

4 Likes
17

this looks great! the OpenNMT guys might be interested too: @jeansenellart

1 Like
18

Great, very nice work, thank you.

1 Like
19

Since there are some people hacking with autograd internals, I’ve created a slack channel #autograd-internals. I’ll be sending @channel messages every time we make a breaking change to our representation so you can be up to date.

@lantiga Awesome!

1 Like
20

Via @mvitez:
For your information, I have created a PyTorch exporter that dumps the execution graph to a pymodel.net file that thnets will be able to read. All the models in torchvision work.

5 Likes