siv
(Siva Sundaramoorthy)
August 31, 2017, 8:31am
1
I tried the seq2seq pytorch implementation available here pytorch-seq2seq . After profiling the evaluation(evaluate.py ) code, the piece of code taking longer time was the decode_minibatch method( github.com/MaximumEntropy/Seq2Seq-PyTorch/blob/master/evaluate.py#L74 )
Trained the model on GPU and loaded the model in CPU mode to make inference. But unfortunately, every sentence seems to take ~10sec. Is slow prediction expected on pytorch?
Any fixes, suggestions to speed up would be much appreciated. Thanks.
hello, I wonder know if you have solved the problem in the speed of seq2seq ?
siv
(Siva Sundaramoorthy)
November 30, 2017, 10:01am
3
Hi,
Looks like pytorch is not Optimized for CPU as it is done for GPU. And also on their high priority list to fix the same
You Can refer the below links on the same.
opened 05:13PM - 23 May 17 UTC
closed 01:45AM - 26 May 17 UTC
**Summary - PyTorch seems to be half as fast as plain Python for very simple net… works.**
### Context
I am learning PyTorch because it promises convenience and easy access to GPU acceleration.
In learning, I re-implemented my previous very simple tutorial 3-layer neural network created from scratch in Python with Numpy.
### Code
The two are here:
- simple python+numpy: [https://github.com/makeyourownneuralnetwork/makeyourownneuralnetwork/blob/master/part2_neural_network_mnist_data.ipynb](https://github.com/makeyourownneuralnetwork/makeyourownneuralnetwork/blob/master/part2_neural_network_mnist_data.ipynb)
- PyTorch: [https://github.com/makeyourownneuralnetwork/pytorch/blob/master/pytorch_neural_network_mnist_data.ipynb](https://github.com/makeyourownneuralnetwork/pytorch/blob/master/pytorch_neural_network_mnist_data.ipynb)
### Results
They are built to be as similar as possible - same size, same loss function, same training data, etc
The result of 2 timing tests show that the **plain python version is about twice as fast as the PyTorch version**:
- home-made simple pure python - 4**40 seconds, 458 seconds**
- simple PyTorch version - **841 seconds, 834 seconds**
### More Detail
Here's a blog which details the experiment conditions in more detail (near the bottom of the post).
- blog [http://makeyourownneuralnetwork.blogspot.co.uk/2017/05/learning-mnist-with-gpu-acceleration.html](http://makeyourownneuralnetwork.blogspot.co.uk/2017/05/learning-mnist-with-gpu-acceleration.html)
### What Am I Doing Wrong?
Or does PyTorch have too much overhead for simple scenarios, and only really shines for larger or GPU accelerated cases?
twitter.com/haldaume3/status/900775899431305217
If you are wondering what I did next after this bottleneck. I switched to the tensorflow implementation.