Is PyTorch slow on the CPU?

1

Recently there was this thread (https://twitter.com/gneubig/status/899826900834689028) on Twitter about how PyTorch is currently much slower than DyNet (between 10x and 20x). Is this exclusively a problem with CPU runs, i.e., the code would be much more competitive on the GPU? Why is it so slow and how likely is to be addressed in the recent future? Soumith promised (https://twitter.com/haldaume3/status/900775899431305217) that it would be addressed soon, but I would like to be sure that it is in fact addressable before investing too much into PyTorch (which I’ve already done a bit). A 20x slowdown is prohibitively slow.

1 Like
2

Torch falls almost at the last of the chart:

soumith_convnet-benchmarks__Easy_benchmarking_of_all_publicly_accessible_implementations_of_convnets
soumith_convnet-benchmarks__Easy_benchmarking_of_all_publicly_accessible_implementations_of_convnets1050×714 146 KB

3

@negrinho we’re addressing these issues in the short term (~1month). We’ll reply on the linked thread once we have a fuller visibility with action items.

2 Likes
4

@QuantScientist no that is irrelevant. these are old benchmarks that are:

  1. not pytorch
  2. for convolutions
  3. dont take cudnn into account.
5

@smth
You are right, my apologies :slight_smile:

6

@smth Has there been follow-up work on this? Thanks!