iOS metal backend expected true but false

I successfully converted a model to use the metal backend and it shows in the opnames metal_prepack::conv2d run like in the example on the pytorch site, but it doesnt show metal::copy_to_host

I try using it an iOS app and definitely call .metal() on the tensors (it has 2 inputs) but get the following error

Expected input1.is_metal() to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)
  Debug info for handle(s): debug_handles:{-1}, was not found.
Exception raised from sub_Tensor at /Users/distiller/project/aten/src/ATen/native/metal/ops/ (most recent call first):

Could anyone point me to a reason this would happen? The cpu version of the model works with no issues