Using PyCharm to debug Pytorch model on GCE, AWS or Azure

deployment
1

Hello everyone,
I know my question is not directly related to deep learning or Pytorch, but I have found this community so helpful and one time more I need your help.

I have implemented a model using Pytorch, but I do not have access to GPU on my local environment and using Google Colab is so frustrating because of lack of access to debugging utilities.

Now I come across this idea to use Google Preemptible Compute Engine or Amazon EC2 Spot or other cloud services, to test my implementation.
The problem is I am not sure about connecting cloud runtime to my local IDE and run debugs by using breakpoints in my local IDE.

I have read about this in the docs on clouds and PyCharm and it seems it is possible, but I am not sure about it.

Thanks for your replies.

2

It is possible to do remote debugging from PyCharm Professional to the cloud instances through ssh connection. And in addition to GCP and AWS, it works with Colab too. Done that.

3

Thanks for your reply.

But, are you sure about Google colab? Because, based on the documentation, you can control your local environment through colab, not controlling colab through your local environment.

I have googled a lot, but I did not find any procedure to make it happen.
Can you please provide any documenation about using Google colab as a remote debugger?

Thank you

4

Yes, I’m sure. Done that several times. Having free K80 GPU right in PyCharm is super handy when you want to make your code GPU proofed.

Configuration is tricky, multistep process, but doable. The main problem is that Colab VM instance is hidden behind a firewall. But inside of the virtual machine lives Linux Ubuntu, co we can do some hocus-pocus.

High level approach would be to:

  1. Create “setup ssh forwarding” jupyter notebook in Google Colab. You can use my configuration notebook as a template for yours - demo_colab_ssh_access. Make sure you generate and then use your RSA access keys, customize port numbers, etc.
  2. Setup and start ssh server on running Colab VM instance.
  3. Establish ssh tunnel from Colab VM instance to some server on the internet, forward traffic from port 22 to some port there and…
  4. Expose this port for external access. portmap.io service allows one port forwarding for free.
  5. From now on you have ability to ssh from your computer to Colab VM instance and you can proceed with configuring remote interpreter via ssh in PyCharm.
  6. Every time you start new Colab session you’ll just replay “setup ssh forwarding” jupyter notebook and voila :slight_smile:

Ping me if you get stuck.

4 Likes
5

Hi again,
Sorry, I got stuck in the final step.

I do not know what I should write in the Allowed IP section. By the way, I get this error Every time I try to make a tunnel from my local machine to google Colab:

Warning: remote port forwarding failed to listen port XXXX

XXXX is the port portmap.io assigned to my rule.

6

Leave it empty for now.

I don’t know which step doesn’t work for you. But… it took me best part of a year to master port forwardings, anyway :wink:

I can help you to debug this interactively, preferably through audio + screen sharing. Would you like to do it? Hangouts or Skype perhaps?

7

Ow, Actually, I am taking this semester final tests, meanwhile I do not have access to any reliable internet connections at the university unfortunately (I know it is weird but it is the truth!).

I will try more next time and if I really have no other options to proceed, I will let you know.

I am so appreciated for your help.
Thanks again.

8

Thank you Wojciech for the resources. I managed to set up a remote PyCharm interpreter.

Is it also possible/did you try to set up PyCharm to use the VM instance as a Jupyter server?

  • Using Managed Server with the remote interpreter does not seem to do what I want - it seems to still run locally
  • with Configured Server I am not sure how to form the URL to use (if even possible) the ssh tunnel
9

Is it also possible/did you try to set up PyCharm to use the VM instance as a Jupyter server?

Yes. I used “Configured Server” option to access dockerized jupyter server, but I don’t remember exact URL form.

If you can access jupyter server from a browser on your computer through ssh tunnel, then it should be straightforward to configure PyCharm.

10

Thank you for your tremendous contribution. I encountered one problem in configuring remote interpreter in Pycharm. Actually, I added remote interpreter but I got an error:

Indicated packet length 1416128883 too large.

I tried removing the line of ls -l /root/.ssh, but not working.
I also added a line of [[ $- == i ]] || return at the beginning of .bashrc. It was not working either.

Can you help me out, please?
Furkan

11

Hello @wojtekcz

Also from me a big thank you for this guide! I tried following your guide, but got stuck and hoped that you could help me out? It seems that I was able to successfully execute the instructions of “demo_colab_ssh_access”. Because when I run the cell with

# ssh tunnel to portmap.io

options = '-oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null'

!ssh $options -i $fn_private_key_pem -f -R 53525:localhost:22 wojtekcz.first@wojtekcz-53525.portmap.io -N

, I get as output

Warning: Permanently added 'wojtekcz-53525.portmap.io,<three digits>.<three digits>.<three digits>.<two digits>' (ECDSA) to the list of known hosts., which I take as it having worked successfully!

(of course, I have replaced “53525”, “wojtekcz” and “first” appropriately, as well as the value for “fn_private_key_pem”)

The next step in your guide was to follow “configuring remote interpreter via ssh in PyCharm” and this is where I got stuck. I tried the “New server configuration” option with <three digits>.<three digits>.<three digits>.<two digits> as my Host, 22 as port and wojtekcz-53525.portmap.io as my username. As my authentication option, I picked “Key pair (OpenSSH or PuTTY):” with my “private_key_pem”, but got

Connection to wojtekcz-53525.portmap.io@<three digits>.<three digits>.<three digits>.<two digits>:22 failed:

Timeout expired

as an error. And thus, I have been scratching my head what I am doing wrong.

12

@Cdfghglz I hope it’s okay to ask you, since wojtekcz doesn’t seem to be active on the pytorh forum anymore. Since it seems that you managed to follow wojtekcz’s guide, could I ask you for help with the problem I encountered, described in the post previous to this one to this topic?