I’m running on a Google Cloud Compute VM. Essentially, when I submit a training command via the gcloud CLI, the program hangs during dataloading, but when I SSH in and run the exact same command interactively, it runs perfectly fine. I’m running a pretty standard loop, using tqdm to track progress
for x, y in tqdm(valloader):
...
with the training command and some command line arguments wrapped up in a shell script, e.g.:
# val.sh
python val.py --batch-size 256 --epochs 30 --lr 2e-5 --num-workers 12
The exact command is something like gcloud compute ssh <instance-name> --command "nohup val.sh > train.out &".
The program ends up spawning a bunch of workers (that I can see with ps) but they all end up hanging, and in the log (train.out) I can see a bunch of logging information from before the validation loop, but no output from tqdm to indicate that dataloading has begun. However, when I run the same command after directly SSHing in, everything works fine. Any thoughts?