Any ideas how one could potentially make a function to measure GPU / CPU utilization during training process?
One can have a separate script launching up nvidia-smi for GPU utilization and reading from proc filesystem for CPU utilization. Another option is enabling GPU profiling in PyTorch and running it with nvprof.
Does anyone have an idea for a simple Python way to do the same but easier?