Hi everyone, I’m not sure if this is the right place to post this issue or if it’s more CUDA-related.
I have a super simple script which should load a torch::jit::script::Module and run some forward passes on a dummy tensor.
However, depending on the model I use, it gets stuck either when it tries to load the model (model1) or when it runs the first forward pass (model2.pt), and I have to manually kill the process.
I’m working on a Windows 10 laptop with a NVIDIA GeForce RTX 3080 and:
- CUDA Toolkit 10.2.
- LibTorch 1.10.1 for CUDA 10.2 (both Release and Debug).
- Visual Studio 2019
The output of nvidia-smi is:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 466.81 Driver Version: 466.81 CUDA Version: 11.3 |
|-------------------------------+----------------------+----------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... WDDM | 00000000:01:00.0 Off | N/A |
| N/A 0C P5 22W / N/A | 183MiB / 16384MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
I built a VS project with this template. The code is the following:
#include <iostream>
#include <ATen/Context.h>
#include <torch/torch.h>
#include <torch/script.h>
int main() {
const torch::Device device = torch::Device(torch::kCUDA, 0);
torch::jit::script::Module model;
std::string MODEL_PATH = "path/to/model1.pt";
// std::string MODEL_PATH = "path/to/model2.pt"
// Load Model
std::cout << "Trying to load model..." << std::endl;
try {
model = torch::jit::load(MODEL_PATH, device);
model.eval();
std::cout << "AI model loaded successfully." << std::endl;
}
catch (const c10::Error& e) {
std::cerr << e.what() << std::endl;
}
std::cout << "Warming up model..." << std::endl;
auto dummy = torch::zeros({ 1, 3, 512, 512}).to(device);
torch::Tensor output;
std::vector<torch::jit::IValue> inputs;
inputs.clear();
inputs.emplace_back(dummy);
std::cout << "\tIteration: ";
for (int i = 0; i < 20; i++) {
std::cout << i << ", ";
output = model.forward(inputs).toTensor();
torch::cuda::synchronize();
}
}
Output for model1
Trying to load model...
Output for model2
Trying to load model...
AI model loaded successfully.
Warming up model...
Iteration: 0,
I can’t figure out what could be the problem or what could I do to further investigate it.