Shapes matched properly, but I get error: RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

Hello,
I have an error and I paste the error message below:

Basically, I was doing a classification task with a model, and I get this error when I try to do back propagation. I was trying to solve this issue, so I checked a few posts that say that this error is due to the shape-matching problem.
However, for my error, I doubt that this is a shape-not-matching problem, because forward propagation works without any error. I can calculate loss just fine on cuda without any problems, as you can see in the screenshot.

FYI, the loss is categorical cross entropy loss! :slight_smile:
I will wait for your replies!
Thank you.

Could you run your code on the CPU and check if you are getting a better error message?
We have unfortunately seen a few errors wrongly reported as cuBLAS or cuDNN issues as the actual error was not properly caught and raised.

Hello, thank you so much for your reply! :smiley:

the one that I got an error was from the sever of my company’s environment.
However, as you could see from the attached screenshot below, I didn’t get the error that I reported, from a different environment from that of my company’s. So I think it’s probably a specific version of a library that is throwing me the error. I’ll check on Monday and figure out which version of which library is problematic.

Thank you so much! :slight_smile:

image

Could you update to the latest stable or nightly release, too?
In case you can still reproduce the issue on the other machine, could you post a minimal and executable code snippet, please?