Performance regression between 2.1.0.dev20230621 (with cuda 1.17.1) and 2.1.0.dev20230628 (with cuda 12.1)

C++
1

I know it isn’t a great topic, but just want to bring this up as it did surprise me.

2

Could you describe what kind of performance regression you were seeing?

3

Runtime of an inference run got like 3x/4x slower.

4

Your information is unfortunately too vague and we didn’t observe any regression in our nightly testing.

5

I understand it’s very vague … I will try to narrow down the problem, at least in term of builds ; we were using the 1.17 cuda build until now (through the docker package), I will try to switch to the 1.18 one when I get a chance, or back to the first cuda 1.12.1 available to see if it has anything to do with this.