I am trying to implement the above algorithm in pytorch since the provided matrix_exp is too slow for my use case. I was able to implement all but the loop at step 11-12 in a fairly efficient way (no loops). Is there any way to implement that last part? I couldn’t find a way to apply matrix power based on integers (the powers) stored in a separate tensor without looping over each which would be too slow. Thanks in advance.
Hi Sam!
The short answer is that trying to write your own version of standard
numerical-analysis algorithms is almost always a bad idea.
Your issue is that pytorch’s matrix_exp is “too slow,” but then you
note that your approach would also be “too slow.” Please consider
the possibility that what you want to calculate – the exponential of a
matrix – represents a substantive piece of computation, and is likely
to be “slow,” whoever implements the algorithm. While it’s not
impossible that pytorch is doing a bad job of this, it’s much more likely
that it’s doing a better job than you would be able to.
Some comments:
Pytorch’s implementation uses an algorithm quite similar to your
reference algorithm. (Your reference algorithm is, to be sure, a
solid, legitimate algorithm for matrix exponentiation.)
People have been doing this kind of computerized numerical analysis
for fifty years now – and, in fact, quite a bit longer. The experts really
do know what they’re doing.
Lastly, pytorch’s matrix_exp supports autograd and backpropagation
out of the box. Your version will too if you make sure to use pytorch
tensor operations throughout, including inside the loops. But doing
so will create a large (one “layer” for each loop iteration) computation
graph, and will likely require excessively more memory and perform
backpropagation significantly more slowly than pytorch’s.
(As an aside, please don’t post screenshots of textual material. Doing
so breaks accessibility, searchability, and copy-paste.)
Best.
K. Frank
Hi Frank,
Thanks for the detailed response, I don’t expect to make a faster implementation of the exact matrix exponential, but I need a more efficient approximate algorithm such as this one as it allows for specifying the precision. I am currently using a naive truncation with ~ 15 first terms and it is several seconds faster per batch and than matrix_exp with results being within ~1e-8 of it’s output in most cases. I wanted to improve on that with the above algorithm which guarantees a certain error margin and from what I have found the algorithm I shared here is supposed to work well for small dimensions (my matrices are < 7x7).