# Dealing with matrix inversion efficiently

**URL:** <https://discuss.pytorch.org/t/dealing-with-matrix-inversion-efficiently/94045>\
**Category:** Uncategorized\
**Created:** [August 25, 2020, 11:59am UTC](https://discuss.pytorch.org/t/dealing-with-matrix-inversion-efficiently/94045 "2020-08-25T11:59:14Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![kl\_divergence](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kl_divergence/32/5069_2.png) [@kl\_divergence](https://discuss.pytorch.org/u/kl_divergence)\
**Post date:** [August 25, 2020, 11:59am UTC](https://discuss.pytorch.org/t/dealing-with-matrix-inversion-efficiently/94045/1 "2020-08-25T11:59:14Z")

</div>

I have two matrix `X` and `Y`. Both are of dimension `[bs, hidden_size`].  
I want to take inverse of matrix obtained by `torch.cat([X, Y], dim=1)`.

Pseudo inverse is not stable (docs say the same). I obtain inverse to deal with this. So I used a `Linear` layer of dimension `(hidden_size, batch_size//2)` to convert X and Y of dimensions `[bs, bs//2`]. After concat, I get a matrix of form `[bs, bs]`. The downside of this approach is that it requires me to use a `batch size = hidden_size/2` to perform well, otherwise the network doesn’t learn anything well.

Is there a mathematical elegant way you can suggest as a workaround ?

---

<div class="post-metadata">

**Author:** ![KFrank](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/32.png) [@KFrank](https://discuss.pytorch.org/u/KFrank)\
**Post date:** [August 25, 2020, 4:19pm UTC](https://discuss.pytorch.org/t/dealing-with-matrix-inversion-efficiently/94045/3 "2020-08-25T16:19:03Z")

</div>

Hello kl!

> [@kl\_divergence](#):
>
> I have two matrix `X` and `Y`. Both are of dimension `[bs, hidden_size`].  
> I want to take inverse of matrix obtained by `torch.cat([X, Y], dim=1)`.

Could you explain conceptually what meaning this matrix inverse  
is supposed to have? Your construction seems odd.

As you have recognized, a matrix must be square to be invertible  
(although not all square matrices are invertible, of course). But  
your batch size (`bs`) is something of a 'technical" parameter that  
doesn’t really have anything to do with your data or the structure of  
your network. As you point out, you have to adjust your batch size  
to match `hidden_size / 2`. But how does your construction make  
sense if it breaks when you change your batch size?

Best.

K. Frank

---

<div class="post-metadata">

**Author:** ![kl\_divergence](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kl_divergence/32/5069_2.png) [@kl\_divergence](https://discuss.pytorch.org/u/kl_divergence)\
**Post date:** [August 26, 2020, 3:04am UTC](https://discuss.pytorch.org/t/dealing-with-matrix-inversion-efficiently/94045/4 "2020-08-26T03:04:14Z")

</div>

Thanks for taking interest. The matrix inversion is part of a formula that is supposed to project features orthogonally.  
 ![Screenshot_20200826_083836](https://discuss.pytorch.org/uploads/default/original/3X/5/f/5f0973eac2b058026a976b5429fe30be2950c0ae.png)

`$F_{G}$` (comes from concatenation of two vectors) is not a square matrix, that’s why I make this modification.

The issue is, that my network performs much better if I reduce second dimensions of X and Y to half of what it was (768-\>384), in all other cases, it doesn’t learn anything. There might be better ways of projection, which I’ve not explored. (For instance, if bs=512, 768-\>512, then I don’t notice any improvement.)

This is an unstable method. The main issue is to perform inversion. That implies I need square matrices. That’s why I’m asking if there’s a better mathematical way to perform this.  
Yes, I realize that my implementation is not batch size agnostic once model is trained. As a result, I am unable to train bigger models (because batch\_size = hidden\_size/2 won’t fit in memory).

So in general sense, what is a better way to perform inversion when the matrix is not square (`pinverse` isn’t stable) ?

---

<div class="post-metadata">

**Author:** ![sprakashdash](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sprakashdash/32/48780_2.png) [@sprakashdash](https://discuss.pytorch.org/u/sprakashdash)\
**Post date:** [June 7, 2024, 12:10pm UTC](https://discuss.pytorch.org/t/dealing-with-matrix-inversion-efficiently/94045/5 "2024-06-07T12:10:26Z")

</div>

`torch.linalg.solve()` is a good alternative for stability purposes.