Hello,
when I use the SparseAdam — PyTorch 1.12 documentation I get the error

``````ValueError: Sparse params at indices [0, 1]: SparseAdam requires dense parameter tensors
``````

when I initialize my `nn.Module`-Layer with a sparse weight matrix (as a `nn.Parameter`).

I wonder wether this is an error or indeed wanted. The second would lead me to the question why you would need a SparseAdam optimizer if it is not working with sparse weights.

That’s expected since `SparseAdam` expects sparse gradients (e.g. from `nn.Embedding(..., sparse=True)`).

inputs,outputs and weights of the model are initialized as `torch.sparse.FloatTensor` and the only model layer is a simple sparse matrix multiplication. Where do I have to change the gradients?

Do I have to define the backward and the forward by myself with an autograd function?

Yes, this might be a possible approach and you could then use `torch.sparse.addmm()` to create the sparse gradients. However, a custom `nn.Module` might be sufficient as the operation should be differentiable and thus you wouldn’t need to implement the `backward` as well. This list shows the supported operations and if the resulting gradient will be sparse.