Sparse attention

[slavavs]

Has anyone seen the implementation of this architecture on pytorch?
https://github.com/openai/sparse_attention/blob/master/attention.py

[dav-ell]

I haven’t, but I’d definitely like to see one!

Any updates on this?

[ptillet (Philippe Tillet)]

FOr anyone who is interested, there is a PyTorch implementation of blocksparse attention here: github.com/ptillet/torch-blocksparse/