Neural Feature Importance

1

Is there any PyTorch-supported work for ‘Neural Feature Importance’ extraction?
I have a trained encoder for 1D spectral data. Now I want to know which features are essential in determining the latent representation of the data in low dimensional space.

I have come across the following:

  1. SHAP
  2. DeepLift.

and other computationally expensive methods for box models like feature permutation.

But SHAP does not support PyTorch fully.

Any idea what could be the workaround?