Hi everyone,
I am training a neural network with a loss function denoted as L. I aim to constrain the network’s outputs (z) such that they lie on a hypersphere of dimension d.
To maintain a valid training framework, I need the backpropagated gradients to respect this constraint, meaning they should be tangent to the hypersphere at the corresponding points.
To achieve this, I plan to project the gradients ∂L/∂z, where z is the network’s output, onto the tangent plane of the hypersphere. Based on my understanding, this projection can be computed as:
g_tangent=g−(g⋅z_normalized) z_normalized,
where g=∂L/∂z and z_normalized is the normalized version of z.
However, I’m having difficulty accessing these gradients, modifying them, and ensuring the loss is properly backpropagated after applying this projection.
I’d greatly appreciate any feedback, suggestions, or guidance on implementing this process. Thank you so much for your help!