It gets in the right direction but x and y are also used in another module so I maybe also need a optimizer for that part, making it more complex.
Is there maybe a way to set the kernel gradients with respect to the prototypes to zero before they are applied?