SwitchNet - a fast transform based neural network

You might like to know about SwitchNet. It uses the fast Walsh Hadamard transform to replace the one-to-all connectivity of a matrix of weights. It then uses parametric (switching based) activation functions as the adjustable system.
Then it uses a final fast transform as a set of set of statistical summary measures acting on the crude parametric function outputs.
Online code example: