How to verify the correctness of network implementation?

How can I verify that the codes I wrote exactly implement the network architecture in my mind?

My idea is re-implementing all the tensor operations in NumPy and feeding in random inputs to see if the results are the same. But this is so dumb!

And how you verify that your numpy implementation is correct.

If the network is totally novel, you can verify the expected shape after each node. Then, feed the network a constant, and check the output after each node. You can also fix the network parameter, start with a small network, check each section of the network on its own…etc.

1 Like

Sounds reasonable. However, how do you know the number is correct when you check the output after each node?

Another way I can think of is to visualize the network and manually inspect if all the nodes and edges are connected correctly. But again, this is not clever. The verification cannot be wrapped into an unit test case and humans make mistakes.