So maybe this was overly terse in the book (entirely my fault), and thank you for asking rather than just being dissatisfied with our book.
I should caution that this, to me, is about having a useful intuition rather than postulating strict and extremely deep “this is a law of nature”-type absolute statements.
The background for this (in the paragraph before the one you quote) is that for the other layers, conventional wisdom says that one would typically reduce mathematical dimension (i.e. number of elements) of the activations in a classification network.
Now the first layer does it differently, and we may ask if we can have an intuition why it takes the form it takes e.g. in ResNet.
There are three parts (because I am splitting the footnote in two):
- As mentioned in the footnote, starting the processing by embedding the image in a high-dimensional space and then working with that is a tried-and-true approach and so is using information in the “neighbourhood” of a given point. This is a parallel to kernel embeddings.
- The other question is then why not use vastly more, like 512 channels directly there. And there the intuition I would suggest is that if the “pointwise/patchwise” embedding is to very high dimensions, one would not expect to gain much. Imagine having 1 input channel and using 1x1 convolutions in the beginning and embedding to N channels. This would mean you take the scalar (1 pixel) and embed it in some N-dimensional space and do this with all pixels. But now you just have all your data in very sparsely populated space and have not gained any insight at all (try linearly embedding a sequence of 1d points to into 2-dimensional space and see if you find it very satisfying). So this line of thinking suggests that it doesn’t make much sense to have more output channels than the patch size. In ResNet 7x7x3 = 147 → 64 this is satisfied, in our example, it is not quite true that we keep the number of output channels smaller than the patch size.
- Finally, kernel machines use this high-dimensional embedding thing and the kernels can be more elaborate than what a conv layer learns. But after the embedding, they kernel machines extremely simple (linear) classifiers, where in deep learning typically have a rather rich structure afterwards. (And to me, this is a bit like a vexation image: You could answer “which part of your model is the feature extractor and which part of the model if the classification head (and where does the classifier end and the loss start)” in many different ways and would get interesting parallels with other methods. Here, I (perhaps outside the usual convention) suggest take a moment to look for at the net as if the first conv layer were the feature extractor and the remainder the classifier (which is then, per the footnote) more fancy than those in the kernel machines.
I hope this elaboration helps clear it up a bit, do not hesitate to ask if there is something to be desired.
Best regards
Thomas