If the first linear layer has in_features = 1 and I input [1, 2, 3] into the model, how will that linear layer be trained? Will it train it independently on 1, 2, and 3 so the layer keeps track of the gradient for each input, and then the optimizer will use the average of all their gradients? If so, is there a way to tell the optimizer to use a custom function instead of the average to combine them?
nn.Linear just wraps matrix multiplication, where right matrix is trainable in_features x out_features map. So, left matrix (input) must have in_features columns (last dimension), and arbitary number of rows (non-last dimensions reshaped into single one).
I assume your [1,2,3] denotes a python list. Normal conversion to a tensor would produce a tensor with 3 columns, not compatible with in_features=1. Adding extra tail dimension (shape (3,1)) would enable independent training, doing same transform to all scalars.
Iâm not sure I understand what you want from gradients, but if your final network output still has that 3 element dimension, you can scale loss along that dimension, this is similar to having weighted samples.
How are the values updated by default? Will it just be an average of the 3 gradients as if they form a batch in gradient descent?
but if your final network output still has that 3 element dimension
What if the output has a 1 element dimension, then how will the previous layers with 3 elements train?
âŚhaving weighted samples
How do you do that in pytorch?
Thank you for your help
Yes
This means that you combine elements somewhere, so theyâre not independent. Your 3 intermediate outputs are just like hidden features, so it makes less sense to train them as independent (1x3 vs 3x3 map) or scale gradients, as theyâll be entangled later anyway.
Pytorch loss classes have reduction=âmeanâ argument. If you use reduction=ânoneâ, you can adjust elementwise losses before computing mean.
1 Like
âŚ(1x3 vs 3x3 map) or scale gradients, as theyâll be entangled later anyway
Where does 3x3 come from?
What does it mean to scale gradients? Do you mean in a weighted sum?
I donât understand what is âentangledâ and why it affects scaling gradients.
Sorry, that was incorrect. With input shape (3,1) you have 1 x out_features map. With usual shape (*,3) you have 3 x out_features map.
If you multiply elementwise losses by some tensor C, all gradients are also multiplied by C. So, thatâs one of the ways to change gradients.
Nevermind. Basically, for 3d â 1d function, expresed with a neural net, 3 inputs usually only work together.
1 Like
But if I do have 3D -> 1D, then how will the training work? If the second layer has 3D, but the output is 1D, how will the weights for the second layer get updated? Will it calculate 3 separate gradients for all 3 elements only for the second layer, and deeper layers will have only 1 shared gradient?
That sounds right.
But this description somehow ignores hidden layer and non-linearity, needed to get a universal function approximator.
So, minimalistic network looks like: 3->64->relu->1, it has linear maps: 3x64,64x1. Thing is, your proposed 1d linear transformations, prepended to this network, wonât do anything beyond what 3x64 map does.
You can of course prepend 1d non-linear transformations to that network, but I donât see a point.
1 Like
My goal is for the network to accept an arbitrary number of inputs and to learn how to do inference with information from all of the inputs. So if I apply a 1x5 linear transformation to input [[1], [2], [3]], the output will be 3x5 tensor [[a,b,c,d,e],[f,g,h,i,j],[k,l,m,n,o]]. Then if I aggregate the 3 outputs into [[v,w,x,y,z]], the later 5xM layers will learn to use information from all inputs, while early layer will learn to represent them in a way that allows for a useful aggregate.
Am I misunderstanding something? Can you elaborate so I can understand better.
Thank you
Ok, with variable input lengths, this makes some sense. But youâll probably need a deep network (MLP) for 1d->5d transformation, as 1x5 linear map just represents a line in 5d space, not sure if thatâs of any use with your aggregation method⌠In most scenarios, another linear map 5xH will be applied to that 5 column tensor, so at least some non-linearity is needed
(otherwise (input @ 1x5) @ 5xH = input @ (1x5 @ 5xH) = input @ (1xH), so first linear layer is not needed).
1 Like
Alright, that makes sense!
Can you point me to the pytorch docs that implies the gradients of the first layers (before aggregation) will be averaged?
Do you have any superior idea for doing this kind of aggregation?
Donât know if such a thing exists. Thatâs just how gradients work with tensor broadcasting (if scalar from one of inputs is used multiple times, you sum relevent derivatives).
Thatâs a bit too abstract. Without positional info, youâre basically limited to mean() and exponential smoothing. Self-attention maybe. If you allow input position to affect output, youâre in the realm of usual sequence handlers: rnn, transformer/attention, causal convolutions etc.
1 Like
Hmm, I guess I will read about tensor broadcasting docs then.
In my problem only the first input matter, then the next inputs (2nd, 3rd, 4th, etc.) order does not matter, so I think we can use a weighted sum. How would exponential smoothing and self-attention work here?
Exponential smoothing is just a recursive weighted mean. But it is not permutation invariant, to model that you should use simple mean(). If first input is special, youâre free to handle is separately.
Self-attention - I donât understand it well enough to elaborate. Its use for aggregation is described in Attentive Neural Processes paper.
1 Like