1d convolution with several identical input channels produced by heavy network

autograd
1

Hi all,

we have a network with this kind of behaviour:

embedding=heavy_shared_net(big_image_embedding)
for all node in nodes : 
    output[node]=output_net(cat(embedding,position(node)))
return output

The number of nodes can reach thousands
heavy_shared_net weights must be updated according to the error on all the output nodes

This solution should work in that regard but it would be nice to parallelize the loop that looks like a conv1d with kernel size 1.

if we use the repeat method on the embedding will the backdrop still work on the heavy_shared_net?

Any suggestions?

8 Likes
2

Yes, repeat is a differentiable operation but will of course increase the memory usage. Based on your code this should already be the case so you could try using it.

1 Like