Incorporate additional ground truth information to segmentation net


I am looking for ways to incorporate anatomy information in CT Data (at what body height a slice is positioned) to my multi organ segmentation net.
I have checked the ways described here Incorporating Side-Channel Information into Convolutional Neural Networks for Robotic Tasks but I am wondering if there is a best practice to do this or if there are other ways. I do not have a fully connected layer (its a UNet like structure), so simply adding channels to the FCN input is not possible.

Thanks for the help!

One simple way is using FiLM (Feature-wise layer modulation)

Sooo if you are using a U-Net this has already been done for audio
Here you have code.
This way you can simply condition inner layers of U-Net with a one-hot vector (or any tensor if you have some spatial info)

Anyway you have the new visual transformers, they just encode the position of the image patches before feeding them into the nn.
You can try and approach like this too.

1 Like