Train cnn simultaneously

How to train theses CNN network simultaneously for the given input and then merge their loss function and then carry out backpropagation on combined loss function in pytorch?
I have attached the network architecture

You can basically define the model exactly as shown in the image.
You would just have to pass the output of the base conv module to both linear layers, and accumulate the losses at the end.
Here is a small example:

class MyModel(nn.Module):
    def __init__(self, nb_vowels=5, nb_consonants=24):
        super(MyModel, self).__init__()
        self.base = nn.Sequential(
            nn.Conv2d(3, 6, 3, 1, 1),
            nn.Conv2d(6, 12, 3, 1, 1),
        self.fc1 = nn.Linear(12 * 6 * 6, 64)
        self.fc_vowels = nn.Linear(64, nb_vowels)
        self.fc_cons = nn.Linear(64, nb_consonants)
    def forward(self, x):
        x = self.base(x)
        x = x.view(x.size(0), -1)
        x = F.relu(self.fc1(x))
        x_vowels = self.fc_vowels(x)
        x_cons = self.fc_cons(x)
        return x_vowels, x_cons

model = MyModel()

# Define your different loss functions here
criterion_vowels = nn.CrossEntropyLoss()
criterion_cons = nn.CrossEntropyLoss()

x = torch.randn(2, 3, 24, 24)
target_vowels = torch.randint(0, 5, (2,))
target_cons = torch.randint(0, 24, (2,))

output_vowels, output_cons = model(x)
loss_vowels = criterion_vowels(output_vowels, target_vowels)
loss_conv = criterion_cons(output_cons, target_cons)

loss = loss_vowels + loss_conv

Thanks @ ptrblck

How to make such layers of vowels and consonats incase of Resnet.Can u share the code for it.@ ptrblck

If you would like to use a ResNet as the base model, just assign it as such and change the number of output features:

self.base = models.resnet50(pretrained=True)
self.base.fc = nn.Linear(self.base.fc.in_features, 64)
self.fc_vowels = nn.Linear(64, ...)