Multi Input Images, with MultiOutput Regression Label

Dear experts, so i just moved from tensorflow to Pytorch and i have a list of images in directories as such


and labels of regression value like so


My Questions is how to load it to Models, creating multi input and multi output single models possibly with transfer learning. I Have used tensorflow in the past, and i cant wrap my mind on how to implemented it in Pytorch.

All of the tutorials only tells me how to do it only on classification and only in one output models, and i have modified Classes, to get label for regression such for only one (ImageA) folders only.

class CustomImageDataset(
    def __init__(self, annotations_file, img_dir, transform=None, target_transform=None):
        self.img_labels = pd.read_csv(annotations_file)
        self.img_dir = img_dir
        self.transform = transform
        self.target_transform = target_transform

    def __len__(self):
        return len(self.img_labels)

    def __getitem__(self, idx):
        img_path = os.path.join(self.img_dir, self.img_labels.iloc[idx, 0])
        image = read_image(img_path)
        label = self.img_labels.iloc[idx, 1]
        if self.transform:
            image = self.transform(image)
        if self.target_transform:
            label = self.target_transform(label)
        return image, label

dataset = CustomImageDataset(
    root_dir = "<Root>",
    transform = my_transform
train_set, validation_set =, [TRAIN_LENGTH, VAL_LENGTH])

train_loader = DataLoader(dataset=train_set, shuffle=shuffle, batch_size=batch_size,num_workers=num_workers,pin_memory=pin_memory)
validation_loader = DataLoader(dataset=validation_set, shuffle=shuffle, batch_size=batch_size,num_workers=num_workers, pin_memory=pin_memory)

and for the models

class CNN(nn.Module):
    def __init__(self, train_CNN=False, num_classes=1):
        super(CNN, self).__init__()
        self.train_CNN = train_CNN
        self.inception = models.inception_v3(pretrained=True, aux_logits=False)
        self.inception.fc = nn.Linear(self.inception.fc.in_features, num_classes)
        self.relu = nn.ReLU()
        self.dropout = nn.Dropout(0.5)
        self.linear = nn.linear()

    def forward(self, images):
        features = self.inception(images)
        return self.linear(self.dropout(self.relu(features))).squeeze(1)

and the training file

model = CNN().to(device)

criterion = nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

for name, param in model.inception.named_parameters():
    if "fc.weight" in name or "fc.bias" in name:
        param.requires_grad = True
        param.requires_grad = train_CNN

def check_accuracy(loader, model):
    if loader == train_loader:
        print("Checking accuracy on training data")
        print("Checking accuracy on validation data")

    num_correct = 0
    num_samples = 0

    with torch.no_grad():
        for x, y in loader:
            x =
            y =

            scores = model(x)
            # predictions = torch.tensor([1.0 if i >= 0.5 else 0.0 for i in scores]).to(device)
            num_correct += (predictions == y).sum()
            num_samples += predictions.size(0)
    return f"{float(num_correct)/float(num_samples)*100:.2f}"
            f"Got {num_correct} / {num_samples} with accuracy {float(num_correct)/float(num_samples)*100:.2f}"

def train():
    for epoch in range(num_epochs):
        loop = tqdm(train_loader, total = len(train_loader), leave = True)
        if epoch % 2 == 0:
            loop.set_postfix(val_acc = check_accuracy(validation_loader, model))
        for imgs, labels in loop:
            imgs =
            labels =
            outputs = model(imgs)
            loss = criterion(outputs, labels)
            loop.set_description(f"Epoch [{epoch}/{num_epochs}]")
            loop.set_postfix(loss = loss.item())

if __name__ == "__main__":

and still i get the errors like so

TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'PIL.Image.Image'>

Is theres any false implementations in my code? and For my other questions, i have only tried to do it only in one Folder Images (ImageA), is theres other implementation to do it for all folder images, with each label regression like so, Or do i just loop over all the folders? Thanks in advance

Based on the error message it seems you are not transforming the PIL.Image to a tensor, so I guess in your code the transform argument might not be set properly. Note that your posted code snippet uses root_dir instead of img_dir, soI assume your real code is not using the same posted code.
To further isolate the issue, check the type(image) in __getitem__ before returning it and make sure it’s a tensor.

Multiple inputs and outputs can be implemented directly in the forward via:

def forward(self, input1, input2):
    out1 = self.layers(input1)
    out2 = self.layers(input2)
    return out1, out2

I’m unsure, if you need to iterate all folders, as it seems you are already using a mapping where the image paths and the corresponding label is given. In this case, I would just reuse the provided mapping.

1 Like

Thankyou sir, you’re literally a live safer. The snipped that you just posted, worked fine, i just needed add another transfer learning layer. Thankyou so much!