Oh, these are quite tough questions and I have to pass. Let’s wait for someone with more intuition about it. 
1 Like
Just curious…
When concatenating 2 different types of features, do we have to do some kind of normalization for both before passing them to the next fc layer? Perhaps a batch_norm layer so all features have 0 mean unit var.
My best guess would be to make sure both activations are in the same range. If the difference in magnitude is large, the smaller part of the tensor might be ignored.
1 Like
where are data in def forward get from?
example in text. i want take into tf-idf features. How to solve?
The forward method will receive all arguments you are passing to the model call:
output = model(arg1, arg2, arg3)
In this example forward would get all three arguments.
I assume you want to calculate the tf-idf features before passing them to the model?
If so, you could use the implementation from e.g. sklearn or alternatively you’ll find some PyTorch implementations on GitHub.
Hi @ptrblck , Am working on resnet image and tabular data, am getting below error, it is working fine until some batch after that am getting weird batch size.
Failed with below error:-
RuntimeError: Sizes of tensors must match except in dimension 0. Got 53 and 64
torch.Size([64, 3, 224, 224]) torch.Size([64, 19])
torch.Size([64, 3, 224, 224]) torch.Size([64, 19])
torch.Size([64, 3, 224, 224]) torch.Size([64, 19])
torch.Size([64, 3, 224, 224]) torch.Size([64, 19])
torch.Size([64, 3, 224, 224]) torch.Size([64, 19])
torch.Size([64, 3, 224, 224]) torch.Size([53, 19])
Am using separate custom datasets for image and tabular data and finally using for loop iterate through the training data.
How to approach this problem, is it right way?
Custom Image Datasets:
class MelanomaDataset(Dataset):
def __init__(self, csv, root_dir, transform=None):
self.annotations = csv.copy()
self.root_dir = root_dir
self.transform = transform
def __len__(self):
return len(self.annotations)
def __getitem__(self, index):
img_path = os.path.join(self.root_dir, self.annotations.iloc[index, 0])
image = Image.open(img_path+'.jpg')
y_label = torch.tensor(self.annotations.iloc[index, 7],dtype=torch.float)
if self.transform:
image = self.transform(image)
return (torch.tensor(image, dtype=torch.float), y_label)
For tabular datasets:
col=['col_0','col_1','col_2','col_3','col_4','col_5'\
,'col_6','col_7','col_8','col_9','col_10','col_11','col_12','col_13','col_14','col_15','col_16','col_17','col_18']
class TabularDataset(Dataset):
def __init__(self, X):
X = X.copy()
self.X = X.loc[:,col].copy().values.astype(np.float32) #categorical columns
self.Y = X.loc[:,'target'].copy().values.astype(np.float) #numerical columns
def __len__(self):
return len(self.Y)
def __getitem__(self, idx):
return torch.tensor(self.X[idx],dtype=torch.float), torch.tensor(self.Y[idx],dtype=torch.float)
Training Loop:
model_final.train()
for e in range(1, EPOCHS+1):
epoch_loss = 0
epoch_acc = 0
for X_batch, y_batch in train_loader:
for X_tab_batch,y_tab_batch in train_tab_loader:
X_batch, y_batch = X_batch.to(device), y_batch.to(device)
X_tab_batch, y_tab_batch = X_tab_batch.to(device), y_tab_batch.to(device)
print(X_batch.shape,X_tab_batch.shape)
optimizer.zero_grad()
y_pred = model_final(X_tab_batch,X_batch)
loss = criterion(y_pred, y_batch.unsqueeze(1))
acc = binary_acc(y_pred, y_batch.unsqueeze(1))
loss.backward()
optimizer.step()
epoch_loss += loss.item()
epoch_acc += acc.item()
print(f'Epoch {e+0:03}: | Loss: {epoch_loss/len(train_loader):.5f} | Acc: {epoch_acc/len(train_loader):.3f}')
DataLoader:
For image:
train_loader = DataLoader(dataset=tensor_dataset, batch_size=64, shuffle=False)
test_loader = DataLoader(dataset=test_dataset, batch_size=64, shuffle=False)
For Tabular data:
train_tab_loader = DataLoader(dataset=tabular_train, batch_size=64, shuffle=False)
test_tab_loader = DataLoader(dataset=tabular_test, batch_size=64, shuffle=False)
NN:
class MelanomaClassification(nn.Module):
def __init__(self):
super(MelanomaClassification, self).__init__()
# Number of input features is 19.
#model = models.resnet50(pretrained=True)
self.cnn=model = models.resnet50(pretrained=True)
for param in self.cnn.parameters():
param.requires_grad = False
self.num_ftrs=self.cnn.fc.in_features
self.cnn.fc=nn.Linear(self.num_ftrs,100)
self.fc2=nn.Linear(119,1)
self.relu = nn.ReLU()
self.dropout = nn.Dropout(p=0.1)
self.batchnorm1 = nn.BatchNorm1d(64)
self.batchnorm2 = nn.BatchNorm1d(64)
def forward(self,inputs,image):
x2 = self.cnn(image)
x2=torch.cat((x2,inputs),dim=1)
x3 = self.relu(self.fc2(x2))
print('net',x3.shape,inputs.shape)
return x3
Thanks
I guess the TabularDataset's length is not divisible by the batch size of 64 without a remainder, so that the last batch could be smaller (53 samples in your case).
A workaround would be to drop the last (smaller) batches using drop_last=True in both DataLoaders.
1 Like
Hi @ptrblck thanks for ur reply😉
problem with custom tabular dataset index, iteration not happened properly in dataloader.
fixed it now👍
Hi @ptrblck. Lets say I have 100 layers which takes data from 100 channels. The output from them needs to be concatenated for further processing. Is there a way to parallelize these layers. If so please guide me though it.
Thank you,
Rashmi Kethireddy
Depending on the workload of each layer, the GPU resources might be fully utilized, so that a parallel execution of multiple layers wouldn’t be possible.
You could try to use custom CUDA streams as described here, which is however an advanced use case, since you would have to make sure to synchronize the code properly in order to avoid race conditions.
Based on similar questions, the achieved speedup was marginal, if the layers were not tiny.
1 Like
Hi @ptrblck, I want to train my ResNet18 model with RGB images and then in the last fc, concatenate with lbp input image and then classify for 2 class.
I modified a little your model because my input RGB images are 600*600 but after put them in the model I don’t know their size after fc.
May you help me to can change the parameter please?
def lbp(x):
radius = 5
n_points = 16
METHOD = 'uniform'
imgUMat = np.float32(x)
gray = cv2.cvtColor(imgUMat, cv2.COLOR_RGB2GRAY)
lbp = local_binary_pattern(gray, n_points, radius, METHOD)
lbp = torch.from_numpy(lbp).float()
return lbp
Here is the model:
class MyModel(nn.Module):
def __init__(self):
super(MyModel, self).__init__()
self.cnn = models.resnet18(pretrained=True)
self.cnn.fc = nn.Linear(
self.cnn.fc.in_features, 20)
self.fc1 = nn.Linear(20 + 10, 60) # I think here should be changed
self.fc2 = nn.Linear(60, 5)
def forward(self, image):
x1 = self.cnn(image)
x2 = lbp(image)
x = torch.cat((x1, x2), dim=1)
x = F.relu(self.fc1(x))
x = self.fc2(x)
return x
model = MyModel()
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
criterion = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr=0.0003, weight_decay=1e-4)
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)
model.to(device)
model = train_model(model, train_dl_orig, valid_dl_orig, criterion, optimizer,scheduler, num_epochs=1)
Here is the error:
error: OpenCV(4.1.1) c:\projects\opencv-python\opencv\modules\imgproc\src\color.simd_helpers.hpp:92: error: (-2:Unspecified error) in function '__cdecl cv::impl::`anonymous-namespace'::CvtHelper<struct cv::impl::`anonymous namespace'::Set<3,4,-1>,struct cv::impl::A0xe227985e::Set<1,-1,-1>,struct cv::impl::A0xe227985e::Set<0,2,5>,2>::CvtHelper(const class cv::_InputArray &,const class cv::_OutputArray &,int)'
> Invalid number of channels in input image:
> 'VScn::contains(scn)'
> where
> 'scn' is 1
Also here is my train part:
def train_one_epoch(model, dataloder, criterion, optimizer, scheduler):
model.train(True)
steps = len(dataloder.dataset) // dataloder.batch_size
running_loss = 0.0
running_corrects = 0
for i, (inputs, labels) in enumerate(dataloder):
inputs, labels = to_var(inputs), to_var(labels)
optimizer.zero_grad()
# forward
outputs = model(inputs)
_, preds = torch.max(outputs.data, 1)
loss = criterion(outputs, labels)
# backward
loss.backward()
# update parameters
optimizer.step()
if scheduler is not None:
scheduler.step()
# statistics
running_loss = (running_loss * i + loss.item()) / (i + 1)
running_corrects += torch.sum(preds == labels).item()
# report
sys.stdout.flush()
sys.stdout.write("\r Step %d/%d | Loss: %.5f" % (i, steps, loss.item()))
epoch_loss = running_loss
epoch_acc = running_corrects / len(dataloder.dataset)
sys.stdout.flush()
print('\r{} Loss: {:.5f} Acc: {:.5f}'.format(' train', epoch_loss, epoch_acc))
return model
1 Like
To get the activation shape you could just add a print statement into the forward method:
def forward(self, image):
x1 = self.cnn(image)
x2 = lbp(image)
print(x1.shape)
print(x2.shape
...
However, the error points to OpenCV, so I guess that local_binary_pattern might be failing, as it’s apparently expecting a different number of channels.
This helped me. Thank you.
For the original question,
can i have a loss function (loss1) for the white part, and different loss (loss2) for the yellow part?
and will the loss1 gradients affect the gradients of the yellow parts as well (loss2 will not be effective)?
Hi, Many thanks for the answer and it’s really helpful! I wondered how could I use torch summary.summary() to summarise the network architecture you showed here?
I think @Raul_Gombru has created the plot manually, but you could use e.g. torchviz to visualize the model.
Many thanks and I will try it asap:)
@ptrblck I am trying to insert the age values as a condition tensor for the first layer of the generator for the cDCGAN. I am following [this] (https://arxiv.org/pdf/1702.01983.pdf) and [this] (https://arxiv.org/pdf/1611.06355.pdf).
how to concatenate the latent vector z of torch.Size([64, 128, 1, 1]) and age values as one hot vector of torch.Size([64, 6]) in first dimension and give to the first layer of G ?
sorry for such a simple query, I am learning PyTorch and simultaneously learning GAN.
If you want to concatenate the two tensors with the given shape, this code should work:
a = torch.randn(64, 128, 1, 1)
b = torch.randn(64, 6)
c = torch.cat((a, b[:, :, None, None]), dim=1)
print(c.shape)
> torch.Size([64, 134, 1, 1])
1 Like
@ptrblck Thanks for your kind attention. I did this b=b.view(64,6,1,1) and then c=torch.cat((a,b),1). I will defintly try your solution too. Thanks once again.