Getting key error when training object detection model

1

Hi! I am getting the error while training my object detection model.
Epoch 1/1
Mini-batch: 1/1457 Loss: 0.1182
Mini-batch: 101/1457 Loss: 0.1295
Mini-batch: 201/1457 Loss: 0.1242
Mini-batch: 301/1457 Loss: 0.1200
Mini-batch: 401/1457 Loss: 0.1188
Mini-batch: 501/1457 Loss: 0.1164

KeyError Traceback (most recent call last)
in ()
8 print("Epoch %i/%i " % (epoch + 1, num_epochs) )
9 average_loss = 0
—> 10 for batch_id, (images, targets) in enumerate(train_data_loader):
11 # Prepare the batch data
12 images, targets = move_batch_to_device(images, targets)

3 frames
/usr/local/lib/python3.6/dist-packages/torch/_utils.py in reraise(self)
393 # (https://bugs.python.org/issue2651), so we work around it.
394 msg = KeyErrorMessage(msg)
–> 395 raise self.exc_type(msg)

KeyError: Caught KeyError in DataLoader worker process 0.
Original Traceback (most recent call last):
File “/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py”, line 185, in _worker_loop
data = fetcher.fetch(index)
File “/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py”, line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File “/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py”, line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File “”, line 43, in getitem
damage_labels.append(damage_dict[label])
KeyError: ‘D0w0’

here is my damage dataset class
from torch.utils.data import Dataset, DataLoader

Inherit from pytorch Dataset for convenience

class DamageDataset(Dataset):

def __init__(self, dataframe):

    super().__init__()

    self.filename = dataframe['filename'].unique()

    self.df = dataframe

def __len__(self) -> int:

    return len(self.filename)



def __getitem__(self, index: int):

    filename = self.filename[index]

    image = read_image_from_train_folder(filename).astype(np.float32)

    # Scale to [0,1] range expected by the pre-trained model

    image /= 255.0

    # Convert the shape from [h,w,c] to [c,h,w] as expected by pytorch

    image = torch.from_numpy(image).permute(2,0,1)

    

    records = self.df[self.df['filename'] == filename]

    

    boxes = records[['xmin', 'ymin', 'xmax', 'ymax']].values

    classes= records['class'].values

    damage_labels=[]

    damage_dict={

        'D00': 1,

        'D01': 2, 

        'D10': 3,

        'D11': 4,

        'D20': 5,

        'D40': 6,

        'D43': 7,

        'D44': 8,

        'D50': 9

        

    }

    for label in classes:

      damage_labels.append(damage_dict[label])

   

    boxes = torch.as_tensor(boxes, dtype=torch.float32)

    n_boxes = boxes.shape[0]

    

    # there is only one foreground class, WHEAT

    labels = torch.as_tensor(damage_labels, dtype=torch.float32)

            

    target = {}

    target['boxes'] = boxes

    target['labels'] = labels

    

    return image, target

and here is my tarining code:
num_epochs = 1

Prepare the model for training

model = model.to(device)

model.train()

for epoch in range(num_epochs):

print("Epoch %i/%i " % (epoch + 1, num_epochs) )

average_loss = 0

for batch_id, (images, targets) in enumerate(train_data_loader):

    # Prepare the batch data

    images, targets = move_batch_to_device(images, targets)

    # Calculate losses

    loss_dict = model(images, targets)

    batch_loss = sum(loss for loss in loss_dict.values()) / len(loss_dict)

    

    # Refresh accumulated optimiser state and minimise losses

    optimizer.zero_grad()

    batch_loss.backward()

    optimizer.step()

    

    # Record stats

    loss_value = batch_loss.item()

    average_loss = average_loss + (loss_value - average_loss) / (batch_id + 1)

    print("Mini-batch: %i/%i Loss: %.4f" % ( batch_id + 1, len(train_data_loader), average_loss), end='\r')

    if batch_id % 100 == 0:

        print("Mini-batch: %i/%i Loss: %.4f" % ( batch_id + 1, len(train_data_loader), average_loss))

I would really appreciate some help into it, thank you.

2

Hy @Bisma_Akram

This happens inside your dataloader Because you don’t have a specific key for that.

The key generating error is this

3

hey, but that isnt a class in my dataset, do you think there is some error in the csv file of the training dataset that contain labels and bounding boxes,how do you think I can find out which index this class exist at so that I can delete it, I am iterating through my dataframe but cant find a class that contains D0w0.

4

Hy @Bisma_Akram

when you iterate over classes, your classes list contains the anomly either check this column or remove it from your dataset. It’s better if you use exception handling here.

5

great! I found out and removed it, could you please help me with the other question I posted too. thank you