I am designing a object detector. I want to ignore images in the dataset that do not have any annotations. Is there any way I can do that?
It depends on your current workflow.
Basically I would try to filter out all files without any annotations.
E.g. if your annotations are stored in a .csv file, you could load it and just pass all files with a valid annotation to the Dataset.
If you could explain your current approach a bit (e.g. what kind of Dataset are you using currently), there might be better suited approaches.
This is the DataLoader:
class dataLoader(Dataset):
def __init__(self, path, root, transforms=None):
self.path = path
self.root = root
self.transforms = transforms
self.img_loc, self.data = get_file_path(self.path)
self.data_len = len(self.data)
def __getitem__(self, index):
img_path = (self.img_loc[index])
img = Image.open(img_path).convert("RGB")
image_id = torch.tensor([index])
data = self.data[index]
xml = ET.parse(data).getroot()
objects = [n.text for n in xml.findall('.//object/name')]
if not objects:
objects.append(('None'))
num_objs = len(objects)
labels = []
for i in range(num_objs):
get_id = get_class_id(objects[i])
labels.append(get_id)
labels = torch.tensor(labels)
box = [[int(box.find('w').text) - 1,
int(box.find('h').text) - 1,
int(box.find('x').text) - 1,
int(box.find('y').text) - 1]
for box in xml.findall('.//object/bndbox')]
boxes = torch.as_tensor(box, dtype=torch.float32)
if not box:
area = 0
else:
area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
iscrowd = torch.zeros((num_objs,), dtype=torch.int64)
keep = (boxes[:, 3]>boxes[:, 1]) & (boxes[:, 2]>boxes[:, 0])
print(all(keep))
boxes = boxes[keep]
labels = labels[keep]
area = area[keep]
iscrowd = iscrowd[keep]
target = {}
target['labels'] = labels
target["image_id"] = image_id
target['boxes'] = boxes
target["area"] = area
target["iscrowd"] = iscrowd
if self.transforms is not None:
img, target = self.transforms(img, target)
return img, target, img_path
def __len__(self):
return self.data_len
The annotations are stored in a xml format. What i want to do is something like when the len(boxes) == 0, the current index should be skipped and move to the next index. I don’t want to return any of the data of the index when there len(boxes) == 0 as that indicates that there is no objects in the image.
Is there a simple way to skip an index?
Would it be possible to compute the invalid indices before and pass them as a list to your Dataset?
If so, you could just add an offset to an index vector using the invalid indices.
E.g. if the invalid indices are at 2 and 4, the indices list could be:
self.indices = [0, 1, 3, 5, ...]
index = self.indices[index]
# use index as before in __getitem__
Alternatively, you could also pass these indices to a Subset, which will apply the same logic.
If you cannot compute these indices beforehand, you could return a pre-defined invalid value, check for it in the training loop, and skip this sample.
Thank you very much. I followed you suggestion and created a script to find all the valid samples in the dataset.
1 Like
Hi there,
I have the same problem with unlabeld pictures. Can you tell me how to create a script to find all the valid samples in the dataset?
Thanks for help.
This was a while ago, so I don’t remember exactly what I did. The way I wrote my script was a for loop of all my data and kept only the data that had a valid annotations. E.g. whether the annotations contained a bounding box or a particular class. If you this doesn’t help, I can have a look for the script I wrote and upload it for you.