False positives hard hats

I’m new to this, so please bare with me. If this is the wrong topic please let me know.

I’ve tried training hard hat detection using both of these datasets:

https//universe.roboflow.com/joseph-nelson/hard-hat-workers/3

The problem I’m seeing is that I get A LOT OF detections of hard hats/helmets for normal heads with no helmet. There are a lot more incorrect classifications than correct. I suspect that is due to the fact that all annotations for helmets include the complete head as well.

I guess there are hundres of ways to screw up the training. I’ve been using this template to train, and just replaced the dataset:

colab.research.google.com/drive/1gDZ2xcTOgR39tGGs-EZ6i3RTs16wmzZQ (link removed as I was not allowed to post multiple links)

Can anyone give any guidance?