Training Yolov3 ffrom scratch with a single image

This is not a Pytorch question per se, although I am using Pytorch for this.

This is not something one wants to do but say I implement the Yolov3 object detection model and I want to train it using a single image which only contains a single object class i.e. I am only interested in this detecting this single object class and on top of it, I have only one labelled instance of it in my training set.

My question is that would such a training scenario even work? I want to train it from scratch i.e. I do not use any pretrained model as a prior input state. I am trying to see if I can get the model to memorize this one instance (which is what should happen as it should overfit massively). However, all the online training code that I have tried cannot really train with this single image as input. What I notice is that the number of detection during training goes to zero after a couple of epochs as training progresses? I wonder if someone more experienced might have some input on this. I was expecting this model with many million parameters to quickly overfit.