When should I stop the object detection model training while mAP are not stable?

Issue

I am re-training the SSD MobileNet with 900 images from the Berkeley Deep Drive dataset, and eval towards 100 images from that dataset.

The problem is that after about 24 hours of training, the totalloss seems unable to go below 2.0:

enter image description here

And the corresponding mAP score is quite unstable:

enter image description here

In fact, I have actually tried to train for about 48 hours, and the TotoalLoss just cannot go below 2.0, something ranging from 2.5~3.0. And during that time, mAP is even lower..

So here is my question, given my situation (I really don’t need any “high-precision” model, as you can see, I pick 900 images for training and would like to simply do a PoC model training/predication and that’s it), when should I stop the training and obtain a reasonably performed model?

Solution

indeed for detection you need to finetune the network, since you are using SSD, there are already some sources out there:

In general your error can be attributed to many factors, the learning rate you are using, the characteristics of the images themselves (are they normalized?) If the ssd network you are using was trained with normalized data and you don’t normalize to retrain then you’ll get stuck while learning. Also what learning rate are they using?

From the model zoo I can see that for SSD there are models trained on COCO
enter image description here

And models trained on Open Images:
enter image description here

If for example you are using ssd_inception_v2_coco, there is a truncated_normal_initializer in the input layers, so take that into consideration, also make sure the input sizes are the same that the ones you provide to the model.

You can get very good detections even with little data if you also include many augmentations and take into account the rest of the things I mentioned, more details on your code would help to see where the problem lies.

Answered By – bpinaya

Answer Checked By – Pedro (AngularFixing Volunteer)

Leave a Reply

Your email address will not be published.