4

I am using Tensorflow Object Detection API for training a CNN from scratch on COCO dataset. I need to use this specific configuration. There is no pre-trained model on COCO with that configuration and this is the reason why I am training from scratch.

However, after 1 week of training and evaluating each checkpoint generated by the training phase this is how my learning phase appears on Tensorboard:

Thus, my questions are:

- does anyone know how many iterations approximately will be necessary? Right now I did more than 500'000 iterations.
- How can be possible that after 500'000 the evaluation is 0,8%? I would expected something like 60-70%.
- Why does there is a sudden drop after 500k iterations? I thought that the eval was supposed to converge to some limit. (this is what SGD should do)
- Is there any 'trick' to speed up the training phase? (ex: increasing the learning rate, etc).

2It might be useful to know a little more about your problem. What would the precision of guessing uniformly at random look like? How much data do you have?

Note too, that SGD is not guaranteed to converge smoothly, because you might update repeatedly based on an unfortunate random sample. It converges only in expectation. – John Doucette – 2018-08-14T01:27:12.517

2I'm training on coco dataset, which is 18GB of labeled data. I've no idea how much the precision of guessing uniformly at random should look like. – Giacomo Bartoli – 2018-08-14T07:36:28.903