What is the way to modify a neural network classifier to deal with sample points from outside of the label set?

5

I am solving an image classification problem. However some photos may not belong to any category, and I'd like not to give any fake information, rather to capture this situation. What are the ways to do it. One idea I have in mind is to give an extra category "OTHER" but then I need to have some data for this category which would be difficult to gather as it's too general.

kmichael08

Posted 2018-09-21T14:28:50.757

Reputation: 63

There are billions possible categories and subcategories... So you aren't doing anything wrong when adding an Unknown class until and unless you want to train on all.... That's the safest and one of the commonly used style... Or if your preds are inconclusive, then vote for that but that's fake.. – Aditya – 2018-09-21T16:06:33.957

You have chosen a good path. Try to do data augmentation. – Media – 2018-09-21T18:16:12.880

Answers

3

This question is a very common question. How to handle samples belong to categories that are not included in the data set. There is no optimal way to handle this:

Creating a separate class is possible but not recommended. First, you need images to start with, second this class will have a very large variation within the class, a property that is not favorable which makes designing a powerful classifier to handle the other classes more challenging and usually results in a low quality classifier.

I prefer to handle this by measuring the confidence. If you are expecting to have anomalies/ outliers, make sure to use classifier that gives both prediction and probabilities ( confidence).

CNN is good enough , you can only accept predictions if the confidence is a higher than a threshold, or the difference between the highest confidence ( label) and the next confidence (label) is higher than a threshold. Most of the probability based classifies can give you confidence values.

Some other classifiers can be changed to give confidence, for example, you can analyze the training data used to give a label for a leave in decision tree, these samples belonging to the same leave can be clustered to get some centroids. In prediction you can predict the label and measure the shortest distance between your target ( testing sample) and each of the centroids belonging to the same leave you end with, that shortest distance can be used as a measure of confidence and can be compared to a threshold to accept / reject the prediction.

Please note that this problem is still under research and there is no clear answer to it. It needs very deep understanding to each classifier to modify it / or add to it to measure confidence and to understand what that measure means

Bashar Haddad

Posted 2018-09-21T14:28:50.757

Reputation: 1 721

yeah great answer! – Francesco Pegoraro – 2018-09-24T20:45:40.623

Thanks! Can you give an example of a classifier, that gives its confidence directly? I only know about measures of confidence/uncertainty build up on predictions such as entropy of predictions. Anything else you could suggest? – kmichael08 – 2018-09-25T07:40:49.607

Updated the answer – Bashar Haddad – 2018-09-25T15:17:00.513