Image segmentation - handcrafted features vs DNN?


Currently working on a project that requires multi-class image segmentation to identify distinct anomaly types in various sheet metals.

Have had moderate success with various NN segmentation architectures, however inference performance is far from ideal given the native resolution of the images (even troublesome with extensive down-sampling).

Have also attempted to hand-craft several features based on filter banks of various convolutional kernels and then subsequently applying clustering algorithms to build a sort of 'anomaly database'. I can readily apply this defined filter bank to extract features and compared to the features to those in the labeled database.

Example anomaly.

Shown in the above image, the center horizontal line needs to be identified as an anomaly. Intuition would lead me to believe simple shadow filters would be able to detect its presence, however given the inherent inconsistencies in the sheet metal - it doesn't appear to be simple (despite human identification being very simple).

I'm not necessarily looking for a perfect solution to solve this, however if anyone was able to provide potential avenues for investigation. I want to exhaust all possible (promising) options of segmentation methods.


Posted 2018-02-24T03:22:37.157

Reputation: 61

Segmentation refers to classifying every single pixel in your image as belonging to a class. Is that what you want to do? Or do you just want to classify the image as containing certain anomalies? – kbrose – 2018-02-26T14:55:13.777

That is somewhat open to ease of implementation/performance. I would like to do what you are describing - as in, being capable of classifying anomalous regions (by way of dividing the original image into patches). However, as some of these anomalies occur over the entire image length, feeding these patches seems to lose the context of the overall image (as the anomalies do have some spatial inconsistencies). Either option is perfectly fine - however, intuitively segmentation makes more sense in my mind (although I may be wrong) as my ground truth masks can be quite subjective. – wubzorz – 2018-02-26T23:03:52.550

Cool. Since performance seems like a big concern it’s worth noting that most segmentation methods are more computationally intensive than just straight up classification. But it seems like you probably already knew that – kbrose – 2018-02-27T00:49:35.873

Yep that's correct - the segmentation performance is less than ideal as you mention. Classification does seem to be appropriate for certain anomaly types that are have little spatial context - however my concern is that I will have to essentially experiment with different hand-crafted features for each type to achieve better inference times. Appreciate your input @kbrose, is your thinking similar? I'm very much looking for alternative methods that might yield better performance without having to laboriously experiment with different feature extraction methods. – wubzorz – 2018-02-27T01:12:50.007



You have an object vs background task, i.e. two classes and instance (or m.b. class segmentation problem). If you have a labelled dataset, you can finetune one of the publicly available networks (e.g. FCN, MaskRCNN) to it to produce an image mask (i.e. an array size of the input image where every pixel is labelled as either object or background). So yeah, in 2018 feature engineering can't beat a (fully convolutional, 140M parameter, 16-layer) ConvNN.


Posted 2018-02-24T03:22:37.157

Reputation: 692

Thanks for your input @Alex, I've had some promising results with various texton features and random forest pixel classification, however as my dataset (inevitably) grows over time - my concern that these hand defined features will be far too time consuming to re-purpose for larger populations. Given the need for performance during inference - I wanted to find an alternative - ideally an implementation not contingent on availability of GPU resources. Is your inclination to try and prune a NN implementation to lower parameter count or further attempt feature engineering? – wubzorz – 2018-02-27T22:32:24.603


I would still go with Deep Neural Networks because they perform very well for segmentation tasks. The only thing that you have to be sure about is that you select the perfect patch size and your DNN should be deep enough to learn all the features.I myself have been doing research on image segmentation at the start I was having lots of issues in segmentation using deep learning but once I got all the parameters right then everything went well.

Syed Nauyan Rashid

Posted 2018-02-24T03:22:37.157

Reputation: 461