## Recognition of lines in a chalkboard

3

I'm trying to develop a real-time application that, from the sequence of chalkboard images captured by a webcam, recognizes the lines being draw on it.

It must be able of recognize the lines from the chalkboard background, filter the presence in the image of the teacher, and translate these lines to some representation, something as a list of basic events like "start of line at xxx,xxx", "continue line at xxx,xxx", ...

After several days looking for references and bibliography, none is found. The most similar are the character recognition applications, in particular when they have a stroke recognition stage.

Any hint ?

Input will be a sequence as this one, this one or this one (just without the presence of the students). I've expect the teacher not hidding his hand. We could imagine a start with an empty chalkboard.

Thanks.

Note: I am looking for more than an answer which says only something similar to "you can use a deep learning training it with two classes", without details or references.

Thanks for the edit. – Neil Slater – 2019-12-14T12:16:49.087

Have you tried Signalprocessing.SE, I don't think DL is a good option here. – DuttaA – 2019-12-14T12:26:51.177

@DuttaA: I was thinking on DataScience, SignalProcessing and this one. If no answers are received, I will move the question. – pasaba por aqui – 2019-12-14T12:28:15.790

I would look into Hough transform https://en.wikipedia.org/wiki/Hough_transform

– George White – 2019-12-14T18:53:38.390

@GeorgeWhite: thanks for your comment. As you know, Hough transform is mainly to detect straight lines. It can be generalized to some other parametric curves, but I do not see promising use it in the case of free forms. – pasaba por aqui – 2019-12-14T19:31:36.593

From ""start of line at xxx,xxx", "continue line at xxx,xxx"" I thought you were assuming straight lines. – George White – 2019-12-14T19:33:40.613

0

I will assume that the camera is stable (no change in position, zoom or other settings during the video recording), otherwise the task becomes markedly more complicated.

Let's say that your dataset is an array of rasters (images in array format). You mention that you want to detect events "start line" and "end line".

One way of doing this would be to compute an approximate time derivative of your image series. For instance, take the image raster at index idx and the one right after, at index idx+1 (captured at instants $$t$$ and $$t + \Delta t$$, where $$\Delta t$$ is the sampling interval).

At coordinates $$(i,j)$$, this derivative could look something like: timeDerivative = (images[idx][j][i] - images[idx+1][j][i])/DeltaT. This is a crude estimate, and there are better ways of computing an approximate discrete derivative, but you get the idea.

Following this, you could declare a state of the recording: drawing line or not drawing line. The states, we assume are always alternating, as a teacher has to take his hand off the blackboard to draw a new line.

When a derivative with large values is detected (a region in the image goes from being black to white suddenly) and the state is "not drawing", the event "start line" is recorded and the state switches to "drawing". While a derivative with large values continues to be detected as time goes on in the vicinity of the previous spot with a large derivative, nothing changes. Once this is no longer true, the state changes to "not drawing" and the event "stop line" is recorded at the last location with a large derivative.

This is the main idea, which can be improved with:

• Defining the area of the blackboard, either by hand or automatically
• Thresholding the images to better isolate the chalk trace from the blackboard
• Using a tracker such as a Kalman filter to know where to look next for the chalk

+1 for Kalman filtering. I was thinking it would probably be required here. – DuttaA – 2019-12-15T17:28:05.517

Thanks for your answer. I think movement of the teacher in front of the chalkboard will cause hard interferences to your proposal of algorithm. – pasaba por aqui – 2019-12-15T19:52:55.777

Indeed. To cut out the teacher, you can threshold then apply a blob extraction algorithm, such as the two pass algorithm. This can identify the teacher if done correctly. You can then cut the teacher out of the image and color his pixels like the blackboard. This comes of course before the start of what I wrote above. – David Cian – 2019-12-15T21:37:16.997