Is it possible to train a neural network to learn something via video footage?


Is it possible to train a neural network to learn something via video footage (which is essentially a sequence of images)?

In other words, if I have a video teaching me how to draw an animal from scratch, can I then use this video to teach the computer to draw the animal in the same way?

There are many machine learning and image processing techniques (such as RNNs) that can be applied to sequences of images or videos. So, I guess the difficult part becomes mapping the activity to an action like moving a pen or something.

Mark Markrowave Charlton

Posted 2017-06-12T22:00:31.630

Reputation: 327



Short answer: No.

Long answer: A neural network is a function that maps input data (e.g. a picture) to output data (e.g. probability that the picture contains a dog). What you propose does not seem like the sort of task this tool is suited for, though I'm not willing to claim that it can't be done.

If you are interested in computer-generated artwork, I encourage you to learn about Generative Adversarial Networks and style transfer.

Jessica Alan

Posted 2017-06-12T22:00:31.630

Reputation: 111

You have only described a neural network as a classification tool.. what about reinforcement learning and regression-based nets? – Mark Markrowave Charlton – 2017-06-13T05:19:55.940

The use case doesn't change the fundamental concept of the net - it's still a function mapping input variables to output variables. – Jessica Alan – 2017-06-13T15:52:19.000

Ok, am i able to map 3d coordinates from a 2d input? lets say i make a 3d object and screenshot it, can i map one to the other? – Mark Markrowave Charlton – 2017-06-13T22:27:41.820

So a network that maps a 2D representation of an object to its 3D vertices? That sounds feasible. – Jessica Alan – 2017-06-14T15:07:59.430

ok cool im on to it! – Mark Markrowave Charlton – 2017-06-14T23:06:34.083


Just like the other answer - it can't really be done like that. However, you might want to take a look at Google Deepdream, which actually enhances images to look more like what they detected. E.g., if a cat is detected, it will make the picture more cat-like.

Thomas W

Posted 2017-06-12T22:00:31.630

Reputation: 967


About the screenshot you mentioned in comments: don't forget that NN build the mappings from learning. With only one screenshot, you'll only be able to construct a NN that outputs the correct 3D for THAT screenshot. With many screenshots, you may come closer to what you want (AFAI understand your question).
But remember that without 2 eyes, you have problems with 3D objects. Doesn't mean it's impossible.
May I suggest you to have a look at convolutional neural networks also


Posted 2017-06-12T22:00:31.630

Reputation: 126

So am i correct in saying that if i can get enough visual information of an object (ie enough screenshots of different angles) and encode it into a higher representation (like identifying that theres a face) i can map that encoded information to a mesh of 3d coordinates? – Mark Markrowave Charlton – 2017-06-13T23:06:11.370

If you know the angles (more precisely the position of the...say camera), you don't even have to use a NN. ;) – jcm69 – 2017-06-13T23:18:56.773

if i have a 3d mesh with geometry (lets say 4 square faces, each face containing 4 vertices) is there a way to represent the polygons edges? becuase if i have 4 vertices i still dont know where the edges are – Mark Markrowave Charlton – 2017-06-13T23:21:30.763

Ok, say you have 2 views from 2 cameras (with known position) of a single point (say a light), then you can reconstruct it's position in a 3D space. Easy. Based on that, you can do many things. That's been the case for years. But it's complicated and specialized. While NN have incredible results that classical approaches can't achieve in a quite simple way. Here, quite means that you've got to be familiar with NN. Often takes time, really ;) – jcm69 – 2017-06-13T23:40:37.727

ok but how can i represent a 3d mesh in terms of its faces instead of its vertices? – Mark Markrowave Charlton – 2017-06-13T23:42:32.853

if i have a cube, how can i represent each face of the cube instead of the vertices of each corner – Mark Markrowave Charlton – 2017-06-13T23:42:58.127

May be by its normal vector ? I'm not a specialist of that domain, really sorry. Sure you'll be interested by

– jcm69 – 2017-06-14T00:03:44.120