Periodic Pattern in Validation Loss Curve


I am currently trying to solve a regression problem using neural networks. I want to detect movement patterns in images over time (video) and output a continuous value. During the training process I noticed a strange behaviour for the validation loss curve and I was wondering if anyone has noticed this kind of periodic pattern on some of their own work. What might cause this?

The model looks like the following:

- TimeDistributed(Conv2D(32, (3,3)))
- TimeDistributed(Conv2D(16, (3,3)))
- TimeDistributed(Flatten())
- GRU(64, stateful=True)
- Dropout(0.5)
- Dense(64, activation='relu')
- Dense(1)

I trained the model using the mean squared error as the loss function, a batch size of 1 and the AdamOptimizer with an initial learning rate of 10^(-6). Obviously, the loss curve for the training data is not very good, but I am currently just wondering about the pattern of the val_loss. The plots below represent the loss of 65 epochs.


Validation Loss

Training Loss

Edit: The way I try to solve my task relies on a sliding window approach where I try to predict a continuous value for the next second based on the last 20 seconds (400 frames) of the time-series input data. But I don't think this information is needed to solve my initial question since the periodic patterns appear over several epochs (one "peak" for about every 15 epochs) which is strange. Although the stateful-version of the GRU is used (btw: using TensorFlow and Keras), the internal state of the GRU is reset after every epoch to maintain a clean start. The stateful keyword is used to indicate a dependency between batches.

Unknown User

Posted 2018-03-01T07:24:56.000

Reputation: 31

1Welcome to AI.SE ...I suggest you give more details in the question like what algorithm you are using, what approach, structure of the NN, etc – DuttaA – 2018-03-01T08:10:05.173

1Hi DuttaA, thank you! I've updated my question, let me know if you need to know anything else. – Unknown User – 2018-03-01T14:51:08.327



your model is giving a high loss for start of the video frame , later its loss is getting decreased as it runs through several frames of the "same" video, but again after a new video occurs its loss again peaks to certain point.


Posted 2018-03-01T07:24:56.000

Reputation: 920