Why do DQNs tend to forget?

4

Why do DQNs tend to forget? Is it because when you feed highly correlated samples, your model (function approximation) doesn't give a general solution?

For example:

  • I use level 1 experiences, my model $p$ is fitted to learn how to play that level.

  • I go to level 2, my weights are updated and fitted to play level 2 meaning I don't know how to play level 1 again.

Chukwudi

Posted 2020-07-27T11:51:00.447

Reputation: 199

1

Please, do not ask the same question in different posts only because you have not yet received an answer. You already asked about ER here: https://ai.stackexchange.com/q/22694/2444. I am sure someone will answer that question.

– nbro – 2020-07-27T13:17:08.527

This question is very related to https://ai.stackexchange.com/q/13289/2444, although I wouldn't say it's a duplicate because yours is specific to DQN.

– nbro – 2020-07-28T13:10:07.943

Answers

3

You are referring to catastrophic forgetting which could be an issue in any neural net. More specifically for DQN refer to this article.

pedrum

Posted 2020-07-27T11:51:00.447

Reputation: 196

1This was incredibly helpful, Thank you – Chukwudi – 2020-07-27T12:20:07.763

1I also have a question, the issue is our replay memory size can’t be too large because of performance issues, too small and it’s irrelevant, so if we have a large space state with multiple tasks, even replay memory wouldn’t be able to help with the catastrophic forgetting, so what can be the solution? – Chukwudi – 2020-07-27T12:23:43.707

@Chukwudi I'm not really sure, sorry. – pedrum – 2020-07-27T13:45:55.803