Past perfect is used when the ‘reference time’ (RT) of your larger discourse, the context within which you mention the past-perfect event, lies entirely in the past. It marks the event it mentions as something which happened before that past RT and gave rise to some (often unspecified) state at that past RT. Thus your (1),
He had known what I had done.
indicates that at the time you are talking about ‘he’ had already come to know about some act which you performed before that time. For instance, it might occur in a context like this, where the RT is established as ‘yesterday’:
John’s complaint yesterday that he was not kept informed about my actions was completely untrue. He had known long before then what I had done; I told him myself more than three weeks ago.
However, the past perfect is rarely obligatory; the time relationships between past events are often perfectly clear without employing a perfect†. That is the case with your (2); in the same context, where both the speaker and the hearer are aware of the time relationships, either or both of those perfects could be replaced with simple pasts.
(2a) He knew what I had done long before that.
(2b) He knew what I did long before that.
The past perfect is used in the same way with a ‘future-in-past’ employing would. In your (3), you are describing the subject’s knowledge at RT: at that time he knew what knowledge he would have to have at some later time. Here's an expansion, with January as the RT.
In January Mike knew what he would want to know, so he started collecting books on the subject.
In (4) you are describing the subject’s knowledge before RT, which gave rise to a state at RT. Here's an expansion, where the RT is April:
When Mike started work on his essay in April he had known since January what he would want to know, so he already had all his reference works on his desk.
† The basic rule is what I call FumbleFingers' Perfect Truism because ELL user FumbleFingers set it out here:
Don't use Past Perfect unless you really have to.