Your proposed sentence is not good because it doesn't provide an object for the verb 'understand'. Consider that you could say "Dudley was a very gifted boy whose teachers didn't understand mathematics." That would be grammatical and meaningful. But it is Dudley they didn't understand, so 'him' is necessary to establish the meaning. Your second pair of sentences isn't analogous to the first.

I think it's worth constructing a different example to illustrate the syntactic principle involved here. Suppose Dudley goes to a small "community-driven" village school with no full-time paid teachers, where both parents of most pupils all put in an hour or two a week teaching at the school. But Dudley's parents don't participate in this teaching...

1: Dudley is a boy whose parents don't teach
2: Dudley is a boy whose parents don't teach him

In #1, teach is an intransitive usage, meaning Dudley's parents don't teach anyone. There's no specific implication from the syntax that they don't teach their own son - that's just a pragmatic assumption given the context.

But we can further imagine that when the village decided to run their own school, they arranged the classes in such a way that no parent ever had to teach a class that included their own children (perhaps to help maintain discipline, for example).

In that scenario, my example #2 wouldn't really make sense (if none of the parents teach their own children, why single out Dudley as an example of this?). But it would make sense if we shift the scenario slightly, to assume that only some parents asked not to be assigned to take classes that included their own children - in which case even though Dudley's parents don't teach their own son, they do take classes with other pupils.

Another example...

3: He is the man whom she loves
4: He is the man who she loves
5: He is the man [who / whom] she loves him

...where strictly speaking only #3 is "correct" (because pronoun whom represents the man - the "object" of the verb; it's She loves him, not She loves he).

But in practice most native speakers today nearly always who instead of whom (except when preceded by a preposition; even today, many people still observe this fast-declining "rule" in contexts such as asking To whom am I speaking? on the telephone).

Finally, note that example #5 is never valid. Whichever of who / whom you choose, it's a pronoun that represents the man (we don't repeat that reference with him as well).

This is just a comment.

Maybe you confuse:

Dudley was a very gifted boy whose teachers didn't understand him

with:

Dudley was a very gifted boy whom teachers didn't understand

The first one is like "He was a very gifted boy. The teachers of the very gifted boy didn't understand him." The second one means "He was a very gifted boy. Teachers didn't understand the very gifted boy".

Generally, a relative clause looks like a full sentence, except that exactly one of the pronouns in it has been replaced with a relative pronoun.

So we can form this relative clause:

His teachers didn't understand him. → ...a boy whose teachers didn't understand him.

Alternatively, we can replace "him" instead of "his" (but this sounds more awkward):

His teachers didn't understand him. → ...a boy whom his teachers didn't understand.

The relative clause that you wrote, on the other hand, corresponds to a sentence that doesn't have "him" on the end (and leaves me asking "didn't understand what?"):

His teachers didn't understand. → ...a boy whose teachers didn't understand.

Your second example follows exactly the same rule: you take a full sentence and replace exactly one of the pronouns in it with a relative pronoun.

His shirt was red. → Dudley, whose shirt was red...

The second part of the sentence basically says

Dudley's teachers didn't understand Dudley.

Since Dudley is already mentioned in the first part of the sentence, pronouns have been used to replace "Dudley" in the second part. But since the second part, written without pronouns, would mention Dudley twice, you need two pronouns ("whose" and "him"). You could use the relative (who) pronoun for the other instance if you wanted, which would produce

Dudley was a very gifted boy whom his teachers didn't understand.

Your other example is simpler: the relative clause, written without pronouns, would be

Dudley's shirt was red.

Dudley only appears once, so only one pronoun is needed.