Absolutely you can use a in that sentence – especially since Stack Exchange users can earn multiple Nice Answer badges. In fact, I think the sentence might even sound a little better if you've accumulated a few other Nice Answer badges already.
However, there is nothing wrong with using the in this context either. In the case of a computer-generated notification, when either article will work, there are some reasons a programmer might opt for the definite article:
You don't have to worry about whether or not the badge name begins with a vowel sound. (Assume the badge changed its name to Excellent Answer – now, in order to stay correct, the system would need to have enough smarts to change the notification from "a Nice Answer badge" to "an Excellent Answer badge".)
For badges that can be earned only once, the might sound better than a. For example, "You have earned an Editor badge" might sound a bit peculiar when users can win that badge only once. For this reason, programmers might go with the article that reads less awkwardly most often.
As for the Oscar parallel being discussed, it's a similar case – either article can be used. It depends on the context, and what the writer wants to focus on. For example, a definite article, especially when the conversation is focused on a particular year's Academy Awards:
When Jack Nicholson opened the envelope and read Rocky as the best-picture winner at the 49th Academy Awards 30 years ago...
However, when looking back on, say, an actor's career, you might see an indefinite article used:
Leonardo DiCaprio finally won an Oscar for his role in The Revenant.
This is one of those areas where the flexibility of English trumps some trivial rule. One could argue that the articles in those two quotations could be swapped and the sentences would still be grammatically sound.
The answer "In English literature" or "In the English literature"? can only earn one singular "Nice Answer" badge. Thus the word the is chosen to indicate that specific case of the award. If you could earn a sequence of identical badges for this one answer then a would be used to imply you earned one of many.
You earned the badge that was available for this answer.
I have to go with:
You've earned a "Nice Answer" badge (Answer score of 10 or more) for ""In English literature" or "In the English literature"?".
In the case of
You won the Oscar for Best Picture.
the is OK because there is only one Oscar in that category.
However, in the badge scenario, "you earned" is irrelevant because that is just the action on the badge.
Regarding the badge itself, while you can only earn one such badge for that question, other people who also answered the question and got an answer score of 10 or more could also get the same badge for the same question. The text of the alert would be exactly the same.
Therefore, more than one of the same badge is possible, so use "a" instead.
However, if only the first person to achieve that score on that question got the badge, then you could use "the".
Bottom line is that the determination is based on how you interpret the logic of the situation.
This answer tries to differentiate the indefinite noun phrase version from the definite noun phrase version, mostly without regard to extra-textual clues; because I think the OP needs to realize first and foremost that assigning the quality of definiteness to a NP makes that phrase inherently different from its indefinite NP . Both versions work here, regardless of what we know or don't know about ELL badges; and it is that difference that I am trying to focus on.
You've earned a "Nice Answer" badge (Answer Score of 10 or more) for this answer of yours].
There is a badge, of the type "Nice Answer," which you've won for your answer. This does not tell you how many "Nice Answer" badges there might be. But you've won one (a) of them for your answer. There could be several kinds of "Nice Answer" badges. Thus this "descriptor" is more of a category than a title or actual name of a specific badge. A (one) implies but does necessitate that there is more than one "Nice Answer" badge that you could win. It could just mean that you've won a "Nice Answer" badge, but then so have many other users.
The information in parentheses is literally parenthetical: it doesn't make the preceding noun phrase any more definite. It's just kind of free floating information connected to "Nice Answer" badge. We don't know for sure if it goes with all possible "Nice Answer" badges or only with the one you've won (which may only be one of many kinds, as far as we know).
This is what an indefinite noun phrase is: it is vague. But it is perfectly grammatical, usable and "sensical" (a word I just made up to mean the opposite of nonsensical) in the context of ELL. What it is not, is that it is not definite. It does not tell the badge winner much about the badge. It, of itself, does not define the badge that well; it gives an indefinite, vague description. ELL could use this sentence instead of the definite noun phrase version. The reason it does not, presumably, is because it prefers to speak more definitely about this badge.
You've earned the "Nice Answer" badge (Answer Score of 10 or more) for [this answer of yours].
Here we have a definite noun phrase. It defines the badge. Because the noun phrase is definite, the most likely way to interpret the terse parenthetical information is that it serves to define this badge. There is only one badge called "Nice Answer" and the info in parentheses gives us the criterion needed for it. Since it is spoken of definitely, it is most certainly the case that there is only one badge called "Nice Answer," and thus "Nice Answer" is this badge's name or title, and not a category.
What we don't know for sure is how many of these badges called "Nice Answer" there are. There could only be one "Nice Answer" badge in existence, just as there is only one Stanley Cup in existence. Perhaps even you when you win it, the last person that won it has to give it up; just like the previous Stanley Cup winner has to relinquish the cup to the new winner. But given the context, and our knowledge of gaming and achievements, we can assume that this specifically named and identified badge is given to whomever achieves the criteria. It's like a Boy Scout badge: any boy scout who meets certain criteria can win the Archery Badge.
So the is there not because of the "for-phrase", but in order to stress the badge's distinctiveness from other badges ("Guru badge", "Teacher badge" etc.)?
1 If the "for-phrase" was not there, nothing in my answer would change.
2 If there were no other ELL badges, nothing in my answer need change.
IOW, no it does not "stress" the badge's distinctiveness from other badges.
With the, we are assigning "Nice Answer" badge the property of definiteness. This is as opposed to not assigning it the property of definiteness (using a). This is true, and there is an inherent difference in meaning in the NA badge and a NA badge, regardless of whether other badges exist.
I am not sure, but perhaps this is the or a fundamental thing an advanced learner/speaker such as you needs to grasp. Or it could be too abstract, I am not sure. The quality of definiteness tells the native speaker something about Badge: Nice Answer that the quality of indefiniteness of Badge: Nice Answer does not. My answer feebly attempts to show this.
It's true that the NA badge is different from the Guru badge, but then a NA badge is different from a Guru badge.
Speaking in a broader than ELL context, I can say four different things:
I won a NA badge/the NA badge today on ELL; now I can add that to a Guru badge/the Guru badge I won on ELU.
You can speak about either badge as having the quality of definiteness or not having that quality.
I'll have to continue later, because I'm due elsewhere.