Take a look at this example:
She couldn't remember which purse was hers.
What it says is that there were a number of purses and one of them was hers, but she couldn't remember which one was it. Do you agree that this example sounds like perfectly valid English? I hope you do because it is perfectly valid English. Now, "She couldn't remember which hers was.", with respect to the example that I just showed you, would actually be equivalent to the following:
She couldn't remember which purses was.
I hope you can see that there are a lot of problems with the English here. First of all, since purses is a word that's in the plural, the past-tense form of the verb to be must also be in its plural form to agree with purses: were. Secondly, wouldn't you say that which purses were would now sound like an incomplete thought? Were the purses hers or were the purses lost? What exactly was happening with them? Not clear at all. And hers in your original example sounds like the plural form of the noun her as there were such a noun. What exactly is a her? Is there really such a noun in English? I don't think so. So, to cut a long story short, your second example grammatically is just a catastrophe. Only your first example is correct:
She couldn't remember which was hers.