In simple terms, the Irish situation is a dilemma that just doesn't exist in other EU countries.
Running through the issues will help, then it'll be clearer why a referendum can't/won't solve them (and isn't viable anyway).
- The British population as a whole voted to leave. That means that in some way or other, they want to drop membership of the common EU rules on borders, and import/export regulations (in order to create their own which may differ over time).
- That in turn implies there must be some kind of border with the EU, where commerce between UK and EU laws meet. The issue is, where to put this border? (Because this is such a hard question, attention has turned to other questions such as "how should it work" or "transition periods", to try and reach a compromise. But that's by the by
The location and nature of the border would not normally be an issue. Many EU countries have borders with non EU countries, and ordinarily, Britain would do the same, with no issues at all. But it can't, because historically, there was social unrest, terrorism, and intense divides in Ireland. Generalising a bit, Protestant Northern Ireland saw itself as an integral part of the UK. Catholic Northern Ireland saw itself as a repressed, colonised, severed part of a united Ireland, virtually under British martial law. The divides were bitter.
The "Troubles" came to an end with an agreement (the "Good Friday" agreement) being reached. The agreement involved shared devolved rule over Northern Ireland, and removal of borders and obstructions between Northern Ireland and Eire (the rest of Ireland). Which means the UK-EU border is in real trouble.....
- If it's put between north and south Ireland, it breaks the Good Friday peace agreement. There isn't a working devolved government to make a decision either, so it would have to be done by the British Government, which would come across as "imposed by Britain". So this risks a renewal of the Troubles, and huge issues in Ireland.
- But if it's not there, presumably it would be put between Northern Ireland and the rest of Britain, which would push northern Ireland away from Britain, and create an internal customs border within and between different parts of Britain itself - also unacceptable to those involved who identify Northern Ireland as an integral part of Britain, as well as politically very difficult at Stormont and Westminster.
So there must be a border, but because NI (in the UK) and Eire (in the non-UK EU) have such a delicate and volatile border situation, an emotionally recent (to those involved) history of conflict, and deep bitter divisions and memories only recently starting the healing process, no border solution appears to exist without huge problems and bitter (possibly violent and at worst socially devastating) objections.
Hence the problem.....
Additional factors making it even more of a headache:
- The desired solution is probably self contradictory - no border controls between Northern Ireland/Eire, no border controls dividing NI and mainland within the UK, but somehow despite these, a customs border between UK and EU.
- Those in favour of leaving ("Brexit") will see retention of EU laws and customs rules, or any lengthy/indeterminate transition, as a betrayal of the referendum result. So "kicking into the long grass" is politically very hard, much more than normal.
- But short term solutions are hard to find. A common political resolution for seeming insolvable political problems is to use vague language and deal with the issues as they arise another day. But here, because it's so pointed and in such sharp awareness, and all sides have very invested views, it's in very sharp focus and that's much harder to do.
- "Technological" solutions are suggested but don't seem really up to the job. Presumably since otherwise we would all use them elsewhere - but we don't. Practically, the EU will want certainty that its customs aren't being worked around/evaded. Without a hard border control of some kind, that's difficult to achieve with enough certainty.
And finally, to come back to your question, why not a referendum?
Because it's simply too divided. It would be akin (in practical terms) to trying to hold a referendum on Northern Ireland staying in the UK with a hard border, or becoming part of Eire.
It's simply too intense a question, to be resolved by a referendum of this kind. Whatever the outcome, it would almost certainly be rejected, and the mere suggestion of such a referendum might bring about the chaos, and, at worst, destroy the fragile peace that has been achieved so far.
Putting it more starkly, feelings on both sides run so deep that even the mere suggestion, made by anyone of standing, that either an internal-to-Ireland or an internal-to-UK stop point/checkpoint/hard-ish border might be created, would probably be enough to bring instability/massive protest/anger at a sense of betrayal.
(And also not to forget, someone has to propose and authorise a referendum. Stormmont is broken right now, and even if working would probably be unable to pass such a decision. Westminster simply doesn't have the legitimacy to do so - if not in reality then in the view of too many Northern Irish citizens. So there is not easy political way for such a referendum to be proposed and put into law, by any party in power, Irish or otherwise, even if the other problems didn't exist)
And this is why Northern Ireland's situation and border is such a big issue/headache/stumbling block in the whole situation. Can't be avoided, can't be left for future, can't be referendum'ed, no plainly viable solution... etc