The word nother is very old and has been used in English since at least eight hundred years ago. It appears, for example, in the fourteenth century Northumbrian poem Cursor Mundi:
In-sted o þi noþer sede, Ne sal þe groue bot thorne and wede.
It also appears in early versions of the bible:
This kynd can by no nother meanes come forth, but by prayer.
Here's an example from David Crockett's An Account of Col. Crockett's Tour to the North and down East, in the year of our Lord one thousand eight hundred and thirty-four … Written by himself, 1835, Carey & Hart:
I'll now answer one nother question about what's the best way of keepin the democratic party in my quarter from splittin.
This word now appears before the word whole in modern American and British English. However, it survives in freer usage in Caribbean English. Here's an excerpt from Richard Allsopp's Dictionary of Caribbean English Usage, 1996, OUP.
Peter and your nother brother were here today.
Notice that in all of these examples the word nother is not preceded by the indefinite article "a". The word nother is not a part of the word another with an adjective stuck in the middle. It's a real word in its own right. It is easy to see how people be confused, though.
Because it is a complete word, nother doesn't need an apostrophe. If we use one, this is a mistake!
The word nother does not appear in very technical or formal writing. We wouldn't see it in highbrow scientific research papers, for example. But you will see it in newspapers and magazines. There is nothing wrong with this word! After all, this word has been used as a modifier in this way for eight centuries now. It is obviously proper grammatical English. Such phrases as a whole nother kettle of fish are, of course, perfectly grammatical too. Of course whatever native speakers generally do is grammatical. That's what grammatical means!
Note: All this information, including quotes, is taken from the Oxford English Dictionary.
References: "nother, adj.2 and pron.2." OED Online. Oxford University Press, June 2015. Web. 21 August 2015.