The meaning of hardly has evolved over the centuries. So has the meaning of the adjective from which it is derived, hard. But where the adjective has accumulated meanings, so most of its meanings are still alive in one context or another, hardly has drifted from meaning to meaning.

I won't trouble you with the entire history (which you may find in the Oxford English Dictionary—look at Hard, both adjective and adverb, and Hardly); but by about 1600 hard had the senses of difficult and close, narrow, and both of these may be detected behind the modern sense of hardly.

When a historian writes that “The King hardly escaped, by charging with his own troop of horse solely, through the body of the enemy” he implies both that Charles escaped only with difficulty and that he escaped only narrowly” — by the skin of my teeth, we also say.

When Captain Corcoran boasts that he “never” curses, and then admits that it’s actually “hardly ever”, he’s claiming that it’s so close to “never” that it’s not worth dwelling on. In fact, it’s hardly even worth mentioning—very close to being not even worth mentioning at all.

In today’s use the difficulty sense has declined, but it’s kept alive by uses like the old song “Johnny I Hardly Knew Ye” in which a woman meets her old love who has been almost unrecognizably disfigured in war.