Word2Vec for Named Entity Recognition



I'm looking to use google's word2vec implementation to build a named entity recognition system. I've heard that recursive neural nets with back propagation through structure are well suited for named entity recognition tasks, but I've been unable to find a decent implementation or a decent tutorial for that type of model. Because I'm working with an atypical corpus, standard NER tools in NLTK and similar have performed very poorly, and it looks like I'll have to train my own system.

In short, what resources are available for this kind of problem? Is there a standard recursive neural net implementation available?

Madison May

Posted 2014-06-19T19:29:57.797

Reputation: 1 959

Have you tried training the Stanford NER on your corpus? There's a tutorial here.

– Emre – 2014-06-19T20:58:53.030

I have not -- should give that a go to see how it fares. – Madison May – 2014-06-19T21:30:42.020

I'd like to use word2vec features or similar, though, since I have access to a relatively small labeled dataset and need to make the most of unlabeled data I have on hand. – Madison May – 2014-06-19T21:37:43.327



Instead of "recursive neural nets with back propagation" you might consider the approach used by Frantzi, et. al. at National Centre for Text Mining (NaCTeM) at University of Manchester for Termine (see: this and this) Instead of deep neural nets, they "combine linguistic and statistical information".


Posted 2014-06-19T19:29:57.797

Reputation: 1 850


Two recent papers use a Deep learning architecture called CharWNN to address this problem. CharWNN was first used to get state of the art results (without handcrafted features) on Part of Speech (POS) tagging on an English corpus.

The second paper by the same author uses the same (or similar) architecture for predicting whether a word belongs to 10 Named Entity classes, with apparent state of the art results.


Posted 2014-06-19T19:29:57.797

Reputation: 336

Interesting. Thanks for sharing this. I wonder why they didn't test this approach on english corpora. – MaticDiba – 2015-09-14T11:33:15.923


Try this. This has an implementation of Word2Vec used instead of Bag of Words for NER and other NLP tasks.


Posted 2014-06-19T19:29:57.797

Reputation: 31

1I've had no problem finding word2vec implementations, but I have been unable to find a working recursive net to use. – Madison May – 2014-08-28T16:57:14.207

Link is no more active, if possible kindly share the new working link – Amandeep – 2019-03-15T10:51:15.313


Here are some ideas about how to use word vectors for NER, which adopts a mostly unsupervised word2vec-centric approach.

  1. Given a set of word vectors (either something you trained, or something off-the-shelf like GoogleNews-vectors-negative300.bin), discover clusters in the vector space. These clusters are basically your definitions for various unnamed concepts.
  2. With minimal supervision, you can map/convert the unnamed clusters to match human knowledge, thus creating named concepts grounded on top of the known word vectors and unnamed concepts. For example, a method findCluster(['joy', 'surprise', 'disgust', 'trust', 'fear', 'sadness', 'anger', 'anticipation']) might return a list containing hundreds of words mostly related to emotion. If you name this list 'emotion', then there you have a named concept 'emotion' defined based on the vector space.
  3. You can also do vector math to find the intermediate concept between two given ones. For example, the vector math might tell you that when given two words 'surprise' and 'disgust', the following are found between the two: dismay, astonishment, disappointment, amazement, bewilderment, exasperation, disbelief, shock, etc. This allows you to build up relationship between concepts.
  4. You can repeat the above to build up various types of named concpts, like: weekdays, all emotions, happy emotions, vehicles, etc.
  5. Once you have built up layers of named concepts, you can then train a RNN on a text corpus that has been augmented with the named concepts, so 'brown fox jumps' is also '{color} {animal} {action}' etc. This way the RNN should be able to learn some rudimentary grammar in an unsupervised manner.
  6. If you have built up a sufficiently powerful grammar from the above, then you should be able to apply it toward some of your NER tasks.

Kaihu Chen

Posted 2014-06-19T19:29:57.797

Reputation: 21