Help regarding NER in NLTK

12

4

I have been working in NLTK for a while using Python. The problem I am facing is that their is no help available on training NER in NLTK with my custom data. They have used MaxEnt and trained it on ACE corpus. I have searched on the web a lot but I could not find any way that can be used to train NLTK's NER.

If anyone can provide me with any link/article/blog etc which can direct me to Training Datasets Format used in training NLTK's NER so I can prepare my Datasets on that particular format. And if I am directed to any link/article/blog etc which can help me TRAIN NLTK's NER for my own data.

This is a question widely searched and least answered. Might be helpful for someone in the future whose working with NER.

Sarmad

Posted 2015-01-29T12:13:01.677

Reputation: 195

Cross-posted on https://www.quora.com/Help-on-training-NLTKs-NER

– Franck Dernoncourt – 2015-09-13T00:01:09.837

Answers

4

Training a model, related to information extraction, in general, and named entity recognition/resolution (NER), in particular, is described in detail in Chapter 7 of the NLTK Book, available online at this URL: http://www.nltk.org/book/ch07.html.

Additionally, I think that you might find useful my related answer on Cross Validated site. It has a lot of references to relevant sources on NER and related topics as well as to various related software tools.

Aleksandr Blekh

Posted 2015-01-29T12:13:01.677

Reputation: 6 438

They do not mention how to train the NER model on custom data, can you tell how to do it? – Hima Varsha – 2016-12-28T12:29:33.997

1

@HimaVarsha I'm not an expert in this area. However, ... I think that NLTK NER model comes pre-trained on the conll2000 corpus, hence no info in NLTK book. Check the following resources: 1. http://nltk-trainer.readthedocs.io (most likely what you need; probably the Training IOB Chunkers section). 2. http://sujitpal.blogspot.com/2012/11/a-consumer-electronics-named-entity.html (might be useful as well). 3. http://nlp.stanford.edu/software/crf-faq.shtml#a (in case you use or will decide to use Stanford NER software).

– Aleksandr Blekh – 2016-12-28T21:22:18.173

I think stanfordcrf implementation does take custom data, but NTLK NER comes just pre-trained. The Training IOB Chunkers is just chunking right? Or does it even do NER? – Hima Varsha – 2016-12-29T05:47:27.630

@HimaVarsha Please pay more attention to advice you're getting. If you'd read post via link #2 above carefully, you'd see that the code there does both NER model training and running. I don't think I can help you beyond the advice above. – Aleksandr Blekh – 2016-12-29T08:37:49.590

3

Is this article good enough? http://www.succeed-project.eu/wiki/index.php/NLTK#Input_format_for_training

There is explanation about how corpus should look like.

Your data needs to be in IOB format (word tag chunktag) to make it work.
Eric NNP B-PERSON
is VB O
the AT B-NP
CEO NN I-NP
of IN O
Google NNP B-ORGANIZATION

MaticDiba

Posted 2015-01-29T12:13:01.677

Reputation: 611

1It would be ideal to post a short summary of the article in this answer. – sheldonkreger – 2015-03-02T23:53:40.360

1

I found this tutorial quite helpful: Complete guide to build your own Named Entity Recognizer with Python He uses the Groningen Meaning Bank (GMB) corpus to train his NER chunk.

After that you can check this tutorial from the same person: Training a NER System Using a Large Dataset Where he uses scikit learn to improve the performance of his system.

Finally some really useful tutorials can be found here: NLTK tutorial This guy has a youtube channel with a lot of tutorial in many subjects (ML,NLP, Python...)

Hope it helps.

Anoroah

Posted 2015-01-29T12:13:01.677

Reputation: 153