Step-by-step NER Model for Bahasa Indonesia with PyTorch and Torchtext

Attention heatmap for a sample news sentence, higher value means more attention.

About a month ago, I felt the need to refresh my memory on PyTorch. At the same time, I wanted to learn more about the recent development in Named Entity Recognition (NER), mainly because the topic is related to my thesis. Strangely, I could not find any comprehensive tutorial. Some tutorials are just an introduction to how PyTorch works in general, which was too simple. Other tutorials contain walls of code, trying to implement every technique that has seen the light of day. If there were something in between, they mixed PyTorch with Keras, rather than using Torchtext (I demand purity!). In the absence of a suitable reference, I start a step-by-step implementation.

To make the learning more concrete, I pick NER for Bahasa Indonesia as the use case, focusing on news articles. If you want to try the final best model in action immediately, I have deployed it to Heroku. Since the model works for Bahasa Indonesia, the website is also written in the same language.

An example of the model in action for a sample news sentence: http://nerindo-simple.herokuapp.com

I split the implementation into eight standalone Google Colab notebooks. The starting point is the Bi-directional LSTM model, which has been proven as a success for NER in many languages for the last 3–4 years. Following the bare minimal model, I added embeddings and Conditional Random Field (CRF). The former is a powerful idea in NLP to represent categorical input as numerical vectors, while the latter is an old concept to impose sequential rules to the model. Additionally, I adopt the attention layer and the transformer, which is based on the attention layer. Finally, the last two parts concern the experimentation to find out which model is the best.

More detail, visit the notebooks:

Final Remark

Comparison of the five configurations. BiLSTM models still perform better than the transformer-based.

As the primary goal of this journey is for learning, the model is by any means imperfect. There is no guarantee that it is state-of-the-art or even close to being one. In terms of methods, I keep everything to the core. Hopefully, the step-by-step notebooks can help you understand each component more clearly and why we should include them.

The complete implementation and dataset can be seen in this repository.

Find me in your nearest Ramen stall.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store