POS Tagging: A review of BIS POS tagset and ILCI-II Malayalam Text Corpus
Posted on September 10, 2019
| Santhosh Thottingal
The Bureau of Indian Standards(BIS) had published a Part of Speech(POS) tagset for Indian languages. POS is the process of assigning a part of speech marker to each word in a given text. In this article, I am reviewing the tag set defined in it. While developing mlmorph project I had explored a candidate POS tagging schema for Malayalam. I did not choose BIS tagset for the reasons I am going to explian in this article.
[Read More]
Markov chain for Malayalam
Posted on June 8, 2019
| Santhosh Thottingal
I have been trying to generate a Markov chain for Malayalam content. A Markov chain is a stochastic model describing a sequence of possible events in which the probability of each event depends only on the state attained in the previous event.(wikipedia). For natural language, it represents a probabilistic model of words- the probability that one word can come after another word. This model can be prepared by feeding large amount of text to system that learns the probabilities of each words.
[Read More]
Lexicon Curation for Mlmorph
Posted on May 26, 2019
| Santhosh Thottingal
One of the key components of Mlmorph is its lexicon. The lexicon contains the root words categorized as nouns, verbs, adjectives, adverbs etc. These are the components used with morphological rules to generate the vocabulary of Malayalam. I collected initial lexicon with about 100,000 words from various sources such as Wikipedia, CLDR and many targeted web crawls. One problem with such collected words is they often contains spelling mistakes. Secondly, classifying these words is not possible without the tedious task of a person going through each and every words.
[Read More]
LibreOffice Malayalam spellchecker using mlmorph
Posted on March 10, 2019
| Santhosh Thottingal
A few months back, I wrote about the spellchecker based on Malayalam morphology analyser. I was also trying to intergrate that spellchecker with LibreOffice. It is not yet ready for any serious usage, but if you are curious and would like to help me in its further development, please read on.
Malayalam spellchecker – a morphology analyser based approach Blog post on spellchecker approach and pla Current status The libreoffice spellchecker for Malayalam is available at https://gitlab.
[Read More]
Malayalam Named Entity Recognition using morphology analyser
Posted on March 10, 2019
| Santhosh Thottingal
Named Entity Recognition, a task of identifying and classifying real world objects such as persons, places, organizations from a given text is a well known NLP problem. For Malayalam, there were several research papers published on this topic, but none are functional or reproducible research.
The morphological characteristics of Malayalam has been always a challenge to solve this problem. When the named entities appear in an inflected or agglutinated complex word, the first step is to analyse such words and arrive at the root words.
[Read More]
Malayalam morphology analyser – First release
Posted on November 25, 2018
| Santhosh Thottingal
I am happy to announce the first version of Malayalam morphology analyser.
After two years of development, I tagged version 1.0.0 . In this release In this release, mlmorph can analyse and generate malayalam words using the morpho-phonotactical rules defined and based on a lexicon. We have a test corpora of Fifty thousand words and 82% of the words in it are recognized by the analyser.
A python interface is released to make the usage of library very easy for developers.
[Read More]
Malayalam morphology analyser – status update
Posted on August 11, 2018
| Santhosh Thottingal
For the last several months, I am actively working on the Malayalam morphology analyser project. In case you are not familiar with the project, my introduction blog post is a good start. I was always skeptical about the approach and the whole project as such looked very ambitious. But, now I am almost confident that the approach is viable. I am making good progress in the project, so this is some updates on that.
[Read More]
The many forms of ചിരി ☺️
Posted on July 15, 2018
| Santhosh Thottingal
This is an attempt to list down all forms of Malayalam word ചിരി(meaning: ☺️, smile, laugh). For those who are unfamiliar with Malayalam, the language is a highly inflectional Dravidian language. I am actively working on a morphology analyser(mlmorph) for the language as outlined in one of my previous blogpost.
I prepared this list as a test case for mlmorph project to evaluate the grammar rule coverage. So I thought of listing it here as well with brief comments.
[Read More]
Talk on ‘Malayalam orthographic reforms’ at Grafematik 2018
Posted on June 21, 2018
| Kavya Manohar
Santhosh and I presented a paper on ‘Malayalam orthographic reforms: impact on language and popular culture’ at Graphematik conference held at IMT Atlantique, Brest, France. Our session was chaired by Dr. Christa Dürscheid.
The paper we presented is available here. The video of our presentation is available in youtube.
Grafematik is a conference, first of its kind, bringing together disciplines concerned with writing systems and their representation in written communication. There were lot of interesting talks on various scripts around the world, their digital representation, role of Unicode, typeface design and so on.
[Read More]
u and uː vowel signs of Malayalam
Posted on April 15, 2018
| Kavya Manohar
The reformed or simplified orthographic script style of Malayalam was introduced in 1971 by this government order. This is what is taught in schools. The text book content is also in reformed style. The prevailing academic situation does not facilitate the students to learn the exhaustive and rich orthographic set of Malayalam script. At the same time they observe a lot of wall writings, graffiti, bill-boards and handwriting sticking to the exhaustive orthographic set.
[Read More]