Mlmorph

In the previous blog post I explained my efforts to modernize SFST. Since I published SFST python binding and modernized to make it compile in all operating systems, next step was to drop HFST dependency of mlmorph and use the new version of SFST 1.5.0. mlmorph 1.3.0 has no dependency on HFST and all installation problems in different operating systems and python versions are solved now. Latest version is also available in pypi.org. It can be installed using pip install mlmorph. You can try it in the collab notebook.

The Malayalam morphology analyser(mlmorph) I am actively developing since 2016 is based on Stuttgart Finite State Transducer(SFST) formalism. SFST was developed by Dr Helmut Schmid. Since SFST had minimal developer apis and no functional python APIs, I was using Helsinki Finite State Transducer(HFST) toolkit. HFST has python binding and better tooling and development support. HFST has SFST backend so there was no issue in using SFST formalism. But HFST development has slowed down in past years. In 2017, I had to switch back to SFST for compiling my automata because HFST’s compile support for SFST was broken. That complicated my build system, but still I was able to use the python binding of HFST. But the development and release cycles of HFST further slowed down and python bindings were not released for python 3.8 or later versions. So mlmorph was stuck with python 3.8 and this was a big pain point since developers trying to use mlmorph found it not working in their python versions. It became more serious when Ubuntu and such popular distros moved to later python versions as default ones. I tried to contact the upstream developers. They were trying to help, but there were no active developers. Helping HFST to release python bindings was not easy because of very complicated build system they have. ...

The test corpus for Malayalam Morphological analysis has many foreign words. They are either written in a non-Malayalam script or written in Malayalam. For example, “ഇലക്ട്രിസിറ്റി”, “ഡോക്സ്”, “ഇന്റർമീഡിയറ്റ്”, “അബ്സ്ട്രാക്റ്റ്”, “ഇല്ലസ്ടേഷൻ”, “ഇല്ലിറ്ററേറ്റ്”, “റെക്കോർഡ്”, “procrastination”, “唐宸禹” - These are all foreign words and it is useless to analyse them using mlmorph. Since mlmorph works based on a root word lexicon, it is practically impossible to have them in lexicon. So there should be a way to identify the words easily and tag them as FW - Foreign word Part of speech. The presence of these foreign words also distorts the coverage statistics of mlmorph. A good part of the test corpus is Malayalam wikipedia corpus and it has so many foreign words when the article is about foreign places or people. ...

I just published a VS Code language extension to support syntax highlighting for Stuttgart Finite State Transducer (SFST) formalism to VS Code. Download and install the extension Source code I learned how to write a language extension when I attempted the opentype feature file support. So I thought of applying that learning to SFST which I regularly use for the Malayalam morphology analyser project.

I presented the Malayalam Morphology Analyser at 2nd Workshop on Technologies for MT of Low Resource Languages- LORESMT Dublin, Ireland as part of 17th MT Summit. Paper: https://www.aclweb.org/anthology/W19-6801 Presentation: https://docs.google.com/presentation/d/… Proceedings from the LORESMT workshop: https://aclweb.org/anthology/volumes/W19-68/

The web interface of Malayalam morphology analyser(mlmorph) is updated. You can see new interface at https://morph.smc.org.in/. The new web application is written in vuejs using vuetify UI framework. The backend is flask. Source code is available at https://gitlab.com/smc/mlmorph-web Morphology analysis Morphology generator Named entity recognition Spellchecker Number spellout

One of the key components of Mlmorph is its lexicon. The lexicon contains the root words categorized as nouns, verbs, adjectives, adverbs etc. These are the components used with morphological rules to generate the vocabulary of Malayalam. I collected initial lexicon with about 100,000 words from various sources such as Wikipedia, CLDR and many targeted web crawls. One problem with such collected words is they often contains spelling mistakes. Secondly, classifying these words is not possible without the tedious task of a person going through each and every words. ...

A few months back, I wrote about the spellchecker based on Malayalam morphology analyser. I was also trying to intergrate that spellchecker with LibreOffice. It is not yet ready for any serious usage, but if you are curious and would like to help me in its further development, please read on. Malayalam spellchecker – a morphology analyser based approach Blog post on spellchecker approach and pla Current status The libreoffice spellchecker for Malayalam is available at https://gitlab.com/smc/mlmorph-libreoffice-spellchecker. You need to get the code using git checkout or download the master version as zip file ...

Named Entity Recognition, a task of identifying and classifying real world objects such as persons, places, organizations from a given text is a well known NLP problem. For Malayalam, there were several research papers published on this topic, but none are functional or reproducible research. The morphological characteristics of Malayalam has been always a challenge to solve this problem. When the named entities appear in an inflected or agglutinated complex word, the first step is to analyse such words and arrive at the root words. ...

I am happy to announce the first version of Malayalam morphology analyser. After two years of development, I tagged version 1.0.0 . In this release In this release, mlmorph can analyse and generate malayalam words using the morpho-phonotactical rules defined and based on a lexicon. We have a test corpora of Fifty thousand words and 82% of the words in it are recognized by the analyser. A python interface is released to make the usage of library very easy for developers. The library is available in pypi.org – https://pypi.org/project/mlmorph/ Installing it is very easy: ...