Sfst

I am excited to announce that the Malayalam morphology analyzer library, mlmorph, is now available as a Rust crate on crates.io. This Rust binding became possible thanks to the availability of Rust bindings for mlmorph’s underlying library - the Stuttgart Finite State Transducer (SFST) formalism. You can find the SFST Rust crate here. The crate comes with comprehensive documentation and example usage to help you get started with Malayalam morphological analysis in your Rust projects.

In the previous blog post I explained my efforts to modernize SFST. Since I published SFST python binding and modernized to make it compile in all operating systems, next step was to drop HFST dependency of mlmorph and use the new version of SFST 1.5.0. mlmorph 1.3.0 has no dependency on HFST and all installation problems in different operating systems and python versions are solved now. Latest version is also available in pypi.org. It can be installed using pip install mlmorph. You can try it in the collab notebook.

The Malayalam morphology analyser(mlmorph) I am actively developing since 2016 is based on Stuttgart Finite State Transducer(SFST) formalism. SFST was developed by Dr Helmut Schmid. Since SFST had minimal developer apis and no functional python APIs, I was using Helsinki Finite State Transducer(HFST) toolkit. HFST has python binding and better tooling and development support. HFST has SFST backend so there was no issue in using SFST formalism. But HFST development has slowed down in past years. In 2017, I had to switch back to SFST for compiling my automata because HFST’s compile support for SFST was broken. That complicated my build system, but still I was able to use the python binding of HFST. But the development and release cycles of HFST further slowed down and python bindings were not released for python 3.8 or later versions. So mlmorph was stuck with python 3.8 and this was a big pain point since developers trying to use mlmorph found it not working in their python versions. It became more serious when Ubuntu and such popular distros moved to later python versions as default ones. I tried to contact the upstream developers. They were trying to help, but there were no active developers. Helping HFST to release python bindings was not easy because of very complicated build system they have. ...