A few months back, I wrote about the spellchecker based on Malayalam morphology analyser. I was also trying to intergrate that spellchecker with LibreOffice. It is not yet ready for any serious usage, but if you are curious and would like to help me in its further development, please read on.
You need LibreOffice 4.1 or later. Latest version is recommended. In the source code directory, run make install to install the extension.
Open libreoffice writer, add some Malayalam text. Make sure to select the language as Malayalam by choosing it from the menu or bottom status bar. You should see the spelling check in action… if everything goes as expected ๐
LibreOffice language settings, You can see mlmorph listed.Spellchecker in action- libreoffice writer.
How can you help?
Theoretically, the extension should work in non-Linux platforms as well. But I have not tested it. The extension need python3 and python-hfst for the operating system. But python-hfst is not available for Windows 64 bit python installation. If you test and get the extension working, please add documentation and if anything missing to make the installation more easy, let me know.
As the mlmorph project get wider support for Malayalam vocabulary, the quality of spellchecker improves automatically.
The morphological characteristics of Malayalam has been always a challenge to solve this problem. When the named entities appear in an inflected or agglutinated complex word, the first step is to analyse such words and arrive at the root words.
As the Malayalam morphology analyser is progressing well, I attempted to build a first version of Malayalam NER on top of it. Since mlmorph gives the POS tagging and analysis, there is not much to do in NER. We just need to look for tags corresponding to proper nouns and report.
Malayalam named entity recognition example using https://morph.smc.org.in/ner
Known Limitations
The recognition is limited by the current lexicon of mlmorph. To recognize out of lexicon entities, a POS guesser would be needed. But this is a general problem not limited to NER. A morphology analyser should also have a POS guesser. In other words as the mlmorph improves, this system also improves automatically.
Currently the recognition is at word level. But sometimes, the entities are written in multiple consecutive words. To resolve that we will need to write a wrapper on top of word level detection system.
The current system is a javascript wrapper on top the mlmorph analyse api. I think NER deserve its own api.
Scribus has a new way to download and use these hyphenation dictionaries. You can now use this feature right away in your installed scribus. The languages with hyphenation support are the following:
Malayalam
Tamil
Telugu
Kannada
Marathi
Hindi
Bengali
Gujarati
Assamese
Panjabi
Odia
How to Add Hyphenation Dictionary?
Navigate to Windows -> Resources in the menu bar. You will see a window as given below. You may want to press “Update Available List”. Then you can see all the languages with hyphenation dictionaries available. Select the download checkbox and press “Download” button. The dictionary will get installed to your system.
Scribus Resource Manager
How to use?
Start a new document. Add text frames and content. You may need narrow columns to have wordbreaking contexts.
Select the text and set appropriate font(Unicode) for your language. Make sure the language is selected as your preferred language.
In Hyphenation properties, set hyphenation character as blank, otherwise visible hyphens will appear.
Set the text justified.
From menu Extras->Hyphenate text. Done.
Hyphenated two column content
How does it work?
The resource manager based hyphenation libraries are easier way to add new hyphenation dictionaries. Earlier, these files need to add to Scribus source code. Now these files are defined in scribus server – http://services.scribus.net/scribus_hyph_dicts.xml. It maps the languages to files to download. So if I update the dictionaries in the github repo, a new installation will take that updated file.
This typeface was financially supported by Kerala Bhasha Institute, a Kerala government agency under cultural department. This is the first time SMC work with Kerala Government to produce a new Malayalam typeface.
Gayathri is a display typeface, available in Regular, Bold, Thin style variants. It is licensed under Open Font License. Source code, including the SVG drawings of each glyph is available in the repository. Gayathri is available for download from smc.org.in/fonts#gayathri
Gayathri has soft, rounded terminals, strokes with varying thickness and good horizontal packing. Gayathri has large glyph set for supporting Malayalam traditional orthography, which is the new trend in contemporary Malayalam. With a total of 1124 glyphs, Gayathri also has basic latin coverage. All Malayalam characters defined till Unicode 11 is supported.
There are not much Malayalam typefaces designed for titles and large displays. We hope Gayathri will fill that gap.
This is also the first typeface by Binoy Dominic. He had proved his lettering skills in his profession as graphic designer, working on branding with Malayalam content for his clients.
Binoy prepared all glyphs in SVGs, our scipts converted it to UFO sources. Trufont was used for small edits. Important glyph information like bearings, names, were defined in yaml configuration. Build scripts generated valid UFO sources and fontmake was used to build OTF output. Of course, there were lot of cycles of design fine tuning. Gitlab CI was used for running the build chain and testing. Fontbakery was used for quality assurance. UFO Normalizer,UFO Lint tools were also part of build system.
DNS over TLS is a security protocol that forces all connections with DNS servers to be made securely using TLS. This effectively keeps ISPs from seeing what website youโre accessing.
For the GNU/Linux distributions using systemd, you can setup this easily by following the below steps.
First, edit the /etc/systemd/resolved.conf and changed the value of DNSOverTLS as :
DNSOverTLS=opportunistic
Now, configure your DNS servers. You need to use DNS server that support DNS over TLS. Examples are Cloudflare DNS 1.1.1.1 or 1.0.0.1. Google DNS 8.8.8.8 also support it. To configure you can use Network manager graphical interface.
Then restart the systemd-resolved using:
sudo systemctl restart systemd-resolved
You are done. To check whether settings are correctly applied, you can try:
$ resolvectl status
Global
LLMNR setting: no
MulticastDNS setting: no
DNSOverTLS setting: opportunistic
If you really want to see how DNS resolution requests are happening, you may use wireshark and inspect port 53 – the usual DNS port. You should not see any traffic on that port. Instead, if you inspect port 853, you can see DNS over TLS requests.
The Swanalekha transliteration based Malayalam input method is now available in Windows and Mac platforms. Thanks to Ramesh Kunnappully, who wrote the keyman implementation.
I wrote this input method in 2008. At those days SCIM was the popular input method for Linux. Later it was rewritten for M17N and used with either IBus or FCITX. A few years later, this input method was made to available in Android using Indic keyboard. Last year, due to requests from Windows and Mac users, Chrome and Firefox extensions were prepared. Thanks to SIL Keyman, now we made it available in those operating systems as well.
By this, Swanalekha Malayalam becomes an input method you can use in all operating systems and phones.
Detailed documentation, downloads are available in Swanalekha website. Source code: gitlab.com/smc/swanalekha. A small video illustrating the installation, configuration and use in Windows 10 given below.
Update: The keyboard is now served by keyman from their website. And the supported platforms also increased.
Download options from https://keyman.com/keyboards/swanalekha_malayalam
I am happy to announce the first version of Malayalam morphology analyser.
After two years of development, I tagged version 1.0.0 .ย
In this release
In this release, mlmorph can analyse and generate malayalam words using the morpho-phonotactical rules defined and based on a lexicon. We have a test corpora of Fifty thousand words and 82% of the words in it are recognized by the analyser.
A python interface is released to make the usage of library very easy for developers. The library is available in pypi.org – https://pypi.org/project/mlmorph/ Installing it is very easy:
Installing it is very easy:
pip install mlmorph
It avoids all difficulties of compiling the sfst formalism and installing the required hfst, sfst packages.
There are lot of known limitations with the current release. I plan to address them in future releases.
Expand lexicon further: The current lexicon was compiled by testing various text and adding missing words found in it. Preparing the coverage test corpora also helped to increase the lexicon. But it still need more improvement
Many language specific constructs which are commonly used, but consisting of multiple conjunctions, adjectives are not well covered. Some examples are เดฎเดฑเตเดฑเตเดฐเต, เดชเดฟเดจเตเดจเตเดเต, เด เดคเตเดชเตเดฒเตเดคเตเดคเดจเตเดจเต, เดเดจเตเดจเดคเดฟเดจเตเดฑเต etc.
Optimizing the weight calculation: As the lexicon size is increased, many rarely used words can become alternate parts in agglutination of the words. For example, เดชเดพเดฒเดเตเดเดพเดเต can have an analysis of เดชเดพเดฒเต, เด เดเตเดเต, เดเดเต -Even though this is grammatically correct, it should get less preference than เดชเดพเดฒเดเตเดเดพเดเต<proper noun>.
Standardization of POS tags: mlmorph has its own pos tags definition. These tags need documentation with examples. I tried to use universal dependencies as much as possible, but it is not enough to cover all required tags for malayalam.
Documentation of formalism and tutorials for developers. So far I am the only developer for the project, which I am not happy about. The learning curve for this project is too steep to attract new developers. Above average understanding of Malayalam grammar is a difficult requirement too. I am planning to write down some tutorials to help new developers to join.
Applications
The project is meaningful only when practical applications are built on top of this.
The Malayalam and Tamil Root Zone Label Generation Rules for International Domain names have been released for public comments. See the announcement from ICANN. This was drafted by the Neo-Brahmi Script Generation Panel (NBGP), in which I am also a member.
Your comments on the proposal for the Malayalam Script Label Generation Rules for the Root Zone (LGR [XML, 18 KB] and supporting documentation [PDF, 998 KB]) can be submitted at the feedback form till Nov 7 2018.
