This typeface was financially supported by Kerala Bhasha Institute, a Kerala government agency under cultural department. This is the first time SMC work with Kerala Government to produce a new Malayalam typeface.
Gayathri is a display typeface, available in Regular, Bold, Thin style variants. It is licensed under Open Font License. Source code, including the SVG drawings of each glyph is available in the repository. Gayathri is available for download from smc.org.in/fonts#gayathri
Gayathri has soft, rounded terminals, strokes with varying thickness and good horizontal packing. Gayathri has large glyph set for supporting Malayalam traditional orthography, which is the new trend in contemporary Malayalam. With a total of 1124 glyphs, Gayathri also has basic latin coverage. All Malayalam characters defined till Unicode 11 is supported.
There are not much Malayalam typefaces designed for titles and large displays. We hope Gayathri will fill that gap.
This is also the first typeface by Binoy Dominic. He had proved his lettering skills in his profession as graphic designer, working on branding with Malayalam content for his clients.
Binoy prepared all glyphs in SVGs, our scipts converted it to UFO sources. Trufont was used for small edits. Important glyph information like bearings, names, were defined in yaml configuration. Build scripts generated valid UFO sources and fontmake was used to build OTF output. Of course, there were lot of cycles of design fine tuning. Gitlab CI was used for running the build chain and testing. Fontbakery was used for quality assurance. UFO Normalizer,UFO Lint tools were also part of build system.
DNS over TLS is a security protocol that forces all connections with DNS servers to be made securely using TLS. This effectively keeps ISPs from seeing what website you’re accessing.
For the GNU/Linux distributions using systemd, you can setup this easily by following the below steps.
First, edit the /etc/systemd/resolved.conf and changed the value of DNSOverTLS as :
Now, configure your DNS servers. You need to use DNS server that support DNS over TLS. Examples are Cloudflare DNS 126.96.36.199 or 188.8.131.52. Google DNS 184.108.40.206 also support it. To configure you can use Network manager graphical interface.
Then restart the systemd-resolved using:
sudo systemctl restart systemd-resolved
You are done. To check whether settings are correctly applied, you can try:
$ resolvectl status
LLMNR setting: no
MulticastDNS setting: no
DNSOverTLS setting: opportunistic
If you really want to see how DNS resolution requests are happening, you may use wireshark and inspect port 53 – the usual DNS port. You should not see any traffic on that port. Instead, if you inspect port 853, you can see DNS over TLS requests.
പതിനെട്ടാം പിറന്നാളാണിന്ന്. അമ്പത്തെട്ടുലക്ഷം ലേഖനങ്ങളോടെ ഇംഗ്ലീഷ്
വിക്കിപീഡിയയും അറുപതിനായിരത്തോളം ലേഖനങ്ങളോടെ മലയാളം വിക്കിപീഡിയയും
ഒരുപാടു പരിമിതികൾക്കും വെല്ലുവിളികൾക്കുമിടയിൽ യാത്ര തുടരുന്നു.
292 ഭാഷകളിൽ വിക്കിപീഡിയ ഉണ്ടെങ്കിലും ഉള്ളടക്കത്തിന്റെ അനുപാതം ഒരുപോലെയല്ല. വിക്കിമീഡിയ ഫൗണ്ടേഷനിൽ കഴിഞ്ഞ നാലുവർഷമായി എന്റെ പ്രധാനജോലി ഭാഷകൾ തമ്മിൽ മെഷീൻ ട്രാൻസ്ലേഷന്റെയും മറ്റും സഹായത്തോടെ ലേഖനങ്ങൾ പരിഭാഷപ്പെടുത്തുന്ന സംവിധാനത്തിന്റെ സാങ്കേതികവിദ്യയ്ക്ക് നേതൃത്വം കൊടുക്കലായിരുന്നു.
ഇന്നലെ ഈ സംവിധാനത്തിന്റെ സഹായത്തോടെ പുതുതായി കൂട്ടിച്ചേർത്ത ലേഖനങ്ങളുടെ എണ്ണം നാലുലക്ഷമായി.
The Swanalekha transliteration based Malayalam input method is now available in Windows and Mac platforms. Thanks to Ramesh Kunnappully, who wrote the keyman implementation.
I wrote this input method in 2008. At those days SCIM was the popular input method for Linux. Later it was rewritten for M17N and used with either IBus or FCITX. A few years later, this input method was made to available in Android using Indic keyboard. Last year, due to requests from Windows and Mac users, Chrome and Firefox extensions were prepared. Thanks to SIL Keyman, now we made it available in those operating systems as well.
By this, Swanalekha Malayalam becomes an input method you can use in all operating systems and phones.
ഭാഷാകമ്പ്യൂട്ടിങ്ങുമായി ബന്ധപ്പെട്ട സ്വതന്ത്ര സോഫ്റ്റ്വെയർ വികസനപ്രവർത്തനങ്ങളിൽ ഏർപ്പെടാൻ തുടങ്ങിയിട്ട് പത്തുവർഷമാകുന്നു. 2008 ൽ ഒക്കെയാണ് ഈ മേഖലയിൽ സജീവമാകുന്നതും പലതരം പ്രോജക്ടുകൾക്കായി സമയം നീക്കിവെച്ചു തുടങ്ങുന്നതും. കഴിഞ്ഞ പത്തുവർഷത്തെ എന്റെ സംഭാവനകൾ ഗിറ്റ്ഹബ്ബിൽ ഉള്ള കോഡിന്റെ അടിസ്ഥാനത്തിൽ ചിത്രീകരിച്ചിരിക്കുകയാണിവിടെ.
ഇതിലെ ഓരോ കള്ളിയും ഒരു ദിവസമാണ്. പച്ച നിറത്തിലുള്ള കള്ളിയുള്ള ദിവസങ്ങളിൽ കോഡ്, ബഗ്ഗ് റിപ്പോർട്ടുകൾ, മറ്റുള്ളവരുടെ കോഡ് റിവ്യൂ ചെയ്യൽ അങ്ങനെയെന്തെങ്കിലും രീതിയിലുള്ള പ്രവർത്തനം ചെയ്തുവെന്നർത്ഥം. ഇളം പച്ചയിൽ നിന്നും കടുംപച്ചയിലേക്ക് പോകുന്തോറും അതിന്റെ എണ്ണം കൂടുന്നു.
ഒരു ഡയറി പോലെത്തന്നെ എന്റെ ജീവിതത്തിലെ മധുരവും കയ്പ്പും എനിക്കിതിൽ വായിച്ചെടുക്കാം. പലപ്പോളായി കാണുന്ന നീണ്ട ഇടവേളകൾ യാത്രകളോ, വ്യക്തിപരമായ നല്ലതോ മോശമോ ആയ വിട്ടുനിൽക്കലുകളാണ്. ഇക്കാര്യത്തിൽ 2016 വളരെ മോശമായിരുന്നെന്നു കാണാം. 2013 ഏപ്രിലിലെ ഇടവേള എന്റെ വിവാഹത്തെ കാണിക്കുന്നു. ഇടയ്ക്ക് ഇടവേളകളില്ലാതെ 100 ദിവസം എന്തെങ്കിലും ചെയ്യുക എന്ന ഒരു ചലഞ്ചും ചെയ്തിരുന്നു- github streak – 2014 സെപ്റ്റംബർ മുതൽ അത് കാണാം.
അഭിമാനിക്കാവുന്ന ഒരു കാര്യം എന്റെ കരിയർ മുന്നോട്ടുപോകുന്തോറും എൻജിനിയറിങ്ങിൽ കൂടുതൽ സംഭാവനകൾ ചെയ്യാൻ കഴിയുന്നുണ്ട് എന്നതാണ്. പൊതുവിൽ ഐടി മേഖലയിൽ പ്രവർത്തിക്കുന്നവർക്കറിയാം, ആദ്യ പത്ത് വർഷങ്ങൾ പിന്നിടുമ്പോൾ മിക്കവാറും എൻജിനിയറിങ്ങ് സ്വഭാവമുള്ള ജോലിയിൽ നിന്നും മാനേജ്മെന്റ് സ്വഭാവമുള്ള ജോലിയിലെത്തിയിരിക്കും. ഞാൻ ആ പാത തിരഞ്ഞെടുത്തില്ല.
2011 ൽ വിക്കിമീഡിയ ഫൌണ്ടേഷനിൽ ഭാഷാ സാങ്കേതികവിദ്യാ വിഭാഗത്തിൽ ജോലിക്ക് ചേർന്നതോടെ, പൊതുജനങ്ങൾക്കായുള്ള കോഡ് എഴുതുന്നത് വളരെയേറെ കൂടി. അതേ സമയം വാരാന്ത്യങ്ങളിലും മറ്റ് ഒഴിവുസമയങ്ങളിലും മലയാളഭാഷയുമായി ബന്ധപ്പെട്ട പ്രവർത്തനങ്ങളിലും ഏർപ്പെട്ടു. അതുകൊണ്ടാണ് ഈ ഗ്രാഫിൽ ശനിയും ഞായറുമൊക്കെ പച്ച നിറം കാണുന്നത്.
അഭിമാനിക്കാവുന്ന മറ്റൊരു കാര്യം എന്റെ പ്രൊഫഷനിൽ, പൊതുജനങ്ങൾക്കായുള്ള കോഡ് എഴുതേണ്ടിവന്നപ്പോഴൊക്കെ അത് സ്വതന്ത്ര സോഫ്റ്റ്വെയറായി ചെയ്യാൻ സാധിച്ചുവെന്നതാണ്. അതായത് ഒരു ലൈൻ കോഡുപോലും ഞാൻ മറച്ചുവെച്ചിട്ടില്ല. ഞാൻ ചെയ്ത ഓരോ സംഭാവനയും കാര്യകാരണസഹിതം തുറന്നുവെച്ചിരിക്കുന്നു. ആർക്കും എപ്പോഴും പരിശോധിക്കാവുന്ന, പഠിക്കാവുന്ന, ഉപയോഗിക്കാവുന്ന വിധം. അതാണ് സ്വതന്ത്ര സോഫ്റ്റ്വെയർ.
ഇതിലെ ചില പ്രവർത്തനങ്ങളുടെ ഫലങ്ങളെങ്കിലും നിങ്ങൾ ഒരു മലയാളിയാണെങ്കിൽ മിക്കവാറും നിത്യജീവിതത്തിൽ ഏതെങ്കിലും വിധത്തിൽ ഉപയോഗിക്കുന്നുണ്ടാവും. അതേസമയം തുടക്കകാലങ്ങളിൽ എഴുതിയ പലതും ഒരു ടെക്നോളജി പരീക്ഷണത്തിൽ നിന്ന് പുറത്ത് കടന്നു ഉപയോഗപ്രദമായ ഒരു സോഫ്റ്റ്വെയർ ആക്കുന്നതിൽ പരാജയപ്പെട്ടിട്ടുമുണ്ട്. പക്ഷേ അതൊക്കെ സ്വാഭാവികമായും പിന്നത്തേക്കുള്ള പാഠങ്ങളായിരുന്നു.
In this release, mlmorph can analyse and generate malayalam words using the morpho-phonotactical rules defined and based on a lexicon. We have a test corpora of Fifty thousand words and 82% of the words in it are recognized by the analyser.
A python interface is released to make the usage of library very easy for developers. The library is available in pypi.org – https://pypi.org/project/mlmorph/ Installing it is very easy:
Installing it is very easy:
pip install mlmorph
It avoids all difficulties of compiling the sfst formalism and installing the required hfst, sfst packages.
There are lot of known limitations with the current release. I plan to address them in future releases.
Expand lexicon further: The current lexicon was compiled by testing various text and adding missing words found in it. Preparing the coverage test corpora also helped to increase the lexicon. But it still need more improvement
Many language specific constructs which are commonly used, but consisting of multiple conjunctions, adjectives are not well covered. Some examples are മറ്റൊരു, പിന്നീട്, അതുപോലെത്തന്നെ, എന്നതിന്റെ etc.
Optimizing the weight calculation: As the lexicon size is increased, many rarely used words can become alternate parts in agglutination of the words. For example, പാലക്കാട് can have an analysis of പാല്, അക്ക്, ആട് -Even though this is grammatically correct, it should get less preference than പാലക്കാട്<proper noun>.
Standardization of POS tags: mlmorph has its own pos tags definition. These tags need documentation with examples. I tried to use universal dependencies as much as possible, but it is not enough to cover all required tags for malayalam.
Documentation of formalism and tutorials for developers. So far I am the only developer for the project, which I am not happy about. The learning curve for this project is too steep to attract new developers. Above average understanding of Malayalam grammar is a difficult requirement too. I am planning to write down some tutorials to help new developers to join.
The project is meaningful only when practical applications are built on top of this.
The Malayalam and Tamil Root Zone Label Generation Rules for International Domain names have been released for public comments. See the announcement from ICANN. This was drafted by the Neo-Brahmi Script Generation Panel (NBGP), in which I am also a member.
My first attempt to develop a spellchecker for Malayalam was in 2007. I was using hunspell and a word list based approach. It was not successful because of rich morphology of Malayalam. Even though I prepared a manually curated 150K words list, it was nowhere near to cover practically infinite words of Malayalam. For languages with productive morphological processes in compounding and derivation that are capable of generating dictionaries of infinite length, a morphology analysis and generation system is required. Since my efforts towards building such a morphology analyser is progressing well, I am proposing a finite state transducer based spellchecker for Malayalam. In this article, I will first analyse the characteristics of Malayalam spelling mistakes and then explain how an FST can be used to implement the solution.
What is a spellchecker?
The spellchecker is an application that tells whether the given word is spelled correctly as per the language or not. If the word is not spelled correctly, the spellchecker often gives possible alternatives as suggestion to correct the misspelled word. The word can be spellchecked independently or in the context of a sentence. For example, in the sentence “അസ്തമയസൂര്യൻ കടലയിൽ മുങ്ങിത്താഴ്ന്നു”, the word “കടലയിൽ” is spelled correctly if considered independently. But in the context of the sentence, it is supposed to be “കടലിൽ”.
The correctness of the word is tested by checking if that word is in the language model. The language model can be simply a list of all known words in the language. Or it can be a system which knows how a word in a language will look like and tell whether the given word is such a word. In the case of Malayalam, we saw that the finite dictionary is not possible. So we will need a system which is ‘aware’ of all words in the language. We will see how a morphology analyser can be such a system.
If the word is misspelled, the system need to give correction. To generate the correctly spelled words from a misspelled word form, an error model is needed. The most common error model is Levenshtein edit distance. In the edit distance algorithm, the misspelling is assumed to be a finite number of operations applied to characters of a string: deletion, insertion, change, or transposition. The number of operations is known as ‘edit distance‘. Any word from the known list of words in the language, with a minimal distance is a candidate for suggestion. Peter Norvig explains such a functional spellchecker in his article “How to Write a spelling corrector?“
There are multiple problems with the edit distance based correction mechanism
For a query word, to generate all candidates after applying the four operations, we can calculate the number of words we need to generate and test its correctness. For a word of length n, an alphabet size a, an edit distance d=1, there will be n deletions, n-1 transpositions, a*n alterations, and a*(n+1) insertions, for a total of 2n+2an+a-1 terms at search time. In the case of Malayalam, a is 117 if we consider all encoded characters in Unicode version 11. If we remove all archaic characters, we still need about 75 characters. So, for edit distance d=1, a=75, for a word with 10 characters, 2*10+2*75*10+75-1 = 1594 and much larger for larger d. So, you will need to do 1594 lookups(spellchecks) in the language model to get possible suggestions.
The concept that the 4 edit operations are the cause for all spelling mistakes is not accurate for Malayalam. There are many common spelling mistakes in Malayalam that are 3 or 4 edit distance from the original word. Usually the edit distance based corrections won’t go beyond d=2 since the number of candidates increases.
The problems with hunspell based spellchecker and Malayalam
Hunspell has a limited compounding support, but limited to two levels. Malayalam can have more than 2 level compounding and sometimes the agglutinated words is also inflected. Hunspell system has an affix dictionary and suffix mapping system. But it is very limited to support complex morphology like Malayalam. With the help of Németh László, Hunspell developer, I had explored this path. But abandoned due to many limitation of Hunspell and lack of programmatic control of the morphological rules.
Nature of Malayalam spelling mistakes
Malayalam uses an alphasyllabary writing system. Each letter you write corresponds to the grapheme representation of a phoneme. In broader sense Malayalam can be considered as a language with one to one grapheme to phoneme correspondence. Where as in English and similar languages, letters might represent a variety of sounds, or the same sounds can be written in different ways. The way a person learns writing a language strongly depends on the writing system.
In Malayalam, since there is one and only one set of characters that can correspond to a syllable, the confusion of letters does not happen. For example, in English, Education, Ship, Machine, Mission all has sh sound [ʃ]. So a person can mix up these combinations. But in Malayalam, if it is sh sound [ʃ], then it is always ഷ.
Because of this, the spelling mistakes that is resulted by four edit operations(deletion, insertion, change, or transposition) may not be an accurate classification of errors in Malayalam. Let us try to classify and analyse the spelling mistake patterns of Malayalam.
Phonetic approximation: The 1:1 grapheme to phoneme correspondence is the theory. But because of this the inaccurate utterance of syllables will cause incorrect spellings. For example, ബൂമി is a relaxed way of reading for ഭൂമി since it is relatively effortless. Since the relaxed way of pronunciation is normal, sometimes people think that they are writing in wrong way and will try to correct it unnecessarily പീഢനം->പീഡനം is one such example.
Consonants: Each consonant in Malayalam has aspirated, unaspirated, voiced and unvoiced variants. Between them, it is very usual to get mixed up
Aspirated and Unaspirated mix-up: Aspirated consonant can be mistakenly written as Unaspirated consonant. For Example, ധ -> ദ, ഢ -> ഡ . Similarly Unaspirated consonant can be mistakenly written as aspirated consonant – Example, ദ ->ധ, ഡ ->ഢ.
Voiced and Voiceless mix-up. Voiced consonants like ഗ, ഘ can be mistakenly written as voiceless forms ക, ഖ. And vice versa.
Gemination of consonants is often relaxed or skipped in the speech, hence it appear in writing too. Gemination in Malayalam script is by combining two consonants using virama. നീലതാമര/നീലത്താമര is an example for this kind of mistakes. There are a few debatable words too, like സ്വർണം/സ്വർണ്ണം, പാർടി/പാർട്ടി. Another way of consonant stress indication is by using Unaspirated Consonant + Virama + Aspirated Consonant. അദ്ധ്യാപകൻ/അധ്യാപകൻ, തീർഥം/തീർത്ഥം, വിഡ്ഡി/വിഡ്ഢി pairs are examples.
Each vowel in Malayalam can be a short vowel or long vowel. Local dialect can confuse people to use one for the other. ചിലപ്പൊൾ/ചിലപ്പോൾ is one example. Since many input tools place the short and long vowels forms with very close keystrokes, it is possible to cause errors. In Inscript keyboard, short and long vowels are in normal and shift position. In transliteration based input methods, long vowel is often typed by repeated keys(i, ii for ി, ീ).
The vowel ഋ is close to റി or റു in pronunciation. Example: ഋതു/റിതു. The vowel sign of ഋ while appearing with a consonant is close to ്ര. Example ഗൃഹം/ഗ്രഹം. ഹൃദയം/ഹ്രുദയം.
Gliding vowels ഐ, ഔ get confused with its constituent vowels. കൈ/കഇ/കയ്, ഔ/അഉ/അവു are example.
In Malayalam, there is a tendency to use എ instead of ഇ, since the reduced effort. Examples: ചിലവ്/ചെലവ്, ഇല/എല, തിരയുക/തെരയുക. Due to wide usage of these variants, it is sometimes very difficult to say one word is wrong. See the discussion about the ‘Standard Malayalam’ at the end of this essay.
Chillus: Chillus are pure consonants. A consonant + virama sequence sometimes has no phonetic difference from a chillu. For example, കല്പന/കൽപന, നിൽക്കുക/നില്ക്കുക combinations. The chillu ർ is sometimes confused with ഋ sign. Examples are: പ്രവർത്തി/പ്രവൃത്തി. The chillu form of മ – ം can appear are as anuswara or ma+virama forms. Examples: പംപ, പമ്പ. But it is not rare to see പംമ്പ for this. Sometimes, the anuswara get confused with ന്, and പമ്പ becomes പന്പ. There were a few buggy fonts that used ന്+പ for മ്പ ligature too.
Weak Phoneme-Grapheme correspondence: Due to historic or evolutionary nature of the script, Malayalam also has some phonemes which has a weak relationship with the graphemes.
ഹ്മ/ മ്മ as in ബ്രഹ്മം/ബ്രമ്മം, ന്ദ/ന്ന as in നന്ദി/നന്നി, ഹ്ന/ന്ന as in ചിഹ്നം/ചിന്നം are some examples where what you pronounce is not exactly same as what you write.
റ്റ, ന്റ – These two highly used conjuncts heavily deviate from the letters and pronunciation. While writing using pen, people don’t make much mistakes since they just draw the shape of these ligatures, but while typing, one need to know the exact key sequence and they get confused. Common mistakes for these conjuncts are ററ, ൻറ, ൻറ്റ , ൻററ
Visual similarity: While using visual input methods such as handwriting based or some onscreen keyboards, either the users or the input tool makes mistakes due to visual similarity
ൃ, ്യ often get confused.
ജ്ഞ, ഞ്ജ is one very common sequence where people are confused. ആദരാജ്ഞലി/ആദരാഞ്ജലി.
ത്സ, ഝ is another combination
The handwriting based input methods like Google handwriting tool is known for recognizing anuswara ം as zero, English o, O etc.
When people don’t know how to insert visarga ഃ, and since there is a very similar key in keyboard- colon : they use it. Example: ദുഃഖം/ദു:ഖം
ള്ള, the geminated form of ള, is very similar to two adjacent ള. This kind of mistakes are very frequent among people whi studied Malayalam inputting informally. Two adjacent റ, is another mistake for റ്റ,
The informal, trial-and-error based Malayalam inputting training also introduced some other mistakes such as using open parenthesis ‘(‘ for ്ര, closing parenthesis ‘)’ for ാ sign.
Ambiguity due to regional dialect: A good example for this is insertion of യ് in verbs. കുറക്കുക/കുറയ്ക്കുക, ചിരിക്കുക/ചിരിയ്ക്കുക, Also in nominal inflections: പൂച്ചയ്ക്ക്/പൂച്ചക്ക്. Usuage of Samvruthokaram to distinguish between a pure consonant and stressed consonant at the end of word is a highly debated topic. For example, അവന്/അവനു്/അവനു. All these forms are common, even though the usage of നു് is less after the script reformation. But since script reformation was not an absolute transformation, it still exist in usage
Spaces: Malayalam is an agglutinative language. Words can be agglutinated, but nothing prevents people to put space and write in simple words. But this should be done carefully since it can alter the meaning. An example is “ആന പുറത്തു കയറി”, ആനപ്പുറത്തു കയറി”, “ആനപ്പുറത്തുകയറി”, “ആനപ്പുറത്ത് കയറി”. Another example: “മലയാള ഭാഷ”, “മലയാളഭാഷ” – Here, there is no valid word “മലയാള”. The anuswara at the end get deleted only when it joins with ഭാഷ as adjective. A morphology analyser can correctly parse “മലയാളഭാഷ” as മലയാളം<proper-noun><adjective>ഭാഷ<noun>. But since language already broke this rule and many people are liberally using space, a spellchecker would need to handle this cases.
Slip of Finger: Accidental insertions or omissions of key presses is the common reason for spelling mistakes. For alphabetic language, mostly this type of errors are addressed. For Malayalam also, this type of accidental slip of finger can happen. For Latin based languages, we can make some analysis since we know a QWERTY keyboard layout and do optimized checks for this kind of issues. Since Malayalam will use another level of mapping on top of QWERTY for inputting(inscript, phonetic, transliteration), it is not easy to analyse this errors. So, in general, we can expect random characters or omission of some characters in the query word. An accidental space insertion has the challenge that it will split the word to two words and if the spellchecking is done by one word at a time, we will miss it.
I must add that the above classification is not based on a systematic study of any test data that I can share. Ideally, this classification should done with real sample of Malayalam written on paper and computer. It should be then manually checked for spelling mistakes, list down the mistakes and analyse the patterns. This exercise would be very beneficial for spellcheck research. In my case, even since I released my word list based spellchecker, noticing spelling errors in internet(social media, mainly) has been my obsession. Sometimes I also tried to point out spelling mistakes to authors and that did not give much pleasant experience to me 😁 . The above list is based on my observation from such patterns.
Malayalam spelling checker
To check if a word is valid, known, correctly spelled word, a simple look up using morphology analyser is enough. If the morphology analyser can parse the word, it is correctly spelled. Note that the word can be an agglutinated at arbitrary levels and inflected at same time.
Out of lexicon words
Compared to the finite set word list, the FST based morphology analyser and generator system covers large number of words using its generation system based on morpho-phonotactics. For a discussion on this see my previous blog post about the coverage test. Since every language vocabulary is a dynamic system, it is still impossible to cover 100% words in a language all the time. New words get added to language every now and then. There are nouns related to places, people names, product names etc that is not in the lexicon of Morphology analyser. So, these words will be reported as unknown words by the spellchecker. Unknown word is interpreted as misspelled word too. This issue is a known problem. But since a spellchecker is often used by a human user, the severity of the issue depends whether the spellchecker does not know about lot of commonly used words or not. Most of the spellcheckers provide an option to add to dictionary to avoid this issue.
As part of the Morphology analyser, the expansion of the lexicon is a never ending task. As the lexicon grows, the spellchecker improves automatically.
Malayalam spelling correction
To provide spelling suggestions, the FST based morphology analyser can be used. This is a three step process
Generate a list of candidate words from the query word. The words in this list may be incorrect too. The words are generated based on the patterns we defined based on the nature of spelling mistakes. We scan the query word for common patterns of errors and apply fix for that pattern. Since there dozens of patterns, we will have many candidate words.
From the candidate list, find out the correctly spelled word using spellcheck method. This will result a very small number of words. These words are the probable replacements for the misspelled query word.
Sort the candidate words to provide more probable suggestion as the first one. For this, we can do a ranking on the suggestion strategies. A very common error pattern get high priority at step 1. So the suggestions from that appear first in the candidate list. A more sophisticated approach would use a frequency model for the words. So candidate words that are very frequent in the language will appear as first candidate.
One thing I observed from the above approach is, in reality the candidate words after all the above steps for Malayalam is most of the time one or two. This make step 3 less relevant. At the same time, an edit distance based approach would have generated more than 5 candidate words for each misspelled word. The candidates from the edit distance based suggestion mechanism would be very diverse, meaning, they won’t have be related to the indented word at all. The following images illustrates the difference.
Context sensitive spellchecking
Usually the spellchecking and suggestion are done at one word at a time. But if we know the context of the word, the spellchecking will be further useful. The context is usually the words before and after the word. An example from English is “I am in Engineer”. Here the word “in” is a correct word, but with in the context, it is wrong. To mark the word “in” wrong, and provide ‘an’ as suggestion, one approach is ngram model of part of speech for the language. In simple words, what kind of word can appear in between a known kind of words. If we build this model for a language, that will surely tell that the a locative POS “in” before Engineer is rare or not seen before.
The Standard Malayalam or lack thereof
How do you determine which is the “correct” or “standard” way of writing a word? Malayalam has lot of orthographic variants for words which were introduced to language as genuine mistakes that later became common words(രാപ്പകൽ/രാപകൽ, ചിലവ്/ചെലവ്), phonetic simplification(അദ്ധ്യാപകൻ/അധ്യാപകൻ, സ്വർണ്ണം/സ്വർണം), or old spelling(കർത്താവ്/കൎത്താവു്) and so on. A debate about the correctness of these words will hardly reach conclusion. For our case, this is more of an issue of selecting words in the lexicon. Which one to include, which one to exclude? It is easy to consider these debates as blocker for the progress of the project and give up: “well, these things are not decided by academics so far, so we cannot do anything about it till they make up their mind”.
I did not want to end up in that deadlock. I decided to be liberal about the lexicon. If people are using some words commonly, they are valid words the project need to recognize as much as possible. That is the very liberal definition I have. I leave the standardization discussion to linguists who care about it.
A Data-Driven Approach to Checking and Correcting Spelling Errors in Sinhala. Asanka Wasala, Ruvan Weerasinghe, Randil Pushpananda, Chamila Liyanage and Eranga Jayalatharachchi [pdf] This paper discuss the phonetic similarity based strategies to create a wordlist, instead of edit distance approach.
Finite-State Spell-Checking with Weighted Language and ErrorModels—Building and Evaluating Spell-Checkers with Wikipedia as Corpus Tommi A Pirinen, Krister Lindén [pdf] This paper outlines the usage of Finite state transducer technique to address the issue of infinite dictionary of morphologically rich languages. They use Finnish as the example language
The common Malayalam spelling mistakes and confusables were presented in great depth by Renowned linguist and author Panmana Ramachandran Nair in his books ‘തെറ്റില്ലാത്ത മലയാളം’, ‘തെറ്റും ശരിയും’, ‘ശുദ്ധ മലയാളം’ and ‘നല്ല മലയാളം’.
Improving Finite-State Spell-Checker Suggestions with Part of Speech N-Grams Tommi A Pirinen and Miikka Silfverberg and Krister Lindén [pdf] – This paper discuss the context sensitive spellchecker approach.
For the last several months, I am actively working on the Malayalam morphology analyser project. In case you are not familiar with the project, my introduction blog post is a good start. I was always skeptical about the approach and the whole project as such looked very ambitious. But, now I am almost confident that the approach is viable. I am making good progress in the project, so this is some updates on that.
Analyser coverage statistics
Recently I added a large corpora to frequently monitor the percentage of words the analyser can parse. The corpora was selected from two large chapters of ഐതിഹ്യമാല, some news reports, an art related essay, my own technical blog posts to have some diversity in the vocabulary.
This is a very encouraging. Achieving a 66% for such a morphologically rich language Malayalam is no small task. From my reading, Turkish and Finnish, languages with same complexity of morphology achieved about 90% coverage. It may be more difficult to increase the coverage for me compared to achieving this much so far. So I am planning some frequency analysis on words that are not parsed by analyser, and find some patterns to improve.
The performance aspect is also notable. Once the automata is loaded to memory, the analysis or generation is super fast. You can see that ~16000 words were analyzed under half of a second.
From the very beginning the project was test driven. I now has 740 test cases for various word forms
The compiled transducer now is 6.2 MB. The transducer is written in SFST-PL and compile using SFST. It used to be compiled using hfst, but hfst is now severely broken for SFST-PL compilation, so I switched to SFST. But the compiled transducer is read using hfst python binding.
Number of states
Number or arcs
Number of final states
The POS tagged lexicon I prepared is from various sources like wiktionary, wikipedia(based on categories), CLDR. While developing I had to improve the lexicon several times since none of the above sources are accurate. The wiktionary also introduced a large amount of archaic or sanskrit terms to the lexicon. As of today, following table illustrates the lexicon status
English borrowed nouns
Affirmations and negations
As you can see, the lexicon is not that big. Especially it is very limited for proper nouns like names, places. I think the verb lexicon is much better. I need to find a way to expand this further.
There is no agreement or standard on the POS tagging schema to be used for Malayalam. But I refused to set this is as a blocker for the project. I defined my own POS tagging schema and worked on the analyser. The general disagreement is about naming, which is very trivial to fix using a tag name mapper. The other issue is classification of features, which I found that there no elaborate schema that can cover Malayalam.
The main challenge I am facing is not technical, it is linguistic. I am often challenged by my limited understanding of Malayalam grammar. Especially about the grammatical classifications, I find it very difficult to come up with an agreement after reading several grammar books. These books were written in a span of 100 years and I miss a common thread in the approach for Malayalam grammar analysis. Sometimes a logical classification is not the purpose of the author too. Thankfully, I am getting some help from Malayalam professors whenever I am stuck.
The other challenge is I hardly got any contributor to the project except some bug reporting. There is a big entry barrier to this kind of projects. The SFST-PL is not something everybody familiar with. I need to write some simple examples for others to practice and join.
I found that some practical applications on top of the morphology analyser is attracting more people. For example, the number spellout application I wrote caught the attention of many people. I am excited to present the upcoming spellchecker that I was working recently. I will write about the theory of that soon.
Now a days GNU/Linux distributions like Ubuntu, Debian, Fedora etc comes with pre-configured fonts for Malayalam. For Sans-serif family, it is Meera and for serif, it is Rachana. If you like to change these fonts, there is no easy way to do with configuration tools in Gnome or KDE. They provide a general font selector for the whole desktop, but not for a given language.
The advantage of setting these preference at system level is, you don’t need to choose this fonts at application level then. For example, you don’t need to set them for firefox, chrome etc. All will follow the system preferences. We will use fontconfig for this
First, create a file named ~/.config/fontconfig/conf.d/50-my-malayalam.conf. If the folders for this file does not exist, just create them. To this file, add the following content.
Save the file and you are done. You can check if the default font for Malayalam changed or not using the following command
$ LANG=ml_IN fc-match
It should list Manjari. The above code we added to the file is not complicated. You can see that we are setting the sans-serif font preference for ml(Malayalam) language as Manjari. Also serif font preference as Rachana. You are free to change the fonts to whatever you prefer.
Note that you may want to close and open your applications to get this preference applied.
You may choose one of the fonts available at smc.org.in/fonts, download and install and use the above configuration with it.