It is your language and your pen

Photo by Joe Shillington on Unsplash

Google recently added voice typing support to more languages. Among the languages Malayalam is also included. The speech recognition is good quality and I see lot of positive comments in my social media stream. Many people started using it as primary input mechanism. This is a big step for Malayalam users without any doubt. Technical difficulties related to writing in Malayalam in mobile devices is getting reduced a lot. This will lead to more content generated and that is one of the stated goals of Google’s Next billion users project. The cloud api for speech recognition will help android developers to build new innovative apps around the speech recognition feature.

Google had added handwriting based input method for many of these languages in 2015. It was also well recieved by Malayalam user community and many chose it as primary input method mechanism for mobile devices.

Google’s machine learning based language tools, including the machine translation is well engineered projects and takes the language technology forward. For a language like Malayalam with relatively less language processing technology, this is a big boost. There is not even a competing product in the above mentioned areas.

All of these above technologies are closed source software, completely controlled by Google. Google’s opensource strategy is a complicated one. Google supports and uses opensource to gain maximum out of it – a pragmatic corporate exploitation. Machine learning based technologies are complex to be defined in the traditional open source definition. Here, for a ML based service provider, the training toolkit might be opensource, tensorflow for example. At the same time, the training data, models might be closed and secret. So, basically the system can be only reproduced by the owners of the data and those who has enough processing capacity. These emerging trends in language technology is also hard for individual opensource developers to catch up because of resourcing issues(data, processing capacity).

Is this model good for language?

Think about this. With no competition, the android operating system with Google’s technology platform is becoming default presence in mobile devices of Malayalam speakers with no doubt. The new language technologies are being quickly accepted as the one and only way to convey a persons expressions to digital world. No, it is not an exaggeration. The availability and quality of these tools is clearly winning its mass user crowd. There is no formal education for Malayalam typing. People discover and try anything that is available. For a new person to the digital world, handwriting was the easiest method to input Malayalam. Now it is speech recognition. And that will be the one and only one way these users know to enter Malayalam content. And these tools are fully owned and controlled by Google with no alternatives.

The open soure alternatives for input methods are still at the traditional typing keyboards. With its peers, they indeed won large user base and it even came to the users before Google entered. For example, the Indic keyboard has 1.4 million installations and actively improved by contributor for 23 languages. But I don’t see any opensource project that is in parallel with handwriting and speech recognition based input methods. As a developer working in Indic language technology based on free software, this is indeed a failure of opensource community.

I contacted a few academic researchers working on speech recognition and handwring recognition and asked what they think about these products by Google. For them, it is more difficult to convince the value of their research. ‘Well, we have products from Google that does this and thousands are using it. Why you want to work again on it?’ This question can’t be answered easily.

But to me, all of these products and its above mentioned nature strongly emphasis the need for free software alternatives. The mediation by closed sourced systems on one of the fundamental language computing task- inputting – with no alternatives puts the whole language and hence its users in heavy risk. Input method technologies, speech recognition, handwriting recognition.. all these are core to the language technology. These technolgies and science behind them should be owned by its speakers. People should be able to study, innovate on top of this technology and should be able to build mechanisms that are free from any corporate control to express their language.

I don’t want to imply or spread fear, uncertainity that Google will one day just start charging for these services or shutdown the tools. That is not my concern. All these language tools I mentioned are not to be built for facing that situation. It is to be developed as fundamental communication tools for the people for the digital age – build, own, learn, use, maintain by the people.

Anniversary of Manjari font release

Today, July 23 marks one year anniversary of Manjari font release.

Out of all my projects, this is the project that gave me highest satisfaction. I see people using it in social media every day for memes, banners, notices. I have seen the font used for Government publishings, notices, reports. I have seen wedding invitations, book covers, Movie titles with Manjari font. I am so happy that Malayalam speakers loved it.

Kavya and me are working on new version of Manjari with more glyphs to support latest Unicode, and optimize some of the ligature rules. The latest version of the font is always available at https://smc.org.in/fonts

ദൃൿസാക്ഷി

സിനിമയെപ്പറ്റിയല്ല, ദൃൿസാക്ഷിയെപ്പറ്റിയാണ്. ദൃൿസാക്ഷി എന്ന വാക്കെങ്ങനെ എഴുതും? അല്ലെങ്കിൽ എങ്ങനെയൊക്കെ എഴുതാം?

ക്+സ എന്നു ചേരുന്നിടത്താണു പ്രശ്നം, രണ്ടുവാക്കുകൾ ഇവിടെ കൂടിച്ചേരുന്നുണ്ടു്, പക്ഷേ കൂടിച്ചേരുന്നിടത്തു് അക്ഷരങ്ങൾ കൂടി കൂട്ടക്ഷരങ്ങളുണ്ടാക്കാൻ പാടില്ലാത്ത ഒരു സവിശേഷതയാണിവിടെയുള്ളതു്.  കയുടെ അടിയിൽ സ എന്ന രൂപം- ഗ്ലിഫ് ഇല്ലാത്ത ഒരു ഫോണ്ടിനെ സംബന്ധിച്ചു് അതു് താഴെക്കൊടുത്തിരിക്കുന്ന സിനിമാ പോസ്റ്ററിലേതുപോലെ നിരത്തിയെഴുതിയാൽ മതി.

പക്ഷേ അങ്ങനെ നിരത്തിയെഴുതിയാൽ മതിയെങ്കിൽ ദൃൿസാക്ഷി എന്ന ഈ 1973 ലെ സിനിമാ പോസ്റ്ററിൽ കയുടെ ചില്ല് ൿ എങ്ങനെ വന്നു?

ക എന്ന വ്യഞ്ജനം പിന്നാലെ വരുന്ന സ-യോടു ചേരാതെ വേറിട്ടുച്ചരിക്കേണ്ട വാക്കാണിതു്. തമിഴ്‌നാട്, കായ്‌കറി, ജോസ്‌തോമസ് തുടങ്ങിയപോലെയൊക്കെ. ഇംഗ്ലീഷിൽ നിന്നു വന്ന ചില വാക്കുകളാണെങ്കിൽ ഹാർഡ്‌വെയർ(ഹാർഡ്വെയർ അല്ല), സോഫ്റ്റ്‌വെയർ, പേയിങ്‌കൌണ്ടർ(പേയിങ്കൌണ്ടർ അല്ല) ഒക്കെ ഉദാഹരണം.

സ്വരം ചേരാത്ത ശുദ്ധരൂപമായ വ്യഞ്ജനമാണു ചില്ലക്ഷരം. ക യുടെ ചില്ല് അത്ര പ്രചാരത്തിലില്ല. ഇലൿട്രോണിക്സ് എന്ന വാക്കിലോ സിവിൿ, സിഡാൿ എന്നിടത്തൊക്കെ ചിലപ്പോൾ കാണാം. 1973 ൽ പക്ഷേ സിനിമാ പോസ്റ്ററിൽ വരാൻ മാത്രം അതിനു പ്രചാരമുണ്ടായിരുന്നെന്നു മനസ്സിലാക്കാം.

ഏതുശരി ഏതുതെറ്റ് എന്ന ചർച്ചയിലേക്ക് പോകാതെ ഈ വാക്കെങ്ങനെയൊക്കെ എഴുതാമെന്നുമാത്രം നമുക്കിവിടെ നോക്കാം.:

  1. ക് + സ: ഇതു് ക യുടെ അടിയിൽ സ അടുക്കിയ ഗ്ലിഫുള്ള ഫോണ്ടുപയോഗിച്ചു് വായിച്ചാൽ ഇങ്ങനെയിരിക്കും: ( This is the most undesired rendering among these examples).ഇതുതന്നെ, അങ്ങനെയൊരു ഗ്ലിഫില്ലാത്ത ഫോണ്ടുപയോഗിച്ച് വായിച്ചാൽ:
  2. ക്+ Zero Width Non Joiner + സ: ക്സ എന്നു നിർബന്ധമായും പിരിഞ്ഞിരിക്കണം എങ്കിൽ യൂണിക്കോഡ് ഉപയോഗിക്കാൻ പറഞ്ഞിരിക്കുന്ന കണ്ട്രോൾ ക്യാരക്ടർ ആണ് Zero width non joiner. മിക്ക ഇൻപുട്ട് മെത്തേഡുകളിലും ഇതു ടൈപ്പു ചെയ്യാൻ സാധിക്കും. ഹാൻഡ് റൈറ്റിങ്ങ് ഇൻപുട്ട് മെത്തേഡിൽ ഇതിനു സാധിക്കില്ല.
  3. ൿ + സ: കയുടെ ചില്ലുപയോഗിച്ച്. കയുടെ ചില്ലിന്റെ യൂണിക്കോഡ് പോയിന്റ് 0D7F ആണ്. ചില്ലുപയോഗിച്ചാൽ പിന്നീട് കയുടെയും സയുടെയും കൂടിച്ചേരൽ നടക്കില്ല.
  4. ക്+ZWJ+സ: കയുടെ അറ്റോമിക് അല്ലാത്ത ചില്ല് – മറ്റു ചില്ലുകൾ പോലെ ഇത് സ്റ്റാൻഡേഡ് അല്ല. എങ്കിലും കുറേ ഫോണ്ടുകളിൽ(Example- in fonts maintained by SMC)

ഒരു വാക്കിങ്ങനെ പലരീതിയിൽ എഴുതുന്നതു് ഭാഷാ ശാസ്ത്രപരമായി ഇത്തിരി കുഴഞ്ഞുമറിഞ്ഞ പ്രശ്നമാണ്. അതിനെപ്പറ്റി തത്കാലം ഇവിടെ വിശദീകരിക്കുന്നില്ല.

Libreoffice Malayalam Hyphenation

I had developed and released hyphenation extension for Malayalam in Openoffice years back. Libreoffice was born later. Eventhough libreoffice supported the openoffice extensions, the extension repository is freshly created for libreoffice. The old extensions were not present in the libreoffice repository.

Now, I have uploaded the Malayalam hyphenation extension in libreoffice extension repository too. I will explain the installation and configuration step by step in this blog post:

All Operating systems

  • Download an extension and save it anywhere on your computer.
  • In LibreOffice, select Tools -> Extension Manager from the menu bar.
  • In the Extension Manager dialog click Add.
  • A file browser window opens. Navigate to the folder where you saved the LibreOffice extension file(s) on your system. The extension’s files have the file extension ‘OXT’.
  • Find and select the extension you want to install and click Open.
  • If this extension is already installed, you’ll be prompted to press OK to confirm whether to overwrite the current version by the new one, or press Cancel to stop the installation.
  • After you are asked whether to install the extension only for your user or for all users. If you choose the Only for me option, the extension will be installed only for your user. If you choose For all users, you need system administrator rights. In this case the extension will be available for all users. In general, choose Only for me, that doesn’t require administration rights on the operating system.

Debian and Ubuntu

The above steps works for Debian and Ubuntu too. But there is a better way. Using your package manager install hyphen-ml package. This will install hyphenation not only for libreoffice, but for typesetting packages like LaTeX.

Using the hyphenation

  • To automatically hyphenate the current or selected paragraphs, choose Format – Paragraph, and then click the Text Flow tab.

    LIbreoffice Hyphenation
  • To manually Hyphenate Single Words, click in the word where you want to add the hyphen, and then press Ctrl+Hyphen(-).
  • To manually Hyphenate Text in a Selection Select the text that you want to hyphenate. Choose Tools – Language – Hyphenation.

For detailed help, read libreoffice hyphenation documentation

A hyphenated paragraph

Known Issues

Malayalam and several other languages does not use visible hypen(-) at the end of line when a word is broken. Currently there is no way to control this in libreoffice.

I had developed hyphenation patterns for 10 other Indian languages too. Yet to upload them to libreoffice repository. But they are readily available in Debian and Ubuntu. You can install them by choosing hyphen-* package.

 

Manjari Font

I am happy to announce the new Malayalam font I was designing for past several months. The new font is named “Manjari”.

Malayalam blogpost: https://blog.smc.org.in/manjari-font/

Malayalam script is known for its curly characters with beautiful loops. Encoded in unicode around 2001, it is relatively new to the digital age. The script has been evolving from rectangle shaped to oval shaped types of varying proportions. The popular culture is more of oval/ellipse shaped curves, mainly because writing methods using stensils or pens demanded less sharp corners. The character or ligature shapes has also been changing gradually towards the shapes that are easy with pens. The Manjari font takes that to another level by smoothening all curves to its maximum.

The curves are constructed along the spiral segments. The resulting shapes are extra smooth. The curve perfection resulted in whitespaces that aquired beautiful leaf and drop shapes between the bowls and loops of the script. It is illustrated in the specimen. The spiral smoothness of curves were complemented by rounded terminals which gives very soft feeling for the eyes.

The design of the curves in Manjari are theoretically based on the PHD thesis by Raph Levien – “From Spiral to Spline: Optimal Techniques in Interactive Curve Design” (http://www.levien.com/phd/thesis.pdf). The Inconsolata monospace humanist latin font known for its clean lines and elegant design by Levien himself is based on this theory.

Normal, Bold, Thin style variants are available. This is the first Malayalam unicode font with thin style variant

Download

The curve strokes in Manjari were drawn in Inkscape using the spiral library written by Raph Levien himself  and opentype feature compilation was done using FontForge. The font is about to release in next few days, SVGs, scripts and source is available at https://gitlab.com/smc/manjari 

Orion Champadiyil prepared some illustrations using the font, you can see them in our font download page

News reports

 

Redesigned font download page of SMC

The font preview and download page of SMC has a fresh look now.

fonts

The intention is to provide a font preview, typography showcase, download site in single page. Every font has multiple illustrations of usage and the text used is editable if you want to try your own text there.

The old page which was also designed by myself was not mobile friendly. It provides a single page view to compare the fonts, each represented as cards. But it did not had enough flexibility to showcase some fine usages of typography.

fonts

The new design is mobile friendly.
fonts

On technical side, I used flexbox, LESS. For carousal style transitions, I used cycle2  More importantly, I did not use bootstrap :).  See code.

New handwriting style font for Malayalam: Chilanka

A new handwriting style font for Malayalam is in development. The font is named as “Chilanka”(ചിലങ്ക).

This is a alpha version release. Following is a sample rendering.

More samples here.

You may try the font using this edtiable page http://smc.org.in/downloads/fonts/chilanka/tests/ -It has the font embedded

Download the latest version: http://smc.org.in/downloads/fonts/chilanka/Chilanka.ttf

Chilanka/ചിലങ്ക is a musical anklet

A brief note on the workflow I used for font development is as follows

  1. Prepared a template svg in Inkscape that has all guidelines and grid setup.
  2. Draw the glyphs. This is the hardest part. For this font, I used bezier tool of inkscape. SVG with stroke alone is saved. Did not prepare outline in Inkscape, this helped me to rework on the drawing several times easily. To visualize how the stroke will look like in outlined version, I set stroke width as 130, with rounded end points. All SVGs are version tracked. SVGs are saved as inkscape svgs so that I can retain my guidelines and grids.
  3. In fontforge, import this svgs and create the outline using expand stroke, with stroke width 130, stroke height 130,  pen angle 45 degree, line cap and line join as round.
  4. Simplify the glyph automatically and manually to reduce the impact of conversion of Cubic bezier to quadratic bezier.
  5. Metrics tuning. Set both left and right bearings as 100 units(In general, there are glyph specfic tuning)
  6. The opentype tables are the complex part. But for this font, it did not take much time since I used SMC’s already existing well maintained feature tables. I could just focus on design part.
  7. Test using test scripts

Some more details:

  • Design: Santhosh Thottingal
  • Technology: Santhosh Thottingal and Kavya Manohar
  • Total number of glyphs: 676. Includes basic latin glyphs.
  • Project started on September 15, 2014
  • Number of svgs prepared: 271
  • Em size: 2048. Ascend: 1434. Descend: 614
  • 242 commits so far.
  • Latest version: 1.0.0-alpha.20141027
  • All drawings are in inkscape. No paper involved, no tracing.

Thanks for all my friends who are helping me testing and for their encouragement.
Stay tuned for first version announcement 🙂

(Cross posted from http://blog.smc.org.in/new-handwriting-style-font-for-malayalam-chilanka/ )

Updated Swanalekha JavaScript Library

Six years back I wrote a javascript version of popular Swanalekha input method. Friends were requesting web based version that they can use in Windows platforms too. Swanalekha was initially written as a SCIM input method and later m17n input method, readily available in GNU/Linux platform. I had provided a bookmarklet version too. Later Rajiv Nair and Nishan Nasir wrote Chrome and Firefox extensions.

Yesterday I noticed that the updated Marunadan Malayali news portal using my javascript library to power Malayalam input method in their search boxes. I don’t track who else or how many use these libraries and tools. But I quickly realized how bad the code is. Or in other words, How better I can write them today. 6 years is a long span of time anyway.

So I did some cleanup and rewrite, added documentation, example and here it is: http://thottingal.in/projects/swanalekha/swanalekha-ml.html

Code: https://github.com/smc/input-methods/tree/master/swanalekha-js

Enjoy.