Releasing the offline version of a wikisource is a challenging project. The technical aspects of the project was designed and implemented by myself. So let me share the details of the project.

As you know a Wikisource contains lot of books, and each book varies in its size, it is divided to chapters or sections. There is no common pattern for books. Each having its own structure. A novel presentation is different from a collection of poems from a Poet. Wikisource also has religious books like Bible, Quran, Bhagavat Geeta, Ramayana etc.  Since books are for continuous reading for a long time, the readabilty and how we present the lengthy chapters in screen also matters. Offline wikipedia tools for example, Kiwix does not do any layout modification of the content and present as it is shown in wikipedia/wikisource. The tool we wrote last year for Malayalam wikipedia offline version also present scrollable vertical content in the screen. Both are not configurable to give different presentation styles depending on the nature of the book.

What we wanted is a book reader kind of application interface.  Readers should be able to easily navigate to books, chapters. The chapter content will be very lengthy. For a long time reading of this content,  a lengthy vertically scrolled text is not a good idea. We also need to take care of the width of the lines.  If each line spans 80-90% of the screen, especially for a wide screen monitor, it is a strain for neck and eyes.

Screenshot of Offline version. Click to enlarge

The selection of books for the offline version was done by the active wikimedians at Wiksource. Some of the selected books was proof read by many volunteers within the last  2 weeks.

The tools used for extracting htmls were adhoc and adapted to meet the good presentation of each book. So there is nothing much to reuse here. Extracting the html and then taking the content part alone using pyquery and removing some unwanted sections from html- basically this is what our scripts did. The content is added to predefined HTML templates with proper CSS for the UI. CSS3 multicolumn feature was used for book like interface. Since IE did not implement this standard even in IE9, for that browser the book like interface was not provided. Chrome browser with version less than 12 could not support, because of these bugs: http://code.google.com/p/chromium/issues/detail?id=45840 and http://code.google.com/p/chromium/issues/detail?id=78155. For easy navigation, mouse wheel support and page navigation buttons provided. For solving non-availability of required fonts, webfonts were integrated with a selection box  to select favorite font. Reader can also select the font size to make the reading comfortable.

Why static html? The variety of platforms and other versions we need to support, necessity to have webfonts, complex script rendering, effort to develop and customize UI, relatively small size of the data, avoiding any installation of software in users system etc made us to choose static html+ jquery + css as the technology choice. The downside is we could not provide full text search.

Apart from the wikisource, we also included a collection of copyleft of images from wikimedia commons. Thanks to Nishan Naseer, for preparing a gallery application using jquery. We selected 4 categories from Commons which are related to Kerala. We hope everybody will like the pictures and it will give  a small introduction to Wikimedia Commons.

Even though the python scripts are not ready to reuse in any projects, if anybody want to have a look at it, please mail me. I am not putting it in public since the script does not make sense outside the context of each book and its existing presentation in Malayalam wikisource.

The CD image is available for download here and one can also browse the CD content here.

Thanks to Shiju Alex for coordinating this project. And thanks to all Malayalam wikisource volunteers for making this happen.  We have included poems, folk songs, devotional songs, novel, grammar book, tales, books on Hinduism, Islam-ism, Christianity, Communism, Philosophy. With this release, it becomes the biggest offline digital archive of Malayalam books.

• Milos Rancic has written a summary of the discussions happened during language committee meeting here : http://lists.wikimedia.org/pipermail/foundation-l/2011-May/065537.html
• Niklas Laxström and Siebrand reviewed the WebFonts extension and enabled at translatewiki.net. Fixed a few bugs that Niklas reported on the extension.
• Purodha Blissenbach was very much interested in the WebFonts and Narayam extensions, we discussed some of the features we need to add. We have it here: Bug 28900 , Bug 28999 and Bug 29000

Recently Sourashtra wikipedia project was started in the wikimedia incubator : http://incubator.wikimedia.org/wiki/Wp/saz and Mediawiki localization started in translatewiki Since the language did not had any proper fonts or input tools, this was not going well.

When we add a  new language support in Mediawiki or start a new language wikipedia,  we need to develop the language technology ecosystem to support its growth. This ecosystem comprises of Unicode code points for the script, proper fonts, rendering support,  input tools, availability of these fonts and input tools in operating systems or alternate ways to get it working in operating system etc.

Sourashtra language had a unicode font developed by Prabu M Rengachari, named ‘Sourashtra’ itself. The font had problems with browsers/operating systems. We fixed to make it work properly. The font was not licensed properly. Prabu agreed to release it in GNU GPLV3 license with font exception. He also agreed to rename the font to another name other than the script name itself.

The font was renamed to Pagul, meaning ‘Footstep’ in Sourashtra and hosted in sourceforge

Once we have a font with proper license, we wanted it to be available in operating systems. I filed a packaging request in Debian. Vasudev Kamath of Debian India Team packaged it and now it is available in debian unstable(sid).  Parag Nemade of Fedora India packaged the font for Fedora and will be avialable in upcoming Fedora 15.

To add a new language support in operating system, we need a locale definition. In GNU Linux this is GLibc locale definition. With the help of Prabu, I prepared the saz_IN locale file for glibc, and filed as bug report to add to glibc. I hope, soon it will be part of Glibc.

Well, all of these was possible since it was GNU/Linux or Free software. Things are a bit difficult on the other side, proprietary operating system world. There is nothing we can do with those operating systems. Since there is no ‘market’ for these minority language, it won’t come to the priority of those companies to add support for these languages. Users will see squares or question marks when they visit sourashtra wikipedia.

We are working on a solution for this, not only for sourashtra, but a common solution for all languages. We are developing a webfonts extension for Mediawiki to provide font embedding in wiki pages to avoid the necessity of having fonts installed in user’s computers. The extension is in development and one can preview it in my test wiki. For Sourashtra, we added webfonts support(preview) .

Input tools needs to be developed and packaged. For mediaiwki, with the help of Narayam extension, we can easily add this support.

With the silpa project, I added a server side, PDF/PNG/SVG rendering support for Sourashtra as well.

Another example showing all inflection forms of the noun പാലക്കാട്, and the same word written in Tamil, Telugu, Hindi. The search shows the results in those languages too. –

You can try it here: http://silpa.org.in/ApproxSearch

This is a Fuzzy string search application. This application illustrates the combined use of Edit distance and Indic Soundex algorithm.

By mixing both written like(edit distance) and sounds like(soundex), we achieve an efficient aproximate string searching. This application is capable of cross language string search too. That means, you can search Hindi words in Malayalam text. If there is any Malayalam word, which is approximate transliteration of hindi word, or sounds alike the Hindi words, it will be returned as an approximate match. The “written like” algorithm used here is a bigram average algorithm. The ratio of common bigrams in two strings and average number of bigrams will give a factor which is greater than zero and less than 1. Similarly the soundex algorithm also gives a weight. By selecting words which has comparison weight more than the threshold weight(which 0.6), we get the search results.

You can read the rules used for Malayalam collation here[PDF document]. Tamil patch was applied to upstream, but the bug is still open since there is some confusion on the results.

Before reading the below discussion, please read the discussion happened in the bug report : [ta_IN] Tamil collation rules are not working in other locales

Since many Tamil friends can give valuable comments on this, I am giving an explanation for my patch here. K Sethu gave some interestin his comments on the patch and I would like to hear from others also. Since collation is a very important component on Tamil support, I feel that an open discussion and consensus  should happen among language speakers outside bug trackers.

This is the logic used currently in Tamil and Malayalam Collation rules also follow the same logic.

1. Consider each consonant as pure consonant + implicit a vowel. ie க= க் + அ   and த= த்+ அ
2. Similarly கா = க்+ ஆ, தி = த்+ இ
3. From #1 and #2, க் < க, த்< த , We get this output for example:அ
   அக்
   அகம்
   அகால
   அக்கம
   அக்கு
   But K Sethu questions this order in his comment here.According to him
   ( consonant1+ virma+ consonant2 ) < ( consonant1+ vowel + [consonant2] )
   or The correct sequence should be அ, அக், அக்கம், அக்கு, அகம், அகால
   But as per my patch
   ( consonant1+ virma+ consonant2 ) > ( consonant1+ vowel + [consontant2] )
   ie, all conjuncts for consonant1 happens after all consonant1+vowel + * sequences.
   So let me try to explain this behaviour.
4. let us take க்த and கத:க்த = க்+ த்+ அ
   கத = க்+ அ+ த்+ அ
   considering the weight comparison logic(decreasing weight from left to right)
   this comparison becomes between
   க்+ த்+ அ and க்+ அ+ த்+ அ
   since க் is common in first weight, removing it. so it becomes
   த்+ அ and அ+ த்+ அ
   Since த் > அ
   த்+ அ > அ+ த்+ அ
   and there by
   க்த > கத
   So conjuncts comes after the cosonant+vowel pairs. hence the result given in #3

Apart from these, equal weights are assigned for ோ (0BCB), ௌ (0BCC), and their canonical equivalent forms.

If anybody interested in testing the patch, get ta_IN and iso14651_t1_common files from here, back up those file in /usr/share/i18n/locales, and place these two files there. reconfigure your locale using “sudo dpkg-reconfigure locales”. Sort some random file using “LANG=ta_IN sort yourfile”. If your distro is not debian based, follow the instructions from here

There is an easy way to test this. Silpa project provides an online application for Indic language collation. You can try it from here. It is a Unicode Collation algorithm implementation. The Unicode collation definition has many mistakes but we have a patched version. You can compare the results between original and patched version.

Feel free to inform this discussion to anybody interested on Tamil Computing. I would be happy to help in the implementation if we  reach  a consensus.

Apart from the existing characters in Indic languages,  Zero width Joiner and Zero width non joiners are widely used in Indic languages to control how the ligatures are formed. For some samples on how they are used, refer the wikipedia links. Being control characters and invisible characters, they are often removed while doing normalization , particularly before doing a string comparison, or collation (sort).

Identifiers, the strings that uniquely represent some data often has a policy on what kind of characters it can contain. For example, email address is an identifier, which unambiguously defines somebody’s email address, does not allow ‘space’ characters in between. Some examples for this kind of identifiers are: email ids, web domain address, variables in programming languages etc.

Gone are the days where identifiers can be represented only using English characters. Python 3.0+ allows  you to define a variable in program using any words that can be represented in Unicode. For more details on this Python feature read PEP 3131 – Supporting Non Ascii Identifiers . Some samples : Program written in Malayalam. In tamil , and In Hindi

Same is the case of Web addresses. With the advent of Internationalized Domain Names(IDN) that allows you register web addresses in your own languages, the English only web address scene is changing.

But this change brings some issues in the definition of ‘Identifiers’ – just like English, what are the characters allowed in using a domain name or programming language identifier that can be used? Standards and specifications are being drafted on this for each language. For Internationalized domain names in Indian languages, CDAC is drafting the policy. For python, the PEP 3131 has specification.

As a general rule, Unicode standard and the standards based on Unicode does not allow you use Unicode control characters such as zwj and zwnj in identifiers. Based on that The Internet Corporation for Assigned Names and Numbers (ICANN) , in RFC 3454 , it prohibits a list of control characters. RFC 3454 is used as a specification for converting a Unicode encoded domain name to its Punicode version for doing the validation.  For example,Thottingal, in Malayalam- തോട്ടിങ്ങല്‍ (0D24 0D4B 0D1F 0D4D 0D1F 0D3F 0D19 0D4D 0D19 0D32 0D4D 200D), when converted to punicode becomes xn--fwcaqax2g2d7dtadc . This conversion excludes the zwj at the end of the word. If I do a reverse conversion from xn--fwcaqax2g2d7dtadc to unicode what I get is തോട്ടിങ്ങല് (0D24 0D4B 0D1F 0D4D 0D1F 0D3F 0D19 0D4D 0D19 0D32 0D4D). Note that codepoint 200D – ZWJ is removed. That means I cannot register my domain thottingal.in in Malayalam properly. You can verify this using ICU online converter.  Now another example, Tamilnadu – in Malayalam തമിഴ്‌നാട് (0D24 0D2E 0D3F 0D34 0D4D 200C 0D28 0D3E 0D1F 0D4D) becomes xn--lwcjmx4a2de7id. When I do a reverse conversion, I getതമിഴ്നാട് (0D24 0D2E 0D3F 0D34 0D4D 0D28 0D3E 0D1F 0D4D) . Now ZWNJ(200C) is missed. Try yourself using the converter . This means one cannot register a website with Tamilnadu written in Malayalam properly. The IDN policies for Indic languages are based on this exclusion rules for zwj, zwnj.

For python 3.0+ ,  you cannot have an identifier in programming language with zwj, zwnj  or any control character in it. See this bug report for more details: Issue 5358

All of the above issues are because of the assumption that zwj,zwnj is prohibited from Identifiers for all cases. But that is not true. Look at the Unicode Standard Annex 31 – “Unicode Identifier and Pattern Syntax”(TR31). TR31 is based on Public Review 96 – “Allowing Special Characters in Identifiers”

This annex describes specifications for recommended defaults for the use of Unicode in the definitions of identifiers and in pattern-based syntax. It also supplies guidelines for use of normalization with identifiers. [...]

default-ignorable characters are normally excluded from Unicode identifiers. However, visible distinctions created by certain format characters (particularly the Join_Control characters) are necessary in certain languages. A blanket exclusion of these characters makes it impossible to create identifiers with the correct visual appearance for common words or phrases in those languages. Identifier systems that attempt to provide more natural representations of terms in modern, customary use should allow these characters in input and display, but limit them to contexts in which they are necessary. [...]

But since the characters are invisible, to meet the security considerations,  It should be clearly defined where and all we can use them. What if a domain is registered with 5 zwnj  continuously in it? It will look same to a string with 4 zwnjs. So TR31 defines 3 valid cases where zwnj and zwj can be used in an Identifier.

• Allow ZWNJ in breaking a cursive connection
• Allow ZWNJ in a conjunct context (example:  തമിഴ്‌നാട് , ദൃക്‌സാക്ഷി)
• Allow ZWJ in a conjunct context (examples:  ന + ് + zwj -> ന്‍ ,  क+  ् +  zwj -> क्‍ )

These 3 cases covers all zwj,zwnj usage patterns in our languages.

So now it is clear that Unicode standard allows them in Identifiers. In that case, there should not be a conflict between Unicode Identifier policy and ICANN policy or any other identifier policy such as PEP 3131. Blanket exclusion of these characters are not allowed. So RFC 3454 should be compatible with TR31. The IDN policy of Indic languages should be based on that new specification and not based on the existing RFC 3454. Since CDAC is responsible of Indic Domain policy, they should take responsibility for bringing this change.

For making a change in PEP 3131, myself and Baiju M started a wiki page explaining what change need to be done. Read it from here.

Having said that, is it desirable to have  two domains,  one with a valid zwj/zwnj usage and another without them? Of course, they will be visually different, avoiding any possibilities for spoofing. Now the question is whether those  two words represent two words in the language?

As far as Malayalam is concerned there are three cases here:

1. Missing ZWJ is considered as a spelling mistake – തമിഴ്‌നാട് (correct), തമിഴ്നാട് (incorrect) pair is an example for that.  Should we allow both domains ? I don’t know any case where a missing ZWNJ form another valid word with different meaning.
2. Missing ZWJ means , the word is a different word with different meaning. This is very rare – വന്‍യവനിക , വന്യവനിക pair is often cited an example for this. But many people argues this is not a valid case.
3. Missing ZWJ never means a spelling mistake, but just a writing style. There are many examples for this. നന്‍മ-നന്മ is one obvious one.

So the question is whether a domain differing by a valid zwj/zwnj use  to an existing registered domain to be allowed or not? I would suggest to use existing policy for domain comparison for this. ie, If the collation weights of existing domain and to-be registered domains are same ,  don’t register the new one. ZWJ, ZWNJ are characters with zero collation weight and in collation or string comparison they are ignored.

Both of these bots were well received  by the users. We have 8000+ users for English-Malayalam Dictionary.  Online blogs/media also gave good publicity. Thanks a lot!.

SMC developers Rajeesh Nambiar, Ershad, Ragsagar, and  Sarath Lakshman had helped in improving the program. You can get the source code from here. It is a small program written using python XMPP library.

We had written this programs one year back, 2009 december itself. We could not launch them for public since we did not had a server to host them.  Usually webhosting providers wont allow to run programs like this in their servers. Recently netdotnet.com provided a VPS server for SMC and we could launch them from that server.

English-Hindi dictionary is reasonably big, but English-Malayalm is very small with only ~10k words. So we just added a Malayalam Wiktionary backend for the bot.

Here is a video on how to use English-Hindi bot prepared by  Varun Verma

• An article about English Malayalam bot in Epathram.com here.
• A blog post by Sailesh in Hindi http://emadad.hindyugm.com/2010/10/know-hindi-meanings-while-chatting.html

We can start this kind of bot for other languages too, if we have dictionaries with Free S/w compatible licenses. If interested, please contact me.

I will be presenting wik2cd, the tool I wrote for Malayalam wikipedia version 1.0 there in a joint workshop with wikipedia offline developers.  I will be joining with Manuel Schneider,  Shiju Alex, Martin Walker in the workshop titled: Creating offline version of Wiki content – Solutions and Challenges. Apart from this, I will be meeting pediapress team, the team behind wikipedia’s latest PDF/Book export feature. There are some issues in this tool for working with Indic languages, mainly because of the PDF rendering engine not capable of rendering complex scripts.

Thanks to Wikimedia foundation for granting me a scholarship to cover travel expenses.

One issue with inkscape when used as DTP for indic script was, a few indic scripts always wanted hyphenation when text is justified. For example Malayalam has lengthy words and often space is wasted in lines if the text is not automatically hyphenated. But this feature was not available in inkscape. There is a wishlist bug for adding this feature to Inkscape.  I tried to develop an extension for Inkscape to achieve this.

It is on top of the python hyphenation code written by Wilbert  Berendsen. The hyphenation rules, also called as patterns is TeX or
Openoffice itself. So  I can support any language which has TeX hyphenation rules. But, since the hyphenation rules are language specific we need a language selection mechanism for the text first. Then only we can select the rules and do the hyphenation. But it is very tricky to implement.  Asking the language of the text every time it is justified is not a good idea. Setting a language for document is another choice, but what if the text contains multiple languages?  But for Indian languages it is very easy, we can automatically detect the scripts using unicode codepoints and load the rules accordingly. So for the time being, my extension support only English and all Indian languages.

Download the extension from http://thottingal.in/projects/inkscape_hyphenation/inkscape-hyphenation.zip . In GNU/Linux machines,  extract the zip file and copy to /usr/share/inkscape/extensions folder. In Windows , extract to [inkscape installation directory]\extensions folder.  After this close and reopen inkscape. You will see a menu named Hyphenate in Effects->Text menu.    In the document, add a text field, enter text in any indian language. Select the text and apply hyphenation by Effects->Text->Hyphenate. Then change the alignment of text to justify. You will see the text get hyphenated and occupying maximum possible space in the text field

I got satisfactory result with Malayalam and Tamil. I did not test other languages. Following images illustrates hyphenated, justified, two column layout of text done in Inkscape

We had a discussion about this in inkscape mailing list . Some developers suggested to have this feature built in, not as extension.  There are few issues to be solved for that. One thing is language selection as I explained. The other issue is regarding the hyphenation character to be used. Unicode standard insists to use soft hyphen – u00AD as hyphenation character. This is an invisible character. For Malayalam, visible hyphens are not required. But some other languages require the hyphen sign where the word is broken at the end of the line. The rules for whether the soft hyphen should be visible or not visible is not clear in Unicode’s specification. Pango never displays a the soft hyphen. There are criticism on this specification of softhyphen

• Jukka Korpela, Soft hyphen (SHY) – a hard problem?  http://www.cs.tut.fi/~jkorpela/shy.html
• Markus Kuhn, Unicode interpretation of SOFT HYPHEN breaks ISO 8859-1   compatibility. Unicode Technical Committee document L2/03-155R, June 2003. http://www.cl.cam.ac.uk/~mgk25/ucs/L2/03155r-kuhn-soft-hyphen.pdf

So I think there is something to be done from Rendering engine or Unicode need to clarify the confusions.  But Openoffice and HTML rendering engines always make soft hyphen at the end of the line, which is not desired for some languages.

Try this extension, let me know the comments. For small scale DTP works, such as pamphlets, notices, brochures  inkscape is enough. But since inkscape is not primarily a DTP software and does not have paging support, for books and large scale DTP works, it may not work well.

]]> http://thottingal.in/blog/2009/10/03/inkscape-hyphenation-extension/feed/ 3