1. Meera font was small compared to other fonts. This was not really a problem in Gnome environment since fontconfig allows you to define a scaling factor to match other font size. But it was an issue in Libreoffice, KDE and mainly in Windows where this kind of scaling feature does not work. Thanks to P Suresh for a rework on glyphs and fixing this issue.
2. Rachana, Meera and Dyuthi had wrong glyphs used as placeholder glyphs. Bugs like these are fixed.
3. Virama 0D4D had a wrong LSB that cause the cursor positioning and glyph boundary go wrong. Fixed that bug
   
4. Atomic Chilu code points introduced in Unicode 5.1 was missing in all the fonts that SMC maintained because of the controversial decision by Unicode and SMC’s stand against that. Issues still exist, but content with code point is present, to avoid any difficulties to users, added those characters to Meera and Rachana fonts.
   
5. Rupee Symbols added to Meera and Rachana. Thanks to Hiran for designing Sans and Serif glyphs for Rupee.
   
6. Dot Reph(0D4E) – The glyphs for this was already present in Meera but unmapped to any unicode point. GSUB Lookup tables added to the glyphs according to unicode specification.
   

For a more detailed change description see this mail thread. There are some minor changes as well.

Thanks to Hussain K H (designer of both Meera and Rachana) , P Suresh, Hiran for their valuable contribution. And thanks to SMC community and font users for using the fonts and reporting bugs. We hope that we can bring this new version in your favorite GNU/Linux distros soon. Wikimedia’s WebFonts extension uses Meera font and the font will be updated there soon. Next release of GNU Freefont is expected to update Malayalam glyphs using Meera and Rachana for freefont-sans and freefont-serif font respectively. We plan to update other fonts we maintain also with these changes in next versions. There are still some glyphs missing in these fonts with respect to the latest unicode version.

SVG is not a webfont format. The purpose of SVG fonts is to be embedded inside of SVG documents  (or linked to them), similar to the way you would embed standard  TrueType or OpenType fonts in a PDF.  SVG fonts are text files that contain the glyph outlines represented  as standard SVG elements and attributes, as if they were single vector  objects in the SVG image. Unlike EOT, WOFF, TTF formats , SVG is plain text uncompressed file.

Even though it is not webfont format, some browsers will accept svg in the @fontface css3 declaration.

Firefox and IE does not support SVG Fonts. Here is the bug on Mozilla bugzilla about this with a lengthy discussion -https://bugzilla.mozilla.org/show_bug.cgi?id=119490  This is one of the reason why Firefox does not score ACID3 test – http://www.itcode.org/why-firefox-is-not-scoring-100-in-the-acid3-test . Developers argue that WOFF is sufficient and SVG Fonts does not give any advantage.  Support for SVG Fonts in the web development and font communities has been declining for some time. There’s already been discussion without objection of dropping SVG fonts from the Acid3 test. The community has put forth a proposal in the SVG Working Group to give SVG Fonts optional status.

Browser Support Matrix for SVG Fonts http://caniuse.com/#feat=svg-fonts  IE and FF do not support it. Webkit based browsers support it – Chrome, Epiphany. For browsers not supporting svg features natively there is a flash based javascript library named svgweb http://code.google.com/p/svgweb/

Limitations of svg fonts:

• Not all of the opentype features are available in SVG specification
• For example Indic fonts require many opentype features for correct rendering – see opentype spec of Malayalam http://www.microsoft.com/typography/otfntdev/malayot/shaping.aspx
• Even though SVG Fonts are support is available in some browsers, practically they cannot render SVG fonts for complex scripts such as Indic – Here is a sample svg file with Meera font defined in it – http://thottingal.in/tests/svg/Meera-fontembedding.svg. As you can see, rendering is wrong.
• The main drawback to SVG fonts is there is no provision for font-hinting. The SVG standard states: “SVG  fonts contain unhinted font outlines. Because of this, on many  implementations there will be limitations regarding the quality and  legibility of text in small font sizes. For increased quality and  legibility in small font sizes, content creators may want to use an  alternate font technology, such as fonts that ship with operating  systems or an alternate WebFont format. – http://www.w3.org/TR/SVG/fonts.html”

There is an alternate proposal to use opentype features of the font and use svg just for the glyphs https://wiki.mozilla.org/SVGOpenTypeFonts

Fontforge can be used for creating SVG Fonts. But the created SVG font works only for simple scripts like Latin. Fails to export GPOS/GSUB tables to the SVG- bug report – http://sourceforge.net/mailarchive/message.php?msg_id=27964229 GSUB issue can be solved either by handcoding the unicode sequences for glyphs or by writing an external script. But , more important opentype features- Vowel sign(matra) reordering issues persists.

Eventhough svg fonts by defining font data inside svg itself does not seem to have much interest from developers, using webfonts inside for svg has some importance. Just like web pages, webfonts can be used to render the text inside the svg. The webfont format depends on the browser. Example: http://thottingal.in/tests/svg/Dyuthi-Webfont.svg (Have a look at the source code of the file)

Releasing the offline version of a wikisource is a challenging project. The technical aspects of the project was designed and implemented by myself. So let me share the details of the project.

As you know a Wikisource contains lot of books, and each book varies in its size, it is divided to chapters or sections. There is no common pattern for books. Each having its own structure. A novel presentation is different from a collection of poems from a Poet. Wikisource also has religious books like Bible, Quran, Bhagavat Geeta, Ramayana etc.  Since books are for continuous reading for a long time, the readabilty and how we present the lengthy chapters in screen also matters. Offline wikipedia tools for example, Kiwix does not do any layout modification of the content and present as it is shown in wikipedia/wikisource. The tool we wrote last year for Malayalam wikipedia offline version also present scrollable vertical content in the screen. Both are not configurable to give different presentation styles depending on the nature of the book.

What we wanted is a book reader kind of application interface.  Readers should be able to easily navigate to books, chapters. The chapter content will be very lengthy. For a long time reading of this content,  a lengthy vertically scrolled text is not a good idea. We also need to take care of the width of the lines.  If each line spans 80-90% of the screen, especially for a wide screen monitor, it is a strain for neck and eyes.

Screenshot of Offline version. Click to enlarge

The selection of books for the offline version was done by the active wikimedians at Wiksource. Some of the selected books was proof read by many volunteers within the last  2 weeks.

The tools used for extracting htmls were adhoc and adapted to meet the good presentation of each book. So there is nothing much to reuse here. Extracting the html and then taking the content part alone using pyquery and removing some unwanted sections from html- basically this is what our scripts did. The content is added to predefined HTML templates with proper CSS for the UI. CSS3 multicolumn feature was used for book like interface. Since IE did not implement this standard even in IE9, for that browser the book like interface was not provided. Chrome browser with version less than 12 could not support, because of these bugs: http://code.google.com/p/chromium/issues/detail?id=45840 and http://code.google.com/p/chromium/issues/detail?id=78155. For easy navigation, mouse wheel support and page navigation buttons provided. For solving non-availability of required fonts, webfonts were integrated with a selection box  to select favorite font. Reader can also select the font size to make the reading comfortable.

Why static html? The variety of platforms and other versions we need to support, necessity to have webfonts, complex script rendering, effort to develop and customize UI, relatively small size of the data, avoiding any installation of software in users system etc made us to choose static html+ jquery + css as the technology choice. The downside is we could not provide full text search.

Apart from the wikisource, we also included a collection of copyleft of images from wikimedia commons. Thanks to Nishan Naseer, for preparing a gallery application using jquery. We selected 4 categories from Commons which are related to Kerala. We hope everybody will like the pictures and it will give  a small introduction to Wikimedia Commons.

Even though the python scripts are not ready to reuse in any projects, if anybody want to have a look at it, please mail me. I am not putting it in public since the script does not make sense outside the context of each book and its existing presentation in Malayalam wikisource.

The CD image is available for download here and one can also browse the CD content here.

Thanks to Shiju Alex for coordinating this project. And thanks to all Malayalam wikisource volunteers for making this happen.  We have included poems, folk songs, devotional songs, novel, grammar book, tales, books on Hinduism, Islam-ism, Christianity, Communism, Philosophy. With this release, it becomes the biggest offline digital archive of Malayalam books.

• Milos Rancic has written a summary of the discussions happened during language committee meeting here : http://lists.wikimedia.org/pipermail/foundation-l/2011-May/065537.html
• Niklas Laxström and Siebrand reviewed the WebFonts extension and enabled at translatewiki.net. Fixed a few bugs that Niklas reported on the extension.
• Purodha Blissenbach was very much interested in the WebFonts and Narayam extensions, we discussed some of the features we need to add. We have it here: Bug 28900 , Bug 28999 and Bug 29000

Recently Sourashtra wikipedia project was started in the wikimedia incubator : http://incubator.wikimedia.org/wiki/Wp/saz and Mediawiki localization started in translatewiki Since the language did not had any proper fonts or input tools, this was not going well.

When we add a  new language support in Mediawiki or start a new language wikipedia,  we need to develop the language technology ecosystem to support its growth. This ecosystem comprises of Unicode code points for the script, proper fonts, rendering support,  input tools, availability of these fonts and input tools in operating systems or alternate ways to get it working in operating system etc.

Sourashtra language had a unicode font developed by Prabu M Rengachari, named ‘Sourashtra’ itself. The font had problems with browsers/operating systems. We fixed to make it work properly. The font was not licensed properly. Prabu agreed to release it in GNU GPLV3 license with font exception. He also agreed to rename the font to another name other than the script name itself.

The font was renamed to Pagul, meaning ‘Footstep’ in Sourashtra and hosted in sourceforge

Once we have a font with proper license, we wanted it to be available in operating systems. I filed a packaging request in Debian. Vasudev Kamath of Debian India Team packaged it and now it is available in debian unstable(sid).  Parag Nemade of Fedora India packaged the font for Fedora and will be avialable in upcoming Fedora 15.

To add a new language support in operating system, we need a locale definition. In GNU Linux this is GLibc locale definition. With the help of Prabu, I prepared the saz_IN locale file for glibc, and filed as bug report to add to glibc. I hope, soon it will be part of Glibc.

Well, all of these was possible since it was GNU/Linux or Free software. Things are a bit difficult on the other side, proprietary operating system world. There is nothing we can do with those operating systems. Since there is no ‘market’ for these minority language, it won’t come to the priority of those companies to add support for these languages. Users will see squares or question marks when they visit sourashtra wikipedia.

We are working on a solution for this, not only for sourashtra, but a common solution for all languages. We are developing a webfonts extension for Mediawiki to provide font embedding in wiki pages to avoid the necessity of having fonts installed in user’s computers. The extension is in development and one can preview it in my test wiki. For Sourashtra, we added webfonts support(preview) .

Input tools needs to be developed and packaged. For mediaiwki, with the help of Narayam extension, we can easily add this support.

With the silpa project, I added a server side, PDF/PNG/SVG rendering support for Sourashtra as well.

Another example showing all inflection forms of the noun പാലക്കാട്, and the same word written in Tamil, Telugu, Hindi. The search shows the results in those languages too. –

You can try it here: http://silpa.org.in/ApproxSearch

This is a Fuzzy string search application. This application illustrates the combined use of Edit distance and Indic Soundex algorithm.

By mixing both written like(edit distance) and sounds like(soundex), we achieve an efficient aproximate string searching. This application is capable of cross language string search too. That means, you can search Hindi words in Malayalam text. If there is any Malayalam word, which is approximate transliteration of hindi word, or sounds alike the Hindi words, it will be returned as an approximate match. The “written like” algorithm used here is a bigram average algorithm. The ratio of common bigrams in two strings and average number of bigrams will give a factor which is greater than zero and less than 1. Similarly the soundex algorithm also gives a weight. By selecting words which has comparison weight more than the threshold weight(which 0.6), we get the search results.

Both of these bots were well received  by the users. We have 8000+ users for English-Malayalam Dictionary.  Online blogs/media also gave good publicity. Thanks a lot!.

SMC developers Rajeesh Nambiar, Ershad, Ragsagar, and  Sarath Lakshman had helped in improving the program. You can get the source code from here. It is a small program written using python XMPP library.

We had written this programs one year back, 2009 december itself. We could not launch them for public since we did not had a server to host them.  Usually webhosting providers wont allow to run programs like this in their servers. Recently netdotnet.com provided a VPS server for SMC and we could launch them from that server.

English-Hindi dictionary is reasonably big, but English-Malayalm is very small with only ~10k words. So we just added a Malayalam Wiktionary backend for the bot.

Here is a video on how to use English-Hindi bot prepared by  Varun Verma

• An article about English Malayalam bot in Epathram.com here.
• A blog post by Sailesh in Hindi http://emadad.hindyugm.com/2010/10/know-hindi-meanings-while-chatting.html

We can start this kind of bot for other languages too, if we have dictionaries with Free S/w compatible licenses. If interested, please contact me.

The conference started on Friday morning. Sue Gardner, Executive Director of the Wikimedia Foundation. talked about the strategies of foundation, and it followed by a QnA with wikimedia board members. We presented the Malayalam CD to Sue Gardner. She remembered the International free software conference she attended at Trivandum on 2008 december.
Our workshop on offline wikipedia versions started at 2.30. Martin Walked introduced the workshop to participants. Manuel Schneider from German wikipedia explained the Openzim format for offline compressed storage of wikipedia and the available readers on desktop computers and mobile phones. Shiju Alex introduced the Malayalam wikipedia offline verision 1.0. I talked about the issues and solutions for providing an offline version, particularly a non-latin complex script wiki to users in CD ROM or DVD. I demonstrated sample offline wikis on Hebru, Tamil, Polish, English, Japanese with the wiki2cd tool. There were a couple of questions on wiki2cd and openzim. Kul Takanao Wadhwa and Tomasz Finc  from wikimedia foundation who are focusing on offline wiki projects attended the workshop and we had a discussion after the talk.

Offline wikipedia people: myself, Shiju Alex, Martin Walker, Manuel Schneider

The offline wiki workshop continued with Pediapress team. They are the people behind the recently added export book/PDF feature of wikipedia. Unfortunately this feature require lots of improvements to get work with Indian scripts.
We met Gerald M, who focus on language support of wikis. He is such a person with amazing passion towards our local language wikipedias. We discussed on the technical issues of the local language wikis. Siebrand joined the discussion and he pointed out some improvements in wiki2cd software.

Discussion with Siebrand on wiki2cd improvements. From left 2 right: Tinu Cherian, myself, Gerard M, Siebrand

On the second day I met Volker Haas, the developer of PDF export/books feature of wikipedia. The library used by the extension of creating PDFs is Reportlab. But it does not support complex scripts such as Indic or Arabic. We have a long discussion on possible solutions. Discussed the Reportlab code. the mwlib code, and the s/w which I am writing now  a days to provide complex script pdf rendering APIs. We will continue to try out some of the options we have to solve this issue soon.

Martin Walker, who presented the Article Selection process of English wikipedia along with us in the workshop  invited me and Shiju for a dinner with his family. And we went for that.

The third day started with plenary session by Jimmy Wales. He talked about small language wikipedia and the issues faced by them. He emphasized the need for offline versions of wikipedia to reach more number of people and talked about the Malayalam Wikipedia offline version.

Jimmy Wales with Malayalam wikipedia CD

Mayooranathan from Tamil wikipedia presented the issues and statistics of Tamil Wikipedia Community. On Monday and Tuesday, we spent time by roaming around the Old Town of Gdansk. Visited St. Marys Church , the biggest brick made church in the world. We climbed the 400 steps of the tower of the church. From the top of the chruch, one can see the entire city. We went to the Baltic sea beach -Westerplatte on a boat and visited Wisłoujście Fortress

Related posts:

* Creating Malayalam Wikipedia CD: http://shijualex.wordpress.com/2010/04/24/creating-malayalam-wikipedia-cd/
* Wiki2cd: http://thottingal.in/blog/2010/04/17/mlwikioncd/
* Wikipedia Sign post: http://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2010-04-19/News_and_notes#Briefly
* Gerard’s Blog: http://ultimategerardm.blogspot.com/2010/04/best-of-malayalam-wikipedia.html
* Gerard’s Blog: http://ultimategerardm.blogspot.com/2010/04/cd-dowloaded-4390-in-10-days.html
* Gerard’s Blog: http://ultimategerardm.blogspot.com/2010/07/malayalamwikipedia-success-story.html

I will be presenting wik2cd, the tool I wrote for Malayalam wikipedia version 1.0 there in a joint workshop with wikipedia offline developers.  I will be joining with Manuel Schneider,  Shiju Alex, Martin Walker in the workshop titled: Creating offline version of Wiki content – Solutions and Challenges. Apart from this, I will be meeting pediapress team, the team behind wikipedia’s latest PDF/Book export feature. There are some issues in this tool for working with Indic languages, mainly because of the PDF rendering engine not capable of rendering complex scripts.

Thanks to Wikimedia foundation for granting me a scholarship to cover travel expenses.

The idea was to get the selected articles in static form to the CD. But this is not easy as we imagine. It is not like saving each  page from browser to the local machine. Following were the challenges:

• Automate the process of getting the page and the images in it. Wikipedia articles changes frequently. So we need the program to fetch the latest article from wiki whenever it is executed.
• Fix all the links, css, javascript, image references so that all resolves within CD itself
• Provide an categorized index of the articles for easily locating the article.
• Provide a search in the article titles.
• ISO 9660 filesystem of CD/DVD has lots of limitations. There are restrictions on unicode names of the files, length of the file names, directory depth, special characters in filenames etc. Wikipedia has its article and image names with unicode, special characters and most of the time they exceeds the filename length. To avoid all these, we should rename most of the files and then fix the cross references in all files.
• It should work on all Operating systems. All the content should be presented with HTML, Javascript and CSS. Being the content in Malayalam, even if the user does not have required fonts in her/his machine, there should not be any problem for reading the content(font embedding required).

Manually solving all these challenges is not the way to go. So I wrote a program, which just takes the article titles and does all the above tasks and finally creates a repository ready for burning to CD ROM.

Wget disappointed me in fetching the content from wiki. There is an open bug in wget which make the download of non-latin URLs impossible.

Have a look at the CD content we created : Malayalam Wikipedia Selected 500 Articles . Hiran helped me with the artworks.

Since entire process is automated, the program can be used for any other language.  I am releasing the program for the benefit of everybody. You can get the program from here. It is written on Python. Jquery was used for the UI.  For details on the usage, customization etc read the wiki page of the project.

For those who can’t read Malayalam, here is a sample wiki created  by the wiki2cd program from English wikipedia by selecting 10 articles.

Malayalam Wikipedia Community  hope that this is a big step to reach the majority of the people who does not have internet access. If printed, this 500 articles will be at least 5000 pages. CDROM also includes information about commonly used free software based tools for Malayalam computing. Some writing tools and fonts are distributed in the same CD ROM.

Thanks to Malayalam Wikipedia for giving this great opportunity to wok on this project.

The ISO image of the CD is available here for download.