The conference started on Friday morning. Sue Gardner, Executive Director of the Wikimedia Foundation. talked about the strategies of foundation, and it followed by a QnA with wikimedia board members. We presented the Malayalam CD to Sue Gardner. She remembered the International free software conference she attended at Trivandum on 2008 december.
Our workshop on offline wikipedia versions started at 2.30. Martin Walked introduced the workshop to participants. Manuel Schneider from German wikipedia explained the Openzim format for offline compressed storage of wikipedia and the available readers on desktop computers and mobile phones. Shiju Alex introduced the Malayalam wikipedia offline verision 1.0. I talked about the issues and solutions for providing an offline version, particularly a non-latin complex script wiki to users in CD ROM or DVD. I demonstrated sample offline wikis on Hebru, Tamil, Polish, English, Japanese with the wiki2cd tool. There were a couple of questions on wiki2cd and openzim. Kul Takanao Wadhwa and Tomasz Finc  from wikimedia foundation who are focusing on offline wiki projects attended the workshop and we had a discussion after the talk.

Offline wikipedia people: myself, Shiju Alex, Martin Walker, Manuel Schneider

The offline wiki workshop continued with Pediapress team. They are the people behind the recently added export book/PDF feature of wikipedia. Unfortunately this feature require lots of improvements to get work with Indian scripts.
We met Gerald M, who focus on language support of wikis. He is such a person with amazing passion towards our local language wikipedias. We discussed on the technical issues of the local language wikis. Siebrand joined the discussion and he pointed out some improvements in wiki2cd software.

Discussion with Siebrand on wiki2cd improvements. From left 2 right: Tinu Cherian, myself, Gerard M, Siebrand

On the second day I met Volker Haas, the developer of PDF export/books feature of wikipedia. The library used by the extension of creating PDFs is Reportlab. But it does not support complex scripts such as Indic or Arabic. We have a long discussion on possible solutions. Discussed the Reportlab code. the mwlib code, and the s/w which I am writing now  a days to provide complex script pdf rendering APIs. We will continue to try out some of the options we have to solve this issue soon.

Martin Walker, who presented the Article Selection process of English wikipedia along with us in the workshop  invited me and Shiju for a dinner with his family. And we went for that.

The third day started with plenary session by Jimmy Wales. He talked about small language wikipedia and the issues faced by them. He emphasized the need for offline versions of wikipedia to reach more number of people and talked about the Malayalam Wikipedia offline version.

Jimmy Wales with Malayalam wikipedia CD

Mayooranathan from Tamil wikipedia presented the issues and statistics of Tamil Wikipedia Community. On Monday and Tuesday, we spent time by roaming around the Old Town of Gdansk. Visited St. Marys Church , the biggest brick made church in the world. We climbed the 400 steps of the tower of the church. From the top of the chruch, one can see the entire city. We went to the Baltic sea beach -Westerplatte on a boat and visited Wisłoujście Fortress

Related posts:

* Creating Malayalam Wikipedia CD: http://shijualex.wordpress.com/2010/04/24/creating-malayalam-wikipedia-cd/
* Wiki2cd: http://thottingal.in/blog/2010/04/17/mlwikioncd/
* Wikipedia Sign post: http://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2010-04-19/News_and_notes#Briefly
* Gerard’s Blog: http://ultimategerardm.blogspot.com/2010/04/best-of-malayalam-wikipedia.html
* Gerard’s Blog: http://ultimategerardm.blogspot.com/2010/04/cd-dowloaded-4390-in-10-days.html
* Gerard’s Blog: http://ultimategerardm.blogspot.com/2010/07/malayalamwikipedia-success-story.html

I will be presenting wik2cd, the tool I wrote for Malayalam wikipedia version 1.0 there in a joint workshop with wikipedia offline developers.  I will be joining with Manuel Schneider,  Shiju Alex, Martin Walker in the workshop titled: Creating offline version of Wiki content – Solutions and Challenges. Apart from this, I will be meeting pediapress team, the team behind wikipedia’s latest PDF/Book export feature. There are some issues in this tool for working with Indic languages, mainly because of the PDF rendering engine not capable of rendering complex scripts.

Thanks to Wikimedia foundation for granting me a scholarship to cover travel expenses.

The idea was to get the selected articles in static form to the CD. But this is not easy as we imagine. It is not like saving each  page from browser to the local machine. Following were the challenges:

• Automate the process of getting the page and the images in it. Wikipedia articles changes frequently. So we need the program to fetch the latest article from wiki whenever it is executed.
• Fix all the links, css, javascript, image references so that all resolves within CD itself
• Provide an categorized index of the articles for easily locating the article.
• Provide a search in the article titles.
• ISO 9660 filesystem of CD/DVD has lots of limitations. There are restrictions on unicode names of the files, length of the file names, directory depth, special characters in filenames etc. Wikipedia has its article and image names with unicode, special characters and most of the time they exceeds the filename length. To avoid all these, we should rename most of the files and then fix the cross references in all files.
• It should work on all Operating systems. All the content should be presented with HTML, Javascript and CSS. Being the content in Malayalam, even if the user does not have required fonts in her/his machine, there should not be any problem for reading the content(font embedding required).

Manually solving all these challenges is not the way to go. So I wrote a program, which just takes the article titles and does all the above tasks and finally creates a repository ready for burning to CD ROM.

Wget disappointed me in fetching the content from wiki. There is an open bug in wget which make the download of non-latin URLs impossible.

Have a look at the CD content we created : Malayalam Wikipedia Selected 500 Articles . Hiran helped me with the artworks.

Since entire process is automated, the program can be used for any other language.  I am releasing the program for the benefit of everybody. You can get the program from here. It is written on Python. Jquery was used for the UI.  For details on the usage, customization etc read the wiki page of the project.

For those who can’t read Malayalam, here is a sample wiki created  by the wiki2cd program from English wikipedia by selecting 10 articles.

Malayalam Wikipedia Community  hope that this is a big step to reach the majority of the people who does not have internet access. If printed, this 500 articles will be at least 5000 pages. CDROM also includes information about commonly used free software based tools for Malayalam computing. Some writing tools and fonts are distributed in the same CD ROM.

Thanks to Malayalam Wikipedia for giving this great opportunity to wok on this project.

The ISO image of the CD is available here for download.

I had a similar project idea during 2009 May and had done some amount of coding for that. The project was to have an IBUS input method which can do letter prediction as well as word prediction. The prediction is based on ngrams.  Since it is based on ibus, it works on all desktop applications.  You can see the screenshots of prototype from here, here and here

The core code was ready. It was written in python and use ibus-python. Unfortunately I did not get time to spend on this project for a long time and currently this project is not there in my top priorities.  Since I see many people interested in auto-completion or predictive text entry, I uploaded the code here http://github.com/santhoshtr/ibus-sulekha .  It is not in a working state as of now, but I would be happy if anybody interested in taking it forward.  I wrote a small documentation on algorithm here, and feel free to contact me if any help is required.

One issue with inkscape when used as DTP for indic script was, a few indic scripts always wanted hyphenation when text is justified. For example Malayalam has lengthy words and often space is wasted in lines if the text is not automatically hyphenated. But this feature was not available in inkscape. There is a wishlist bug for adding this feature to Inkscape.  I tried to develop an extension for Inkscape to achieve this.

It is on top of the python hyphenation code written by Wilbert  Berendsen. The hyphenation rules, also called as patterns is TeX or
Openoffice itself. So  I can support any language which has TeX hyphenation rules. But, since the hyphenation rules are language specific we need a language selection mechanism for the text first. Then only we can select the rules and do the hyphenation. But it is very tricky to implement.  Asking the language of the text every time it is justified is not a good idea. Setting a language for document is another choice, but what if the text contains multiple languages?  But for Indian languages it is very easy, we can automatically detect the scripts using unicode codepoints and load the rules accordingly. So for the time being, my extension support only English and all Indian languages.

Download the extension from http://thottingal.in/projects/inkscape_hyphenation/inkscape-hyphenation.zip . In GNU/Linux machines,  extract the zip file and copy to /usr/share/inkscape/extensions folder. In Windows , extract to [inkscape installation directory]\extensions folder.  After this close and reopen inkscape. You will see a menu named Hyphenate in Effects->Text menu.    In the document, add a text field, enter text in any indian language. Select the text and apply hyphenation by Effects->Text->Hyphenate. Then change the alignment of text to justify. You will see the text get hyphenated and occupying maximum possible space in the text field

I got satisfactory result with Malayalam and Tamil. I did not test other languages. Following images illustrates hyphenated, justified, two column layout of text done in Inkscape

We had a discussion about this in inkscape mailing list . Some developers suggested to have this feature built in, not as extension.  There are few issues to be solved for that. One thing is language selection as I explained. The other issue is regarding the hyphenation character to be used. Unicode standard insists to use soft hyphen – u00AD as hyphenation character. This is an invisible character. For Malayalam, visible hyphens are not required. But some other languages require the hyphen sign where the word is broken at the end of the line. The rules for whether the soft hyphen should be visible or not visible is not clear in Unicode’s specification. Pango never displays a the soft hyphen. There are criticism on this specification of softhyphen

• Jukka Korpela, Soft hyphen (SHY) – a hard problem?  http://www.cs.tut.fi/~jkorpela/shy.html
• Markus Kuhn, Unicode interpretation of SOFT HYPHEN breaks ISO 8859-1   compatibility. Unicode Technical Committee document L2/03-155R, June 2003. http://www.cl.cam.ac.uk/~mgk25/ucs/L2/03155r-kuhn-soft-hyphen.pdf

So I think there is something to be done from Rendering engine or Unicode need to clarify the confusions.  But Openoffice and HTML rendering engines always make soft hyphen at the end of the line, which is not desired for some languages.

Try this extension, let me know the comments. For small scale DTP works, such as pamphlets, notices, brochures  inkscape is enough. But since inkscape is not primarily a DTP software and does not have paging support, for books and large scale DTP works, it may not work well.

]]> http://thottingal.in/blog/2009/10/03/inkscape-hyphenation-extension/feed/ 2 http://thottingal.in/blog/2009/08/15/ooo_hyphenation_extensions/ http://thottingal.in/blog/2009/08/15/ooo_hyphenation_extensions/#comments Sat, 15 Aug 2009 09:35:45 +0000 Santhosh http://thottingal.in/blog/?p=221 Openoffice Indic Natural Language group announces the availability of the following Openoffice hyphenation dictionary extensions.

1. Malayalam Hyphenation Rules version 1.2
2. Kannada Hyphenation Rules version 1.1
3. Bengali Hyphenation Rules verson 1.1
4. Hindi Hyphenation Rules version 1.1
5. Telugu Hyphenation Rules version 1.0
6. Tamil Hyphenation Rules version 1.0
7. Gujarati Hyphenation Rules version 1.0
8. Panjabi Hyphenation Rules version 1.0
9. Oriya Hyphenation Rules version 1.0
10. Marathi Hyphenation Rules version 1.0

Spellchecker extension for Malayalam is also ready.

For a complete list of writing aids for Openoffice in Indic Languages is available here

Hyphenation Rules for Languages other than Marathi is already packages in Fedora 11. This releases contains updates and bug fixes. Fedora 12 will contains these updates. These extensions are yet to be packaged for Debian/Ubuntu.

More details about hyphenation rules are available here

Project silpa is getting ready for a 0.1 version.

1. The web framework got many changes to support JSON based RPC calls from external applications. That means,  web/desktop applications can use the APIs of Silpa through RPC calls.
2. Page rendering logic is moved from server to client. Web interface use javascript based synchronous JSON based RPC calls to get the results from server. Jquery is used for render the page.
3. Uses PyMeld Templating Engine for modules having web interface(Not all modules will not have web interface)
4. Framework is now Python WSGI application. Initially it was plain CGI. WSGI reduces the response time and allows the server to be executed as daemon
5. Many new modules are getting added- Spellchecker : which is not based on aspell or hunspell  and I am going to try out some algorithms to get optimal suggestions. Not completed.
6. Soundex Algorithm- webbased demo and APIs as I explained in my  previous blog post
7. An Inexact search algorithm and its implementation based on visual and phonetic distance between two words is getting ready. I will explain it in another blogpost
8. Hyphenation – Online tool as well as APIs
9. N-gram for Indic languages- API, web interface
10. API documentation is going on, but not completed. I have plans to make silpa as a python library for offline use too.
11. Moved from SMC‘s git repo to a seperate git repo. After 0.1 baseline, I will create branches for stable and development.
12. Application is running on a git controlled deployment workflow. Thanks to Joe Maller  for nice documentation on this.

That’s all for now!.  There are too many things to be done. Some of the modules does not support all languages as of now.  If anybody interested in contributing to the project, please contact me.  Try out the application, read the code and let me know your comments.

The project is named as Silpa, may be an acronym of Swathanthra(Mukth, Free as in Freedom) Indian Language Processing Applications. It is a web framework and a set of applications for processing Indian Languages in many ways. Or in other words, it is a platform for porting existing and upcoming language processing applications to the web.

Before going to the details, you can have a quick preview of the application here : http://smc.org.in/silpa

The project is designed for adding applications/utilities as plugins. The framework is written from scratch using python language. As you can see in the development version, there are number of modules already written.  Most of the modules requires some more work to make it _complete_. The application is free software and there is a link to the source code at the bottom of the application.

As it is meant for covering all languages of India, all modules should be capable of handling all scripts from India(Sometimes English too). At the same time , the language of input data is transparent , meaning, user need not mention that _this_ is the language in which she is entering the data. Unlike desktop applications which asks to specify the language along with the input data(for eg: Spell checker) , the modules should try to detect the language them self. And if possible, modules try to process the data even if the input data is in multiple Indic scripts.

The modules may be General purpose(eg: Dictionary, Spellcheck,Sort. Transliteration, Font conversion..) or Technology/Algorithm  Demonstration purpose (eg: Hyphenation, Stemmer, Search algorithms)

Some of the modules are usable  as of now, while some of them are in development. You may just try out them. User’s data will not be logged  except when a crash occurs(at that time user data and exception trace will be logged for later debugging).

And, this is also a call for contributors. You may propose new ideas for modules, feature suggestion etc.. A few  students showed interest in the project. Unfortunately python is not a language in their  college syllabus. So if you are good in python and have interest in contributing to the project, drop me a mail . There is no separate version for development and the one which is present at http://smc.org.in/silpa . All development happens there itself and any change in the code is immediately available for use!(or immediately starts crashing for user data)

I will write on some interesting algorithms I used for some modules later. If you are curious to know them, read the code!

ചങ്ങാതിമാരേ,
കേരളത്തിലെ രസകരമായ പ്രാദേശിക ഭാഷാ ഭേദങ്ങളെക്കുറിച്ചു് നിങ്ങള്‍ക്കെല്ലാമറിയാമല്ലോ? തിരുവനന്തപുരം, കോട്ടയം, തൃശ്ശൂര്‍, ഷൊര്‍ണ്ണൂര്‍, പാലക്കാട്, കോഴിക്കോട് കണ്ണൂര്‍, വയനാട് തുടങ്ങി നമുക്കു് വ്യത്യസ്തങ്ങളായ മലയാളത്തിന്റെ രൂപഭേദങ്ങളുണ്ടു്. അച്ചടി മലയാളത്തില്‍ നിന്നും വളരെയേറെ വ്യത്യസ്തമാണു് അവ. അച്ചടി മലയാളം കൊടുത്തു് സ്ഥലത്തിന്റെ പേരു കൊടുത്താല്‍ ആ പ്രദേശത്തെ മലയാളത്തിന്റെ രീതിയിലേക്കു അതിനെ മാറ്റിത്തരുന്ന ഒരു സോഫ്റ്റ്‌വെയര്‍ രസകരമാവില്ലേ?

അത്തരത്തിലൊരു ശ്രമമാണു് “ക്ടാവ്” Slang converter എന്നു പേരിട്ടിരിക്കുന്ന പ്രൊജക്ട്. ഇതിന്റെ കൂടെ കൊടുത്തിരിക്കുന്ന സ്ക്രീന്‍ഷോട്ട് നോക്കൂ. ഡെവലപ്മെന്റ് പതിപ്പിന്റെ ചിത്രമാണതു്. കുറച്ചു നിയമങ്ങളുടെ അടിസ്ഥാനത്തില്‍ Natural Language Processing ന്റെ പുതിയ ശാഖയായ AMP(Ambiguous Language Processing) എന്ന വിദ്യ ഉപയോഗിച്ചാണു് ഇതു ചെയ്തിരിക്കുന്നതു്. Qt/C++ ആണു് കോഡ്. UI ചെയ്യാന്‍ Qt Creator ഉപയോഗിച്ചു.

ഒരു മലയാളം ഫയലില്‍ പല സ്ലാങ്ങില്‍ തിരയാനുള്ള സംവിധാനവും തയ്യാറാക്കാന്‍ പദ്ധതിയുണ്ടു് . അതായതു് ഗഡി എന്നു തിരഞ്ഞാല്‍ സുഹൃത്തു് , ചങ്ങാതി എന്നൊക്കെ കിട്ടണം. പിന്നെ ഗഡി എന്നു സ്പെല്ലിങ്ങ് തെറ്റിച്ചെഴുതിയാല്‍ സുഹൃത്ത്, ചങ്ങാതി എന്നൊക്കെ സ്പെല്‍ചെക്കറില്‍ സൂചന വരാനുള്ള ഫീച്ചറും നമുക്കു് ചെയ്യണം. GPL V3 ലൈസന്‍സിലുള്ള ഈ അപ്ലിക്കേഷനു് ഇതിന്റെ നിയമങ്ങള്‍ വിപുലപ്പെടുത്താനും ടെസ്റ്റ് ചെയ്യാനും വിവിധ ജില്ലകളില്‍ താമസിക്കുന്നവരില്‍ നിന്നുള്ള സഹായം ആവശ്യമുണ്ടു്.

സഹകരിക്കുമല്ലോ.

അഭിപ്രായങ്ങളറിയിക്കുക.

• Hacking the GLMatrix screensaver
• Screensavers in your language

Download the binary packages: Deb package, and RPM package

There are 6 screensavers in that package, for Malayalam, Hindi, Oriya , Bengali, Tamil and Gujarati. After installation, goto KDE system settings->Desktop->Screensaver and select any of this.

Screenshots(click to get the image in original size):




KDE Screensaver configuration for Hindi:




Enjoy…!