Indic

Recently we released two Jabber buddy bots for dictionary lookup. By adding [email protected] as a chat contact one can ask for the meaning of an English word in Malayalam by just sending a chat message. Similarly for English-Hindi or Hindi-English dictionary, we have another bot [email protected]. Both of these dictionaries use Dict databases based on DICT protocol. Both of these bots were well received by the users. We have 8000+ users for English-Malayalam Dictionary. Online blogs/media also gave good publicity. Thanks a lot!. ...

On 22nd August, I conducted a workout session with Praveen on Indic Language Computing at Red Hat Office, Pune. The plan was to solve some of the issues in Devanagari support for the encoding converter Payyans. But most of the time was spent on Introducing the concepts of Indic language computing to participants. Project Silpa was also introduced and demonstrated. Students from College of Engg, Pune and other colleges attended. Red Hat sponsored the venue at their office. It was very interesting to interact with energetic and enthusiastic students.

I will be attending Wikimania 2010, Gdansk, Poland. This annual international conference of the Wikimedia community is from July 9 to July 11. I will be presenting wik2cd, the tool I wrote for Malayalam wikipedia version 1.0 there in a joint workshop with wikipedia offline developers. I will be joining with Manuel Schneider, Shiju Alex, Martin Walker in the workshop titled: Creating offline version of Wiki content – Solutions and Challenges. Apart from this, I will be meeting pediapress team, the team behind wikipedia’s latest PDF/Book export feature. There are some issues in this tool for working with Indic languages, mainly because of the PDF rendering engine not capable of rendering complex scripts. ...

One year back I wrote about how to use Inkscape as a workaround solution for DTP in indic scripts. Still we don’t have any DTP software which supports Indic scripts in Unicode. Scribus still does not have the Indic support. One issue with inkscape when used as DTP for indic script was, a few indic scripts always wanted hyphenation when text is justified. For example Malayalam has lengthy words and often space is wasted in lines if the text is not automatically hyphenated. But this feature was not available in inkscape. There is a wishlist bug for adding this feature to Inkscape. I tried to develop an extension for Inkscape to achieve this. ...

Openoffice Indic Natural Language group announces the availability of the following Openoffice hyphenation dictionary extensions. Malayalam Hyphenation Rules version 1.2 Kannada Hyphenation Rules version 1.1 Bengali Hyphenation Rules verson 1.1 Hindi Hyphenation Rules version 1.1 Telugu Hyphenation Rules version 1.0 Tamil Hyphenation Rules version 1.0 Gujarati Hyphenation Rules version 1.0 Panjabi Hyphenation Rules version 1.0 Oriya Hyphenation Rules version 1.0 Marathi Hyphenation Rules version 1.0 Spellchecker extension for Malayalam is also ready. ...

[Please read the Silpa project annoucement before reading this blogpost] Project silpa is getting ready for a 0.1 version. The web framework got many changes to support JSON based RPC calls from external applications. That means, web/desktop applications can use the APIs of Silpa through RPC calls. Page rendering logic is moved from server to client. Web interface use javascript based synchronous JSON based RPC calls to get the results from server. Jquery is used for render the page. Uses PyMeld Templating Engine for modules having web interface(Not all modules will not have web interface) Framework is now Python WSGI application. Initially it was plain CGI. WSGI reduces the response time and allows the server to be executed as daemon Many new modules are getting added- Spellchecker : which is not based on aspell or hunspell and I am going to try out some algorithms to get optimal suggestions. Not completed. Soundex Algorithm- webbased demo and APIs as I explained in my previous blog post An Inexact search algorithm and its implementation based on visual and phonetic distance between two words is getting ready. I will explain it in another blogpost Hyphenation – Online tool as well as APIs N-gram for Indic languages– API, web interface API documentation is going on, but not completed. I have plans to make silpa as a python library for offline use too. Moved from SMC‘s git repo to a seperate git repo. After 0.1 baseline, I will create branches for stable and development. Application is running on a git controlled deployment workflow. Thanks to Joe Maller for nice documentation on this. That’s all for now!. There are too many things to be done. Some of the modules does not support all languages as of now. If anybody interested in contributing to the project, please contact me. Try out the application, read the code and let me know your comments.

Soundex is a phonetic indexing algorithm. It is used to search/retrieve words having similar pronunciation but slightly different spelling. Soundex was developed by Robert C. Russell and Margaret K. Odell. A variation called American Soundex was used in the 1930s for a retrospective analysis of the US censuses from 1890 through 1920. It is also described in Donald Knuth’s The Art of Computer Programming. The National Archives and Records Administration (NARA) maintains the current rule set for the official implementation of Soundex used by the U.S. Government. ...

Many of my friends already know about a project I am working on, this is a public announcement of that. The project is named as Silpa, may be an acronym of Swathanthra(Mukth, Free as in Freedom) Indian Language Processing Applications. It is a web framework and a set of applications for processing Indian Languages in many ways. Or in other words, it is a platform for porting existing and upcoming language processing applications to the web. ...

We just formed Indic Regional Language group for Openoffice. This is as per the Openoffice Native Language Consortium Plans. The objectives of such groups can be read from here. Basically the group is meant for better coordination among Indic languages to make Openoffice experience in our language better. The announcement of this group is here Thanks to Charles-H. Schulz, we got a mailing list [email protected]. To subscribe login to http://native-lang.openoffice.org We just started, and I will soon setup a wiki page there. To start with , I will collect the list of issues pending for Indian languages from people from various languages and will find out people from various languages as point of contacts. Feel free to contact me for anything related to Openoffice in your language. ...

Following image shows the python-graphviz generated visualization of N-Gram representation of first paragraph this article from Hindi wikipedia. The image represents the possible paths through which a sentence can be constructed if we start from a word भारत. Click to view the enlarged image