Blogs

Openoffice Indic Natural Language group announces the availability of the following Openoffice hyphenation dictionary extensions. Malayalam Hyphenation Rules version 1.2 Kannada Hyphenation Rules version 1.1 Bengali Hyphenation Rules verson 1.1 Hindi Hyphenation Rules version 1.1 Telugu Hyphenation Rules version 1.0 Tamil Hyphenation Rules version 1.0 Gujarati Hyphenation Rules version 1.0 Panjabi Hyphenation Rules version 1.0 Oriya Hyphenation Rules version 1.0 Marathi Hyphenation Rules version 1.0 Spellchecker extension for Malayalam is also ready. ...

[Please read the Silpa project annoucement before reading this blogpost] Project silpa is getting ready for a 0.1 version. The web framework got many changes to support JSON based RPC calls from external applications. That means, web/desktop applications can use the APIs of Silpa through RPC calls. Page rendering logic is moved from server to client. Web interface use javascript based synchronous JSON based RPC calls to get the results from server. Jquery is used for render the page. Uses PyMeld Templating Engine for modules having web interface(Not all modules will not have web interface) Framework is now Python WSGI application. Initially it was plain CGI. WSGI reduces the response time and allows the server to be executed as daemon Many new modules are getting added- Spellchecker : which is not based on aspell or hunspell and I am going to try out some algorithms to get optimal suggestions. Not completed. Soundex Algorithm- webbased demo and APIs as I explained in my previous blog post An Inexact search algorithm and its implementation based on visual and phonetic distance between two words is getting ready. I will explain it in another blogpost Hyphenation – Online tool as well as APIs N-gram for Indic languages– API, web interface API documentation is going on, but not completed. I have plans to make silpa as a python library for offline use too. Moved from SMC‘s git repo to a seperate git repo. After 0.1 baseline, I will create branches for stable and development. Application is running on a git controlled deployment workflow. Thanks to Joe Maller for nice documentation on this. That’s all for now!. There are too many things to be done. Some of the modules does not support all languages as of now. If anybody interested in contributing to the project, please contact me. Try out the application, read the code and let me know your comments.

Soundex is a phonetic indexing algorithm. It is used to search/retrieve words having similar pronunciation but slightly different spelling. Soundex was developed by Robert C. Russell and Margaret K. Odell. A variation called American Soundex was used in the 1930s for a retrospective analysis of the US censuses from 1890 through 1920. It is also described in Donald Knuth’s The Art of Computer Programming. The National Archives and Records Administration (NARA) maintains the current rule set for the official implementation of Soundex used by the U.S. Government. ...

I was reading an article named “Why Can’t a Computer Translate More Like a Person?” by Alan K. Melby. The article is about the challenges that machine translation technology face to reach a acceptable quality of translation. He explains the importance of culture sensitivity required for machine translation programs. Article lists a number of examples where MT can go wrong if context , culture etc are not taken into consideration. There are very interesting arguments about how reductionalism becomes a wrong choice while designing MT. If you are interested in natural language processing or machine translation and wondering if there is any limit for computer programs to reach human’s language capabilities, please read it. ...

Recently I had to extract text from PDF files for indexing the content using Apache Lucene. Apache PDFBox was the obvious choice for the java library to be used. Apache PDFBox is an opensource java library for working with PDF files. The PDFBox library allows creation of new PDF documents, manipulation of existing documents and the ability to extract content from documents. PDFBox also includes several command line utilities. ...

Many of my friends already know about a project I am working on, this is a public announcement of that. The project is named as Silpa, may be an acronym of Swathanthra(Mukth, Free as in Freedom) Indian Language Processing Applications. It is a web framework and a set of applications for processing Indian Languages in many ways. Or in other words, it is a platform for porting existing and upcoming language processing applications to the web. ...

We just formed Indic Regional Language group for Openoffice. This is as per the Openoffice Native Language Consortium Plans. The objectives of such groups can be read from here. Basically the group is meant for better coordination among Indic languages to make Openoffice experience in our language better. The announcement of this group is here Thanks to Charles-H. Schulz, we got a mailing list [email protected]. To subscribe login to http://native-lang.openoffice.org We just started, and I will soon setup a wiki page there. To start with , I will collect the list of issues pending for Indian languages from people from various languages and will find out people from various languages as point of contacts. Feel free to contact me for anything related to Openoffice in your language. ...

ചങ്ങാതിമാരേ, കേരളത്തിലെ രസകരമായ പ്രാദേശിക ഭാഷാ ഭേദങ്ങളെക്കുറിച്ചു് നിങ്ങള്‍ക്കെല്ലാമറിയാമല്ലോ? തിരുവനന്തപുരം, കോട്ടയം, തൃശ്ശൂര്‍, ഷൊര്‍ണ്ണൂര്‍, പാലക്കാട്, കോഴിക്കോട് കണ്ണൂര്‍, വയനാട് തുടങ്ങി നമുക്കു് വ്യത്യസ്തങ്ങളായ മലയാളത്തിന്റെ രൂപഭേദങ്ങളുണ്ടു്. അച്ചടി മലയാളത്തില്‍ നിന്നും വളരെയേറെ വ്യത്യസ്തമാണു് അവ. അച്ചടി മലയാളം കൊടുത്തു് സ്ഥലത്തിന്റെ പേരു കൊടുത്താല്‍ ആ പ്രദേശത്തെ മലയാളത്തിന്റെ രീതിയിലേക്കു അതിനെ മാറ്റിത്തരുന്ന ഒരു സോഫ്റ്റ്‌വെയര്‍ രസകരമാവില്ലേ? അത്തരത്തിലൊരു ശ്രമമാണു് “ക്ടാവ്” Slang converter എന്നു പേരിട്ടിരിക്കുന്ന പ്രൊജക്ട്. ഇതിന്റെ കൂടെ കൊടുത്തിരിക്കുന്ന സ്ക്രീന്‍ഷോട്ട് നോക്കൂ. ഡെവലപ്മെന്റ് പതിപ്പിന്റെ ചിത്രമാണതു്. കുറച്ചു നിയമങ്ങളുടെ അടിസ്ഥാനത്തില്‍ Natural Language Processing ന്റെ പുതിയ ശാഖയായ AMP(Ambiguous Language Processing) എന്ന വിദ്യ ഉപയോഗിച്ചാണു് ഇതു ചെയ്തിരിക്കുന്നതു്. Qt/C++ ആണു് കോഡ്. UI ചെയ്യാന്‍ Qt Creator ഉപയോഗിച്ചു. ...

This code #!/usr/bin/env python # -*- coding: utf-8 -*- ml_string=u"സന്തോഷ് हिन्दी" for ch in ml_string: if(ch.isalpha()): print ch gives this output സ ന ത ഷ ह न द And fails for all mathra signs of Indian languages. This is a known bug in glibc. Does anybody know whether python internally use glibc functions for this basic string operations or use separate character database llke QT does?