See you at foss.in!

One issue with inkscape when used as DTP for indic script was, a few indic scripts always wanted hyphenation when text is justified. For example Malayalam has lengthy words and often space is wasted in lines if the text is not automatically hyphenated. But this feature was not available in inkscape. There is a wishlist bug for adding this feature to Inkscape.  I tried to develop an extension for Inkscape to achieve this.

It is on top of the python hyphenation code written by Wilbert  Berendsen. The hyphenation rules, also called as patterns is TeX or
Openoffice itself. So  I can support any language which has TeX hyphenation rules. But, since the hyphenation rules are language specific we need a language selection mechanism for the text first. Then only we can select the rules and do the hyphenation. But it is very tricky to implement.  Asking the language of the text every time it is justified is not a good idea. Setting a language for document is another choice, but what if the text contains multiple languages?  But for Indian languages it is very easy, we can automatically detect the scripts using unicode codepoints and load the rules accordingly. So for the time being, my extension support only English and all Indian languages.

Download the extension from http://thottingal.in/projects/inkscape_hyphenation/inkscape-hyphenation.zip . In GNU/Linux machines,  extract the zip file and copy to /usr/share/inkscape/extensions folder. In Windows , extract to [inkscape installation directory]\extensions folder.  After this close and reopen inkscape. You will see a menu named Hyphenate in Effects->Text menu.    In the document, add a text field, enter text in any indian language. Select the text and apply hyphenation by Effects->Text->Hyphenate. Then change the alignment of text to justify. You will see the text get hyphenated and occupying maximum possible space in the text field

I got satisfactory result with Malayalam and Tamil. I did not test other languages. Following images illustrates hyphenated, justified, two column layout of text done in Inkscape

We had a discussion about this in inkscape mailing list . Some developers suggested to have this feature built in, not as extension.  There are few issues to be solved for that. One thing is language selection as I explained. The other issue is regarding the hyphenation character to be used. Unicode standard insists to use soft hyphen – u00AD as hyphenation character. This is an invisible character. For Malayalam, visible hyphens are not required. But some other languages require the hyphen sign where the word is broken at the end of the line. The rules for whether the soft hyphen should be visible or not visible is not clear in Unicode’s specification. Pango never displays a the soft hyphen. There are criticism on this specification of softhyphen

• Jukka Korpela, Soft hyphen (SHY) – a hard problem?  http://www.cs.tut.fi/~jkorpela/shy.html
• Markus Kuhn, Unicode interpretation of SOFT HYPHEN breaks ISO 8859-1   compatibility. Unicode Technical Committee document L2/03-155R, June 2003. http://www.cl.cam.ac.uk/~mgk25/ucs/L2/03155r-kuhn-soft-hyphen.pdf

So I think there is something to be done from Rendering engine or Unicode need to clarify the confusions.  But Openoffice and HTML rendering engines always make soft hyphen at the end of the line, which is not desired for some languages.

Try this extension, let me know the comments. For small scale DTP works, such as pamphlets, notices, brochures  inkscape is enough. But since inkscape is not primarily a DTP software and does not have paging support, for books and large scale DTP works, it may not work well.

]]> http://thottingal.in/blog/2009/10/03/inkscape-hyphenation-extension/feed/ 2 http://thottingal.in/blog/2009/08/15/ooo_hyphenation_extensions/ http://thottingal.in/blog/2009/08/15/ooo_hyphenation_extensions/#comments Sat, 15 Aug 2009 09:35:45 +0000 Santhosh http://thottingal.in/blog/?p=221 Openoffice Indic Natural Language group announces the availability of the following Openoffice hyphenation dictionary extensions.

1. Malayalam Hyphenation Rules version 1.2
2. Kannada Hyphenation Rules version 1.1
3. Bengali Hyphenation Rules verson 1.1
4. Hindi Hyphenation Rules version 1.1
5. Telugu Hyphenation Rules version 1.0
6. Tamil Hyphenation Rules version 1.0
7. Gujarati Hyphenation Rules version 1.0
8. Panjabi Hyphenation Rules version 1.0
9. Oriya Hyphenation Rules version 1.0
10. Marathi Hyphenation Rules version 1.0

Spellchecker extension for Malayalam is also ready.

For a complete list of writing aids for Openoffice in Indic Languages is available here

Hyphenation Rules for Languages other than Marathi is already packages in Fedora 11. This releases contains updates and bug fixes. Fedora 12 will contains these updates. These extensions are yet to be packaged for Debian/Ubuntu.

More details about hyphenation rules are available here

Project silpa is getting ready for a 0.1 version.

1. The web framework got many changes to support JSON based RPC calls from external applications. That means,  web/desktop applications can use the APIs of Silpa through RPC calls.
2. Page rendering logic is moved from server to client. Web interface use javascript based synchronous JSON based RPC calls to get the results from server. Jquery is used for render the page.
3. Uses PyMeld Templating Engine for modules having web interface(Not all modules will not have web interface)
4. Framework is now Python WSGI application. Initially it was plain CGI. WSGI reduces the response time and allows the server to be executed as daemon
5. Many new modules are getting added- Spellchecker : which is not based on aspell or hunspell  and I am going to try out some algorithms to get optimal suggestions. Not completed.
6. Soundex Algorithm- webbased demo and APIs as I explained in my  previous blog post
7. An Inexact search algorithm and its implementation based on visual and phonetic distance between two words is getting ready. I will explain it in another blogpost
8. Hyphenation – Online tool as well as APIs
9. N-gram for Indic languages- API, web interface
10. API documentation is going on, but not completed. I have plans to make silpa as a python library for offline use too.
11. Moved from SMC’s git repo to a seperate git repo. After 0.1 baseline, I will create branches for stable and development.
12. Application is running on a git controlled deployment workflow. Thanks to Joe Maller  for nice documentation on this.

That’s all for now!.  There are too many things to be done. Some of the modules does not support all languages as of now.  If anybody interested in contributing to the project, please contact me.  Try out the application, read the code and let me know your comments.

The soundex code for a word is an english alphabet followed by a number of digits. The algorithm is explained  with examples here

By this algorithm, if my name is written as Santhosh , Santosh , Santhos or Santos , the soundex code remains same and it is S532

Soundex had many limitations and sometimes creates false positive or false negative errors. There are variants for soundex and one important variant is Metaphone alogirthm by Lawrence Philip . Metaphone have another improved version called Double Metaphone.

Well,  it works fine, but only for English. Just like our languages also have varying spelling. But more than spelling, in India , we have another issue to be addressed: Words(often nouns) getting transliterated among Indian Languages. Let me give some examples: In railway reservation chart, your name will be written in English as well as in Hindi. You are from Kerala(or some other state) and your name is transliterated to some other language. The only thing remain same is its pronunciation.  It will be great if we can search on this data based on pronunciation, right?

We see a lot of discussions happening around e-governance and other computerization projects such as national UID etc now a days. Projects that handle Indic text heavily will require efficient search and string processing algorithms.

You got a list of names in Bengali and you dont know Bengali but you know Malayalam. Obviously you can’t search on Bengali text using Malayalam. But can’t we develop such algorithm?

So our requirement is this:

1. Language Independent Search on Indic text.
2. Comparison should be based on pronunciation
3. Should be tolerant to spelling variation

The above discussed soundex algorithm just fits to the solution. Only issue is that algorithm is defined only for English. So time to design soundex for our languages!.

Original Soundex algorithm is not multilingual. And not designed for language independent indexing.  And the digits to represent phonetic categories for Indian languages will not fit into 6. We have more phonetical families. So our algorithm will not be exact “localization” of English soundex, but we will use the concept. Let us call it as IndicSoundex . Infact Metaphone algorithm uses 16 families for English.

One of the characteristcs of Indian languages is that all languages share same phonetic features. We have vowels a, aa, i, ii, u…. then consonant families ka, cha, ta, tha, pa..  etc.  What we need to do is mapping all these sets to a common representation which is independent of language. And we call that representation as soundex for an Indic word.

While grouping and mapping Indic letters to phonetic codes, the following are facts are taken into consideration.

1. Group short and long vowel to a single code. e and ee is considered as equal
2. Consider half consonants as full consonants. For this ignore halants.
3. Group consonant families. ka, kha,ga,gha, nga becomes a single family. Same is the case of cha, ta, tha,pa
4. Group ra, Ra,
5. Group la,La, zha
6. Group sa,Sa,sha

When I grouped like this I got 20 groups.  I have prepared a table for all Indic letters and corresponding code here http://thottingal.in/soundex/soundex.html

Algorithm:

1. For each letter in the word except first letter, get the corresponding soundex digit from the character map, which is nothing but a table like this.
2. If the letter is not found in character map, the  soundex digit for that letter is 0
3. Duplicate consecutive soundex codes are skipped. ie, effectively क्क will be considered as क.
4. Replace first digit with first alpha character.
5. remove all 0s from the soundex code.
6. Return soundex code padded to the required length (ie, if required length of code is 5 and soundex is സBCD, then soundex returned will be സBCD0.

Of cource, we need an implementation. Get the python code for this from here. If you don’t care about the code,  try the online soundex converter from here: http://smc.org.in/silpa/Soundex

From the above algorithm , the soundex code for സന്തോഷ് is സLKES000. and for सन्तौष , it is सLKES000 . So if I need to compare सन्तौष and സന്തോഷ് and need a positive result, we should do something in comparison logic and there by making the comparison language independent

An example :
കാര്‍തിക്, കാര്‍ത്തിക്,  കാര്‍തിഗ് = കAPKBF00
கார்திக்= கAPKBF00

Comparison

1. Compare the two string without calculating the soundex codes. If there are same return 0
2. Calculate the sounedx codes for both strings, if the match return 1. ie, both strings are from same language and sounds alike
3. If the soundex of strings has different fist letter and rest of the part is same, check whether the first letters are with same soundex digit. If so, both words from different languages, but sounds alike. Return code for this will be 2
4. If none of the above conditions match, return -1, indicating both strings are completely different.

Use http://smc.org.in/silpa/Soundex to experiment with this algorithm. Following screenshot shows how കാര്‍ത്തിക് and કાર્તિક્ are found to be phonetically similar or “sounds alike”

Ok,  we got the soundex code. The soundex code itself is not useful. We need to implement a search utility based on this. As I explained above, an intra-indic search program. We will see it later.

Feel free to comment on algorithm and please suggest any improvement we can do.

The article is written long time back, and Machine Translation technologies improved a lot. There are commercial as well as free translation products for many languages. There are research going on in intra-indic as well as english-indic translations.  I am not sure how far these technologies solved the challenges mentioned in the above mentioned article, but I believe that the questions are still valid.

The question is whether the programs can understand our culture, language usage , emotions etc. For translating limited domain or dry content, the machine translation may be effective, but in a general purpose use, I don’t know how effective they are.

Melby argues :

That key factor which is missing from current theories is agency. By agency, I mean the capacity to make real choices by exercising our will, ethical choices for which we are responsible. [...]. Any ‘choice’ that is a rigid and unavoidable consequence of the circumstances is not a real choice that could have gone either way and is thus not an example of agency. A computer has no real choice in what it will do next. Its next action is an unavoidable consequence of the machine language it is executing and the values of data presented to it. I am proposing that any approach to meaning that discounts agency will amount to no more than the mechanical manipulation of symbols such as words, that is, moving words around and linking them together in various ways instead of understanding them. Computers can already manipulate symbols. In fact, that is what they mostly do. But manipulating symbols does not give them agency and it will not let them handle language like humans. Symbol manipulation works only within a specific domain, and any attempt to move beyond a domain through symbol manipulation is doomed, for manipulation of symbols involves no true surprises, only the strict application of rules. General vocabulary, as we have seen, involves true surprises that could not have been predicted.

With all these advanced technologies, can we develop a universal , any-to-any language translation program? We have seen many examples where human beings are failing miserably in sensible translation. If you are looking for  english->hindi translation effectiveness, try this using google Translation

आप हिन्दी समझते है ? ==> You understand English?

So do you think that if there is any such universal translation tool,  it is nearly impossible and “only god can create such a tool” ?! . Heard about Babel fish (of The Hitchhiker’s Guide to the Galaxy)? .  The babel fish is small, yellow, leech-like, and is a universal translator which simultaneously translates from one spoken language to another. When inserted into the ear, its nutrition processes convert sound waves into brain waves, neatly crossing the language divide between any species you should happen to meet whilst travelling in space. According to the Hitchhiker’s Guide, the Babel fish was put forth as an example for the non-existence of God: .

“I refuse to prove that I exist,” says God, “for proof denies faith, and without faith I am nothing.”

“But,” says Man, “the Babel fish is a dead giveaway isn’t it? It could not have evolved by chance. It proves that you exist, and so therefore, by your own arguments, you don’t. Q.E.D.

“Oh dear,” says God, “I hadn’t thought of that,” and promptly vanishes in a puff of logic

Alan K Melby argues that Douglas Adams was saying that there can’t be any such fish.

The silliness of the above argument is intended, I believe, to show the futility of trying to prove the existence of God, through physics or any other route. Belief in God is a starting point, not a conclusion. If it were a conclusion, then that conclusion would have to be based on something else that is firmer than our belief in God. If that something else forces everyone to believe in God, then faith is denied. If that something else does not force us to believe in God, then it may not be a sufficiently solid foundation for our belief.

Adams may also be saying something about translation and the nature of language. I can speculate on what Adams had in mind to say about translation when he dreamed up the Babel fish. My own bias would have him saying indirectly that there could be no such fish since there is no universal set of thought patterns underlying all languages. Even with direct brain to brain communication, we would still need shared concepts in order to communicate. Words do not really fail us. If two people share a concept, they can eventually agree on a word to express it. Ineffable experiences are those that are not shared by others.

I have some friends studying on machine translation with Indian Languages. They are evaluating shallow transfer method(Statistical methods to the words surrounding the ambiguous word.) for this using tools like apertium. Let us hope that they will succeed in their efforts.

Let me give one example translation between Tamil and Malayalam where context matters.

In Malayalam, for ‘wait, wait’, we usually say, “നില്ക്കു് നില്ക്ക്”(Literal meaning:  ’stand, stand’ ) . For the same purpose , I have noticed that my Tamil speaking friends  use “இரு இரு” (Literal meaning: ’sit, sit’  ). Now if the translation is done without knowing this usage, it is going to be funny. Shallow transfer methods use multiple intermediate  languages for translation. For eg: If there is a translation  tool available for a->b and b->c and then a->c is possible through a->b->c . I feel that this is going to be a big challenge.. to keep the word meaning, context, common usage…etc.. Let us wait/sit/stand and see

Since we saw “a nonexistence of God proof”, let me give another one, that I read sometime back.

1. God is so powerful, he can do any thing,
2. God can create anything , if #1 is true
3. If #2 is is true, he can create a big stone that he cannot lift!
4. If he cannot lift a stone, then #1 is wrong, hence #2 also wrong. So God does not exist!

Looks very silly, right? or “Logical” ?

Apache PDFBox is an opensource java library for working with PDF files. The PDFBox library allows creation of new PDF documents, manipulation of existing documents and the ability to extract content from documents. PDFBox also includes several command line utilities.

There is no latest build available for PDFBox. Sourceforge has very old binaries. But  the old version fails to work with PDF 1.5 specification. So one need to compile the latest code from SVN. 

I am sharing the latest jar file built from svn here

The following example explains how to extract the text from a pdf file using PDFBox.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import org.apache.pdfbox.cos.COSDocument;
import org.apache.pdfbox.pdfparser.PDFParser;
import org.apache.pdfbox.pdmodel.PDDocument;
import org.apache.pdfbox.util.PDFTextStripper;
 
public class PDFTextParser {
 
	// Extract text from PDF Document
	static String pdftoText(String fileName) {
		PDFParser parser;
		String parsedText = null;;
		PDFTextStripper pdfStripper = null;
		PDDocument pdDoc = null;
		COSDocument cosDoc = null;
		File file = new File(fileName);
		if (!file.isFile()) {
			System.err.println("File " + fileName + " does not exist.");
			return null;
		}
		try {
			parser = new PDFParser(new FileInputStream(file));
		} catch (IOException e) {
			System.err.println("Unable to open PDF Parser. " + e.getMessage());
			return null;
		}
		try {
			parser.parse();
			cosDoc = parser.getDocument();
			pdfStripper = new PDFTextStripper();
			pdDoc = new PDDocument(cosDoc);
			pdfStripper.setStartPage(1);
			pdfStripper.setEndPage(5);
			parsedText = pdfStripper.getText(pdDoc);
		} catch (Exception e) {
			System.err
					.println("An exception occured in parsing the PDF Document."
							+ e.getMessage());
		} finally {
			try {
				if (cosDoc != null)
					cosDoc.close();
				if (pdDoc != null)
					pdDoc.close();
			} catch (Exception e) {
				e.printStackTrace();
			}
		}
		return parsedText;
	}
	public static void main(String args[]){
		System.out.println(pdftoText("/home/santhosh/pdfbox/test.pdf"));
	}
 
}
 
 

More details on the APIs can be read from here

The project is named as Silpa, may be an acronym of Swathanthra(Mukth, Free as in Freedom) Indian Language Processing Applications. It is a web framework and a set of applications for processing Indian Languages in many ways. Or in other words, it is a platform for porting existing and upcoming language processing applications to the web.

Before going to the details, you can have a quick preview of the application here : http://smc.org.in/silpa

The project is designed for adding applications/utilities as plugins. The framework is written from scratch using python language. As you can see in the development version, there are number of modules already written.  Most of the modules requires some more work to make it _complete_. The application is free software and there is a link to the source code at the bottom of the application.

As it is meant for covering all languages of India, all modules should be capable of handling all scripts from India(Sometimes English too). At the same time , the language of input data is transparent , meaning, user need not mention that _this_ is the language in which she is entering the data. Unlike desktop applications which asks to specify the language along with the input data(for eg: Spell checker) , the modules should try to detect the language them self. And if possible, modules try to process the data even if the input data is in multiple Indic scripts.

The modules may be General purpose(eg: Dictionary, Spellcheck,Sort. Transliteration, Font conversion..) or Technology/Algorithm  Demonstration purpose (eg: Hyphenation, Stemmer, Search algorithms)

Some of the modules are usable  as of now, while some of them are in development. You may just try out them. User’s data will not be logged  except when a crash occurs(at that time user data and exception trace will be logged for later debugging).

And, this is also a call for contributors. You may propose new ideas for modules, feature suggestion etc.. A few  students showed interest in the project. Unfortunately python is not a language in their  college syllabus. So if you are good in python and have interest in contributing to the project, drop me a mail . There is no separate version for development and the one which is present at http://smc.org.in/silpa . All development happens there itself and any change in the code is immediately available for use!(or immediately starts crashing for user data)

I will write on some interesting algorithms I used for some modules later. If you are curious to know them, read the code!

Thanks to Charles-H. Schulz, we got a mailing list indic@native-lang.openoffice.org. To subscribe login to http://native-lang.openoffice.org

We just started, and I will soon setup a wiki page there. To start with , I will collect the list of issues pending for Indian languages from people from various languages and will find out people from various languages as point of contacts. Feel free to contact me for anything related to Openoffice in your language.

Update: June 3, 2009: This is our wiki page