Indesign CC automatic hyphenation for Indian languages

More and more publishers are starting to use Indesign CC and Unicode. One of the many adavantages the publishers get with unicode and Indesign cc is automatic hyphenation. A few of my friends told me that they don’t know how to use hyphenation. Eventhough I never used Indesign before, I decided to figure out. In my Windows 10 virtual machine, I installed Indesign CC 2018.

Following is a tutorial on how to get perfect hyphenation for text in Indian languages in Indesign. I use Malayalam as example.

Indesign CC 2018 comes with Hunspell hyphenation dictionaries. These hyphenation dictionaries are written by me long time back. See https://github.com/smc/hyphenation

From menu Edit-> Preferences->Dictionary, set Language and Hyphenation as “Hunspell”

Create a text frame and add content to it. Make sure that the composer is set as Adobe World-Ready paragraph composer. You can access it from Paragraph settings as shown below. Without this settings, the Indic text won’t render correctly.

Tick the “Hyphenation” from the paragraph settings. Select an appropriate font for the content. Choose the language of the content as Malayalam or other Indic language you are working on. See screenshot below. Justify the content.

The content will get automatically hyphenated. If you resize the column width or insert more content, text will get automatically hyphenated.

The exported PDF will look like:

You can see the hyphenation rules in Installation folder: C:\Program Files\Adobe\Adobe InDesign CC 2018\Resources\Dictionaries\LILO\Linguistics\Providers\Plugins2\AdobeHunspellPlugin\Dictionaries

Patterns are available for Assamese, Bengali, Panjabi, Gujarati, Assamese, Marathi, Tamil, Telugu, Odia, Kannada and Malayalam.

I have not tried older Indesign versions, so I don’t know from which version this feature is available. But I don’t see a reason for not using latest version either.

Scribus gets Malayalam Hyphenation support

Scribus now has support for Malayalam hyphenation.

I filed a bug report to add Malayalam hyphenation rules to Scribus and it is now added to scribus. The hyphenation rules are based on the TeX hyphenation patterns I wrote.

How to use

You need scribus 1.5.4 or later. It is not yet available as release while I am writing this. But once released you can get from https://www.scribus.net/downloads/

  • Start a new document. Add text frames and content. You will need narrow columns to have wordbreaking contexts. For example 2 columns as I use for demo here.
  • Select the text and set font as a Malayalam font like Manjari, Set the language as Malayalam.
  • In Hyphenation properties, set hyphenation character as blank, otherwise visible hyphens will appear.
  • Set the text justified.
  • From menu Extras->Hyphenate text. Done.

Here is the output:

Hyphenated two column content

 

Libreoffice Malayalam Hyphenation

I had developed and released hyphenation extension for Malayalam in Openoffice years back. Libreoffice was born later. Eventhough libreoffice supported the openoffice extensions, the extension repository is freshly created for libreoffice. The old extensions were not present in the libreoffice repository.

Now, I have uploaded the Malayalam hyphenation extension in libreoffice extension repository too. I will explain the installation and configuration step by step in this blog post:

All Operating systems

  • Download an extension and save it anywhere on your computer.
  • In LibreOffice, select Tools -> Extension Manager from the menu bar.
  • In the Extension Manager dialog click Add.
  • A file browser window opens. Navigate to the folder where you saved the LibreOffice extension file(s) on your system. The extension’s files have the file extension ‘OXT’.
  • Find and select the extension you want to install and click Open.
  • If this extension is already installed, you’ll be prompted to press OK to confirm whether to overwrite the current version by the new one, or press Cancel to stop the installation.
  • After you are asked whether to install the extension only for your user or for all users. If you choose the Only for me option, the extension will be installed only for your user. If you choose For all users, you need system administrator rights. In this case the extension will be available for all users. In general, choose Only for me, that doesn’t require administration rights on the operating system.

Debian and Ubuntu

The above steps works for Debian and Ubuntu too. But there is a better way. Using your package manager install hyphen-ml package. This will install hyphenation not only for libreoffice, but for typesetting packages like LaTeX.

Using the hyphenation

  • To automatically hyphenate the current or selected paragraphs, choose Format – Paragraph, and then click the Text Flow tab.

    LIbreoffice Hyphenation
  • To manually Hyphenate Single Words, click in the word where you want to add the hyphen, and then press Ctrl+Hyphen(-).
  • To manually Hyphenate Text in a Selection Select the text that you want to hyphenate. Choose Tools – Language – Hyphenation.

For detailed help, read libreoffice hyphenation documentation

A hyphenated paragraph

Known Issues

Malayalam and several other languages does not use visible hypen(-) at the end of line when a word is broken. Currently there is no way to control this in libreoffice.

I had developed hyphenation patterns for 10 other Indian languages too. Yet to upload them to libreoffice repository. But they are readily available in Debian and Ubuntu. You can install them by choosing hyphen-* package.

 

Hyphenation in web

This is a follow up of a 4 year old blog post about hyphenation. Hyphenation allows the controlled splitting of words to improve the layout of paragraphs, typically splitting words at syllabic or morphemic boundaries and visually indicating the split (usually with a hyphen).

I wrote about how a webpage can use Hyphenator javascript library to achieve hyphenation for a text with ‘justify‘ style. Along with the hyphenation rules I wrote for many Indian languages, this solution works and some websites already use it. The Hyphenator library helps to insert Soft hyphens in appropriate positions inside the text.

Example showing the difference between Malayalam text hyphenated and not hyphenated. You can see lot of line space wasted with white space in non-hyphenated text
Example showing the difference between Malayalam text hyphenated and not hyphenated. You can see lot of line space wasted with white space in non-hyphenated text

 

More recently browsers such as Firefox, Safari and Chrome have begun to support the CSS3 hyphens property, with hyphenation dictionaries for a range of languages, to support automatic hyphenation.

For hyphenation to work correctly, the text must be marked up with language information, using the language tags described earlier. This is because hyphenation rules vary by language, not by script. The description of the hyphens property in CSS says “Correct automatic hyphenation requires a hyphenation resource appropriate to the language of the text being broken. The user agents is therefore only required to automatically hyphenate text for which the author has declared a language (e.g. via HTML lang or XML xml:lang) and for which it has an appropriate hyphenation resource.”

CSS Example

-webkit-hyphens: auto;
-moz-hyphens: auto;
-ms-hyphens: auto;
-o-hyphens: auto;
hyphens: auto;

Browser Compatibility

  • Chrome 13+ with -webkit prefix
  • Firefox 6.0+ with -moz prefix
  • IE 10+ with -ms prefix.

Hyphenation rules

CSS Text Level 3 does not define the exact rules for hyphenation, however user agents are strongly encouraged to optimize their line-breaking implementation to choose good break points and appropriate hyphenation points.

Firefox has hyphenation rules for about 40 languages. A complete list of languages supported in FF and IE is available at Mozilla wiki

You can see that none of the Indian languages are listed there. Hyphenation rules can be reused from the TeX hyphenation rules.  Jonathan Kew was importing the hyphenation rules from TeX and I had requested importing the hyphenation rules for Indian languages too.  But that was more than a year back, not much progress in that. Apparently there was a licensing issue with derived work but looks like it is resolved already.

CSS4 Text

While this is all well and good, it doesn’t provide the fine grain control you may require to get professional results. For this CSS4 Text introduce more features.

  • Limiting the number of hyphens in a row using hyphenate-limit-lines. This property is currently supported by IE10 and Safari, using the -ms- and -webkit- prefix respectively.
  • Limiting the word length, and number of characters before and after the hyphen using hyphenate-limit-chars
  • Setting the hyphenation character using hyphenate-character. Helps to override the default soft hyphen character

More reading

PS: Sometimes hyphenation can be very challenging. For example hyphenating the 746 letter long name of Wolfe+585, Senior.

Inkscape hyphenation extension

One year back I wrote about how to use Inkscape as a workaround solution for DTP in indic scripts. Still we don’t have any DTP software which supports Indic scripts in Unicode. Scribus still does not have the Indic support.

One issue with inkscape when used as DTP for indic script was, a few indic scripts always wanted hyphenation when text is justified. For example Malayalam has lengthy words and often space is wasted in lines if the text is not automatically hyphenated. But this feature was not available in inkscape. There is a wishlist bug for adding this feature to Inkscape.  I tried to develop an extension for Inkscape to achieve this.

It is on top of the python hyphenation code written by Wilbert  Berendsen. The hyphenation rules, also called as patterns is TeX or
Openoffice itself. So  I can support any language which has TeX hyphenation rules. But, since the hyphenation rules are language specific we need a language selection mechanism for the text first. Then only we can select the rules and do the hyphenation. But it is very tricky to implement.  Asking the language of the text every time it is justified is not a good idea. Setting a language for document is another choice, but what if the text contains multiple languages?  But for Indian languages it is very easy, we can automatically detect the scripts using unicode codepoints and load the rules accordingly. So for the time being, my extension support only English and all Indian languages.

Download the extension from http://thottingal.in/projects/inkscape_hyphenation/inkscape-hyphenation.zip . In GNU/Linux machines,  extract the zip file and copy to /usr/share/inkscape/extensions folder. In Windows , extract to [inkscape installation directory]\extensions folder.  After this close and reopen inkscape. You will see a menu named Hyphenate in Effects->Text menu.    In the document, add a text field, enter text in any indian language. Select the text and apply hyphenation by Effects->Text->Hyphenate. Then change the alignment of text to justify. You will see the text get hyphenated and occupying maximum possible space in the text field

I got satisfactory result with Malayalam and Tamil. I did not test other languages. Following images illustrates hyphenated, justified, two column layout of text done in Inkscape

Malayalam Hyphenation In inkscape
Malayalam Hyphenation In inkscape
Tamil Hyphenation in Inkscape
Tamil Hyphenation in Inkscape

We had a discussion about this in inkscape mailing list . Some developers suggested to have this feature built in, not as extension.  There are few issues to be solved for that. One thing is language selection as I explained. The other issue is regarding the hyphenation character to be used. Unicode standard insists to use soft hyphen – u00AD as hyphenation character. This is an invisible character. For Malayalam, visible hyphens are not required. But some other languages require the hyphen sign where the word is broken at the end of the line. The rules for whether the soft hyphen should be visible or not visible is not clear in Unicode’s specification. Pango never displays a the soft hyphen. There are criticism on this specification of softhyphen

So I think there is something to be done from Rendering engine or Unicode need to clarify the confusions.  But Openoffice and HTML rendering engines always make soft hyphen at the end of the line, which is not desired for some languages.

Try this extension, let me know the comments. For small scale DTP works, such as pamphlets, notices, brochures  inkscape is enough. But since inkscape is not primarily a DTP software and does not have paging support, for books and large scale DTP works, it may not work well.

New Hyphenation Pattern Extensions for Openoffice

Openoffice Indic Natural Language group announces the availability of the following Openoffice hyphenation dictionary extensions.

  1. Malayalam Hyphenation Rules version 1.2
  2. Kannada Hyphenation Rules version 1.1
  3. Bengali Hyphenation Rules verson 1.1
  4. Hindi Hyphenation Rules version 1.1
  5. Telugu Hyphenation Rules version 1.0
  6. Tamil Hyphenation Rules version 1.0
  7. Gujarati Hyphenation Rules version 1.0
  8. Panjabi Hyphenation Rules version 1.0
  9. Oriya Hyphenation Rules version 1.0
  10. Marathi Hyphenation Rules version 1.0

Spellchecker extension for Malayalam is also ready.

For a complete list of writing aids for Openoffice in Indic Languages is available here

Hyphenation Rules for Languages other than Marathi is already packages in Fedora 11. This releases contains updates and bug fixes. Fedora 12 will contains these updates. These extensions are yet to be packaged for Debian/Ubuntu.

More details about hyphenation rules are available here

Hyphenation of Indian Languages in Webpages

In my last blogpost I explained hyphenation of Indian language text in openoffice. In this blogpost I will explain how hyphenation can be done in webpages.

As I explained importance of hyphenation come into picture when we justify the text. The length of the lines are controlled by the parent tags…. Unicode had defined a special character called soft hyphen for hyphenation denoted by ­ . In HTML, the plain hy­phen is rep­re­sent­ed by the “-” char­ac­ter (- or-). The soft hy­phen is rep­re­sent­ed by the char­ac­ter en­ti­ty ref­er­ence ­ (­ or ­)

User agents-browsers can break the line whenever a soft hyphen is found. So if we have a javascript based implemenation, which insert the softhyphen in between the words based on language specific rules, we can achieve hyphenation in webpages too.

Hyphenator is a project which does exactly the same. “Hyphenator.js brings client-side hyphenation of HTML-Documents on to every browser by inserting soft hyphens using hyphenation patterns and Frank M. Liangs hyphenation algorithm commonly known from LaTeX and Openoffice. “

Hyphenator was not tested for any non-latin languages so far. I tried to add support for Indian languages and the result was satisfactory. I used the
same rules I defined for openoffice. Unlike latin languages, the number of hyphenation patterns for Indian languages is very less and the performance is good because of that.

I have added Malayalam, Tamil, Hindi, Oriya, Kannda, Telugu, Bengali, Gujarati and Panjabi support to it. You can see a working example here. (I wanted to embed one example here. But livejournal doesnot allow javascript inside blog body ). The column layout is done by CSS. Try resizing the browser windows and try a print preview too..

Don’t forget to read the source code of that page. It is very simple. If you want hyphenation in your webpage, all you need is to include the javascript as done in the example. We need to provide the lang attributes for nodes so that the required patterns for that language can be loaded. I placed the new language patterns temporarily in download area of SMC. I will ask the author of Hyphenator to include it in upstream itself. Code is available here


Update(18-Dec-2008):Thanks to Mathias Nater, author of hyphenator, the patterns were added to upstream.

Hyphenation of Indian Languages and Openoffice

What is Hiphenation?

Hyphenation is the process inserting hyphens in between the syllables of a word so that when the text is justified, maximum space is utilized.

Hiphenation is an important feature that DTP softwares provide. For Indian languages there is no good DTP softwares available. XeTex is the only choice to work with unicode and professional quality page layout. But xetex and DTP are not exactly same. Inkscape can be used as temporary solution. But only for small scale works. There is a project going on to add Harfbuzz backend to Scribus, the freedomware DTP package.

Hiphenation is also requred in many other places. Actually it is required where ever we ‘justify’ a block of text in openoffice or any wordprocessors. Same is the case of webpages. If we justify a block of text in ml_IN, let is see what is happening now

Note the long gaps between words. This is a screenshot taken from firefox. The default hiphenation just breaking the lines in space characters. And no doubt that it makes the pages ugly. The problem becomes worse if the length of the word is more and column width is less.

So what is the solution?

Ideal solution : Applications should be aware of the language, its hiphenation rules and should to the hiphenation wherever required.

Openoffice can take hiphenation dictionaries just like spell checkers. But for Indian languages, we are yet to prepare hiphenation dictionaries(more on that later.) . CSS3 draft of w3c has a provision for hyphenate. But it is stil in draft stage

Algorithm For Hiphenation

The basic for all hyphenation algorithms is the hyphenation algorithm, designed by Frank Liang in 1983, which is adopted in TeX. Wikipedia artcle of TeX explain this with very simple example

If TeX must find the acceptable hyphenation positions in the word encyclopedia, for example, it will consider all the subwords of the extended word .encyclopedia., where . is a special marker to indicate the beginning or end of the word. The list of subwords include all the subwords of length 1 (., e, n, c, y, etc), of length 2 (.e, en, nc, etc), etc, up to the subword of length 14, which is the word itself, including the markers. TeX will then look into its list of hyphenation patterns, and find subwords for which it has calculated the desirability of hyphenation at each position. In the case of our word, 11 such patterns can be matched, namely 1c4l4, 1cy, 1d4i3a, 4edi, e3dia, 2i1a, ope5d, 2p2ed, 3pedi, pedia4, y1c. For each position in the word, TeX will calculate the maximum value obtained among all matching pattern, yielding en1cy1c4l4o3p4e5d4i3a4. Finally, the acceptable positions are those indicated by an odd number, yielding the acceptable hyphenations en-cy-clo-pe-di-a. This system based on subwords allows the definition of very general patterns (such as 2i1a), with low indicative numbers (either odd or even), which can then be superseded by more specific patterns (such as 1d4i3a) if necessary. These patterns find about 90% of the hyphens in the original dictionary; more importantly, they do not insert any spurious hyphen. In addition, a list of exceptions (words for which the patterns do not predict the correct hyphenation) are included with the Plain TeX format; additional ones can be specified by the user.

For more details about the algorithm used in Openoffice read this paper by Nemeth Laszlo

Hiphenation in Indian languages.

Unlike English or any other languages, hiphenation in Indian languages are not that much complex. In general following are the rules

  • [consonant][vowel][consonat] can be hiphenated as [consonant][vowel] – [consonat] if vowel is not a virama or halant
  • Dont split a word after ZWJ
  • We can split a word after ZWNJ
  • plus any language specific rules. For eg: in ml_IN a line should not start with a chillu letter.

Hiphenation Dictionaries for Indian languages.

Based on the above mentioned rules, Let us try to create hiphenation dictionaries for Indian languages. I will explain this with the help of a Hindi word example: अनुपल्ब्ध.
We have to define the following rules in the dictionary for this
अ1 -> 1 is odd number , ie. word can be splitterd after अ
ु1 -> 1 is odd number , ie. word can be splitterd after ु
1ल -> 1 is odd number , ie. word can be splitterd before ल
1प -> 1 is odd number , ie. word can be splitterd before प
1ब -> 1 is odd number , ie. word can be splitterd before ब
्2 -> 2 is even number , ie. word can NOT be splitterd after ्
1ध -> 1 is odd number , ie. word can be splitterd before ध
So the end result is अ+नु+प+ल्ब्ध

Same way we can create the Hyphenation dictionaries for all other languages. I have prepared the Hyphenation dictionaries for 8 Indian Languages. Download it from the git repo of the SMC.
How to Install a xx_IN hyphenation dictionary.

  • Copy the hyphenation dictionay hyph_xx_IN to /usr/share/myspell/dicts folder.
  • Create a file at /usr/share/myspell/infos/ooo/ folder named openoffice.org-hyphenation-xx with one line content
    HYPH xx IN hyph_xx_IN
  • Run this command sudo update-openoffice-dicts

Open the openoffice writer, Open some fille in your language or type some text. Justify the text. Set the language of the selection by using Tools->Language menu Hiphenate it by using Tools->Language->Hiphenation menu.

Hope it works :). I tested only Hindi and Malayalam. For other languages , inform me if you see any problems or if it is not working . Here is the hyphenated Malayalam paragraph. Compare it with the image I showed at the beginning of this blogpost

Ok. so after testing these hyphenation dictionaries, if we provide them to upstream and packaged, hyiphenation problems in openoffice is solved. 🙂

But…. How to solve this problem in web pages?!. We will discuss it in next blogpost!
PS: Thanks to Nemeth Laszlo , author of Hunspell and Openoffice Hyphenation for helping me to prepare the hyphenation tables.


Update(Apr 16,2009) The hyphenation dictionaries were packaged for Fedora and will be part of Fedora 11