zabeehkhan was trying to code a Pashto (ps_AF) module for dhvani. And he told me that “it is not saying anything” :). So I took the code and found the problem. Dhvani has a UTF-8 decoder and UTF-16 converter. It was written by Dr. Ramesh Hariharan and was tested only with the unicode range of the languages in India. It was buggy for most of the other languages and there by the language detection logic and text parsing logic was failing. So I did some googling, went through the code tables of gucharmap and got some helpful information from here and here

So here is my new UTF8Decoder and converter

This program converts a utf-8 encoded string to utf-16 hexadecimal code sequence

UTF-8 is a variable-width encoding of Unicode.
UTF-16 is a fixed width encoding of two bytes

A UTF-8 decoder must not accept UTF-8 sequences that are longer than necessary to
encode a character. For example, the character U+000A (line feed) must be accepted from
a UTF-8 stream only in the form 0x0A, but not in any of the following five possible overlong forms:

  0xC0 0x8A
  0xE0 0x80 0x8A
  0xF0 0x80 0x80 0x8A
  0xF8 0x80 0x80 0x80 0x8A
  0xFC 0x80 0x80 0x80 0x80 0x8A

Ref: UTF-8 and Unicode FAQ for Unix/Linux http://www.cl.cam.ac.uk/~mgk25/unicode.html

Author: Santhosh Thottingal <santhosh.thottingal at gmail.com>
License: This program is licensed under GPLv3 or later version(at your choice)
unsigned short
utf8_to_utf16 (unsigned char *text, int *ptr)

  unsigned short c;		/*utf-16 character */
  int i = 0;
  int trailing = 0;
  if (text[*ptr] < 0x80)	/*ascii character till 128 */
      trailing = 0;
      c = text[(*ptr)++];
  else if (text[*ptr] >> 7)
      if (text[*ptr] < 0xE0)
	  c = text[*ptr] & 0x1F;
	  trailing = 1;
      else if (text[*ptr] < 0xF8)
	  c = text[*ptr] & 0x07;
	  trailing = 3;

      for (; trailing; trailing--)
	  if ((((text[++*ptr]) & 0xC0) != 0x80))
	  c <<= 6;
	  c |= text[*ptr] & 0x3F;

  return c;


/* for testing */
main ()
  char *instr = "സന്തോഷ് തോട്ടിങ്ങല്‍";	/* my name :) */
  int length = strlen (instr);
  int i = 0;

  for (; i < length;)
      printf ("0x%.4x ", utf8_to_utf16 (instr, &i));
  printf ("\n");
/* output is:
0x0d38 0x0d28 0x0d4d 0x0d24 0x0d4b 0x0d37 0x0d4d 0x0020 0x0d24 0x0d4b 0x0d1f 0x0d4d 0x0d1f 0x0d3f 0x0d19 0x0d4d 0x0d19 0x0d32 0x0d4d 0x200d 

  return 0;

There may be already existing libraries for this, but writing a simple one ourself is fun and good learning experience.

For example, in python, to get the UTF-16 code sequence for a unicode string, we can use this:

print repr(str)

This gives the following output


1 thought on “UTF8Decoder”

  1. iconv

    $ echo “സന്തോഷ്‌” | iconv -f utf-8 -t UNICODE| hexdump
    0000000 feff 0d38 0d28 0d4d 0d24 0d4b 0d37 0d4d < <========
    0000010 200c 000a

    — Evuraan.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.