Zwj

Recently, while preparing a critique for IDN Policy for Malayalam language prepared by CDAC, I noticed that ICANN does not allow control characters in the domain names. Sometime back I noticed Python 3 identifiers also does not allow control characters in the Identifiers. This blog post attempts to analyze the issue by looking at the Unicode and ICANN specifications about these special characters. Apart from the existing characters in Indic languages, Zero width Joiner and Zero width non joiners are widely used in Indic languages to control how the ligatures are formed. For some samples on how they are used, refer the wikipedia links. Being control characters and invisible characters, they are often removed while doing normalization , particularly before doing a string comparison, or collation (sort). ...