Blogs

In October 2023, we announced sentencex — a sentence segmentation library built for the wide language diversity of Wikipedia and Wikimedia projects. The original release comprised two separate libraries: one in Python (used by MinT, our machine translation service) and one in JavaScript (used by Content Translation). Both were rule-based, practical, and got the job done. But maintaining two implementations of the same logic was a constant source of drift. sentencex 1.0 is a ground-up rewrite in Rust, with the Python and JavaScript libraries replaced by thin bindings over a single, shared core. ...

Ever wondered how a computer learns to generate text that actually looks like Malayalam? Not just random characters, but something with actual structure? I’m not talking about Large Language Models here. I’m talking about Small Language Models that are efficient and explainable. something you can build and run on your own laptop. In my previous “The Broken Token” article, I presented a Malayalam unigram tokenizer and analysed its strengths and weaknesses. I did fertility rate evaluation and then analysed the tokenization in the context of Malayalam language characteristics. A common evaluation method for tokenizers is using them in downstream tasks—so I decided to build a text generator. That’s where things got interesting. ...

Tokenization is the first cut a language model makes on text. For Malayalam, that cut has almost always been in the wrong place — and the consequences reach much further than efficiency. Most large language models use tokenizers trained primarily on English and related Latin-script languages. When those tokenizers encounter Malayalam, the results range from inefficient to broken. Andrej Karpathy captured this as follows: This post is in two connected parts. The first is a technical exercise: I trained two subword tokenizers for Malayalam — one BPE, one Unigram — on the SMC Malayalam corpus and published them on HuggingFace. I explain the algorithms, the design choices that make them suitable for Malayalam, and measure their fertility against a cross-section of major LLMs including GPT-4, Gemma, Sarvam, and LLaMA. ...

I am a type designer using non-conventional approaches to type design. Since 2020, I have been experimenting with parametric type design approaches based on a modernized MetaPost approach. I have written extensively about this approach in this blog. I have also published a paper1 on that. One of the drawbacks of the MetaPost-based approach is that MetaPost is quite old, actually older than me. It was written in the 1980s. MetaPost was actually a modernization on top of MetaFont by Knuth. It did not have SVG support, but that was added in 2024. MetaPost is written in literate programming language WEB, then generating Pascal code from it. Hence the tooling and developer experience around it is quite suboptimal. Extending it is also almost impossible. ...

I published svg2glif - a Rust crate to convert SVG vector graphics into UFO (Unified Font Object) GLIF format, making it easy to incorporate SVG artwork into font development workflows. Source code Library documentation I am migrating my python based type engineering workflow to rust (fontmake -> fontc) and this is one of the tool I need. ❯ target/release/svg2glif -i examples/A.svg -o examples/A.glif --em-size 1000 --descent 250 Wrote glif to examples/A.glif (took 187.36µs) Yes, 187.36µs 🚀😄 ...

On November 3rd, I spoke at Malayalam University on “How AI learns Malayalam,” tracing how humans acquire language versus how Large Language Models learn from patterns at scale. This invited session was part of Symbosium on modernization of Malayalam language, organized by Sahitya Akademi and Malayalam University. Using Chomsky’s Syntactic Structures and the “colorless green ideas” example to frame structure vs. meaning, I contrasted innate rules with data-driven learning, engaged Emily M. Bender et al.’s “Stochastic Parrots” critique of ungrounded fluency, and reflected on Richard Sutton’s “Bitter Lesson” and his call for world models - asking what written text alone can and cannot teach machines. To contrast meaning and Syntax, I used a scene from 1985 Malayalam movie “Boeing Boeing” (..ഇനിയും വരില്ലേ ആനകളേയും തെളിച്ചുകൊണ്ട്.. 😃 ) ...

I am excited to announce that the Malayalam morphology analyzer library, mlmorph, is now available as a Rust crate on crates.io. This Rust binding became possible thanks to the availability of Rust bindings for mlmorph’s underlying library - the Stuttgart Finite State Transducer (SFST) formalism. You can find the SFST Rust crate here. The crate comes with comprehensive documentation and example usage to help you get started with Malayalam morphological analysis in your Rust projects.

Integrating Knowledge Graphs (KGs) with Large Language Models (LLMs) is a well-explored research field. KGs are vast, structured databases storing factual associations as graph edges. KGs can help LLMs for tasks like question answering, drawing on the structured, often up-to-date information within KGs, thereby mitigating the risk of hallucination. For instance, an LLM that can query Wikidata—a prominent KG project—instead of solely depending on its training data becomes significantly more reliable and useful. ...

Wikidata is a rich knowledge graph, but its raw data format can be challenging for both humans and AI to process effectively. This blog post explores how I addressed these challenges by creating qrender, a tool for rendering Wikidata items in more human-readable and AI-friendly formats. In my previous article about qjson, I explained the importance of retrieving all information about a Wikidata Item. I write qjsonas an easy API to fetch all such information in one API call instead of multiple SPARQL queries or API calls. ...

In April 2024, I took a two-month sabbatical from work. During this period, and in the year that followed, I dedicated significant effort to improving my skills. Having spent nearly 20 years in software engineering, I’ve witnessed substantial changes in the field. Opportunities to learn and practice new skills don’t always arise naturally at work. While I made a conscious effort to dedicate time each day to reading and learning, finding time for hands-on practice remained a challenge. I set specific goals: to deepen my understanding of system architecture and modern design approaches, explore high-performance programming, and integrate AI into my development process. Here’s an update on my progress. ...