Natural Language Processing

As a programmer, I spend most of my time in a terminal application like Kitty. I use Neovim as my code editor. I use CLI based AI agents. But the biggest pain, even in 2026, is that there is no terminal that can render complex scripts like Indic languages or Arabic. This is a significant limitation for me, as most of my work involves language processing. In this article, I will give a brief overview of why this issue remains unsolved—covering the character-cell grid model, width measurement, and the distinction between text shaping and rendering—along with ongoing efforts and a small tool I built recently that illustrates a solution path. ...

We are witnessing a resurgence and evolution of Command Line Interfaces (CLIs), accelerated by AI agents. Text-based, scriptable CLI tools work very well with LLM-based workflows. Accessing Wikipedia articles during an agent session is common. Usually, a webfetch call is used to get the HTML for a page from a URL like https://en.wikipedia.org/wiki/2026_Winter_Olympics. That works, and LLMs are smart enough to read HTML. But there is a cost: HTML is for rendering, so the model must ignore a lot of non-content markup to get to the useful text. i That increases token usage and adds context noise. Can we improve this? ...

I’m excited to announce two new resources for natural language processing researchers and developers: wikisentences - A Rust-based tool for extracting sentence datasets from Wikipedia dumps in any language ml-wiki-sentences - A dataset of 2.25 million Malayalam sentences extracted from Wikipedia, now available on HuggingFace, prepared using the above tool. The Wikisentences Tool The wikisentences project provides a complete pipeline for creating sentence datasets from Wikipedia content: Core Technology wiki-html-text-extractor (Rust) - Uses tree-sitter-html to parse article HTML and extract clean plain text sentencex (Rust) - Handles accurate sentence segmentation across languages. See my recent article about this library Four-Stage Pipeline Download enterprise HTML dumps from WikimediaThere is no recent html dumps for wikipedia, except this one year old dump Convert JSON dumps to Parquet format (id, name, url, language, html) Extract plain text from HTML (id, url, name, text) Segment text into sentences (id, url, name, sentence, sentence_index) Each stage is handled by a separate Python script, with the heavy lifting done by efficient Rust binaries. The pipeline is designed to be memory-efficient, streaming data between stages without writing intermediate files to disk. ...

If you have ever needed to work with sentences inside an HTML page — highlight them, translate them, read them aloud — you quickly run into a deceptively awkward problem. The text is not plain text. It is interspersed with tags, attributes, inline elements, and markup that your sentence detector has no business reading. This post walks through an exploratory JavaScript project — html-sentence-segmenter — that I built to figure out how to do this properly. It is not a polished, reusable library. Think of it as a working proof-of-concept that demonstrates the approach, with a live demo using Wikipedia articles. ...

In October 2023, we announced sentencex — a sentence segmentation library built for the wide language diversity of Wikipedia and Wikimedia projects. The original release comprised two separate libraries: one in Python (used by MinT, our machine translation service) and one in JavaScript (used by Content Translation). Both were rule-based, practical, and got the job done. But maintaining two implementations of the same logic was a constant source of drift. sentencex 1.0 is a ground-up rewrite in Rust, with the Python and JavaScript libraries replaced by thin bindings over a single, shared core. ...

Ever wondered how a computer learns to generate text that actually looks like Malayalam? Not just random characters, but something with actual structure? I’m not talking about Large Language Models here. I’m talking about Small Language Models that are efficient and explainable. something you can build and run on your own laptop. In my previous “The Broken Token” article, I presented a Malayalam unigram tokenizer and analysed its strengths and weaknesses. I did fertility rate evaluation and then analysed the tokenization in the context of Malayalam language characteristics. A common evaluation method for tokenizers is using them in downstream tasks—so I decided to build a text generator. That’s where things got interesting. ...

Tokenization is the first cut a language model makes on text. For Malayalam, that cut has almost always been in the wrong place — and the consequences reach much further than efficiency. Most large language models use tokenizers trained primarily on English and related Latin-script languages. When those tokenizers encounter Malayalam, the results range from inefficient to broken. Andrej Karpathy captured this as follows: This post is in two connected parts. The first is a technical exercise: I trained two subword tokenizers for Malayalam — one BPE, one Unigram — on the SMC Malayalam corpus and published them on HuggingFace. I explain the algorithms, the design choices that make them suitable for Malayalam, and measure their fertility against a cross-section of major LLMs including GPT-4, Gemma, Sarvam, and LLaMA. ...