Why your slug generator is dropping Malayalam vowels and how to fix it

TL;DR: Most slug libraries only allow \p{L} (letters), \p{N} (numbers), and \p{S} (symbols). In Indic scripts like Malayalam, vowels are often combining marks (\p{M}), not letters. The fix is simple: add \p{M} to your allowed character class. I’ll show you exactly how.

I recently fixed this memos PR #6051. The bug was simple but devastating: Malayalam tags like #കവിത were being truncated to #കവ. The last two characters — ി (vowel sign I) and ത — were rendered outside tag.

Before	After

The ി character is U+0D3F MALAYALAM VOWEL SIGN I. Its Unicode category is Mc (Mark, Spacing Combining). The old tag grammar allowed letters, numbers, and symbols, but not marks. So the lexer stopped right there.

This isn’t an isolated bug. I’ve seen it in slug generators across multiple languages and frameworks.

It shows up in two shapes.

• A tag lexer or an allow-list regex treats the mark as a separator, so the word loses its vowel and splits: കവിത → കവ-ത. This was the case with Memos
• An accent stripper deletes the mark outright, leaving no separator behind: കവിത → കവത.

An example - Django

The deleting variant isn’t only hand-rolled code. Here is the relevant line of django.utils.text.slugify:

value = re.sub(r"[^\w\s-]", "", value.lower())

Python’s \w is Unicode-aware, but it means alphanumeric or underscore — and marks are not alphanumeric:

>>> re.fullmatch(r'\w', 'ി')   # U+0D3F, category Mc
None
>>> from django.utils.text import slugify
>>> slugify('തിരുവനന്തപുരം', allow_unicode=True)
'തരവനനതപര'

Five of those thirteen code points are marks. All five are gone.

Note that the allow_unicode=True is documented as the way to keep non-ASCII slugs, so it is what you reach for when you are already trying to do the right thing. And the eight-line body gets copy-pasted verbatim — docstring and all — into projects that never install Django, so grepping for the literal [^\w\s-] finds more instances than grepping for Django does. Github code search

One trap when porting between ecosystems: Python’s \w drops marks, while JavaScript’s \w is ASCII-only. The same regex fails differently in each language.

Why this keeps happening

Unicode isn’t just “letters with accents.” It has a whole separate category for characters that combine with other characters: \p{M} (Marks).

In many writing systems — Indic scripts (Malayalam, Devanagari, Tamil, Telugu, Kannada), Arabic, Hebrew, and even decomposed Latin — vowels and diacritics are combining marks, not letters.

When you write a regex like [^\p{L}\p{N}]+ to strip “non-alphanumeric” characters, you’re accidentally deleting the vowels from half the world’s languages.

How to test if your library has this bug

Run this quick test in your language of choice:

Input:  കവിത  (Malayalam for "poem")
Expected: കവിത
Buggy output: കവത  (vowels dropped)

More test cases:

If your slug library returns anything shorter than the input, it’s dropping marks.

What if you’re writing it yourself?

Here’s the fix for each ecosystem.

JavaScript / Node.js

function slugify(input) {
  return input
    .normalize('NFD')                          // decompose
    .toLowerCase()
    .replace(/[^\p{L}\p{N}\p{M}]+/gu, '-')    // keep letters, numbers, MARKS
    .replace(/^-+|-+$/g, '')
    .normalize('NFC');                         // recompose
}

Key change: \p{M} added to the character class. The /u flag enables Unicode mode .

Don’t skip the final normalize('NFC'). Without it the slug keeps whatever NFD produced, so café ends up as e + U+0301 and two visually identical titles give you two different slugs.

Python

import regex  # pip install regex

def slugify(input):
    return regex.sub(
        r'[^\p{L}\p{N}\p{M}]+',  # keep letters, numbers, MARKS
        '-',
        input.lower()
    ).strip('-')

Key change: \p{M} in the character class — and regex rather than the standard library. re has no \p{...} support at all and raises error: bad escape \p.

Stdlib only? Filter on the category directly:

import re, unicodedata

def slugify(input):
    out = ''.join(
        ch if unicodedata.category(ch)[0] in 'LNM' else '-'
        for ch in unicodedata.normalize('NFC', input.lower())
    )
    return re.sub(r'-+', '-', out).strip('-')

Rust

use regex::Regex;
use std::sync::LazyLock;

static NON_SLUG: LazyLock<Regex> =
    LazyLock::new(|| Regex::new(r"[^\p{L}\p{N}\p{M}]+").unwrap());

fn slugify(input: &str) -> String {
    NON_SLUG
        .replace_all(&input.to_lowercase(), "-")
        .trim_matches('-')
        .to_string()
}

Key change: \p{M} in the pattern.

\p{...} needs the crate’s unicode-gencat feature. It ships in the default unicode feature, so plain regex = "1" works — but if anything in your build sets default-features = false, the pattern fails to compile at runtime with error: Unicode property not found, and the unwrap() panics. Ask for it back explicitly:

regex = { version = "1", default-features = false, features = ["std", "unicode-gencat"] }

When to use a library instead

If you’re using a library, check its defaults. Many libraries transliterate to ASCII by default, which will mangle Indic scripts. Look for a “preserve Unicode” or “Unicode-first” option.

But even with a library, always test with your target scripts. The bug is subtle enough that it ships silently.