Sentences

I’m excited to announce two new resources for natural language processing researchers and developers: wikisentences - A Rust-based tool for extracting sentence datasets from Wikipedia dumps in any language ml-wiki-sentences - A dataset of 2.25 million Malayalam sentences extracted from Wikipedia, now available on HuggingFace, prepared using the above tool. The Wikisentences Tool The wikisentences project provides a complete pipeline for creating sentence datasets from Wikipedia content: Core Technology wiki-html-text-extractor (Rust) - Uses tree-sitter-html to parse article HTML and extract clean plain text sentencex (Rust) - Handles accurate sentence segmentation across languages. See my recent article about this library Four-Stage Pipeline Download enterprise HTML dumps from WikimediaThere is no recent html dumps for wikipedia, except this one year old dump Convert JSON dumps to Parquet format (id, name, url, language, html) Extract plain text from HTML (id, url, name, text) Segment text into sentences (id, url, name, sentence, sentence_index) Each stage is handled by a separate Python script, with the heavy lifting done by efficient Rust binaries. The pipeline is designed to be memory-efficient, streaming data between stages without writing intermediate files to disk. ...

If you have ever needed to work with sentences inside an HTML page — highlight them, translate them, read them aloud — you quickly run into a deceptively awkward problem. The text is not plain text. It is interspersed with tags, attributes, inline elements, and markup that your sentence detector has no business reading. This post walks through an exploratory JavaScript project — html-sentence-segmenter — that I built to figure out how to do this properly. It is not a polished, reusable library. Think of it as a working proof-of-concept that demonstrates the approach, with a live demo using Wikipedia articles. ...

In October 2023, we announced sentencex — a sentence segmentation library built for the wide language diversity of Wikipedia and Wikimedia projects. The original release comprised two separate libraries: one in Python (used by MinT, our machine translation service) and one in JavaScript (used by Content Translation). Both were rule-based, practical, and got the job done. But maintaining two implementations of the same logic was a constant source of drift. sentencex 1.0 is a ground-up rewrite in Rust, with the Python and JavaScript libraries replaced by thin bindings over a single, shared core. ...