I’m excited to announce two new resources for natural language processing researchers and developers:

  1. wikisentences - A Rust-based tool for extracting sentence datasets from Wikipedia dumps in any language
  2. ml-wiki-sentences - A dataset of 2.25 million Malayalam sentences extracted from Wikipedia, now available on HuggingFace, prepared using the above tool.

The Wikisentences Tool

The wikisentences project provides a complete pipeline for creating sentence datasets from Wikipedia content:

Core Technology

Four-Stage Pipeline

  1. Download enterprise HTML dumps from WikimediaThere is no recent html dumps for wikipedia, except this one year old dump
  2. Convert JSON dumps to Parquet format (id, name, url, language, html)
  3. Extract plain text from HTML (id, url, name, text)
  4. Segment text into sentences (id, url, name, sentence, sentence_index)

Each stage is handled by a separate Python script, with the heavy lifting done by efficient Rust binaries. The pipeline is designed to be memory-efficient, streaming data between stages without writing intermediate files to disk.

You can drive the entire pipeline with a single command:

make data/ml.sentences.parquet

Processing all articles from wikipedia, extracting text and then segmenting to sentences took me two days when I tried this in 2023. But now it takes less than 30mins.

Malayalam Wikipedia Sentence Dataset

I’ve processed the Malayalam Wikipedia dump using this pipeline to create a high-quality sentence dataset:

  • 2.25 million sentences extracted from Malayalam Wikipedia
  • Available in Parquet format for efficient processing
  • Licensed under CC-BY-SA-3.0 (same as Wikipedia)
  • Hosted on HuggingFace

The dataset includes:

  • id: Wikipedia page ID
  • url: Canonical URL of the Wikipedia article
  • name: Article title
  • sentence: The extracted sentence text
  • sentence_index: Position of sentence within the article

Getting Started

You can load the Malayalam sentence dataset directly from HuggingFace using the 🤗 Datasets library:

from datasets import load_dataset

dataset = load_dataset("smcproject/ml-wiki-sentences", split="train")
print(f"Loaded {len(dataset)} Malayalam sentences")
print(dataset[0])  # First sentence example

For developers who want to create sentence datasets for other languages, the wikisentences tool is available on GitHub with detailed usage instructions in the README.

Applications

Sentence datasets like this are valuable for:

  • Training language models
  • Developing NLP applications (tokenization, sentence boundary detection)
  • Linguistic research
  • Creating evaluation benchmarks for Malayalam NLP tasks

The wikisentences pipeline makes it straightforward to create similar datasets for any of the 300+ language editions of Wikipedia.

Future Work

I invite researchers to process other language Wikipedias using the same pipeline and release them on HuggingFace.

Disclaimer

I work at the Wikimedia Foundation. However, this project, its exploration, and the opinions expressed are entirely my own and do not reflect my employer’s views. This is not an official Wikimedia Foundation project.