We are witnessing a resurgence and evolution of Command Line Interfaces (CLIs), accelerated by AI agents. Text-based, scriptable CLI tools work very well with LLM-based workflows. Accessing Wikipedia articles during an agent session is common. Usually, a webfetch call is used to get the HTML for a page from a URL like https://en.wikipedia.org/wiki/2026_Winter_Olympics.
That works, and LLMs are smart enough to read HTML. But there is a cost: HTML is for rendering, so the model must ignore a lot of non-content markup to get to the useful text. i That increases token usage and adds context noise. Can we improve this?
What if a text-only version of a Wikipedia article is available, or even a markdown version? A JSON format is also useful when structured information is needed. To make this work, an LLM needs the skill to use a CLI that prepares these formats. Output is cleaner, tokens are saved, and context stays focused.
So I built a CLI to transform a Wikipedia article into text, markdown, and JSON formats.
How to install and use
With uvx
uvx wikipedia-article-transform fetch --language en --title Oxygen --format markdown
Alternatively, you can install it with Cargo:
cargo install wikipedia-article-transform --features cli
Fetch an article:
# Plain text (default)
wikipedia-article-transform fetch --language en --title "Rust_(programming_language)"
# Semantic JSON section tree
wikipedia-article-transform fetch --language ml --title "കേരളം" --format json
# Markdown with inline formatting
wikipedia-article-transform fetch --language en --title "Liquid_oxygen" --format markdown
How it works
It fetches the article HTML and parses it using tree-sitter with the tree-sitter-html grammar.
Non-content blocks like infoboxes, nav elements, and hatnotes are skipped.
Then the parser extracts clean text plus structured metadata, and renders the result as plain text, markdown, or JSON.
Source code: https://github.com/santhoshtr/wikipedia-article-transform
Why not use wikitext directly?
LLMs can read wikitext, but it is usually a worse intermediate format for question-answering and summarization workflows.
- Wikitext is authoring syntax, not reader-facing content. A lot of meaning is hidden in templates/transclusions (
{{...}}), parser functions, and Lua modules that must be expanded first. - Without MediaWiki expansion, the model sees scaffolding instead of final prose.** It spends tokens reconstructing structure instead of answering the question.
In this tool, text, markdown, and JSON are extracted from the final article rendering shown to readers. That rendering is produced by the MediaWiki engine after internal resolution of templates and modules.
Web interface
If you just want to try the output formats in a browser, there is a simple hosted interface. Pick a language code and page title, then change only the extension to choose the format. Edit the URL path as needed.
How to tell your AI agent to use it
This tool is also available as an agentic skill. You may install it with:
npx skills add https://github.com/santhoshtr/wikipedia-article-transform
Now, this skill will be picked up by your agentic AI tool (Claude code, OpenCode etc). Let me illustrate with a prompt:
What does Wikipedia say about Marie Curie's early life and education?
Include relevant citations.
With the skill
- Marie Curie (born Maria Salomea Skłodowska) was born on 7 November 1867 in Warsaw (then in Congre
ss Poland under the Russian Empire), the youngest of five children in a family of teachers. Citation
: "Marie Curie– Polish Girlhood (1867–1891) Part 1" (American Institute of Physics, archived sou
rce, retrieved 7 November 2011); Estreicher, Tadeusz, *Polski słownik biograficzny* vol. 4 (1938),
p. 111.
- Her family experienced financial hardship linked to Polish patriotic activity and Russian repressi
on, and she also faced personal losses in childhood (her sister Zofia died, then her mother Bronisł
awa died), events that shaped her outlook in youth. Citation: "Marie Curie– Polish Girlhood (1867�
��1891) Part 1" (American Institute of Physics, retrieved 7 November 2011); Goldsmith, Barbara, *Obs
essive Genius: The Inner World of Marie Curie* (2005), p. 149.
- Because women were barred from regular higher education institutions in partitioned Poland, she st
udied through Warsaw's clandestine Flying University while supporting herself through tutoring and g
overness work. Citation: "Marie Curie– Polish Girlhood (1867–1891) Part 2" (American Institute o
f Physics, retrieved 7 November 2011); Estreicher, Tadeusz, *Polski słownik biograficzny* vol. 4 (1
938), p. 112.
- Before leaving for France, she also began practical scientific training in Warsaw (1890-1891) in a
chemistry laboratory at the Museum of Industry and Agriculture, under the direction of her cousin J
ozef Boguski. Citation: "Marie Curie– Polish Girlhood (1867–1891) Part 2" (American Institute of
Physics, retrieved 7 November 2011); Wierzewski, Wojciech A., "Mazowieckie korzenie Marii" (2008).
- In late 1891 she moved to Paris, enrolled at the University of Paris, studied physics/chemistry/ma
thematics under difficult living conditions, earned a physics degree in 1893, and completed a second
degree in 1894. Citation: "Marie Curie– Student in Paris (1891–1897) Part 1" and Part 2 (Americ
an Institute of Physics, retrieved 7 November 2011); Pasachoff, Naomi, *Marie Curie: And the Science
of Radioactivity* (1996), p. 93.
The answer generated with the skill is usually denser and better grounded in specific sources. Because the tool provides a cleaner article representation first, the model spends less effort parsing navigation markup and more effort on the actual content and references.
Without the skill (webfetch)
According to Wikipedia, Marie Curie (born Maria Salomea Skłodowska) was born in Warsaw on 7 Novembe
r 1867, the youngest child in a family of educators. Her father taught mathematics and physics, and
her mother ran a girls' boarding school.[1][2]
Wikipedia says her childhood was marked by hardship: family financial decline under Russian rule, th
e death of her sister Zofia from typhus, and the death of her mother from tuberculosis when Maria wa
s ten.[2][3]
For education, she attended J. Sikorska's boarding school, then a girls' gymnasium, graduating in 18
83 with a gold medal.[1] Because women were barred from regular higher education in Russian-controll
ed Poland, she studied through Warsaw's clandestine Flying University.[1][2]
To fund future study, she worked as a tutor and governess and supported her sister Bronisława's med
ical education in Paris under a mutual support arrangement. She also continued self-study and began
practical scientific work in Warsaw (1890-1891).[1][4]
Wikipedia further states that in 1891, at age 24, she moved to Paris to continue formal higher educa
tion, where she earned advanced degrees and began the research career that later made her famous.[5]
## Citations
[1] Wikipedia, *Marie Curie*, "Early years" section, inline refs [15], [16], [23]. https://en.wikipe
dia.org/w/index.php?title=Marie_Curie&oldid=1341820133#Early_years
[2] Wikipedia inline reference [16] ("Marie Curie – Polish Girlhood (1867-1891)"). https://en.wiki
pedia.org/w/index.php?title=Marie_Curie&oldid=1341820133#cite_note-Marie_Curie_%E2%80%93_Polish_Girl
hood_(1867%E2%80%931891)-18
[3] Wikipedia inline references [21], [22] in "Early years" (family religion and deaths of mother/si
ster). https://en.wikipedia.org/w/index.php?title=Marie_Curie&oldid=1341820133#Early_years
[4] Wikipedia inline reference [23] (support arrangement with Bronisława; governess period). https:
//en.wikipedia.org/w/index.php?title=Marie_Curie&oldid=1341820133#cite_note-Marie_Curie_%E2%80%93_Po
lish_Girlhood_(1867%E2%80%931891)1-25
[5] Wikipedia lead paragraph on move to Paris in 1891 and higher degrees, inline citations [6], [7].
https://en.wikipedia.org/w/index.php?title=Marie_Curie&oldid=1341820133
Which is better? 😄
Let us look at the tokens in markdown version and html version: we are measuring using tiktoken. I used https://token-count.streamlit.app/
As you can clearly see, we only need 10% of tokens compared to original html. Token is 💰.
Use cases beyond LLM agent workflows
This transformer is useful in many data and product workflows even without an interactive AI agent.
- Build NLP corpora from clean article text for sentence segmentation, topic analysis, and summarization experiments.
- Generate embeddings for retrieval and QA pipelines where HTML boilerplate adds no value and only increases noise.
- Feed markdown output into publishing systems, docs pipelines, and static-site workflows that already expect markdown input.
- Use JSON section trees for structural analysis, heading-level comparisons, and multilingual content studies.
- Improve factual verification workflows by keeping parsed article text and references together in a structured output, making claim-to-citation checks easier to automate.
Disclaimer
I work at the Wikimedia Foundation. However, this project, its exploration, and the opinions expressed are entirely my own and do not reflect my employer’s views. This is not an official Wikimedia Foundation project.