For those deeply involved with Wikidata, the richness of its interconnected data is both a blessing and a challenge when it comes to programmatic access. While the standard wbgetentities API endpoint is fundamental, retrieving the complete set of properties, including labels and values, for a given item often leads to a cascade of recursive API calls. For example, suppose we fetch all properties for Q42 using wbgetentities API - https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q42. In the response, if well lookup the “country of citizenship” (P27) for Q42 (Douglas Adams): the initial response only provides the target QID (Q145), necessitating further queries to resolve both P27 and Q145 into human-readable labels. This quickly becomes inefficient for comprehensive data extraction.
The need to retrieve all information visible on a Wikidata item’s page – properties, statements, qualifiers, and their corresponding labels – is a common requirement. However, the inherent limitations of the standard API make this a cumbersome process.
The standard API’s approach of returning only the QID for properties and values means that developers often find themselves in a loop of making multiple API calls to resolve these IDs into meaningful labels. This not only increases the complexity of the code but also leads to performance issues due to the high number of requests made to the API.
The solution lies in leveraging the expressive power of SPARQL. A carefully constructed SPARQL query can efficiently fetch all the required details in a single request. The following query demonstrates this capability:
SELECT
?property
?propertyLabel
?statementValue
?statementValueLabel
?statementValueImage
?qualifierProperty
?qualifierPropertyLabel
?qualifierValue
?qualifierValueLabel
?unitOfMeasure
?unitOfMeasureLabel
?statementRankLabel
WHERE {
VALUES ?item {wd:%s}
?item ?propertyPredicate ?statement .
?statement ?statementPropertyPredicate ?statementValue .
?property wikibase:claim ?propertyPredicate .
?property wikibase:statementProperty ?statementPropertyPredicate .
?statement wikibase:rank ?statementRank .
BIND(IF(?statementRank = wikibase:NormalRank, "Normal", IF(?statementRank = wikibase:PreferredRank, "Preferred", IF(?statementRank = wikibase:DeprecatedRank, "Deprecated", "Unknown"))) AS ?statementRankLabel)
OPTIONAL { ?statementValue wdt:P18 ?statementValueImage . }
OPTIONAL { ?statement ?qualifierPredicate ?qualifierValue . ?qualifierProperty wikibase:qualifier ?qualifierPredicate . }
OPTIONAL { ?statement ?statementValueNodePredicate ?valueNode . ?valueNode wikibase:quantityUnit ?unitOfMeasure . }
SERVICE wikibase:label { bd:serviceParam wikibase:language "%s, en" . ?property rdfs:label ?propertyLabel . ?statementValue rdfs:label ?statementValueLabel . ?qualifierProperty rdfs:label ?qualifierPropertyLabel . ?qualifierValue rdfs:label ?qualifierValueLabel . ?unitOfMeasure rdfs:label ?unitOfMeasureLabel . }
}
ORDER BY ?property ?statementValue ?qualifierProperty ?qualifierValue
Executing this query (replace %s with the desired QID, e.g., Q42) on the Wikidata Query Service yields the raw data.
Example for Q42: query
qjson
While the SPARQL query provides the necessary data, the direct output often requires significant post-processing to achieve a usable, structured format. To address this, I’ve developed qjson, a Toolforge API designed for efficient and structured retrieval of all properties for a given Wikidata item.
qjson simplifies this process by handling the SPARQL query execution and structuring the response into a clean JSON format, complete with labels in the specified language.
Usage Examples:
- Fetching all data for Q405 (Moon) in English: https://qjson.toolforge.org/Q405.json
- Retrieving the same data in French: https://qjson.toolforge.org/Q405.json?lang=fr
To minimize unnecessary load on the Wikidata infrastructure, qjson implements response caching with a defined TTL. This ensures efficient data retrieval for frequently accessed items.
Source code: https://gitlab.wikimedia.org/santhosh/qjson. This tools is written in Go programming language.
Disclaimer
I work at the Wikimedia Foundation. However, this project, exploration, and the opinions expressed are entirely my own and do not reflect my employer’s views. This is not an official Wikimedia Foundation project.