The allpages API module allows you to do just that. Its limit (when you set aplimit=max) is 500, so to query all 4.5M articles, you would need about 9000 requests. But a dump is a better choice, because there are many different dumps, including all-titles-in-ns0 which, as its name suggests, contains exactly what you want … Read more
Here are a few different possible approaches; use whichever works for you. All my code examples below use requests for HTTP requests to the API; you can install requests with pip install requests if you have Pip. They also all use the Mediawiki API, and two use the query endpoint; follow those links if you … Read more
You need to use the urllib2 that superseedes urllib in the python std library in order to change the user agent. Straight from the examples import urllib2 opener = urllib2.build_opener() opener.addheaders = [(‘User-agent’, ‘Mozilla/5.0’)] infile = opener.open(‘http://en.wikipedia.org/w/index.php?title=Albert_Einstein&printable=yes’) page = infile.read()
I wrote a Python library that aims to make this very easy. Check it out at Github. To install it, run $ pip install wikipedia Then to get the first paragraph of an article, just use the wikipedia.summary function. >>> import wikipedia >>> print wikipedia.summary(“Albert Einstein”, sentences=2) prints Albert Einstein (/ˈælbərt ˈaɪnstaɪn/; German: [ˈalbɐt ˈaɪnʃtaɪn] … Read more
You can just use a URL like this: http://en.wikipedia.org/?curid=18630637 This is the shortest form, others are also possible: http://en.wikipedia.org/wiki?curid=18630637 http://en.wikipedia.org/wiki/Translation?curid=18630637 http://en.wikipedia.org/w/index.php?curid=18630637 Note that MediaWiki ignores the page title if you specify a curid, so even http://en.wikipedia.org/wiki/FooBar?curid=18630637 leads to the same page.
MediaWiki’s API is running on Wikipedia (docs). You can also use the Special:Export feature to dump data and parse it yourself. More information.
There’s a way to get the entire “introduction section” without any HTML parsing! Similar to AnthonyS’s answer with an additional explaintext parameter, you can get the introduction section text in plain text. Query Getting Stack Overflow’s introduction in plain text: Using the page title: https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro&explaintext&redirects=1&titles=Stack%20Overflow Or use pageids: https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro&explaintext&redirects=1&pageids=21721040 JSON Response (warnings stripped) { “query”: … Read more