Wikipedia Scraper

What this Actor scrapes

Wikipedia is the world's most-cited knowledge base and the go-to seed corpus for RAG pipelines, NLP benchmarks, and knowledge graphs. The official REST API at en.wikipedia.org/api/rest_v1/ is reliable, but it hands you one article at a time — no bulk mode, no redirect-following, no scheduling, no structured output. This Actor takes a list of titles or URLs (in any Wikipedia language), normalises them, follows redirects, retries on transient failures, and writes one clean row per article: summary, plain-text body, infobox data, references, categories, and lead image.

Infobox-preservation is the feature most Wikipedia scraper tools skip because parsing them is genuinely fiddly. Structured facts — birth dates, populations, capitals, taxonomic ranks — are what make a Wikipedia-grounded RAG useful for question-answering, not just paragraph retrieval. We keep them.

What we handle for you

• 🛡️ Browser fingerprint rotation — curl-cffi impersonates real Chrome / Firefox / Safari TLS handshakes so the upstream sees a real browser client, not a Python script.
• 🌐 Proxy rotation via Apify Proxy — fresh session and exit IP on every block or throttle response.
• 🔁 Retries with exponential backoff — up to 5 attempts per article on 408 / 429 / 5xx, with Retry-After headers honoured precisely.
• 🧱 Rate-limit-aware pacing — when the upstream pushes back we slow down and surface partial progress; we never silently return an empty dataset.
• 🧊 Clean, typed dataset rows — Pydantic-validated, ISO-8601 timestamps, stable IDs, JSON / CSV / Excel export straight from the Apify Console.
• 💰 Pay-Per-Event pricing — you pay only for results that land in your dataset. No data, no charge.

Use cases

• RAG corpus seeding — download Wikipedia articles bulk for a domain-specific knowledge base (Renaissance art, medical terminology, legal concepts) and load directly into LangChain, LlamaIndex, Chroma, or Weaviate.
• Wikipedia dataset for RAG — build a refreshable article corpus that stays current without downloading the 100 GB monthly XML dump.
• Multilingual Wikipedia dataset — fetch the same article across 10+ languages, ID-aligned, for cross-lingual evaluation or translation benchmarks.
• Wikipedia infobox extraction — pull structured facts (dates, coordinates, taxonomy, population) that most Wikipedia scraper tools discard.
• Definition harvesting — pull the lead sentence for every term in a glossary or ontology.
• Change monitoring — schedule weekly runs and diff last_modified timestamps to detect article updates.
• Wikipedia text extraction API — replace ad-hoc wikipedia Python library calls with a managed, scalable pipeline that handles retries and output formatting for you.

Input

Paste this into the Apify Console, or send it as the run input over the API. Proxy settings are on by default; you rarely need to touch them.

{
  "titles": [
    "Web scraping",
    "Natural language processing",
    "Retrieval-augmented generation"
  ],
  "language": "en",
  "includeFullText": true,
  "includeReferences": false,
  "concurrency": 4,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Output

One row per result, schema-validated before it is written. Export JSON, CSV, Excel or XML from the run, or read it over the API.

titlepageidlanguageurlsummarydescriptionextract_htmlfulltextthumbnail_urloriginal_image_urlcategoriesreferenceslast_modifiedscraped_at

{
  "title": "Web scraping",
  "pageid": 1323566,
  "language": "en",
  "url": "https://en.wikipedia.org/wiki/Web_scraping",
  "description": "Data extraction from websites",
  "summary": "Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites.",
  "thumbnail_url": "https://upload.wikimedia.org/wikipedia/commons/thumb/a/ae/Web_scraping.png/320px-Web_scraping.png",
  "categories": [
    "Web scraping",
    "Data mining",
    "Internet privacy"
  ],
  "last_modified": "2025-04-12T08:34:21Z",
  "scraped_at": "2026-06-01T10:00:00Z"
}

Pricing

You pay only for results that land. Cap any run with maxTotalChargeUsd. See pricing & billing for worked examples.

Limitations

• Current revision only — we pull the live version of each article. Version history and revision diffs are a separate API surface and are out of scope here.
• Infobox structured parsing — the Wikipedia REST API does not expose infobox fields as clean key-value JSON. We capture the infobox HTML where present; for deeply structured infobox facts use the Wikidata API alongside this Actor.
• Concurrency ceiling — we cap concurrency at 16 to stay within polite bot-policy bounds. Very large batches (50k+ articles) run fine; they just take longer than an aggressive parallelised approach would.
• Redirect chains — we follow one redirect hop. Circular redirects or chains longer than three hops are logged and skipped; the article title is still written to the dataset with a null body so you can see what was missed.
• Not for real-time monitoring — Apify runs are asynchronous. For live change detection, schedule runs via the Apify Scheduler rather than polling the Actor directly.