DevilScrapes
Wayback Machine CDX Scraper icon

Developer tools

Wayback Machine CDX Scraper

Query the Internet Archive's Wayback Machine CDX index at scale -- every URL a domain ever had captured, filtered by status code, MIME type, and date range, paginated honestly with a stable validated schema. Index discovery, not single-snapshot fetching.

Run it on Apify ↗ See input & output

Free Apify credit covers a first run. No credit card to try.

What this Actor scrapes

The Internet Archive publishes a CDX Server API (<code>web.archive.org/cdx/search/cdx</code>) that indexes every snapshot it has ever taken of a URL, host, or domain — timestamp, original URL, MIME type, HTTP status, content digest, and length. This Actor drives that API for you: it builds the right <code>matchType</code>/<code>filter</code>/<code>collapse</code>/date-range query, walks <code>resumeKey</code> pagination across tens of thousands of rows without losing a page, and writes one clean row per capture. Point it at a domain to rebuild your full historical URL inventory (handy after a migration, when pages that used to rank 301 into oblivion), or at a single page to trace every version the Archive ever saw.

What we handle for you

  • 🔁 We retry with exponential backoff on 408 / 429 / 503 and honour Retry-After — up to 5 attempts per page.
  • 🧭 We paginate correctly — resumeKey chained page by page, stopping the instant a short page or your maxResults cap says we're done, so you never get truncated or duplicated rows.
  • 🧱 We fault-isolate bad pages — one page that fails after retries gets skipped and logged; it never kills the whole run.
  • 🧊 We keep the dataset clean — Pydantic-validated rows, fixed-position column mapping (never guessed from headers), ISO-8601 timestamps.
  • 💰 You pay only for results that land. No data → no charge beyond the small actor-start warm-up fee.

Use cases

  • Post-migration URL recovery — rebuild the full list of URLs a domain ever had so you can 301 every one of them correctly instead of guessing.
  • OSINT / domain diligence — reconstruct a site's publishing history: when pages appeared, what they served, when they changed.
  • Historical content audits — filter by status code and MIME type to find every HTML page that was once live under a domain.
  • Change monitoring — pull digests over a date range to see exactly when a page's content last changed.

Input

Paste this into the Apify Console, or send it as the run input over the API. Proxy settings are on by default; you rarely need to touch them.

FieldTypeRequiredWhat it does
url string yes Target URL, host, or domain to look up in the Wayback Machine CDX index, e.g. "stripe.com" or "https://stripe.com/pricing".
matchType string no How the URL is matched: an exact URL, a URL prefix, an exact host, or the whole domain (every subdomain and path — the "recover my whole site's history" default).
dateFrom string no Start of the capture date range, digits only (YYYY through YYYYMMDDhhmmss, partial allowed). Leave blank for no lower bound.
dateTo string no End of the capture date range, digits only (YYYY through YYYYMMDDhhmmss, partial allowed). Leave blank for no upper bound.
filters array no Repeatable CDX filter expressions, e.g. "statuscode:200", "!statuscode:404", "mimetype:text/html".
collapse string no CDX collapse expression to de-duplicate adjacent rows sharing a field, e.g. "urlkey" or "timestamp:8". Leave blank to disable.
pageSize integer no Rows requested per CDX page (the API's "limit" parameter).
maxResults integer no Hard cap on total rows fetched across all pages — protects spend and compute. 
{
  "url": "stripe.com",
  "matchType": "domain",
  "pageSize": 5,
  "maxResults": 5,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Output

One row per result, schema-validated before it is written. Export JSON, CSV, Excel or XML from the run, or read it over the API.

urlkeytimestampcapture_dateoriginal_urlmimetypestatuscodedigestlengtharchive_urlscraped_at 

{
  "urlkey": "com,stripe)/",
  "timestamp": "20260815120133",
  "capture_date": "2026-08-15T12:01:33+00:00",
  "original_url": "https://stripe.com/",
  "mimetype": "text/html",
  "statuscode": "200",
  "digest": "3I42H3S6NNFQ2MSVX7XZKYAYSCX5QBYJ",
  "length": 18452,
  "archive_url": "https://web.archive.org/web/20260815120133/https://stripe.com/",
  "scraped_at": "2026-09-01T12:00:00+00:00"
}

Pricing

EventPriceWhen
Actor start$0.05Once per run, covers warm-up and proxy session setup.
Snapshot scraped$0.0020Per result written to the dataset.

You pay only for results that land. Cap any run with maxTotalChargeUsd. See pricing & billing for worked examples.

FAQ

Does this fetch the archived page content?
No — this Actor returns the CDX index (capture metadata: timestamp, status, MIME type, digest, length), not rendered HTML. Each row's archive_url is a direct link you can fetch yourself or feed into a follow-up scrape.
What does "domain" match type actually match?
domain returns every URL under the target domain across all subdomains and paths — the flagship "recover my whole site's history" use case. Use host to stay on one hostname, prefix for a URL prefix, or exact for one exact URL.
Why are some `statuscode` or `length` values null?
The Internet Archive records - for some captures (redirect-only or malformed responses). The Actor converts those to null rather than treating them as errors.
What happens if my query matches nothing?
The run finishes successfully with zero rows and a status message describing exactly what was searched — a narrow filter with no matches is not a failure.

Ready to run it?

Open the listing on Apify, paste the input above, and watch rows land. If it ever breaks, it is our problem before it is yours.

Run it on Apify ↗ Ask a question

Related Actors

Teams that run this also run