XML Sitemap URL Extractor

What this Actor scrapes

Give it a list of site URLs and it discovers every sitemap the site publishes — first by trying /sitemap.xml directly, then by parsing Sitemap: lines out of robots.txt (a 200 response on either is not treated as proof of a real sitemap; the Actor checks the Content-Type and looks like real sitemap XML before it commits to parsing it). Sitemap index files (<sitemapindex>) are expanded recursively up to a configurable depth, gzip-compressed sitemaps (.xml.gz) are decompressed transparently, and redirects are followed. Every leaf <url> entry becomes one dataset row tagged with the sitemap file and source site it came from.

What we handle for you

• 🛡️ Browser fingerprint rotation — curl-cffi impersonates real Chrome / Firefox / Safari TLS handshakes so the target sees a browser, not Python.
• 🌐 Residential proxy rotation via Apify Proxy — fresh session and exit IP on every block.
• 🔁 Retries with exponential backoff on 408 / 429 / 5xx — up to 5 attempts per page, Retry-After honoured.
• 🧱 Rate-limit-aware pacing — when the target pushes back, we slow down instead of getting banned.
• 🧊 Clean, typed dataset rows — Pydantic-validated, ISO-8601 timestamps, stable IDs, JSON / CSV / Excel export straight from the Apify Console.
• 💰 Pay-Per-Event pricing — you only pay for results that hit your dataset. No data, no charge.

Use cases

• SEO content audits — pull a site's full published-URL inventory in one pass, no crawling required.
• Site migration checks — diff sitemap output before/after a replatform to confirm nothing silently dropped.
• Competitive research — see how many pages a competitor publishes and how often they update (lastmod, changefreq).
• Freshness monitoring — schedule runs and diff lastmod values to catch stale or newly-published pages.

Input

Paste this into the Apify Console, or send it as the run input over the API. Proxy settings are on by default; you rarely need to touch them.

{
  "siteUrls": [
    "https://www.cloudflare.com",
    "https://apify.com"
  ],
  "maxUrlsPerSite": 10,
  "maxSitemapDepth": 3,
  "maxTotalUrls": 20,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

Output

One row per result, schema-validated before it is written. Export JSON, CSV, Excel or XML from the run, or read it over the API.

loclastmodchangefreqprioritysitemap_urlsource_site

{
  "loc": "https://apify.com/about",
  "lastmod": null,
  "changefreq": null,
  "priority": null,
  "sitemap_url": "https://apify.com/sitemap/pages.xml",
  "source_site": "https://apify.com"
}

Pricing

You pay only for results that land. Cap any run with maxTotalChargeUsd. See pricing & billing for worked examples.

Limitations

• We read what a site's sitemap files declare — we don't crawl or discover pages a site doesn't list in a sitemap.
• The standard sitemap protocol caps a single leaf file at 50,000 URLs; large sites split across many files via a sitemap index, which we follow up to maxSitemapDepth.
• maxUrlsPerSite and maxTotalUrls are hard caps by design (bandwidth control) — raise them if you need a fuller export, within the input limits.