News & feeds · Social media

Substack Newsletter Scraper

Q: Can I get paywalled post bodies?

No. Paywalled posts require a paid subscription; body_html returns null for those. You get all public metadata (title, date, reaction count, etc.) regardless of paywall status.

Q: What if a publication has thousands of posts?

Set maxPostsPerPublication to 0 and the Actor will paginate to the full archive. Runs are priced per result, so longer archives cost proportionally more.

Full Substack publication archive via the public API — every post with body, reactions, comments and a paywall flag, with complete pagination handling. RAG-ready.

Run it on Apify ↗ See input & output

Free Apify credit covers a first run. No credit card to try.

What this Actor scrapes

Substack hosts thousands of newsletters across every niche — from tech and finance to culture and investigative journalism. This Actor taps each publication's public JSON API to extract the full post archive: metadata for every post, plus body HTML for free-tier content. Works with native Substack domains (author.substack.com) and custom domains alike.

What we handle for you

🛡️ Browser fingerprint rotation — curl-cffi impersonates real Chrome / Firefox / Safari TLS handshakes so the target sees a browser, not Python.
🌐 Residential proxy rotation via Apify Proxy — fresh session and exit IP on every block.
🔁 Retries with exponential backoff on 408 / 429 / 5xx — up to 5 attempts per page, Retry-After honoured.
🧱 Rate-limit-aware pacing — when the target pushes back, we slow down instead of getting banned.
🧊 Clean, typed dataset rows — Pydantic-validated, ISO-8601 timestamps, stable IDs, JSON / CSV / Excel export straight from the Apify Console.
💰 Pay-Per-Event pricing — you only pay for results that hit your dataset. No data, no charge.

Use cases

Newsletter intelligence — track a competitor publication's cadence, topics, and engagement over time.
Content research — aggregate posts across multiple newsletters on a niche to find trends and gaps.
Lead gen — identify high-engagement free posts to understand what resonates with a target audience.
Archive backfill — pull the full post history into a warehouse for analysis or ML training.
Paywall mapping — understand the free-vs-paid content split for any publication.

Input

Paste this into the Apify Console, or send it as the run input over the API. Proxy settings are on by default; you rarely need to touch them.

Field	Type	Required	What it does
`publications`	array	yes	One or more Substack publication URLs or custom domains. Accepts both native substack.com URLs and custom domains (e.g. https://www.lennysnewsletter.com).
`maxPostsPerPublication`	integer	no	Maximum number of posts to collect per publication. Set to 0 to collect the full archive.
`includeBody`	boolean	no	When enabled, body_html is included for free posts. Paywalled posts return null body.
`postedAfter`	string	no	ISO YYYY-MM-DD. Drop posts dated before this. The archive is walked newest-first, so pagination for a publication stops early once crossed.
`postedBefore`	string	no	ISO YYYY-MM-DD. Drop posts dated after this, without stopping pagination early — older, in-range posts may still follow. Must be on or after Posted after when both are set.

{
  "publications": [
    "https://www.lennysnewsletter.com"
  ],
  "maxPostsPerPublication": 5,
  "includeBody": false,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

Output

One row per result, schema-validated before it is written. Export JSON, CSV, Excel or XML from the run, or read it over the API.

publicationpost_idtitlesubtitleurlpost_dateaudienceis_paywalledreaction_countcomment_countword_countbody_html

{
  "publication": "https://www.lennysnewsletter.com",
  "post_id": 123456,
  "title": "The most important metric you're ignoring",
  "subtitle": "How to think about retention in your product",
  "url": "https://www.lennysnewsletter.com/p/the-most-important-metric",
  "post_date": "2024-03-15T09:00:00.000Z",
  "audience": "free",
  "is_paywalled": false,
  "reaction_count": 847,
  "comment_count": 43,
  "word_count": 2100,
  "body_html": null,
  "scraped_at": "2026-06-07T10:00:00Z"
}

Pricing

Event	Price	When
Actor start	$0.20	Once per run, covers warm-up and proxy session setup.
Result emitted	$0.0025	Per result written to the dataset.

You pay only for results that land. Cap any run with maxTotalChargeUsd. See pricing & billing for worked examples.

FAQ

Does this work with custom domains?

Yes. Pass the full base URL of the publication — whether it's author.substack.com or a custom domain like https://www.lennysnewsletter.com — and the Actor resolves the correct API endpoint.

Can I get paywalled post bodies?

No. Paywalled posts require a paid subscription; body_html returns null for those. You get all public metadata (title, date, reaction count, etc.) regardless of paywall status.

How far back does the archive go?

As far as the publication's API returns. We paginate until we hit maxPostsPerPublication or exhaust the archive, whichever comes first.

What if a publication has thousands of posts?

Set maxPostsPerPublication to 0 and the Actor will paginate to the full archive. Runs are priced per result, so longer archives cost proportionally more.

Ready to run it?

Open the listing on Apify, paste the input above, and watch rows land. If it ever breaks, it is our problem before it is yours.

Run it on Apify ↗ Ask a question

Related Actors

Teams that run this also run

$5.00/ 1k results