PubMed Scraper — Papers, DOI & MeSH to JSON

What this Actor scrapes

NCBI's E-utilities is the canonical gateway to PubMed's 36+ million records — and it punishes naive callers with hard rate limits, chained esearch → efetch calls, XML quirks, and intermittent 500s. This pubmed scraper turns a free-form search query into a fully typed dataset (PMID, title, authors, abstract, journal, DOI, MeSH terms, publication types, author-supplied keywords, full citation URL) and absorbs every piece of upstream friction: paged fetches, backoff on 429s, transient-error retries, XML-to-JSON coercion. Provide your NCBI API key to lift throughput from 3 req/s to 10 req/s; either way the rows come out identical — clean and consistent.

This is a research and metadata tool only — abstracts, titles, identifiers, and controlled vocabulary. It never touches patient records, never claims to be clinical decision support, and deliberately does not fetch full text (full-text licensing lives on the publisher's side, not PubMed's). We scrape what PubMed openly indexes.

What we handle for you

• 🛡️ Browser fingerprint rotation — curl-cffi impersonates real Chrome / Firefox / Safari TLS handshakes so the target sees a browser, not Python.
• 🌐 Residential proxy rotation via Apify Proxy — fresh session and exit IP on every block.
• 🔁 Retries with exponential backoff on 408 / 429 / 5xx — up to 5 attempts per page, Retry-After honoured.
• 🧱 Rate-limit-aware pacing — when NCBI pushes back, we slow down, surface a status message, and keep going. You never get a silent empty dataset.
• 🧊 Clean, typed dataset rows — Pydantic-validated, ISO-8601 timestamps, stable IDs, JSON / CSV / Excel export straight from the Apify Console.
• 💰 Pay-Per-Event pricing — you only pay for results that hit your dataset. No data, no charge.

Use cases

• Clinical RAG pipelines — pull fresh PubMed metadata on a schedule and embed abstracts into a vector store for a medical-literature chatbot or pharmacovigilance alert.
• Literature reviews and meta-analyses — retrieve every paper matching a topic + date range in one run; export to CSV for your review management tool.
• Pharma competitive intel — track new mentions of a drug, compound, or trial ID across PubMed as they appear.
• Author publication monitoring — daily [Author] diff to feed a personal or departmental RSS-style alert.
• MeSH-based corpus assembly — extract every paper tagged with specific MeSH headings to build a training corpus or annotation benchmark.
• Bulk PubMed dataset download — run a broad query (e.g. "CRISPR"[MeSH] AND 2020:2025[PDat]) and export thousands of records in a single job.

Input

Paste this into the Apify Console, or send it as the run input over the API. Proxy settings are on by default; you rarely need to touch them.

{
  "searchQuery": "crispr review 2024",
  "maxResults": 50,
  "sortBy": "most_recent",
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Output

One row per result, schema-validated before it is written. Export JSON, CSV, Excel or XML from the run, or read it over the API.

pmidpmciddoititleabstractauthorsjournaljournal_isopublication_typesmesh_termskeywordspub_datepubmed_urlscraped_at

{
  "pmid": "39000123",
  "pmcid": "PMC11234567",
  "doi": "10.1038/s41573-024-00901-3",
  "title": "Advances in CRISPR-Cas12a therapeutics \u2014 a 2024 review",
  "abstract": "CRISPR-based gene editing has matured rapidly ...",
  "authors": [
    "Smith J",
    "Patel R",
    "Chen W"
  ],
  "journal": "Nature Reviews Drug Discovery",
  "journal_iso": "Nat Rev Drug Discov",
  "publication_types": [
    "Review",
    "Journal Article"
  ],
  "mesh_terms": [
    "CRISPR-Cas Systems",
    "Gene Editing",
    "Therapeutics"
  ],
  "keywords": [
    "CRISPR",
    "gene therapy",
    "Cas12a"
  ],
  "pub_date": "2024-03-01",
  "pubmed_url": "https://pubmed.ncbi.nlm.nih.gov/39000123/",
  "scraped_at": "2026-06-01T09:12:00Z"
}

Pricing

You pay only for results that land. Cap any run with maxTotalChargeUsd. See pricing & billing for worked examples.

Limitations

• Metadata only — this Actor hits E-utilities (esearch + efetch). Full text lives on publisher sites and is out of scope. The pmcid field gives you a pointer to PubMed Central when the paper is openly available there.
• Citation graphs — which papers cite which — are not in scope. Use the iCite API for that.
• Older records — some fields (especially abstract, doi, mesh_terms) may be absent for pre-1970 records. The Actor surfaces null rather than fabricating data.
• NCBI rate limits — the Actor honours NCBI's stated quota (3 req/s without an API key, 10 req/s with one). We will not race past these limits; doing so gets the entire endpoint burned for everyone. Provide an apiKey for high-volume jobs.
• Patient data and PHI — PubMed indexes abstracts and metadata only. There is no patient data here, and this tool must not be used as clinical decision support.