Google AI Overview Citation Scraper

What this Actor scrapes

For each query you pass in, this Actor:

1. Opens https://www.google.com/search?q=<query>&hl=<language>&gl=<country> in a fresh Camoufox page. 2. Dismisses any EU consent dialog automatically. 3. Waits 4–15 seconds for the AI Overview block to lazy-render. 4. Probes an 8-selector priority battery to find the AI Overview container, recording which selector matched on every row — so you can detect Google rotating their markup with a single GROUP BY selector_used query. 5. Extracts every citation link inside the carousel: URL, registrable domain, anchor text, and 1-based position. 6. Emits one row per citation, or — when AI Overview did not appear — a single marker row with ai_overview_appeared=false. Absence of AI Overview is itself a meaningful AEO data point.

What we handle for you

• Camoufox-rendered, not headless Chromium — we use a Firefox fork with anti-detection patches. Standard Playwright and Selenium emit fingerprints that Google's defences pick off instantly; Camoufox is the only browser we allow for scraping (ADR-0002).
• We rotate residential proxies on every block — fresh session_id, fresh exit IP via Apify Proxy RESIDENTIAL. You never fight for bandwidth against someone else's CAPTCHA loop.
• We handle CAPTCHA interstitials — when Google serves the sorry/index reCAPTCHA page, we rotate the proxy session and retry before emitting a marker row. The run never silently returns an empty dataset.
• 8-selector priority battery with drift telemetry — Google has rotated the AI Overview DOM label at least three times since launch. We maintain an ordered selector list, probe all eight on each page, and record the winner in selector_used so you can chart selector drift over time.
• We retry with exponential backoff on network failures and rate-limit responses. Up to five attempts per query before we surface a partial-success status.
• Per-query session isolation — fresh proxy session and fresh browser page per query. Cookies and rate-limit state from one query never bleed into the next.
• Pydantic v2 input + output validation — invalid input fails fast before any browser starts; row schema is enforced at push time. You get typed columns, not a bag of string fields.
• Pay-Per-Event pricing — $0.05 start + $0.005 per row. No data, no charge beyond the warm-up fee.

Use cases

• AEO dashboard — schedule a weekly run for your 50 highest-priority queries; chart source_domain share-of-citation over time alongside ai_overview_appeared rate. Detect when AI Overview starts citing a new competitor in your space.
• Pre-launch content gap analysis — feed in the queries you want to rank for, see which domains Google currently cites, and target outreach to publishers in the cite list rather than chasing pure backlink volume.
• Brand citation monitoring — does AI Overview cite your domain for queries where your brand is the answer? Most brands have zero instrumentation here today; this is the direct way to find out.
• Competitive intelligence — track exactly which 3–5 sources Google's generative system trusts for each of your category's head queries; compare to traditional SERP rank.
• Selector drift monitoring — the selector_used column is a leading indicator of Google rotating AI Overview markup. Useful for SaaS observability of generative search behaviour even if you don't mine the citation data directly.
• Localised AEO — pair country=us and country=gb runs over the same query list to detect locale-specific citation behaviour.

Input

Paste this into the Apify Console, or send it as the run input over the API. Proxy settings are on by default; you rarely need to touch them.

{
  "queries": [
    "best running shoes 2026",
    "what causes inflation"
  ],
  "country": "us",
  "language": "en",
  "maxQueries": 25,
  "waitMsAfterLoad": 8000,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}

Output

One row per result, schema-validated before it is written. Export JSON, CSV, Excel or XML from the run, or read it over the API.

querycountrylanguageai_overview_appearedai_overview_text_excerptcitation_positionsource_domainsource_urlsource_titleselector_usedblocked_by_captchaquery_errorscraped_at

{
  "query": "what causes inflation",
  "country": "us",
  "language": "en",
  "ai_overview_appeared": true,
  "ai_overview_text_excerpt": "Inflation is caused by a combination of demand-pull factors, cost-push factors...",
  "citation_position": 1,
  "source_domain": "imf.org",
  "source_url": "https://www.imf.org/en/Publications/fandd/issues/Series/Back-to-Basics/Inflation",
  "source_title": "Inflation: Prices on the Rise",
  "selector_used": "div[aria-label=\"AI Overview\"]",
  "blocked_by_captcha": false,
  "scraped_at": "2026-05-16T20:50:00.000Z"
}

Pricing

You pay only for results that land. Cap any run with maxTotalChargeUsd. See pricing & billing for worked examples.

Limitations

• AI Overview triggers on ~30% of queries today. Queries that look transactional, navigational, or trademark-heavy will mostly produce ai_overview_appeared=false marker rows. That absence data is still valuable — and you're charged the same per row either way.
• v0.1 is English-tuned. The text-based selector fallback looks for the literal string AI Overview. Non-English locales (e.g. gl=de) may emit false negatives on the fallback path. The CSS selector battery is locale-agnostic.
• Apify Proxy is required. Google blocks datacenter IPs without proxy enrichment. We retry proxy resolution automatically on transient exhaustion; if every attempt still fails, the run finishes with a clear status message instead of leaving you guessing.
• Mobile SERP is out of scope. Mobile AI Overview uses a different DOM structure; a separate Actor variant is planned.
• No following citation links. This Actor records the cited URL but does not visit it. Pair with a downstream HTTP scraper when you need destination content.