How Devil Scrapes Handles Proxies and Blocking

Scraping is an adversarial process: the sites we target actively try to detect and stop automated traffic. We built our whole fleet around absorbing that fight so you don’t have to think about it. This page explains, concretely, what we do on every run, what the proxyConfiguration field in an Actor’s input actually controls, and what to expect when a run comes back partial.

What we handle on every run

• 🛡️ We rotate browser fingerprints. Requests replay real Chrome, Firefox, and Safari TLS and HTTP/2 signatures in rotation, so the target sees traffic that looks like an actual browser, not a bare script.
• 🌐 We rotate proxies through Apify Proxy. On a block, we drop the session and pick up a fresh exit IP and a fresh session ID rather than hammering the same one.
• 🔁 We retry with exponential backoff. Transient failures (408, 429, 5xx, network errors) get retried automatically, honoring any Retry-After header the target sends, up to a bounded number of attempts.
• 🧱 We back off, not push through, when a target is genuinely rate-limiting. A run that hits a hard wall reports what it managed rather than silently truncating your dataset.
• 🧊 We keep the output clean regardless of what happened upstream — Pydantic-validated rows, ISO-8601 timestamps, stable IDs, no half-parsed junk rows from a page that came back in an unexpected shape.

You don’t configure any of this. It’s built into every Actor’s request layer before it ever ships.

What proxyConfiguration in the input actually does

Most of our Actors expose a proxyConfiguration field in the input form (the proxy editor type in Apify’s input schema). It’s how you tell the run which proxy pool to draw from:

{
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}

• useApifyProxy: true routes traffic through Apify Proxy instead of the run’s container IP directly.
• apifyProxyGroups picks which pool: ["RESIDENTIAL"] for IPs that look like real home connections, or a datacenter group for faster, cheaper IPs on targets that don’t scrutinize origin as closely.
• Leaving the field at its default is usually the right call — we set the default to whichever tier the target actually needs, based on what we measured while building the Actor, not a guess.

Residential vs. datacenter: when it matters

Datacenter IPs are faster and cheaper to route through; plenty of targets don’t care where a request originates as long as nothing else about it looks automated. Residential IPs cost more but present as ordinary consumer connections, which matters against targets that flag datacenter ranges regardless of how clean the rest of the request looks.

We pick the default tier per Actor based on what actually clears against that specific target, not a blanket “always residential” policy that would waste your proxy budget on targets that never needed it. Hitting an unusually strict corner of a target? Switching apifyProxyGroups to ["RESIDENTIAL"] is the first thing to try before reaching out to us.

Why a run can come back partial, and how we report it

Some runs finish with fewer results than you asked for — not because of a bug, but because the target pushed back partway through and we chose to stop cleanly rather than keep hammering a wall. When that happens, the run still reports SUCCEEDED (you got real data, just not the full amount), and the status message says exactly what happened:

Scraped 142/200 results before the target rate-limited.

That’s a deliberate design choice: silently returning less data than requested with no explanation would leave you debugging blind. A specific status message means you know whether to re-run later, narrow your input, or just accept the partial set as-is.

The other side of this rule matters just as much: a search that runs completely and legitimately matches nothing finishes SUCCEEDED with zero rows and a status message describing what was searched. Zero results is not the same thing as a block — treating a clean, empty answer as a failure would be its own kind of dishonesty.

What to do if you hit a block

1. Check the status message first — it tells you whether the run finished clean, partial, or failed outright.
2. Try widening the proxy tier to ["RESIDENTIAL"] if the input exposes proxyConfiguration and you’re not already there.
3. Re-run later if the block looks time-based rather than input-specific.
4. Report it if the same input fails consistently — see Maintenance and support, including the Apify Issues tab on the Actor’s Store page and our /contact/ form.

FAQ

Do I need to configure proxies myself?

No — every Actor ships with a sensible default proxyConfiguration for the target it scrapes. You only need to touch the field if you’re hitting an unusually strict corner of a site and want to try a stronger proxy tier.

Why did my run return fewer results than I asked for?

Most likely the target rate-limited partway through and we stopped cleanly instead of pushing further. Check the run’s status message — it states exactly how many results landed and why the run stopped there.

Is a zero-result run the same as a blocked run?

No. A genuinely empty match (the target answered, nothing matched your input) finishes SUCCEEDED with zero rows and a status message describing the search. A block that prevented the run from completing at all finishes differently and is treated as a bug we investigate — see Maintenance and support.

Can I choose which country my proxy exits from?

Some Actors expose country pinning inside proxyConfiguration (an apifyProxyCountry field) when geography affects the data returned. Check that Actor’s Input table on its Store page for whether the option is exposed.

What’s the difference between a “recoverable” retry and a real failure?

We retry automatically on transient signals — 408, 429, 5xx, dropped connections — up to a bounded number of attempts with backoff. A real failure is something the retry loop can’t fix on its own: a target that’s changed its page structure entirely, or a hard, sustained block. Those surface as a FAILED run with a specific error in the log, not a silent empty dataset.