Export Formats and Data Quality in Devil Scrapes Datasets

Every Devil Scrapes Actor writes to the same kind of storage — an Apify dataset, one row per result — and every row is built from a Pydantic model before it’s written, not assembled as a loose dict. That discipline is what makes the output consistent enough to script against without reading each Actor’s source first. This page covers the export formats available and the conventions the rows follow.

Export formats

GET /v2/datasets/{datasetId}/items?format=<format> supports:

Format	Best for
json	Nested structures, scripting, APIs
jsonl	Streaming large exports line-by-line without loading the whole file
csv	Spreadsheets, quick manual review
xlsx	Sharing with non-technical stakeholders
xml	Legacy systems that expect XML feeds
rss	Feed readers, monitoring a dataset as a subscribable feed
html	A human-readable table view, no tooling required

Add clean=1 to any of these to drop Apify’s internal bookkeeping fields (#debug, request metadata) and get back just your result rows. The Console’s Export button on a dataset’s page wraps the same endpoint — picking a format there and downloading is equivalent to hitting the URL yourself.

curl "https://api.apify.com/v2/datasets/<id>/items?format=jsonl&clean=1" -o results.jsonl

Field naming conventions

Every Actor defines its output rows as a Pydantic ResultRow model, and the field names on that model are the field names you get back — there’s no separate “display name” layer that diverges from the raw data. Across our fleet you’ll see:

• camelCase or snake_case, consistently within a single Actor’s schema (check that Actor’s README Output table for the exact casing it uses).
• Descriptive, unabbreviated names — advertiserName, landingUrl, firstShownAt, not adv_nm or url1.
• Nested objects for structured sub-data (a list of tags, a badges array) rather than flattened numbered columns like tag1, tag2, tag3 — this is most visible in the JSON/JSONL exports; the CSV/XLSX exports flatten nested fields into a serialized string in a single column, since spreadsheet formats don’t have a native nested-array shape.

Timestamps: always ISO-8601

Any date or datetime field in a Devil Scrapes dataset row is ISO-8601 (2026-09-09T14:32:00.000Z), always UTC unless the field name says otherwise. This is a direct consequence of model_dump(mode="json") on a Pydantic model with a native datetime field — Pydantic serializes datetimes to ISO-8601 strings automatically, so every Actor gets the same behavior for free rather than each one hand-rolling its own date formatting. If you’re parsing rows in Python, datetime.fromisoformat() (Python 3.11+) reads these directly; in JavaScript, new Date(field) does the same.

Stable IDs

Where the source data has a natural unique identifier — a post ID, a video ID, an advertiser ID, a listing ID — we carry it through into the row rather than inventing our own. That’s what makes deduplication across repeated runs straightforward: match on the source ID, not on a hash of the whole row (which breaks the moment any field changes value, e.g. a view count or a last-updated timestamp that legitimately changes between runs of the same item).

Where no natural ID exists on the source (a scraped page with no obvious primary key), the Actor’s README Output table will say so explicitly — check there before assuming a field is stable across runs.

Deduplication tips

• Prefer the source ID as your dedup key over any locally hashed value, for the reason above.
• Use a named dataset (Actor.open_dataset(name="...") under the hood, but from your side this just means asking the Actor to reuse a fixed dataset across scheduled runs rather than getting a fresh one every time) if you’re accumulating results over multiple scheduled runs and want Apify itself, rather than your downstream code, to be the single place holding the running set.
• De-dup on load, not on write, if you’re loading into a warehouse — an INSERT ... ON CONFLICT (or your database’s equivalent upsert) keyed on the source ID handles re-runs cleanly without the Actor needing to know your history.

Handling nulls

Pydantic models mark genuinely optional fields as T | None = None rather than omitting them from the row entirely — so a field that’s sometimes missing from the source still appears in every row, just as null when absent, rather than sometimes-present-sometimes-missing-from-the-JSON-object. That consistency matters for anything that loads rows into a fixed-schema destination (a SQL table, a strict JSON schema validator downstream) — you can rely on the key always being there, even when the value is null.

Don’t confuse a null field with a failed row: a null on an optional field means the source genuinely didn’t have that data (a listing with no price, a profile with no bio) — the row itself is still a validated, real result.

FAQ

Which export format should I use for a spreadsheet?

CSV or XLSX. XLSX is friendlier for non-technical stakeholders (native Excel formatting); CSV is the more universal lowest-common-denominator if the destination tool just needs plain text.

Why are nested fields flattened into a single column in my CSV export?

CSV has no native representation for nested arrays or objects. Use the JSON or JSONL export instead if you need the nested structure intact.

How do I know if a field is safe to use as a stable dedup key?

Check the Actor’s README Output table — it documents which field, if any, carries the source’s natural unique ID. If none is documented, treat row identity as tied to that specific run rather than durable across runs.

Are all timestamps in UTC?

Yes, unless a field name explicitly says otherwise (e.g. a localTime field derived from the source’s own stated timezone). Default assume UTC and convert on your side if you need a different zone.

What does a null value mean in a result row?

It means the field is genuinely optional and the source didn’t have a value for it on that particular row — not that the row failed to scrape. Required fields are never null; a row missing a required field wouldn’t pass the Actor’s output validation and wouldn’t be written at all.