Robots.txt & Security.txt Compliance Auditor

What this Actor scrapes

Feed this Actor a list of sites and it fetches four well-known metadata files per site — robots.txt, security.txt (RFC 9116), ads.txt, and humans.txt — and parses them into one structured audit row: crawl-policy rules, named AI-crawler posture (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider), security-contact/expiry data, and ads.txt record counts. Buyers are SEO/compliance teams, ad-ops teams vetting ad-tech supply chains, and publishers checking their own AI-bot policy — a recurring audit teams re-run on a schedule. No API key needed anywhere; all four paths are plain-text, no-auth files.

What we handle for you

• 🔍 Soft-404 detection — a custom-branded "page not found" that still answers HTTP 200 doesn't fool us; we check Content-Type and the file's own directive syntax before calling it present.
• 🤖 Named AI-crawler posture — one line per site tells you exactly whether GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, and Bytespider are blocked, allowed, or simply never mentioned.
• 🔁 RFC 9116 well-known-then-legacy fallback — we check /.well-known/security.txt first and fall back to the legacy /security.txt path automatically.
• 🧱 Per-file fault isolation — one file missing or erroring never sabotages the other three; you get a full row with clear markers instead of a failed run.
• 🧊 Clean, typed dataset rows — Pydantic-validated, ISO-8601 timestamps, ready for CSV/JSON/Excel export.
• 💰 Pay-Per-Event pricing — you only pay for sites where at least one file resolved. No data, no charge beyond the warm-up fee.

Use cases

• SEO / technical-SEO audits — check crawl-policy and AI-crawler exposure across your whole site portfolio in one run.
• Ad-tech / ad-ops compliance — validate ads.txt supply-chain records before flagging fraud risk.
• Security compliance — track vulnerability-disclosure contact and expiry dates across your infrastructure.
• Competitive / publisher research — see which AI crawlers competitors block or allow.

Input

Paste this into the Apify Console, or send it as the run input over the API. Proxy settings are on by default; you rarely need to touch them.

{
  "sites": [
    "www.cloudflare.com",
    "github.com"
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

Output

One row per result, schema-validated before it is written. Export JSON, CSV, Excel or XML from the run, or read it over the API.

siteaudited_atrobots_txtsecurity_txtads_txthumans_txtai_crawler_summarysecurity_txt_expiredhas_wildcard_disallowfetch_errors

{
  "site": "www.cloudflare.com",
  "audited_at": "2026-09-05T12:00:00Z",
  "robots_txt": {
    "exists": true,
    "fetch_status": "present",
    "user_agent_groups": [
      {
        "agent": "GPTBot",
        "allow": [],
        "disallow": [
          "/"
        ],
        "crawl_delay": null
      }
    ],
    "sitemaps": [
      "https://www.cloudflare.com/sitemap.xml"
    ],
    "ai_crawler_posture": {
      "GPTBot": "blocked",
      "ClaudeBot": "not_mentioned",
      "CCBot": "not_mentioned",
      "Google-Extended": "not_mentioned",
      "PerplexityBot": "not_mentioned",
      "Bytespider": "not_mentioned"
    }
  },
  "security_txt": {
    "exists": true,
    "location": "well_known",
    "fetch_status": "present",
    "contacts": [
      "mailto:security@cloudflare.com"
    ],
    "expires": null,
    "expires_parsed": null,
    "expires_in_past": false,
    "policy_urls": []
  },
  "ads_txt": {
    "exists": false,
    "fetch_status": "absent",
    "record_count": 0
  },
  "humans_txt": {
    "exists": false,
    "fetch_status": "absent",
    "excerpt": null
  },
  "ai_crawler_summary": {
    "blocked": [
      "GPTBot"
    ],
    "allowed": [],
    "not_mentioned": [
      "ClaudeBot",
      "CCBot",
      "Google-Extended",
      "PerplexityBot",
      "Bytespider"
    ],
    "blocked_count": 1,
    "allowed_count": 0,
    "not_mentioned_count": 5
  },
  "security_txt_expired": false,
  "has_wildcard_disallow": false,
  "fetch_errors": []
}

Pricing

You pay only for results that land. Cap any run with maxTotalChargeUsd. See pricing & billing for worked examples.

Limitations

• No path-level Allow/Disallow matching against arbitrary customer URLs — only the raw rule dump and named-crawler posture.
• No parsing of non-standard robots.txt directives (Host:, Clean-param:, Noindex:).
• No security.txt PGP Encryption: fetch or signature verification.
• No following of Sitemap:, Policy:, or ads.txt owner/manager URLs — recorded as strings, never crawled.
• No plain-HTTP fallback — a host reachable only on port 80/no-TLS is treated as unreachable.
• No historical diffing/change-alerting between runs — each run is a stateless snapshot; schedule your own recurring runs to track change.
• Up to 500 sites per run — larger audits split across runs.