Published 2026-08-20 · Tested 2026-08-19

Arcade.dev

F

Arcade.dev received 10 PASS votes and passed 2 of five agent surface checks. The clearest finding came from the find the exact limits task.

Panel: GPT 5.6 Sol, Opus 5, DeepSeek v4F Battery: v1 Read as markdown (opens in a new tab)

Three AI models, GPT 5.6 Sol, Claude Opus 5, and DeepSeek v4 Flash, each read Arcade.dev’s public documentation independently and attempted five first-hour developer jobs: authorize and call a tool, find the exact limits, recover from throttling, verify an access webhook, use the Python SDK.

No accounts, API calls, or code execution were used. Every verdict came from public pages and every published quotation passed a live verification check.

Freshness

How rechecks work
Tested
Quotes verified
Surface rechecked

No change since the test.

5 of 5 quoted passages still appear on the live pages.

Arcade.dev Nextra · published
F

53.8% · 43/80 · AI Agent Readiness Score · reading 30 pts · surface 50 pts

llms.txt PASS
llms-full.txt FAIL
markdown mirror PASS
MCP server FAIL
docs AI FAIL
Task GPT 5.6 SolOpus 5DeepSeek v4F Consensus
Authorize and call a tool PASSPASSPASS PASS
Find the exact limits FAILFAILPARTIAL FAIL
Recover from throttling PASSPASSPASS PASS
Verify an access webhook PASSPARTIALPASS PASS
Use the Python SDK PASSPARTIALPASS PASS

docs platform: Nextra (unscored) · verified 2026-08-19

What to fix first

These 4 fixes could add up to 35 points to the AI Agent Readiness Score. The list ranks each fix by the points it would add. How the ranking works

  1. 1
    +10 points llms-full.txt check failed

    Found: HTTP 404 HTML, not plausible llms content.

    Fix: Publish llms-full.txt at the docs root with the full text of every docs page in one plain-text file.

    Evidence: docs.arcade.dev/llms-full.txt (opens in a new tab)

  2. 2
    +10 points MCP server check failed

    Found: Initialize returned HTTP 404 HTML.

    Fix: Run a public MCP server for the docs that answers a JSON-RPC initialize call and offers a docs search tool.

    Evidence: docs.arcade.dev/mcp (opens in a new tab)

  3. 3
    +10 points Docs AI check failed

    Found: The live docs expose search but no embedded Ask AI control.

    Fix: Add an assistant to the docs site that answers questions from the docs and links to its sources.

  4. 4
    +5 points Find the exact limits FAIL

    Found: No docs page gives a general API rate limit or plan quotas. Plan quotas appear only on marketing pages.

    Fix: Publish the API rate limit and each plan's quotas on one docs page, matching the current pricing page.

    Evidence: www.arcade.dev/pricing (opens in a new tab)

What the docs get right

  • Authorize and call a tool: 3 PASS votes. The quickstart covers account and API-key prerequisites, Python installation, authorization waiting, execution, and uv run main.py.
  • Recover from throttling: 3 PASS votes. The example identifies the error, converts retry_after_ms from milliseconds to seconds, and tells the caller to wait and retry.
  • 2 of 5 agent surface checks. Present: llms.txt, markdown mirrors. Missing: llms-full.txt, an MCP server, docs AI.

Authorize and call a tool

PASS

PASS consensus from 3 PASS.

The quickstart covers account and API-key prerequisites, Python installation, authorization waiting, execution, and uv run main.py. One page carries the whole path in Python, TypeScript, and Java: account signup, API key, uv add arcadepy, client construction, client.tools.authorize, client.auth.wait_for_completion, client.tools.execute, uv run main.py, and the expected terminal output. The quickstart provides a complete from-zero path: sign up, get API key via the linked page, install arcadepy, and call any Arcade-hosted tool via authorize/wait_for_completion/execute with a full working example.

Find the exact limits

FAIL

FAIL consensus from 1 PARTIAL, 2 FAIL.

The docs mention shared provider limits but give no exact API or plan numbers, and llms-full.txt returned 404. No page on docs.arcade.dev states a rate-limit number or a plan quota: the Arcade API Reference is three sentences with no limits section, api.arcade.dev/v1/swagger defines no 429 response for /v1/tools/execute or /v1/tools/authorize, and the only published figure (1,000 API calls per minute) lives on the marketing blog, whose free-tier quotas contradict the current pricing page. Only the audit-logs API rate limit (100 req/60s/IP) and MCP notification rate limit (60/min/client configurable 1-1000) are documented in the docs. No general tool-execution API rate limit or plan-level quotas appear anywhere in the docs; plan quotas are on the marketing site only.

Recover from throttling

PASS

PASS consensus from 3 PASS.

The example identifies the error, converts retry_after_ms from milliseconds to seconds, and tells the caller to wait and retry. The error kind, the retry_after_ms field, the millisecond-to-second conversion, and the retry action are all on one page in three languages, though retry_after_ms is optional in the tool.Error schema and only the Java sample guards against its absence. The error-handling docs document UPSTREAM_RUNTIME_RATE_LIMIT as an OutputError.Kind enum value and show how to use retry_after_ms to wait before retrying, with code examples in Python, TypeScript, and Java.

Verify an access webhook

PASS

PASS consensus from 2 PASS, 1 PARTIAL.

The dashboard guide creates the extension and hook configuration, while the webhook contract defines POST /access and both auth forms. Endpoints, payload tables, response codes, failure modes, and retry rules are fully specified and the bearer-token path is usable, but mTLS is one sentence with no certificate issuance, trust, or rotation detail, the canonical OpenAPI spec declares only bearerAuth, and the two setup pages name the dashboard location differently (Contextual Access versus Logic Extensions → Hook Points). The contextual access documentation covers the complete webhook extension lifecycle: OpenAPI spec, endpoint implementation (POST /access, /pre, /post), bearer token or mTLS authentication configurable in the Dashboard, hook configurations with scoping and failure modes, and runnable Go example servers.

Use the Python SDK

PASS

PASS consensus from 2 PASS, 1 PARTIAL.

The quickstart and OpenAPI agree on tool_name, input, and user_id; tested .md variants returned HTML fallbacks. The official Python client is arcadepy (pip install arcadepy on the References page) and its tool_name/input/user_id arguments match the REST schemas.ExecuteToolRequest and schemas.AuthorizeToolRequest exactly, but the quickstart passes auth_response.id to wait_for_completion while the Authorized Tool Calling, FAQ, and error-handling pages pass the auth_response object, and no page says which is correct. The Python SDK (arcadepy) is documented with an install command (uv add arcadepy), import example, and a complete authorize-and-execute flow that matches the tool API form used consistently throughout the docs.

The receipt

your authentications will share any rate limits from those providers with other Arcade customers.

The docs mention shared provider limits but give no exact API or plan numbers, and llms-full.txt returned 404.

Agent surface notes

Initialize returned HTTP 404 HTML.

The live docs expose search but no embedded Ask AI control.

Show the score

AI Agent Readiness Score 53.8%, grade F

Paste this into a readme:

[![AI Agent Readiness Score 53.8%](https://docsforagents.com/badge/arcade-dev.svg)](https://docsforagents.com/reports/arcade-dev-docs-ai-agent-readiness/)

Method note

This is a reading test of public documentation, not an execution test. No accounts were created and no API calls were run. The AI Agent Readiness Score counts fifteen reading votes at PASS 2, PARTIAL 1, and FAIL 0, for 30 possible points. Five agent surface checks add 10 points each. The total is 80. Consensus chips show each row majority and do not affect scoring. The panel split on 3 of five tasks. Quotes shown here were re-fetched and confirmed verbatim on 2026-08-19.

Read the full methodology

Put another docs site through the battery.

Nominate a docs site