Published 2026-08-20 · Tested 2026-08-19
Arcade.dev
FArcade.dev received 10 PASS votes and passed 2 of five agent surface checks. The clearest finding came from the find the exact limits task.
Three AI models, GPT 5.6 Sol, Claude Opus 5, and DeepSeek v4 Flash, each read Arcade.dev’s public documentation independently and attempted five first-hour developer jobs: authorize and call a tool, find the exact limits, recover from throttling, verify an access webhook, use the Python SDK.
No accounts, API calls, or code execution were used. Every verdict came from public pages and every published quotation passed a live verification check.
Freshness
How rechecks work- Category
- Agent frameworks & memory
- Tested
- Quotes verified
- Surface rechecked
No change since the test.
5 of 5 quoted passages still appear on the live pages.
53.8% · 43/80 · AI Agent Readiness Score · reading 30 pts · surface 50 pts
Agent surface checks · 20/50
The Reading Test
| Task | GPT 5.6 Sol | Opus 5 | DeepSeek v4F | Consensus |
|---|---|---|---|---|
| Authorize and call a tool | PASS | PASS | PASS | PASS |
| Find the exact limits | FAIL | FAIL | PARTIAL | FAIL |
| Recover from throttling | PASS | PASS | PASS | PASS |
| Verify an access webhook | PASS | PARTIAL | PASS | PASS |
| Use the Python SDK | PASS | PARTIAL | PASS | PASS |
docs platform: Nextra (unscored) · verified 2026-08-19
What to fix first
These 4 fixes could add up to 35 points to the AI Agent Readiness Score. The list ranks each fix by the points it would add. How the ranking works
- 1 +10 points llms-full.txt check failed
Found: HTTP 404 HTML, not plausible llms content.
Fix: Publish llms-full.txt at the docs root with the full text of every docs page in one plain-text file.
Evidence: docs.arcade.dev/
llms-full.txt (opens in a new tab) - 2 +10 points MCP server check failed
Found: Initialize returned HTTP 404 HTML.
Fix: Run a public MCP server for the docs that answers a JSON-RPC initialize call and offers a docs search tool.
Evidence: docs.arcade.dev/
mcp (opens in a new tab) - 3 +10 points Docs AI check failed
Found: The live docs expose search but no embedded Ask AI control.
Fix: Add an assistant to the docs site that answers questions from the docs and links to its sources.
- 4
Found: No docs page gives a general API rate limit or plan quotas. Plan quotas appear only on marketing pages.
Fix: Publish the API rate limit and each plan's quotas on one docs page, matching the current pricing page.
What the docs get right
- Authorize and call a tool: 3 PASS votes. The quickstart covers account and API-key prerequisites, Python installation, authorization waiting, execution, and uv run main.py.
- Recover from throttling: 3 PASS votes. The example identifies the error, converts retry_after_ms from milliseconds to seconds, and tells the caller to wait and retry.
- 2 of 5 agent surface checks. Present: llms.txt, markdown mirrors. Missing: llms-full.txt, an MCP server, docs AI.
Authorize and call a tool
PASSPASS consensus from 3 PASS.
The quickstart covers account and API-key prerequisites, Python installation, authorization waiting, execution, and uv run main.py. One page carries the whole path in Python, TypeScript, and Java: account signup, API key, uv add arcadepy, client construction, client.tools.authorize, client.auth.wait_for_completion, client.tools.execute, uv run main.py, and the expected terminal output. The quickstart provides a complete from-zero path: sign up, get API key via the linked page, install arcadepy, and call any Arcade-hosted tool via authorize/wait_for_completion/execute with a full working example.
Find the exact limits
FAILFAIL consensus from 1 PARTIAL, 2 FAIL.
The docs mention shared provider limits but give no exact API or plan numbers, and llms-full.txt returned 404. No page on docs.arcade.dev states a rate-limit number or a plan quota: the Arcade API Reference is three sentences with no limits section, api.arcade.dev/v1/swagger defines no 429 response for /v1/tools/execute or /v1/tools/authorize, and the only published figure (1,000 API calls per minute) lives on the marketing blog, whose free-tier quotas contradict the current pricing page. Only the audit-logs API rate limit (100 req/60s/IP) and MCP notification rate limit (60/min/client configurable 1-1000) are documented in the docs. No general tool-execution API rate limit or plan-level quotas appear anywhere in the docs; plan quotas are on the marketing site only.
Recover from throttling
PASSPASS consensus from 3 PASS.
The example identifies the error, converts retry_after_ms from milliseconds to seconds, and tells the caller to wait and retry. The error kind, the retry_after_ms field, the millisecond-to-second conversion, and the retry action are all on one page in three languages, though retry_after_ms is optional in the tool.Error schema and only the Java sample guards against its absence. The error-handling docs document UPSTREAM_RUNTIME_RATE_LIMIT as an OutputError.Kind enum value and show how to use retry_after_ms to wait before retrying, with code examples in Python, TypeScript, and Java.
Verify an access webhook
PASSPASS consensus from 2 PASS, 1 PARTIAL.
The dashboard guide creates the extension and hook configuration, while the webhook contract defines POST /access and both auth forms. Endpoints, payload tables, response codes, failure modes, and retry rules are fully specified and the bearer-token path is usable, but mTLS is one sentence with no certificate issuance, trust, or rotation detail, the canonical OpenAPI spec declares only bearerAuth, and the two setup pages name the dashboard location differently (Contextual Access versus Logic Extensions → Hook Points). The contextual access documentation covers the complete webhook extension lifecycle: OpenAPI spec, endpoint implementation (POST /access, /pre, /post), bearer token or mTLS authentication configurable in the Dashboard, hook configurations with scoping and failure modes, and runnable Go example servers.
Use the Python SDK
PASSPASS consensus from 2 PASS, 1 PARTIAL.
The quickstart and OpenAPI agree on tool_name, input, and user_id; tested .md variants returned HTML fallbacks. The official Python client is arcadepy (pip install arcadepy on the References page) and its tool_name/input/user_id arguments match the REST schemas.ExecuteToolRequest and schemas.AuthorizeToolRequest exactly, but the quickstart passes auth_response.id to wait_for_completion while the Authorized Tool Calling, FAQ, and error-handling pages pass the auth_response object, and no page says which is correct. The Python SDK (arcadepy) is documented with an install command (uv add arcadepy), import example, and a complete authorize-and-execute flow that matches the tool API form used consistently throughout the docs.
The receipt
your authentications will share any rate limits from those providers with other Arcade customers.
The docs mention shared provider limits but give no exact API or plan numbers, and llms-full.txt returned 404.
Agent surface notes
Initialize returned HTTP 404 HTML.
The live docs expose search but no embedded Ask AI control.
Show the score
Paste this into a readme:
[](https://docsforagents.com/reports/arcade-dev-docs-ai-agent-readiness/) Method note
This is a reading test of public documentation, not an execution test. No accounts were created and no API calls were run. The AI Agent Readiness Score counts fifteen reading votes at PASS 2, PARTIAL 1, and FAIL 0, for 30 possible points. Five agent surface checks add 10 points each. The total is 80. Consensus chips show each row majority and do not affect scoring. The panel split on 3 of five tasks. Quotes shown here were re-fetched and confirmed verbatim on 2026-08-19.
Put another docs site through the battery.
Nominate a docs site