Published 2026-08-20 · Tested 2026-08-19

Arcade.dev

F

Arcade.dev received 10 PASS votes and passed 2 of five readiness checks. The clearest finding came from the find the exact limits task.

Panel: GPT 5.6 Sol, Opus 5, DeepSeek v4F Battery: v1 Verified: 2026-08-19 Read as markdown (opens in a new tab)

Three AI models, GPT 5.6 Sol, Claude Opus 5, and DeepSeek v4 Flash, each read Arcade.dev’s public documentation independently and attempted five first-hour developer jobs: authorize and call a tool, find the exact limits, recover from throttling, verify an access webhook, use the Python SDK.

No accounts, API calls, or code execution were used. Every verdict came from public pages and every published quotation passed a live verification check.

Arcade.dev Nextra · published 2026-08-20
F

53.8% · 43/80 · AI Agent Readiness Score · reading 30 pts · surface 50 pts

llms.txt PASS
llms-full.txt FAIL
markdown mirror PASS
MCP server FAIL
docs AI FAIL
Task GPT 5.6 SolOpus 5DeepSeek v4F Consensus
Authorize and call a tool PASSPASSPASS PASS
Find the exact limits FAILFAILPARTIAL FAIL
Recover from throttling PASSPASSPASS PASS
Verify an access webhook PASSPARTIALPASS PASS
Use the python sdk PASSPARTIALPASS PASS

docs platform: Nextra (unscored) · verified 2026-08-19

What the docs get right

  • Authorize and call a tool: 3 PASS votes. The quickstart covers account and API-key prerequisites, Python installation, authorization waiting, execution, and `uv run main.py`.
  • Recover from throttling: 3 PASS votes. The example identifies the error, converts `retry_after_ms` from milliseconds to seconds, and tells the caller to wait and retry.
  • 2 of 5 readiness checks. Present: llms.txt, markdown mirrors. Missing: llms-full.txt, an MCP server, docs AI.

Authorize and call a tool

PASS

PASS consensus from 3 PASS.

The quickstart covers account and API-key prerequisites, Python installation, authorization waiting, execution, and `uv run main.py`. One page carries the whole path in Python, TypeScript, and Java: account signup, API key, uv add arcadepy, client construction, client.tools.authorize, client.auth.wait_for_completion, client.tools.execute, uv run main.py, and the expected terminal output. The quickstart provides a complete from-zero path: sign up, get API key via the linked page, install arcadepy, and call any Arcade-hosted tool via authorize/wait_for_completion/execute with a full working example.

Find the exact limits

FAIL

FAIL consensus from 1 PARTIAL, 2 FAIL.

The docs mention shared provider limits but give no exact API or plan numbers, and `llms-full.txt` returned 404. No page on docs.arcade.dev states a rate-limit number or a plan quota: the Arcade API Reference is three sentences with no limits section, api.arcade.dev/v1/swagger defines no 429 response for /v1/tools/execute or /v1/tools/authorize, and the only published figure (1,000 API calls per minute) lives on the marketing blog, whose free-tier quotas contradict the current pricing page. Only the audit-logs API rate limit (100 req/60s/IP) and MCP notification rate limit (60/min/client configurable 1-1000) are documented in the docs. No general tool-execution API rate limit or plan-level quotas appear anywhere in the docs; plan quotas are on the marketing site only.

Recover from throttling

PASS

PASS consensus from 3 PASS.

The example identifies the error, converts `retry_after_ms` from milliseconds to seconds, and tells the caller to wait and retry. The error kind, the retry_after_ms field, the millisecond-to-second conversion, and the retry action are all on one page in three languages, though retry_after_ms is optional in the tool.Error schema and only the Java sample guards against its absence. The error-handling docs document UPSTREAM_RUNTIME_RATE_LIMIT as an OutputError.Kind enum value and show how to use retry_after_ms to wait before retrying, with code examples in Python, TypeScript, and Java.

Verify an access webhook

PASS

PASS consensus from 2 PASS, 1 PARTIAL.

The dashboard guide creates the extension and hook configuration, while the webhook contract defines `POST /access` and both auth forms. Endpoints, payload tables, response codes, failure modes, and retry rules are fully specified and the bearer-token path is usable, but mTLS is one sentence with no certificate issuance, trust, or rotation detail, the canonical OpenAPI spec declares only bearerAuth, and the two setup pages name the dashboard location differently (Contextual Access versus Logic Extensions → Hook Points). The contextual access documentation covers the complete webhook extension lifecycle: OpenAPI spec, endpoint implementation (POST /access, /pre, /post), bearer token or mTLS authentication configurable in the Dashboard, hook configurations with scoping and failure modes, and runnable Go example servers.

Use the python sdk

PASS

PASS consensus from 2 PASS, 1 PARTIAL.

The quickstart and OpenAPI agree on `tool_name`, `input`, and `user_id`; tested `.md` variants returned HTML fallbacks. The official Python client is arcadepy (pip install arcadepy on the References page) and its tool_name/input/user_id arguments match the REST schemas.ExecuteToolRequest and schemas.AuthorizeToolRequest exactly, but the quickstart passes auth_response.id to wait_for_completion while the Authorized Tool Calling, FAQ, and error-handling pages pass the auth_response object, and no page says which is correct. The Python SDK (arcadepy) is documented with an install command (uv add arcadepy), import example, and a complete authorize-and-execute flow that matches the tool API form used consistently throughout the docs.

The receipt

your authentications will share any rate limits from those providers with other Arcade customers.

The docs mention shared provider limits but give no exact API or plan numbers, and `llms-full.txt` returned 404.

Agent surface notes

Initialize returned HTTP 404 HTML.

The live docs expose search but no embedded Ask AI control.

Method note

This is a reading test of public documentation, not an execution test. No accounts were created and no API calls were run. The AI Agent Readiness Score counts fifteen reading votes at PASS 2, PARTIAL 1, and FAIL 0, for 30 possible points. Five readiness checks add 10 points each. The total is 80. Consensus chips show each row majority and do not affect scoring. The panel split on 3 of five tasks. Quotes shown here were re-fetched and confirmed verbatim on 2026-08-19.

Read the full methodology

Put another docs site through the battery.

Nominate a docs site