Published 2026-08-20 · Tested 2026-08-19
Composio
B-Composio received 9 PASS votes and passed 4 of five readiness checks. The clearest finding came from the use the Python SDK task.
Three AI models, GPT 5.6 Sol, Claude Opus 5, and DeepSeek v4 Flash, each read Composio’s public documentation independently and attempted five first-hour developer jobs: connect an agent tool, find the exact limits, recover from a 429, verify a trigger webhook, use the Python SDK.
No accounts, API calls, or code execution were used. Every verdict came from public pages and every published quotation passed a live verification check.
80.0% · 64/80 · AI Agent Readiness Score · reading 30 pts · surface 50 pts
Agent surface checks · 40/50
The Reading Test
| Task | GPT 5.6 Sol | Opus 5 | DeepSeek v4F | Consensus |
|---|---|---|---|---|
| Connect an agent tool | PASS | PARTIAL | PASS | PASS |
| Find the exact limits | PASS | PASS | PARTIAL | PASS |
| Recover from a 429 | PARTIAL | PASS | PASS | PASS |
| Verify a trigger webhook | PARTIAL | PASS | PASS | PASS |
| Use the python sdk | PARTIAL | PARTIAL | PASS | PARTIAL |
docs platform: Fumadocs on Next.js (unscored) · verified 2026-08-19
What the docs get right
- Connect an agent tool: 2 PASS votes. The quickstart covers API keys, a session, in-chat Connect Link authentication, and execution; the optional manual-auth Markdown URL returned 404.
- Find the exact limits: 2 PASS votes. Starter and Hobby allow 2,000 requests per minute, Growth allows 10,000, and Enterprise is custom on both current and v3 pages.
- 4 of 5 readiness checks. Present: llms.txt, llms-full.txt, markdown mirrors, docs AI. Missing: an MCP server.
Connect an agent tool
PASSPASS consensus from 2 PASS, 1 PARTIAL.
The quickstart covers API keys, a session, in-chat Connect Link authentication, and execution; the optional manual-auth Markdown URL returned 404. The credentials-to-connection-to-invocation path is fully documented (COMPOSIO_API_KEY from dashboard settings, session creation, runtime Connect Link, then session.execute()), but Python session creation is written two contradictory ways: quickstart and how-composio-works use composio.sessions.create(user_id=...) while authentication, the Anthropic provider page, llms.txt, and the AI-generator note appended to every page use composio.create(user_id=...), and the Python SDK reference for the Composio class documents neither a sessions property nor a create method. The quickstart page, provider guides (OpenAI, Anthropic), and authentication docs together provide an unambiguous path: install the SDK, create a session with composio.sessions.create(), call session.tools() for meta tools, and execute via session.execute() or a provider handle_tool_calls helper.
Find the exact limits
PASSPASS consensus from 2 PASS, 1 PARTIAL.
Starter and Hobby allow 2,000 requests per minute, Growth allows 10,000, and Enterprise is custom on both current and v3 pages. One table gives all four plans exactly , Starter 2,000 requests/1 minute, Hobby 2,000 requests/1 minute, Growth 10,000 requests/1 minute, Enterprise Custom with no window , and the legacy v3 page at /reference/v3/rate-limits.md carries a byte-identical table, so the fixed-window rule and the quotas do not disagree anywhere in the docs. Exact per-minute API rate limits are documented for Starter/Hobby/Growth/Enterprise, but the main marketing pricing page (composio.dev/pricing) uses entirely different plan names (Free/Pro/Enterprise) and the pro-tools page uses yet another naming scheme (Totally Free/Ridiculously Cheap/Serious Business) with a contradictory 'Free: 100/min standard tool calls' rate limit that does not square with the rate-limits page's 'Starter: 2,000/min'.
Recover from a 429
PASSPASS consensus from 2 PASS, 1 PARTIAL.
Errors gives backoff, Rate Limits supplies Retry-After and plan quotas, and Proxy Execute separately identifies upstream 429 responses. Retry-After is defined precisely as "Seconds until the window resets (only on 429 responses)", X-RateLimit-Remaining and X-RateLimit-Window-Size are documented for pre-emptive throttling, the 429 body is shown verbatim, and /docs/extending-sessions/proxy-execute.md separately covers the upstream 429 case with "Composio doesn't retry automatically." The rate-limits and errors pages document the 429 response format, the Retry-After header, the X-RateLimit-* headers, and three specific recovery steps (monitor remaining, honor Retry-After, cache static data). The errors page additionally recommends exponential backoff.
Verify a trigger webhook
PASSPASS consensus from 2 PASS, 1 PARTIAL.
The linked creation and receiving pages cover the full path, while the trigger section root URL returned 404. The whole chain is covered end to end: inspect the trigger type's config schema, composio.triggers.create() with the slug and trigger_config, composio.triggers.set_webhook_subscription() to register the project webhook URL and capture the secret, the V3 envelope shown as literal JSON, and both an SDK parse(verify_secret=...) path and a from-scratch HMAC implementation with a stated 300-second replay tolerance. The triggers documentation covers the complete path: creating a trigger (composio.triggers.create()), registering a project webhook (composio.triggers.set_webhook_subscription()), receiving events, and verifying signatures via parse()/verifyWebhook() with HMAC-SHA256, all with Python SDK code examples for FastAPI and Flask.
Use the python sdk
PARTIALPARTIAL consensus from 1 PASS, 2 PARTIAL.
The Python SDK reference executes composio.tools.execute() without the required version or opt-in flag, while the agent quickstart uses current sessions. The package name, the Python 3.10-or-newer floor, a class-by-class reference with source links, and a complete runnable Anthropic tool-calling example are all present, but the Installation section of that reference page is an empty heading in the markdown mirror and its Quick Start demonstrates composio.tools.get() / composio.tools.execute() direct execution, which the same site's AI-generator note describes as the lower-level alternative to the recommended composio.create(user_id) session entry point. The Python SDK reference documents installation (uv add composio or pip install composio, Python 3.10+), initialization with an API key, tool discovery (composio.tools.get()), tool execution (composio.tools.execute()), and the session API (composio.sessions.create()), all consistent with the session-based concepts described in how-composio-works.
The receipt
Starting from Python SDK v0.9.0 and TypeScript SDK v0.2.0, specifying versions is required for manual tool execution , and "latest" is not accepted there on its own.
The Python SDK reference executes composio.tools.execute() without the required version or opt-in flag, while the agent quickstart uses current sessions.
Agent surface notes
Initialize returned product-MCP HTML or an authentication error, not a documentation MCP result.
The live docs expose visible Ask AI controls.
Method note
This is a reading test of public documentation, not an execution test. No accounts were created and no API calls were run. The AI Agent Readiness Score counts fifteen reading votes at PASS 2, PARTIAL 1, and FAIL 0, for 30 possible points. Five readiness checks add 10 points each. The total is 80. Consensus chips show each row majority and do not affect scoring. The panel split on 5 of five tasks. Quotes shown here were re-fetched and confirmed verbatim on 2026-08-19.
Put another docs site through the battery.
Nominate a docs site