Published 2026-09-21 · Tested 2026-08-19
Ably
C-Ably received 11 PASS votes and passed 3 of five agent surface checks. The clearest finding came from the recover from error 42910 task.
Three AI models, GPT 5.6 Sol, Claude Opus 5, and DeepSeek v4 Flash, each read Ably’s public documentation independently and attempted five first-hour developer jobs: publish the first message, find the exact limits, recover from error 42910, verify a message webhook, use the JavaScript SDK.
No accounts, API calls, or code execution were used. Every verdict came from public pages and every published quotation passed a live verification check.
Freshness
How rechecks work- Category
- Email & messaging
- Tested
- Quotes verified
- Surface rechecked
No change since the test.
5 of 5 quoted passages still appear on the live pages.
70.0% · 56/80 · AI Agent Readiness Score · reading 30 pts · surface 50 pts
Agent surface checks · 30/50
The Reading Test
| Task | GPT 5.6 Sol | Opus 5 | DeepSeek v4F | Consensus |
|---|---|---|---|---|
| Publish the first message | PASS | PASS | PASS | PASS |
| Find the exact limits | PASS | PASS | PASS | PASS |
| Recover from error 42910 | PARTIAL | PARTIAL | PARTIAL | PARTIAL |
| Verify a message webhook | PARTIAL | PASS | PASS | PASS |
| Use the JavaScript SDK | PASS | PASS | PASS | PASS |
docs platform: Gatsby 5.16.1 (unscored) · verified 2026-08-19
What to fix first
These 3 fixes could add up to 23 points to the AI Agent Readiness Score. The list ranks each fix by the points it would add. How the ranking works
- 1 +10 points llms-full.txt check failed
Found: HTTP 404 at the tested site and documentation roots.
Fix: Publish llms-full.txt at the docs root with the full text of every docs page in one plain-text file.
- 2 +10 points MCP server check failed
Found: Initialize returned HTTP 404 with no valid MCP response.
Fix: Run a public MCP server for the docs that answers a JSON-RPC initialize call and offers a docs search tool.
Evidence: ably.com/
docs/ mcp (opens in a new tab) - 3
Found: The 42910 docs page calls it a system-wide protection, while the Ably FAQ calls it an account hard limit.
Fix: Rewrite the 42910 FAQ entry to match the docs page scope, and state a starting retry delay and maximum.
Evidence: ably.com/
docs/ platform/ errors/ codes/ 42910-rate-limit-exceeded-generic (opens in a new tab)
What the docs get right
- Publish the first message: 3 PASS votes. The guide waits for connection, subscribes once, publishes after subscribing, and logs the received message.
- Find the exact limits: 3 PASS votes. In task order, Free lists 6,000,000, 200, 500/s, and 50/s; Standard lists Unlimited, 10,000, 2,500/s, and 50/s.
- 3 of 5 agent surface checks. Present: llms.txt, markdown mirrors, docs AI. Missing: llms-full.txt, an MCP server.
Publish the first message
PASSPASS consensus from 3 PASS.
The guide waits for connection, subscribes once, publishes after subscribing, and logs the received message. One page carries the whole path in order: CDN script tag, new Ably.Realtime({key, clientId}), await connection.once('connected'), channels.get, await channel.subscribe with a logging callback, then await channel.publish, so receipt is confirmed by the subscriber the guide already registered. Four-step guide with CDN include, client instantiation, channel subscribe, and channel publish in a linear progression with no branching or ambiguity.
Find the exact limits
PASSPASS consensus from 3 PASS.
In task order, Free lists 6,000,000, 200, 500/s, and 50/s; Standard lists Unlimited, 10,000, 2,500/s, and 50/s. All four figures sit in one page with inline definitions per row (Free/Standard: 6,000,000 vs Unlimited monthly messages, 200 vs 10,000 concurrent connections, 500 vs 2,500 account message rate, 50 vs 50 per-channel publish rate) and the public pricing page repeats the same numbers. All four requested metrics are tabulated on the limits page and agree with the Free and Standard individual plan pages.
Recover from error 42910
PARTIALPARTIAL consensus from 3 PARTIAL.
The page gives complete recovery guidance; an older FAQ contradicts its scope, and a legacy markdown URL returned 404. CONTRADICTION: the docs page scopes 42910 to a system-wide protection that is not account-specific, while Ably's own help site says 42910 means "you have hit a hard limit on your account" and illustrates it with metric = channel.maxRate, which the current docs assign to error 42913; the docs page also gives no numeric backoff, only "increasing the delay if it is rejected again". Error 42910 is not listed in the official error codes documentation at /docs/platform/errors/codes; it is only documented in the separate FAQ knowledge base, requiring the agent to leave the primary docs to find recovery details.
Verify a message webhook
PASSPASS consensus from 2 PASS, 1 PARTIAL.
Creation, batching, signing, retry, and ordering are documented, but the decode prose says fromEncoded while its code calls fromEncodedArray. Every part is documented without guessing: generic.md gives the dashboard rule steps and the channel.message event type plus the Sign with key and Request Mode settings, and webhooks.md gives the batched payload JSON, the five HMAC-SHA256 signature steps keyed off X-Ably-Key, retries (single: 4s then 20s; batched: delay = delay * sqrt(2) capped at 60s), and ordering (batched preserves order for the same publisher on the same channel, single request cannot). One page covers rule creation via dashboard or API, batched payload structure, signature verification with explicit steps, exponential-backoff retry with formula and example delays, and ordering guarantees per publisher-channel pair.
Use the JavaScript SDK
PASSPASS consensus from 3 PASS.
The CDN guide uses Realtime with awaited subscribe and publish forms that match the current JavaScript reference. The getting-started guide, the pub-sub concept page, the connect page, and the API reference all agree on the same promise-based v2 surface (new Ably.Realtime, channels.get, await channel.subscribe, await channel.publish), though the docs pin only the major version via cdn.ably.com/lib/ably.min-2.js and never state an exact SDK version number. The getting-started page shows the CDN include, client construction, subscribe, and publish in sequence, matching the current SDK 2.x API reference.
The receipt
Retry the operation after a short delay, increasing the delay if it is rejected again.
The page gives complete recovery guidance; an older FAQ contradicts its scope, and a legacy markdown URL returned 404.
Agent surface notes
Initialize returned HTTP 404 with no valid MCP response.
The live docs expose a public Ask AI control.
Show the score
Paste this into a readme:
[](https://docsforagents.com/reports/ably-docs-ai-agent-readiness/) Method note
This is a reading test of public documentation, not an execution test. No accounts were created and no API calls were run. The AI Agent Readiness Score counts fifteen reading votes at PASS 2, PARTIAL 1, and FAIL 0, for 30 possible points. Five agent surface checks add 10 points each. The total is 80. Consensus chips show each row majority and do not affect scoring. The panel split on 1 of five tasks. Quotes shown here were re-fetched and confirmed verbatim on 2026-08-19.
Put another docs site through the battery.
Nominate a docs site