Published 2026-08-06 · Tested 2026-08-05
Mailgun
CAll three models passed the quickstart, the webhook pages never show the full payload, and the only FAIL vote of the run was about rate limits.
Three AI models, GPT 5.6 Sol, Claude Opus 5, and DeepSeek v4 Flash, each read Mailgun’s public documentation independently and attempted the same five first-hour developer jobs: send the first email, find the rate limits, recover from a 429, set up webhooks end to end, and use the Python SDK.
No accounts, no API calls, no code execution. A verdict means the docs got an agent to a confident answer, backed by a verbatim quote from a live page. Where the panel disagreed, the matrix shows the split.
76.3% · 61/80 · Agent-Ready Grade · reading 30 pts · readiness 50 pts
AI agent readiness · 40/50
The Reading Test
| Task | GPT 5.6 Sol | Opus 5 | DeepSeek v4F | Consensus |
|---|---|---|---|---|
| Send the first email | PASS | PASS | PASS | PASS |
| Find the rate limits | FAIL | PARTIAL | PARTIAL | PARTIAL |
| Recover from a 429 | PASS | PARTIAL | PASS | PASS |
| Webhooks end to end | PARTIAL | PARTIAL | PARTIAL | PARTIAL |
| Use the Python path | PASS | PASS | PARTIAL | PASS |
docs platform: Redocly (unscored) · verified 2026-08-05 and 2026-08-06
What the docs get right
- One quickstart, eight languages. The sandbox quickstart walks from signup to a delivered email in five steps, shows the identical example in eight languages, and prints the exact 200 response body, so the send call needs no reconstruction.
- A real markdown mirror. Every page publishes a companion .md endpoint, and llms.txt indexes all 144 pages, so an agent can read the entire corpus verbatim instead of scraping a JavaScript shell.
- One consolidated error table. The API overview collects response codes 400 through 500 with per-code troubleshooting and defines the three X-RateLimit headers alongside them.
- Production-grade webhook security. Webhook verification is documented as a complete HMAC-SHA256 recipe with a runnable Node snippet and anti-replay guidance.
Send the first email
PASSThe only unanimous PASS of the run. The path runs sign up, copy the private API key, authorize a recipient, send, check the inbox, and the same example appears in eight languages with the exact 200 response body.
Opus 5 logged two blemishes: a dangling cross-reference to a nonexistent Step 2B, and the page never says where to find your sandbox domain. DeepSeek hit a 404 on one quickstart URL variant, and the canonical path worked.
Find the rate limits
PARTIALThe docs document the mechanism thoroughly and the numbers almost not at all. The one concrete figure, 500 requests every 10 seconds, appears under a Rate Limits and Quotas heading nested inside the Metrics reporting page, and no page says whether it is Metrics-scoped or account-wide. No plan-quota table exists anywhere in the docs map. The only quota figures the panel found were 5 sandbox recipients and 300 emails per day for unverified domains.
GPT graded FAIL, Opus 5 and DeepSeek PARTIAL. DeepSeek added a structural note: the dedicated rate-limits page is a JavaScript-rendered OpenAPI spec that would not fetch, so its one number came from a search snippet rather than a readable page.
Recover from a 429
PASSThe API overview’s response-code table covers 429 and defines X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset, which is enough to build a retry loop, and two panelists passed it.
Opus 5 graded PARTIAL for two reasons: the reset header’s definition conflates an absolute epoch timestamp with a relative duration, and the send endpoint’s own page documents only 200 and 400 responses, so an agent holding a 429 from that endpoint finds nothing about it there.
Webhooks end to end
PARTIALAll three legs exist: a create endpoint whose event list includes delivered, a delivered-event payload example, and the HMAC verification recipe. All three panelists still graded PARTIAL, for converging reasons. The security page shows the signature object and says it is posted alongside the webhook’s event data. The payloads page shows the event body and never mentions a signature. The combined POST envelope an agent must actually parse appears on no page.
Opus 5 also flagged that two pages disagree on retry intervals, and DeepSeek could not fetch the account-webhooks API reference at all, leaving the programmatic create step to guesswork.
Use the Python path
PASSThe page is labelled Official Mailgun Python SDK, installs with pip install mailgun-python, links a repository that resolves, and its send example matches the REST shape of the messages endpoint.
DeepSeek graded PARTIAL: the page defers to the repository readme for full details, and the SDK’s client wrapper differs from the quickstart’s raw requests calls with nothing reconciling the two.
The receipt
API calls are subject to a rate limit of 500 requests every 10 seconds
The only concrete rate-limit number in the docs, nested under a Metrics reporting page. No plan-quota table exists in the docs map.
Readiness notes
Mailgun’s MCP server rejects a plain GET with a 405 and answered a JSON-RPC initialize on POST at documentation.mailgun.com/mcp. A status-code check alone would have read it as absent.
The markdown mirror is real: one panelist navigated the entire test through the llms.txt map and the .md endpoints.
Method note
This is a reading test of public documentation, not an execution test. No accounts were created and no API calls were run. The Agent-Ready Grade counts the fifteen reading votes at PASS 2, PARTIAL 1, FAIL 0 (30 points) and the five readiness checks at 10 points each (50 points), out of 80, shown as a US school letter grade with its percentage. Consensus chips show each row’s majority and play no scoring role. The panel split on three of the five tasks, and the matrix shows every vote. Quotes shown here were re-fetched and confirmed verbatim on the dates above.
Put another docs site through the battery.
Nominate a docs site