Methodology

A reading test, not an execution test

Each report asks one question of one product: can the public documentation get an AI agent through five first-hour developer jobs? The panel reads the docs and attempts each job using only what the fetched pages say. No accounts are created, no API calls are executed, no code runs.

A PASS therefore means the docs got an agent to a confident, unambiguous answer. It does not mean the integration was executed live. This constraint is also the point: every product gets identical treatment regardless of pricing tier, and every report is reproducible from public pages.

The battery

The battery is fixed across reports so grades are comparable. Changing a task bumps the battery version, and the change is disclosed in the affected report. The current battery is v1.

• Quickstart. From zero, reach the product’s core action, such as sending the first email or accepting a test payment. Is there one clear path from credentials to a successful call?
• Limits. Find the exact rate limits and plan quotas. Are they stated, and stated consistently everywhere they appear?
• Error recovery. Starting from a common error such as a 429, can the docs get the agent from error to fix? Is there an error reference?
• Webhooks end to end. Create, receive, and verify a webhook. Is the full path documented, and do the pages agree with each other?
• SDK coverage. Find the primary-language SDK and a minimal working example. Does the example match the API shape shown elsewhere in the docs?

Products without a webhook surface or an SDK get a substitute task chosen before any panelist runs: authentication end to end, or pagination on the main list endpoint. Substitutions are printed on the scorecard.

The panel

Three models run the identical brief independently: GPT 5.6 Sol, Claude Opus 5, and DeepSeek v4 Flash. No panelist is told about the others, and no panelist sees another’s output. DeepSeek is deliberately a budget-tier panelist: its verdicts indicate what docs look like to the model tier companies actually deploy at scale.

The panel is exactly three, every run, so grades stay comparable. A panelist that errors out gets one retry with the identical brief, and if it still fails the run waits rather than shipping a reduced panel. Any permanent change of membership is disclosed on the first affected scorecard.

Model self-reports are unreliable, so the run record, not the model’s own claim, names the panelists. Column headers on the scorecard print the versions as invoked.

Verdicts

• PASS. The agent completed the task with one clear documented path and no guessing. Minor page-hopping is fine.
• PARTIAL. The agent completed the task but had to guess, resolve an ambiguity, or reconcile pages that disagree. Internal contradictions are automatically at most PARTIAL.
• FAIL. The docs could not get the agent to a confident answer.

Panelists are instructed to grade generously. The series leads with what docs get right, and a contradiction between two live pages is reported as drift, a systems problem every docs site accumulates, not as a team being sloppy.

The Agent-Ready Grade

The headline result is a US school letter grade. Reading points: the fifteen votes, three judges across five tasks, at PASS 2, PARTIAL 1, FAIL 0, for a maximum of 30. Readiness points: the five surface checks at 10 each, for a maximum of 50. The total out of 80 becomes a percentage, and the percentage becomes the grade, A+ at 97 and above down to F below 60.

The matrix is the breakdown: every vote appears as its own tile, and no aggregate tally is shown. Consensus tiles mark each row’s majority, play no scoring role, and a true three-way split reads SPLIT rather than pretending to be a verdict. Disagreement is data: the same pages reading three different ways is often the most useful finding in the report. Single runs are noise, and the panel, never a single run, is the unit of truth.

Evidence and verification

Every verdict requires one verbatim evidence quote and the URL it came from, plus a log of every URL the panelist consulted. A verdict without a quote is invalid.

Before anything is published, every quote destined for a scorecard or report is independently re-fetched and confirmed verbatim on the live page. Anything that fails is dropped or replaced. Each report prints its verification dates, because pages change.

The readiness survey

Alongside the reading test, each report records the agent-facing surface the product ships. These are verifiable facts with evidence, not judgments, so they are never put to a panel vote. Each of the five checks is worth 10 points of the grade; the docs platform is recorded as metadata and never graded.

• Docs platform. The tool serving the docs, identified from page markers, asset hosts, or generator tags.
• llms.txt and llms-full.txt. Checked by HTTP request at the docs roots. A 200 with plausible content counts.
• Markdown mirror. Confirmed by appending .md to a real docs page URL, or by following llms.txt links, and checking that it serves markdown.
• MCP server. Probed with a JSON-RPC initialize POST against candidate endpoints. A 200 on GET may be a marketing page, and a 405 on GET may still be a live server, so only a valid MCP response counts.
• Docs AI. An embedded assistant users can interact with on the docs site, identified from widget markers.

Limits of the method

Strong models can infer around documentation gaps, and popular products’ docs may appear in training data, so the no-prior-knowledge instruction cannot be fully enforced. The design compensates in three ways: panelists must ground every verdict in a fetched quote, the panel spans model tiers, and disagreement is published rather than smoothed over.

The test reads documentation. It does not measure API design, SDK quality, support, or anything behind a login. A product with excellent docs and a difficult API will score well here, and that is the intended scope.