# Blacksmith docs: AI Agent Readiness Score 87.5% (B+)

**87.5% · 70/80 · AI Agent Readiness Score · 20/30 reading points · 50/50 agent surface points**

Blacksmith received 5 PASS votes and passed 5 of five agent surface checks. The clearest finding came from the find the exact limits task.

- Tested: 2026-08-19
- Published: 2026-09-21
- Battery: v1
- Scoring: reading 30 pts · surface 50 pts
- Docs: https://docs.blacksmith.sh/

Three AI models, GPT 5.6 Sol, Claude Opus 5, and DeepSeek v4 Flash, each read Blacksmith’s public documentation independently and attempted five first-hour developer jobs: run the first workflow, find the exact limits, recover a queued job, authorize the GitHub integration, filter workflow runs.

No accounts, API calls, or code execution were used. Every verdict came from public pages and every published quotation passed a live verification check. Preflight substitutions: T3: A queued runner job replaced HTTP 429 because it is the common documented recovery path. T4: GitHub App authentication replaced webhooks because no developer-facing webhook interface is documented. T5: Run History filtering replaced SDK coverage because no official SDK or public list endpoint is documented.

## Freshness

[How rechecks work](https://docsforagents.com/methodology/#freshness)

- Category: [Cloud & developer platforms](https://docsforagents.com/grades/?category=cloud-platforms)
- Tested: 2026-08-19
- Quotes verified: 2026-08-19
- Surface rechecked: 2026-09-21

No change since the test.

5 of 5 quoted passages still appear on the live pages.

## Agent surface checks · 50/50

| Check | Verdict | Points |
| --- | --- | --- |
| llms.txt | PASS | 10 |
| llms-full.txt | PASS | 10 |
| Markdown mirror | PASS | 10 |
| MCP server | PASS | 10 |
| Docs AI | PASS | 10 |

## The Reading Test · 20/30

| Task | GPT 5.6 Sol | Opus 5 | DeepSeek v4F | Consensus |
| --- | --- | --- | --- | --- |
| Run the first workflow | PARTIAL | PASS | PARTIAL | PARTIAL |
| Find the exact limits | PARTIAL | PARTIAL | PARTIAL | PARTIAL |
| Recover a queued job | PASS | PASS | PARTIAL | PASS |
| Authorize the GitHub integration | PARTIAL | PARTIAL | PARTIAL | PARTIAL |
| Filter workflow runs | PASS | PARTIAL | PASS | PASS |

Docs platform: Mintlify (unscored) · verified 2026-08-19

## What to fix first

These 3 fixes could add up to 8 points to the AI Agent Readiness Score. The list ranks each fix by the points it would add. [How the ranking works](https://docsforagents.com/methodology/#what-to-fix-first)

1. **+3 points · Find the exact limits · PARTIAL**

   **Found:** The runners overview says runners cost half of GitHub's per-minute price, while the pricing page says 33% less.

   **Fix:** State one per-minute price comparison with GitHub on both the runners overview and the pricing page.

   **Evidence:** [docs.blacksmith.sh/blacksmith-runners/overview](https://docs.blacksmith.sh/blacksmith-runners/overview)

2. **+3 points · Authorize the GitHub integration · PARTIAL**

   **Found:** The quickstart defers GitHub App authorization to console steps, and no page lists the permissions the app requests.

   **Fix:** Name and link the GitHub App, list the permissions it requests, and document its install steps.

   **Evidence:** [docs.blacksmith.sh/introduction/quickstart](https://docs.blacksmith.sh/introduction/quickstart)

3. **+2 points · Run the first workflow · PARTIAL**

   **Found:** The quickstart covers authorization and the runs-on swap but has no step that confirms a job ran on Blacksmith.

   **Fix:** Add a final quickstart step that checks Run History to confirm the job ran on a Blacksmith runner.

   **Evidence:** [docs.blacksmith.sh/introduction/quickstart](https://docs.blacksmith.sh/introduction/quickstart)

## What the docs get right

- **Recover a queued job: 2 PASS votes.** A dedicated FAQ explains that GitHub's org-level runner registration lets repositories without the app adopt runners provisioned for repositories that have it, names the resulting under-provisioning, and gives the single corrective action, and the Network & IP Allowlisting page separately covers the other documented cause of jobs not being picked up.
- **Filter workflow runs: 2 PASS votes.** Screenshots expose run filters, while text documents the global search action, query syntax, and Blacksmith-only result scope.
- **5 of 5 agent surface checks.** Present: llms.txt, llms-full.txt, markdown mirrors, an MCP server, docs AI.

## Run the first workflow

**PARTIAL**

PARTIAL consensus from 1 PASS, 2 PARTIAL.

The quickstart covers authorization and runner-tag migration, but confirmation requires cross-referencing Run History. The Quickstart gives an ordered path from GitHub authorization to a Migration Wizard or a manual runs-on swap, with a GitHub-tag-to-Blacksmith-tag mapping table and a diff example, though the step that confirms a job actually ran on a Blacksmith runner is only implied by the Run History page rather than stated as a verification step. The migration path (authorize app, replace runs-on) is clearly documented, but no explicit step tells the user how to confirm a workflow ran successfully on Blacksmith. Step 5 links to Run History for debugging, and the Run History FAQ says 'Only job runs on Blacksmith runners are visible', but the quickstart never connects those dots into a confirmation step.

## Find the exact limits

**PARTIAL**

PARTIAL consensus from 3 PARTIAL.

Resource, cache, concurrency, and platform limits are exact, but the runner and pricing pages disagree on per-minute savings. Concurrency (none), resource ceilings (per-tag vCPU, memory, storage tables), cache allowance (25GB per repo per week, 7-day LRU eviction), platform limits (5 sticky disks per job, no Docker Linux containers on Windows, no nested virtualization on ARM, no personal repositories), and free-tier conversion ratios are all stated exactly, but the docs say Blacksmith runners are exactly half GitHub's per-minute price while the pricing page says 33% cheaper per minute, and the Bazel cache storage cap is never given a number in the docs. Concurrency limits, runner resource ceilings, cache allowance (25GB/week/repo), and pricing tiers are documented, but the pricing page claims '33% cheaper than GitHub per-minute pricing' while the runners overview claims 'exactly half the cost of GitHub's per minute' (50% cheaper) and the actual listed price ($0.004/min) confirms 50%, a contradiction between pages.

## Recover a queued job

**PASS**

PASS consensus from 2 PASS, 1 PARTIAL.

The FAQ explains the under-provisioning cause and directs installation on every repository using a Blacksmith runner label. A dedicated FAQ explains that GitHub's org-level runner registration lets repositories without the app adopt runners provisioned for repositories that have it, names the resulting under-provisioning, and gives the single corrective action, and the Network & IP Allowlisting page separately covers the other documented cause of jobs not being picked up. The runners FAQ addresses one queuing cause (GitHub App not installed on all repos using blacksmith runners). The Network & IP Allowlisting page addresses another (IP allowlists blocking runner registration). The Scale Apps page addresses rate-limit throttling. No single page consolidates all three recovery paths into a diagnostic flow.

## Authorize the GitHub integration

**PARTIAL**

PARTIAL consensus from 3 PARTIAL.

Permission synchronization is explicit, but the docs delegate GitHub App authorization to undocumented console steps. Permission synchronization and the org-admin/repo-role capability matrix are documented in full, and the Quickstart lists three concrete reasons an organization fails to appear (unauthenticated SSO session, app not installed, user not a member), but the authorization step itself is delegated to the console with "follow the steps" and the primary Blacksmith GitHub App is never named, linked, or given a requested-permission list the way the auxiliary Scale Apps are. The quickstart tells the user where to go and what permissions the app needs, and the Permissions page documents post-installation access mapping. But the actual GitHub App installation flow, the specific permissions requested, and the OAuth authorization steps are not documented, the agent must navigate the dashboard to discover the flow.

## Filter workflow runs

**PASS**

PASS consensus from 2 PASS, 1 PARTIAL.

Screenshots expose run filters, while text documents the global search action, query syntax, and Blacksmith-only result scope. The Logs page fully documents the query grammar with twelve property filters, substring search, negation, and escaping rules, but the Run History page documents no filter controls of its own beyond the sentence that the page exists for filtering, the global search is described only as selecting a log line with no named control or scope boundary, and no log or run retention window appears anywhere in the documentation. The Run History and Logs pages together document the full filtering capability: property filters (run_id, workflow, branch, repo, level, etc.), substring search, exclusion syntax, global log search across all CI runs, and the scope ('Only job runs on Blacksmith runners are visible').

## The receipt

> Blacksmith does not impose any concurrency limits i.e., limits on how many vCPUs or jobs that can be run simultaneously.

Resource, cache, concurrency, and platform limits are exact, but the runner and pricing pages disagree on per-minute savings.

- [docs.blacksmith.sh/blacksmith-runners/overview](https://docs.blacksmith.sh/blacksmith-runners/overview)

## Agent surface notes

Initialize returned JSON-RPC protocol 2025-06-18 and Blacksmith server information.

The live docs expose a public assistant panel.

## Method note

This is a reading test of public documentation, not an execution test. No accounts were created and no API calls were run. The AI Agent Readiness Score counts fifteen reading votes at PASS 2, PARTIAL 1, and FAIL 0, for 30 possible points. Five agent surface checks add 10 points each. The total is 80. Consensus chips show each row majority and do not affect scoring. The panel split on 3 of five tasks. Quotes shown here were re-fetched and confirmed verbatim on 2026-08-19.

Methodology: https://docsforagents.com/methodology/

Canonical URL: https://docsforagents.com/reports/blacksmith-docs-ai-agent-readiness/