Published 2026-08-20 · Tested 2026-08-19

Railway

C

Railway received 6 PASS votes and passed 4 of five agent surface checks. The clearest finding came from the find the exact limits task.

Panel: GPT 5.6 Sol, Opus 5, DeepSeek v4F Battery: v1 Read as markdown (opens in a new tab)

Three AI models, GPT 5.6 Sol, Claude Opus 5, and DeepSeek v4 Flash, each read Railway’s public documentation independently and attempted five first-hour developer jobs: deploy an application, find the exact limits, recover from a 429, verify a project webhook, use the TypeScript SDK.

No accounts, API calls, or code execution were used. Every verdict came from public pages and every published quotation passed a live verification check.

Freshness

How rechecks work
Tested
Quotes verified
Surface rechecked

No change since the test.

5 of 5 quoted passages still appear on the live pages.

Railway Custom Next.js · published
C

73.8% · 59/80 · AI Agent Readiness Score · reading 30 pts · surface 50 pts

llms.txt PASS
llms-full.txt PASS
markdown mirror PASS
MCP server PASS
docs AI FAIL
Task GPT 5.6 SolOpus 5DeepSeek v4F Consensus
Deploy an application PASSPASSPASS PASS
Find the exact limits PARTIALPARTIALPASS PARTIAL
Recover from a 429 PARTIALPARTIALPASS PARTIAL
Verify a project webhook FAILFAILPARTIAL FAIL
Use the TypeScript SDK PASSPARTIALPARTIAL PARTIAL

docs platform: Custom Next.js (unscored) · verified 2026-08-19

What to fix first

These 5 fixes could add up to 21 points to the AI Agent Readiness Score. The list ranks each fix by the points it would add. How the ranking works

  1. 1
    +10 points Docs AI check failed

    Found: The live docs expose no embedded Ask AI control.

    Fix: Add an assistant to the docs site that answers questions from the docs and links to its sources.

  2. 2
    +5 points Verify a project webhook FAIL

    Found: The Webhooks page covers setup, payload, and test sends, but names no signing secret or authenticity check.

    Fix: State on the Webhooks page how, or whether, a receiver can verify that a delivery came from Railway.

    Evidence: docs.railway.com/observability/webhooks (opens in a new tab)

  3. 3
    +2 points Find the exact limits PARTIAL

    Found: The API page defines X-RateLimit-Limit as a daily maximum beside hourly limits. Opus 5 also found that the Plans and Scaling pages state different Pro resource ceilings.

    Fix: State which window X-RateLimit-Limit counts, and explain how Pro per-service and per-replica resource limits relate.

    Evidence: docs.railway.com/integrations/api (opens in a new tab)

  4. 4
    +2 points Recover from a 429 PARTIAL

    Found: The API page defines Retry-After but never mentions 429 or the header value's unit. Opus 5 also found no throttled response shape or backoff guidance.

    Fix: State the status code for an exhausted request window and the Retry-After unit, and add a backoff example.

    Evidence: docs.railway.com/integrations/api (opens in a new tab)

  5. 5
    +2 points Use the TypeScript SDK PARTIAL

    Found: The Public API pages never mention the TypeScript SDK. Opus 5 also found that two SDK pages recommend different token variables.

    Fix: Link the TypeScript SDK from the Public API pages, and explain when to use RAILWAY_TOKEN or RAILWAY_API_TOKEN.

    Evidence: docs.railway.com/feature-flags (opens in a new tab)

What the docs get right

  • Deploy an application: 3 PASS votes. The quick start names a sample repository, deployment steps, and the domain action in one dashboard path.
  • Find the exact limits: 1 PASS votes. Both tables are findable, but the API page defines X-RateLimit-Limit as daily beside hourly limits.
  • 4 of 5 agent surface checks. Present: llms.txt, llms-full.txt, markdown mirrors, an MCP server. Missing: docs AI.

Deploy an application

PASS

PASS consensus from 3 PASS.

The quick start names a sample repository, deployment steps, and the domain action in one dashboard path. The quick start supplies a concrete forkable sample repo (railwayapp-templates/nextjs-basic), names each dashboard control verbatim, and offers three independent paths (GitHub, CLI, Docker image), with the domain step spelled out on the linked networking page. Quick Start provides a step-by-step path from zero to deployed app with a Railway-provided domain via three deployment options (GitHub, CLI, Docker image).

Find the exact limits

PARTIAL

PARTIAL consensus from 1 PASS, 2 PARTIAL.

Both tables are findable, but the API page defines X-RateLimit-Limit as daily beside hourly limits. The plans page states Pro allows 1,000 vCPU and 1 TB RAM per service and that these 'include replica multiplication', while the scaling page states each Pro replica gets 24 vCPU / 24 GB and that two replicas cap at 48 vCPU / 48 GB, so the two pages return different answers for the same service-level ceiling; separately the API rate-limit list omits the Trial plan that the plans page defines and gives no requests-per-second figure for Free. API rate limits are tabulated by plan in one page; per-service resource quotas (RAM, CPU, replicas, ephemeral storage, volume storage, image size) are tabulated by plan in a separate Plans page. The values are consistent across both pages.

Recover from a 429

PARTIAL

PARTIAL consensus from 1 PASS, 2 PARTIAL.

The page says when to retry, but it never identifies 429 or defines the header value's unit or format. Retry-After is defined in a single table row and is the only Retry-After reference in the entire documentation corpus, but no page documents the HTTP 429 status code for the Railway API, the units of the Retry-After value, the shape of the throttled response, or any backoff guidance, and the same table describes X-RateLimit-Limit as a per-day maximum while the limits above it are stated per hour and per second. The API documentation explains rate-limit response headers including Retry-After, which tells the caller how long to wait before the next request.

Verify a project webhook

FAIL

FAIL consensus from 1 PARTIAL, 2 FAIL.

Setup, payload, and testing are documented, but the page provides no signature or authenticity-verification method. Creation and payload shape are documented, but the docs describe no signing secret, signature header, or any other authenticity mechanism for Railway-sent webhooks anywhere in the corpus, so a receiver cannot distinguish a genuine Railway delivery from a forged POST, and the documented test-payload path is itself disclosed as likely to fail. Webhook setup and test-payload button are documented, but there is no documented HMAC signature, shared secret, or any mechanism to verify the authenticity of Railway's own webhook payloads. The 'Example payload' heading exists but contains no example body.

Use the TypeScript SDK

PARTIAL

PARTIAL consensus from 1 PASS, 2 PARTIAL.

The quick start uses RAILWAY_API_TOKEN and RAILWAY_ENVIRONMENT_ID, while the API page explains account-token authentication. An official SDK exists (npm package railway, repo railwayapp/railway-ts-sdk) with runnable authenticated examples, but it is never mentioned on any Public API page, it covers only sandboxes, feature flags and infrastructure-as-code rather than the GraphQL concepts the API docs teach (projects, services, deployments, variables, domains, volumes), and its two documented entry points disagree on which environment variable authenticates. The railway npm package is a full TypeScript SDK with Sandbox and IaC APIs, detailed README, and configuration that matches the GraphQL endpoint and token types documented on docs.railway.com. However, docs.railway.com does not link to or mention this SDK, so an agent starting from the docs site would not discover it.

The receipt

Requests per hour: 100 RPH for Free customers, 1000 RPH for Hobby customers, 10000 RPH for Pro customers; custom for Enterprise.

Both tables are findable, but the API page defines X-RateLimit-Limit as daily beside hourly limits.

Agent surface notes

Initialize returned a valid OAuth-protected MCP authentication challenge.

The live docs expose no embedded Ask AI control.

Show the score

AI Agent Readiness Score 73.8%, grade C

Paste this into a readme:

[![AI Agent Readiness Score 73.8%](https://docsforagents.com/badge/railway.svg)](https://docsforagents.com/reports/railway-docs-ai-agent-readiness/)

Method note

This is a reading test of public documentation, not an execution test. No accounts were created and no API calls were run. The AI Agent Readiness Score counts fifteen reading votes at PASS 2, PARTIAL 1, and FAIL 0, for 30 possible points. Five agent surface checks add 10 points each. The total is 80. Consensus chips show each row majority and do not affect scoring. The panel split on 4 of five tasks. Quotes shown here were re-fetched and confirmed verbatim on 2026-08-19.

Read the full methodology

Put another docs site through the battery.

Nominate a docs site