Published 2026-08-20 · Tested 2026-08-19

Railway

C

Railway received 6 PASS votes and passed 4 of five readiness checks. The clearest finding came from the find the exact limits task.

Panel: GPT 5.6 Sol, Opus 5, DeepSeek v4F Battery: v1 Verified: 2026-08-19 Read as markdown (opens in a new tab)

Three AI models, GPT 5.6 Sol, Claude Opus 5, and DeepSeek v4 Flash, each read Railway’s public documentation independently and attempted five first-hour developer jobs: deploy an application, find the exact limits, recover from a 429, verify a project webhook, use the TypeScript SDK.

No accounts, API calls, or code execution were used. Every verdict came from public pages and every published quotation passed a live verification check.

Railway Custom Next.js · published 2026-08-20
C

73.8% · 59/80 · AI Agent Readiness Score · reading 30 pts · surface 50 pts

llms.txt PASS
llms-full.txt PASS
markdown mirror PASS
MCP server PASS
docs AI FAIL
Task GPT 5.6 SolOpus 5DeepSeek v4F Consensus
Deploy an application PASSPASSPASS PASS
Find the exact limits PARTIALPARTIALPASS PARTIAL
Recover from a 429 PARTIALPARTIALPASS PARTIAL
Verify a project webhook FAILFAILPARTIAL FAIL
Use the typescript sdk PASSPARTIALPARTIAL PARTIAL

docs platform: Custom Next.js (unscored) · verified 2026-08-19

What the docs get right

  • Deploy an application: 3 PASS votes. The quick start names a sample repository, deployment steps, and the domain action in one dashboard path.
  • Find the exact limits: 1 PASS votes. Both tables are findable, but the API page defines X-RateLimit-Limit as daily beside hourly limits.
  • 4 of 5 readiness checks. Present: llms.txt, llms-full.txt, markdown mirrors, an MCP server. Missing: docs AI.

Deploy an application

PASS

PASS consensus from 3 PASS.

The quick start names a sample repository, deployment steps, and the domain action in one dashboard path. The quick start supplies a concrete forkable sample repo (railwayapp-templates/nextjs-basic), names each dashboard control verbatim, and offers three independent paths (GitHub, CLI, Docker image), with the domain step spelled out on the linked networking page. Quick Start provides a step-by-step path from zero to deployed app with a Railway-provided domain via three deployment options (GitHub, CLI, Docker image).

Find the exact limits

PARTIAL

PARTIAL consensus from 1 PASS, 2 PARTIAL.

Both tables are findable, but the API page defines X-RateLimit-Limit as daily beside hourly limits. The plans page states Pro allows 1,000 vCPU and 1 TB RAM per service and that these 'include replica multiplication', while the scaling page states each Pro replica gets 24 vCPU / 24 GB and that two replicas cap at 48 vCPU / 48 GB, so the two pages return different answers for the same service-level ceiling; separately the API rate-limit list omits the Trial plan that the plans page defines and gives no requests-per-second figure for Free. API rate limits are tabulated by plan in one page; per-service resource quotas (RAM, CPU, replicas, ephemeral storage, volume storage, image size) are tabulated by plan in a separate Plans page. The values are consistent across both pages.

Recover from a 429

PARTIAL

PARTIAL consensus from 1 PASS, 2 PARTIAL.

The page says when to retry, but it never identifies 429 or defines the header value's unit or format. Retry-After is defined in a single table row and is the only Retry-After reference in the entire documentation corpus, but no page documents the HTTP 429 status code for the Railway API, the units of the Retry-After value, the shape of the throttled response, or any backoff guidance, and the same table describes X-RateLimit-Limit as a per-day maximum while the limits above it are stated per hour and per second. The API documentation explains rate-limit response headers including Retry-After, which tells the caller how long to wait before the next request.

Verify a project webhook

FAIL

FAIL consensus from 1 PARTIAL, 2 FAIL.

Setup, payload, and testing are documented, but the page provides no signature or authenticity-verification method. Creation and payload shape are documented, but the docs describe no signing secret, signature header, or any other authenticity mechanism for Railway-sent webhooks anywhere in the corpus, so a receiver cannot distinguish a genuine Railway delivery from a forged POST, and the documented test-payload path is itself disclosed as likely to fail. Webhook setup and test-payload button are documented, but there is no documented HMAC signature, shared secret, or any mechanism to verify the authenticity of Railway's own webhook payloads. The 'Example payload' heading exists but contains no example body.

Use the typescript sdk

PARTIAL

PARTIAL consensus from 1 PASS, 2 PARTIAL.

The quick start uses RAILWAY_API_TOKEN and RAILWAY_ENVIRONMENT_ID, while the API page explains account-token authentication. An official SDK exists (npm package `railway`, repo railwayapp/railway-ts-sdk) with runnable authenticated examples, but it is never mentioned on any Public API page, it covers only sandboxes, feature flags and infrastructure-as-code rather than the GraphQL concepts the API docs teach (projects, services, deployments, variables, domains, volumes), and its two documented entry points disagree on which environment variable authenticates. The railway npm package is a full TypeScript SDK with Sandbox and IaC APIs, detailed README, and configuration that matches the GraphQL endpoint and token types documented on docs.railway.com. However, docs.railway.com does not link to or mention this SDK, so an agent starting from the docs site would not discover it.

The receipt

Requests per hour: 100 RPH for Free customers, 1000 RPH for Hobby customers, 10000 RPH for Pro customers; custom for Enterprise.

Both tables are findable, but the API page defines X-RateLimit-Limit as daily beside hourly limits.

Agent surface notes

Initialize returned a valid OAuth-protected MCP authentication challenge.

The live docs expose no embedded Ask AI control.

Method note

This is a reading test of public documentation, not an execution test. No accounts were created and no API calls were run. The AI Agent Readiness Score counts fifteen reading votes at PASS 2, PARTIAL 1, and FAIL 0, for 30 possible points. Five readiness checks add 10 points each. The total is 80. Consensus chips show each row majority and do not affect scoring. The panel split on 4 of five tasks. Quotes shown here were re-fetched and confirmed verbatim on 2026-08-19.

Read the full methodology

Put another docs site through the battery.

Nominate a docs site