Published 2026-08-20 · Tested 2026-08-19

LangChain

B+

LangChain received 5 PASS votes and passed 5 of five readiness checks. The clearest finding came from the find the exact limits task.

Panel: GPT 5.6 Sol, Opus 5, DeepSeek v4F Battery: v1 Verified: 2026-08-19 Read as markdown (opens in a new tab)

Three AI models, GPT 5.6 Sol, Claude Opus 5, and DeepSeek v4 Flash, each read LangChain’s public documentation independently and attempted five first-hour developer jobs: build the first agent, find the exact limits, recover from model throttling, authenticate a model, use the Python SDK.

No accounts, API calls, or code execution were used. Every verdict came from public pages and every published quotation passed a live verification check. Preflight substitutions: T4: Model-provider authentication end to end replaced webhooks because the LangChain OSS docs expose no product-webhook interface.

LangChain Mintlify · published 2026-08-20
B+

87.5% · 70/80 · AI Agent Readiness Score · reading 30 pts · surface 50 pts

llms.txt PASS
llms-full.txt PASS
markdown mirror PASS
MCP server PASS
docs AI PASS
Task GPT 5.6 SolOpus 5DeepSeek v4F Consensus
Build the first agent PARTIALPARTIALPASS PARTIAL
Find the exact limits PASSPARTIALPARTIAL PARTIAL
Recover from model throttling PARTIALPARTIALPARTIAL PARTIAL
Authenticate a model PARTIALPASSPARTIAL PARTIAL
Use the python sdk PARTIALPASSPASS PASS

docs platform: Mintlify (unscored) · verified 2026-08-19

What the docs get right

  • Use the python sdk: 2 PASS votes. The langchain package and create_agent callable-tool API agree across pages, but the minimum OpenAI example omits langchain-openai.
  • Build the first agent: 1 PASS votes. The quickstart defines and invokes get_weather, but its installation step omits the independent langchain-openai package required by its OpenAI example.
  • 5 of 5 readiness checks. Present: llms.txt, llms-full.txt, markdown mirrors, an MCP server, docs AI.

Build the first agent

PARTIAL

PARTIAL consensus from 1 PASS, 2 PARTIAL.

The quickstart defines and invokes get_weather, but its installation step omits the independent langchain-openai package required by its OpenAI example. The code path is exact and self-contained , create_agent, a plain get_weather callable, and agent.invoke , but the quickstart's own install step lists only `pip install -U langchain`, while the install page states provider integrations live in separate packages, so the default OpenAI tab cannot run as written without cross-referencing a second page. Quickstart provides a complete code example with create_agent, a get_weather Python function passed as a tool, and agent.invoke() , install, API key, code, and invocation are all in one page.

Find the exact limits

PARTIAL

PARTIAL consensus from 1 PASS, 2 PARTIAL.

The LangSmith tracing path documents exact endpoint limits, plan-tier event limits, data-ingest limits, and monthly trace quotas with scopes and windows. Exact values, windows, and 429 semantics are published for LangSmith (5000 POST/PATCH /runs per minute, 50,000/250,000/500,000 trace events per hour by plan, 500MB/2.5GB/5.0GB hourly ingest, 5,000 traces per month), but the OSS agent path from T1 has no stated product limits anywhere, and the plan labels used to scope the quotas do not match the plan names used elsewhere in the docs. LangSmith plan quotas are stated (5k traces/month Developer, 10k/month Plus) but these are platform trace limits, not rate limits on the OSS agent path. The LLM Gateway rate-limit policies doc shows only configurable example values (100 req/min, 1,000,000 tokens/hr), not hard product-enforced quotas. No product-enforced rate limit applies to the documented OSS agent path because the user brings their own provider key.

Recover from model throttling

PARTIAL

PARTIAL consensus from 3 PARTIAL.

/oss/python/langchain/rate-limiting returned 404; the error page links complete Python limiter code but labels MODEL_RATE_LIMIT JavaScript-only. The error page enumerates all four remedies but links only the rate limiter; response caching and multi-provider distribution are unlinked prose with no matching OSS page (the Python index contains no fallback documentation at all, and the only caching section documents prompt caching, which lowers token cost rather than request count), and the page omits that Python chat models already retry 429s automatically. The error page lists four high-level approaches and links to the rate-limiting section of Models, which provides an InMemoryRateLimiter code example and max_retries with exponential backoff. However the error page notes it is JS-only. Caching and multi-provider distribution are named but lack complete code fixes on these pages. The fix is spread across two pages with no single copy-paste correction.

Authenticate a model

PARTIAL

PARTIAL consensus from 1 PASS, 2 PARTIAL.

OpenAI setup and invocation are complete, but the MODEL_AUTHENTICATION page labels the error JavaScript-only while showing Python correction code. The full path is documented without guessing: the credential name OPENAI_API_KEY, where to obtain it, three configuration methods (environment variable, getpass, explicit api_key kwarg), invocation through create_agent, and a dedicated MODEL_AUTHENTICATION page with a spelling, loading, and proxy checklist. Credential setup is well-documented in Quickstart (eight provider tabs with env vars) and Models page (init examples for seven providers). The MODEL_AUTHENTICATION error page provides troubleshooting steps and the explicit api_key bypass pattern. But the error page says it is JS-only, and there is no code example showing how to catch MODEL_AUTHENTICATION from within a running agent and respond to it.

Use the python sdk

PASS

PASS consensus from 2 PASS, 1 PARTIAL.

The langchain package and create_agent callable-tool API agree across pages, but the minimum OpenAI example omits langchain-openai. The package is unambiguously `langchain` on Python 3.10+, the minimum agent with a custom tool is on the quickstart, and the API surface agrees across the quickstart, the agents page, the tools page, and the JavaScript quickstart, which uses the same createAgent shape and the same get_weather tool. Install page shows pip install -U langchain (Python 3.10+). Quickstart shows the complete agent with create_agent, a plain Python function as a tool, and agent.invoke(). The API form is consistent across install, quickstart, agents, tools, and models pages.

The receipt

Personal organizations are limited to 5,000 traces per month until a credit card is added.

The LangSmith tracing path documents exact endpoint limits, plan-tier event limits, data-ingest limits, and monthly trace quotas with scopes and windows.

Agent surface notes

Initialize returned JSON-RPC protocol 2025-06-18 and server name Docs by LangChain.

The live docs expose a visible Ask AI control linked to the LangChain docs assistant.

Method note

This is a reading test of public documentation, not an execution test. No accounts were created and no API calls were run. The AI Agent Readiness Score counts fifteen reading votes at PASS 2, PARTIAL 1, and FAIL 0, for 30 possible points. Five readiness checks add 10 points each. The total is 80. Consensus chips show each row majority and do not affect scoring. The panel split on 4 of five tasks. Quotes shown here were re-fetched and confirmed verbatim on 2026-08-19.

Read the full methodology

Put another docs site through the battery.

Nominate a docs site