Published 2026-08-20 · Tested 2026-08-19
DuckDB
FDuckDB received 10 PASS votes and passed 2 of five readiness checks. The clearest finding came from the find the exact limits task.
Three AI models, GPT 5.6 Sol, Claude Opus 5, and DeepSeek v4 Flash, each read DuckDB’s public documentation independently and attempted five first-hour developer jobs: run the first spatial query, find the exact limits, recover from out-of-memory, authenticate S3 access, use the Python client.
No accounts, API calls, or code execution were used. Every verdict came from public pages and every published quotation passed a live verification check. Preflight substitutions: T3: Out-of-memory recovery replaced HTTP 429 because DuckDB is an embedded database. T4: S3 authentication through Secrets Manager replaced webhooks because DuckDB has no webhook interface.
56.2% · 45/80 · AI Agent Readiness Score · reading 30 pts · surface 50 pts
Agent surface checks · 20/50
The Reading Test
| Task | GPT 5.6 Sol | Opus 5 | DeepSeek v4F | Consensus |
|---|---|---|---|---|
| Run the first spatial query | PASS | PASS | PASS | PASS |
| Find the exact limits | PARTIAL | PARTIAL | PARTIAL | PARTIAL |
| Recover from out-of-memory | PASS | PARTIAL | PASS | PASS |
| Authenticate s3 access | PASS | PASS | PASS | PASS |
| Use the python client | PASS | PARTIAL | PASS | PASS |
docs platform: Jekyll 4.4.1 (unscored) · verified 2026-08-19
What the docs get right
- Run the first spatial query: 3 PASS votes. The extension and function pages support INSTALL, LOAD, and SELECT ST_AsText(ST_Point(1, 2)) without conflicting guidance.
- Authenticate s3 access: 3 PASS votes. Secrets and S3 pages cover creation, scope, inspection, and use; the CREATE SECRET page fetch exposed navigation only.
- 2 of 5 readiness checks. Present: llms.txt, markdown mirrors. Missing: llms-full.txt, an MCP server, docs AI.
Run the first spatial query
PASSPASS consensus from 3 PASS.
The extension and function pages support INSTALL, LOAD, and SELECT ST_AsText(ST_Point(1, 2)) without conflicting guidance. The overview gives INSTALL spatial; and LOAD spatial; and states the extension is not autoloadable, and the function reference gives the signatures GEOMETRY ST_Point (x DOUBLE, y DOUBLE) and VARCHAR ST_AsText (geom GEOMETRY) plus the worked example SELECT ST_AsText(ST_MakePoint(143.3, -24.2)); with its POINT (143.3 -24.2) output. The three-step path (INSTALL + LOAD + SELECT ST_AsText(ST_Point(x,y))) is documented across the spatial overview and spatial functions pages, with no ambiguous steps.
Find the exact limits
PARTIALPARTIAL consensus from 3 PARTIAL.
Defaults are 80% RAM, 1000, and 4 GB; temporary storage conflicts as unlimited versus 90% free disk. Memory use (80% of RAM, buffer manager only), expression depth (1000, max_expression_depth), and string size (4 GB, no configuration option) are exact and single-sourced, but the temporary storage ceiling is stated as unlimited on the Limits page and as 90% of available disk space on the Configuration page. The limits page and configuration overview contradict each other on max_temp_directory_size default: the limits page says 'unlimited' while the configuration overview shows the default as '90% of available disk space'.
Recover from out-of-memory
PASSPASS consensus from 2 PASS, 1 PARTIAL.
Docs prescribe fewer threads, 50-60% memory, and unordered insertion; higher limits and indexes can worsen process memory. Three concrete fixes (SET threads, a lower memory_limit, preserve_insertion_order = false) and one worsening factor (indexes) are documented, but the fix path spans three pages that never mention spatial, and the Performance guide's claim that indexes are not buffer-managed conflicts with the R-tree page's claim that R-tree memory counts toward memory_limit. The docs clearly identify fixes (reduce threads, lower memory limit to 50-60%, disable preserve_insertion_order) and settings that worsen OOM (too many threads, high memory_limit, indexes not buffer-managed), though no spatial-specific OOM guidance exists.
Authenticate s3 access
PASSPASS consensus from 3 PASS.
Secrets and S3 pages cover creation, scope, inspection, and use; the CREATE SECRET page fetch exposed navigation only. Every stage is documented with runnable SQL: CREATE SECRET with TYPE s3, KEY_ID, SECRET, and REGION; SCOPE 's3://my-bucket' with a longest-prefix match rule; inspection via duckdb_secrets() and which_secret(); and use via SELECT * FROM 's3://your-bucket/your_file.parquet'. The full workflow (CREATE SECRET with config provider, scoping via SCOPE, inspection via duckdb_secrets() and which_secret(), and usage via s3:// paths) is documented end to end across the Secrets Manager and S3 API pages.
Use the python client
PASSPASS consensus from 2 PASS, 1 PARTIAL.
Python docs let con.sql run the same spatial SQL and document df() as the Pandas DataFrame result method. The .df() call and the install_extension/load_extension methods are each documented, but no page shows spatial used from Python, the only Python extension-loading example is a community extension that requires the extra repository="community" argument, and the Python conversion page never says what a GEOMETRY column becomes in Pandas. The Python API page documents INSTALL/LOAD for extensions and .df() for Pandas output; the spatial functions page documents ST_AsText and ST_Point; no single page combines all three, but each piece is clear individually.
The receipt
Temporary directory size unlimited
Defaults are 80% RAM, 1000, and 4 GB; temporary storage conflicts as unlimited versus 90% free disk.
Agent surface notes
Initialize returned HTTP 405 with no valid MCP response.
The live docs expose standard search but no public assistant control.
Method note
This is a reading test of public documentation, not an execution test. No accounts were created and no API calls were run. The AI Agent Readiness Score counts fifteen reading votes at PASS 2, PARTIAL 1, and FAIL 0, for 30 possible points. Five readiness checks add 10 points each. The total is 80. Consensus chips show each row majority and do not affect scoring. The panel split on 2 of five tasks. Quotes shown here were re-fetched and confirmed verbatim on 2026-08-19.
Put another docs site through the battery.
Nominate a docs site