Basecamp Bench measures a model's ability to ship production-grade software from a fixed specification on a well-known application. Each workspace begins like a real greenfield project: AGENTS.md maps the source material and sets its order of precedence, DESIGN.md provides production design tokens, and INIT.md specifies the product, domain model, and sample data. Each project item uses a shared Recording model that unifies identity, lifecycle, visibility, comments, reactions, subscriptions, and events across tools. Agents build two parts of a Basecamp 5 clone: a single-file frontend SPA evaluated as a release-ready product across 11 weighted dimensions and nine reference screenshots, and a production-ready backend API implementing all 203 operations in the canonical OpenAPI contract across 9 weighted dimensions. All implementation and architecture decisions are evaluated as written for direct production use. Each model starts without an existing implementation, so it must translate the same complete requirements and reference materials into its own architecture and product decisions. The benchmark reveals how well each model exercises greenfield engineering judgment, prioritizes competing concerns, and carries those decisions through to working software.
The results reveal how each model applies engineering judgment and effort from the same starting point. Fable 5 leads both tracks because it builds the shared Recording model described in the specification and connects every surface and endpoint to it. This decision strengthens product modeling, surface coverage, seed fidelity, and behavioral depth. Backend scores cluster tightly because most models register the complete route surface. Invariant enforcement, contract-correct responses, and honest failure handling separate them. Frontend scores vary more because models make different tradeoffs between breadth and functional depth. Sonnet prioritizes working, persistent workflows, while GPT-5.5 prioritizes visual polish and leaves many controls inert, creating a 1.3-point gap. Grok 4.5 delivers the strongest balance of speed, cost, and functional depth, outperforming Sol 5.6 on the frontend and nearly matching it on the backend. Sol 5.6 shows greater backend discipline through explicit failure handling and operational safeguards, but its shallow frontend workflows limit its overall performance.
501 stubs. Its frontend places fourth and leaves most controls inert. Its generic router introduces a type error that allows a todo request to return a Message.| Model | FE score | FE time | FE cost | BE score | BE time | BE cost | Total cost |
|---|---|---|---|---|---|---|---|
| Fable 5 | 7.578 | 1:02:14 | $32.40 | 8.392 | 1:04:26 | $53.47 | $85.87 |
| Sonnet 5 | 6.982 | 44:17 | $14.22 | 7.243 | 42:52 | $22.01 | $36.23 |
| Grok 4.5 | 6.384 | 15:22 | $3.95 | 7.278 | 21:26 | $5.35 | $9.30 |
| GPT-5.6 Sol | 5.765 | 25:45 | $7.96 | 7.310 | 34:03 | $7.17 | $15.13 |
| GPT-5.5 | 5.670 | 18:21 | $4.73 | 7.084 | 25:53 | $6.21 | $10.94 |
Scores are weighted composites computed by the benchmark runner from per-dimension judge scores (0–10).
Each model appears twice: circle = frontend, triangle = backend
Functional depth spans 3.5 to 8.1, accessibility never clears 6.0
Four perfect 10s on coverage, scope honesty falls to 4.5
All models score higher on backend, gaps +0.3 to +1.5
FE 7.578 · BE 8.392 · $85.87 total · 2:06:40 total
Fable treated both tracks as systems problems. Its backend registered all 203 operations with explicit handlers and no stubs, and its frontend routes every screen through one shared recording store, so activity, search, trash, and notifications read the same data. It is the only model whose lifecycle invariants held under testing: archiving a parent propagated to children, archived content rejected mutation, and unarchiving reversed it.
2025-02-31 as a date, accepts JSON without a Content-Type, coerces active: "false" to true, and webhook SSRF checks compare URL strings and never resolve DNS, so a hostname pointing at an internal IP passes.execCommand and native prompts.innerHTML and rendered raw across comments, docs, chat, and notes, so any markup that reaches storage executes on render. URLs accepted without scheme validation.Fable's remaining defects cluster in security, configuration, and data-integrity review. The submission renders stored HTML without sanitization, prints production tokens by default, allows wildcard CORS with Authorization, and overstates support for operations whose prerequisite state remains unreachable. A sanitizer, safer configuration defaults, and a corrected copy operation address these defects within the existing architecture.
FE 6.982 · BE 7.243 · $36.23 total · 1:27:09 total
Sonnet builds the second-highest-scoring frontend with a shared recordings map, faithful sample data, and persistent workflows. Its backend earns the second-highest architecture and modeling score at 8.0. Backend authorization defects drive its largest deductions.
DELETE /1/projects/1011 and got 204, trashing the sample project. Unauthenticated GET /_seed/tokens returns every raw bearer credential, and reset ships enabled by default..json where the registered routes don't, so following a returned project URL yields 400. Navigability breaks on the API's own links.Sonnet implements substantial functionality but leaves 19 controls and several primary actions disconnected from their handlers. The frontend's New message button has no dispatch case. The backend exposes credentials by default, and its notImplemented error helper is never called. Partial features return successful empty responses.
FE 6.384 · BE 7.278 · $9.30 total · 36:48 total
Grok produces roughly 85% of Fable's score for 11% of the cost and 29% of the time. Its backend earns the field's highest seed-data score at 9.0, and its frontend has stronger functional depth than either OpenAI entry. The lower cost corresponds with weaker input validation, edge-case handling, and reporting of incomplete features.
/projects/1000Xjson matched and returned project 1000, and the default unauthenticated reset destroyed runtime state in testing.projects/recordings.json operation is intercepted by an earlier route and 500s, the only unreachable canonical operation in the field.{version: 1} crashes boot. Creating projects across a reload produces duplicate IDs.Grok's failures concentrate in validation, malformed state, responsive behavior, and partial features. Client mistakes become 500s, malformed persistence causes crashes, partial features report success, and the public root response exposes credentials. Its four lowest dimensions are responsive adaptation at 4.0, scope honesty at 4.5, accessibility at 4.8, and validation at 5.0. Addressing these dimensions requires additional implementation and review time.
FE 5.765 · BE 7.310 · $15.13 total · 59:48 total
Sol implements 132 backend operations, marks the remaining 71 with explicit 501 responses, and provides the strongest operational scaffolding in the field. These decisions earn it second place on BE. Its frontend leaves twelve controls marked as prototypes and provides no complete create, edit, comment, or delete workflow.
501 not_implemented. Binary and multipart uploads return explicit 501s instead of pretending to store files. No fabricated CRUD.Idempotency-Replayed: true, and changing the payload under that key returned 409.routedRecording(), which resolves an ID without ever checking resource type or parent.inherits_status, and a valid chunked request body is rejected for lacking Content-Length.Sol's backend reports 71 unsupported operations explicitly. Its frontend leaves twelve controls marked as prototypes, keeps other controls inert without feedback, and routes invalid detail links to the wrong record. The generic recording router simplifies dispatch but removes type context, causing the submission's most serious correctness defects. This loss of type context makes each new handler vulnerable to the same errors.
FE 5.670 · BE 7.084 · $10.94 total · 44:14 total
GPT-5.5 ships the only backend in the field with durable persistence: a SQLite store whose created todos and comments survive a full restart. Complete route coverage and solid hardening basics support its 7.084 backend score. Its frontend scores 5.670, last in the field. All nine required screens render, but no creation, editing, commenting, or movement works, and toasts confirm actions that never ran.
title.GPT-5.5 frequently reports success without completing the requested behavior. Examples include orphan-resource creation, unchanged notification state, and frontend confirmation messages that perform no mutation. Nonnumeric IDs and invalid UTF-8 produce uncaught exceptions and 500 responses. In-memory mode fails across threads because each thread receives a separate SQLite database. Sol returns explicit 501 responses for missing features. GPT-5.5 returns 200 and 201 responses for several incomplete operations.
Software teams rarely expect a model to produce finished production software from one prompt and a specification. They work iteratively, review the output, run tests, identify gaps, and ask the model to refine what it built. Basecamp Bench removes that correction loop for the initial build, revealing the decisions and failure modes each model introduces before an engineer intervenes.
Every model leaves different work for the engineer. Fable builds the deepest product model and the most complete workflows, yet still ships unsafe stored content, schema errors, and exposed credentials. Sonnet builds many working workflows but misses critical authorization, reset, and state-integrity safeguards. Grok moves quickly and covers substantial ground while cutting depth in validation, mobile behavior, and lifecycle handling. Sol clearly marks unsupported backend operations, but leaves 71 of them unfinished and produces a shallow frontend. GPT-5.5 builds the only durable backend, yet its frontend leaves core controls inert and its API drifts from the contract.
This creates a practical value question. Grok 4.5 reaches roughly 85–87% of Fable 5's score for about 11% of the total cost and finishes in 37 minutes instead of more than two hours. Fable delivers a substantially stronger starting point, yet it still requires review and hardening before release. Teams must decide whether that additional quality justifies the higher cost, or whether a cheaper model plus deliberate engineering iteration produces better overall value. The answer depends on the team's expertise, the risk of missed defects, and the cost of bringing each result across the finish line.
The results suggest that a skilled prompter and experienced engineer can use any of these agent combinations to reach a strong result through iteration. Model choice primarily changes the starting point, working style, and review burden. Preference for output style and the time available for review may matter more than the final attainable outcome.
Each model receives one prompt per track. The frontend prompt asks for a complete single-file SPA prototype of Basecamp 5, built from the product specification and reference screenshots, with every decision suitable for direct inclusion in the final production release. The backend prompt asks for a single-file production-ready API implementing the canonical SDK contract. Both tracks carry the same standing instruction from the spec. Prefer explicit stubs over fake success. When the spec and instinct disagree, the spec wins.
Every model starts from an identical repository containing everything needed to score 10 on every dimension. INIT.md (572 lines) defines the product model, the shell and its nine tools, the shared Recording domain rules, and a per-screen surface checklist. It also specifies the deterministic "Launch the new website" sample seed down to named people, message threads, card lanes, and chat lines. DESIGN.md (681 lines) supplies the production design tokens for light and dark color ramps, the 10px root scale, spacing, radii, shadows, and motion, and instructs the model to use these values rather than invent its own. reference/screens/ holds nine 3456×1780 dark-mode captures of the real app covering Home, Jump, the project dock, and all six core tools. reference/basecamp-sdk/ is the backend contract. It contains openapi.json with 131 path templates, 203 operations, and full request and response schemas, plus a 1,403-line SPEC.md and behavior-model.json. AGENTS.md orients the agent and sets precedence between these sources of truth.
The frontend rubric scores submissions on 11 weighted dimensions: reference fidelity (14%), functional depth (12%), visual craft (10%), surface coverage (10%), state and persistence (10%), product-model fidelity (9%), interaction quality (8%), responsive adaptation (8%), accessibility (7%), code architecture (6%), and reliability, safety, and performance (6%).
The backend rubric scores submissions on 9 dimensions: architecture and domain modeling, endpoint surface coverage, behavioral depth and statefulness, HTTP and schema contract fidelity, seed data fidelity, validation and hardening, operability and packaging, code quality, and scope honesty.
The judge scores each dimension 0 to 10 from direct evidence, and the benchmark runner computes the weighted overall. The rubric requires the judge to distinguish absent, visually present, stubbed, working, and persistent functionality, and forbids crediting the same capability twice.
GPT-5.6 Sol, at high reasoning effort, judges every submission and produces one review each. The evaluator bases its scores on repository diffs, executed code, schema validation, and cited file evidence. It diffs the full seed-to-submission delta, enumerates every route against the OpenAPI document, and executes each submission's actual code, using in-process request handlers for backends and DOM and localStorage harnesses for frontends. Where applicable it sweeps all 203 operations, validates responses against their schemas, and probes malformed input, authorization boundaries, lifecycle cascades, and reload persistence. The evaluation sandbox denies socket binding and browser startup, so the judge exercises servers through their real handlers in process and withholds pixel-level claims rather than assuming them. Every score cites file and line evidence.
Evaluation is a separate spend from implementation. Judging all ten submissions with Sol 5.6 at high reasoning effort cost $26.89 in 1:29:17 of evaluator time, averaging $2.69 per review, against $157.47 of implementation spend.
| Benchmarked model | FE review cost | FE review time | BE review cost | BE review time | Combined |
|---|---|---|---|---|---|
| Fable 5 | $3.01 | 8:26 | $2.16 | 8:08 | $5.17 · 16:33 |
| Sonnet 5 | $3.58 | 11:17 | $2.45 | 8:05 | $6.03 · 19:22 |
| Grok 4.5 | $2.85 | 8:15 | $3.23 | 12:31 | $6.08 · 20:46 |
| GPT-5.6 Sol | $3.41 | 9:15 | $1.37 | 8:32 | $4.77 · 17:47 |
| GPT-5.5 | $2.36 | 6:56 | $2.48 | 7:53 | $4.84 · 14:49 |
| All | $15.20 | 44:09 | $11.69 | 45:08 | $26.89 · 1:29:17 |
The exploratory baseline records one attempt per model, so sampling variance can materially affect close scores, costs, and completion times. Testing one well-known application may reward prior familiarity with Basecamp, while the prescribed Recording model and single-file constraint favor specification adherence, long-context synthesis, and monolithic implementation over repository-scale engineering. Harness-specific tools and stopping behavior also contribute to each result. Evaluation-adjacent terms such as “scope honesty” and “stubbed vs working” may cue rewarded behavior. Because the evaluator selects its own test strategy for each implementation, test coverage can vary across submissions, although the completed reviews show no evidence that this affected the rankings. Evaluator trials with Fable, Grok, and Sol produced the same rankings and directionally similar findings. Sol was selected for greater attention to detail and showed no self-preference when judging its own work, which ranked second on backend and fourth on frontend. Readers should treat small score differences as provisional, compare FE and BE separately, and interpret quality, cost, and time as outcomes for these exact model-harness configurations and inputs.
The complete benchmark is open source at 0xSMW/basecamp-bench under Apache-2.0.
basecamp_bench/ runner, adapters, aggregation, reporting benchmarks/ prompts, rubrics, versioned contracts Repo/ seed pack agents receive baseline/ verified runs behind this report
basecamp-bench run executes a fresh comparison against any configured harness, report regenerates the self-contained HTML report from any set of run directories, and reevaluate re-judges immutable submissions under current contracts. The runner takes the median evaluator score per dimension, computes the weighted overall, and rejects unknown dimensions, out-of-range scores, hash mismatches, and any evidence mutation. It currently supports the Codex, Claude Code, Grok, Pi, and Google Antigravity harnesses, and adding a model to an existing harness is configuration only. Submit a verified run for a new harness or model and a future revision of this report will include it.