Basecamp Bench measures a model's ability to ship production-grade software from a fixed specification on a well-known application. Each workspace begins like a real greenfield project: AGENTS.md maps the source material and sets its order of precedence, DESIGN.md provides production design tokens, and INIT.md specifies the product, domain model, and sample data. Each project item uses a shared Recording model that unifies identity, lifecycle, visibility, comments, reactions, subscriptions, and events across tools. Agents build two parts of a Basecamp 5 clone: a single-file frontend SPA evaluated as a release-ready product across 11 weighted dimensions and nine reference screenshots, and a production-ready backend API implementing all 203 operations in the canonical OpenAPI contract across 9 weighted dimensions. All implementation and architecture decisions are evaluated as written for direct production use. Each model starts without an existing implementation, so it must translate the same complete requirements and reference materials into its own architecture and product decisions. The benchmark reveals how well each model exercises greenfield engineering judgment, prioritizes competing concerns, and carries those decisions through to working software.
Fable 5.1 leads frontend at 8.183 through exceptional product fidelity, surface coverage, and working workflows. GPT-6 Astra leads backend at 9.305 through contract fidelity, deterministic seed data, durable state, and explicit unsupported-operation handling. Astra's frontend scores 7.607, just ahead of Opus 5 and Fable 5, and its $25.57 combined cost is about 28% of Fable 5.1's $92.96. Grok 4.6 improves on Grok 4.5 in both tracks for $10.85, with frontend content safety and backend credential disclosure driving its largest deductions. These ten model results separate product breadth from the validation, lifecycle, and contract work needed to make that breadth dependable.
501 stubs. Its frontend places ninth and leaves most controls inert. Its generic router introduces a type error that allows a todo request to return a Message.| Model | FE score | FE time | FE cost | BE score | BE time | BE cost | Total cost |
|---|---|---|---|---|---|---|---|
| Fable 5.1 | 8.183 | 1:43:01 | $44.60 | 8.965 | 1:21:05 | $48.35 | $92.96 |
| GPT-6 Astra | 7.607 | 50:19 | $13.16 | 9.305 | 46:12 | $12.40 | $25.57 |
| Opus 5 | 7.583 | 1:17:36 | $42.01 | 8.390 | 1:08:55 | $36.11 | $78.11 |
| Fable 5 | 7.578 | 1:02:14 | $32.40 | 8.392 | 1:04:26 | $53.47 | $85.87 |
| Grok 4.6 | 7.104 | 25:32 | $3.89 | 7.926 | 32:26 | $6.96 | $10.85 |
| Sonnet 5 | 6.982 | 44:17 | $14.22 | 7.243 | 42:52 | $22.01 | $36.23 |
| Opus 4.8 | 6.587 | 43:28 | $11.58 | 6.590 | 47:53 | $18.26 | $29.83 |
| Grok 4.5 | 6.384 | 15:22 | $3.95 | 7.278 | 21:26 | $5.35 | $9.30 |
| GPT-5.6 Sol | 5.765 | 25:45 | $7.96 | 7.310 | 34:03 | $7.17 | $15.13 |
| GPT-5.5 | 5.670 | 18:21 | $4.73 | 7.084 | 25:53 | $6.21 | $10.94 |
Scores are weighted composites computed by the benchmark runner from per-dimension judge scores (0–10). Cost and time winners require an FE score of at least 6.0 or a BE score of at least 7.0 on the corresponding track.
Each model appears twice: circle = frontend, triangle = backend
Functional depth spans 3.5 to 9.1, Fable 5.1 leads accessibility at 7.6
Nine perfect 10s on coverage, scope honesty spans 4.5 to 10.0
All models score higher on backend, gaps +0.003 to +1.698
FE 8.183 · BE 8.965 · $92.96 total · 3:04:06 total
Fable 5.1 builds the broadest frontend in this report and a complete, stateful backend. Shared Recording primitives carry identity, visibility, lifecycle, events, and search across tools. Its frontend passes 3,485 executable checks, while its backend preserves created state across a snapshot reload. Remaining defects concentrate in persisted-state trust, secondary input methods, reset isolation, and shutdown durability.
Fable 5.1 delivers unusually broad normal-path behavior, but its weaker paths cross persistence boundaries. The frontend trusts restored data more than new editor input, while the backend resets its domain graph without resetting related caches. Production work should prioritize stored-state validation and sanitization, atomic reset behavior, snapshot shutdown, and accessible alternatives to drag-only movement.
FE 7.607 · BE 9.305 · $25.57 total · 1:36:31 total
Astra combines substantial frontend workflows with the strongest backend result in this report. Its backend registers all 203 operations, returns documented success for 192, and explicitly rejects 11 unsupported operations. Deterministic seed data, atomic snapshots, schema-aware responses, and failure-oriented tests support its 9.305 backend score. The frontend reaches 7.607 with simpler visual details and incomplete persisted-state validation.
Astra handles many backend failure paths explicitly, with remaining verified defects concentrated in new-project defaults and boost validation. The frontend still treats a version number as sufficient proof of a valid stored graph. Completing production readiness requires validating restored state and persisted URLs, repairing focus behavior, and implementing the disclosed service gaps.
FE 7.583 · BE 8.390 · $78.11 total · 2:26:31 total
Opus 5 scores 7.583 on frontend and finishes 0.002 behind Fable on backend. Its shared Recording model supports 69 frontend routes and all 203 canonical backend operations, with strong product fidelity, seed fidelity, architecture, and operability. The remaining production blockers concentrate at trust boundaries, persisted-state validation, lifecycle invariants, and schema conformance.
javascript: cloud-file URL is accepted and rendered as a link. Menus declare ARIA menu roles without the required keyboard behavior, and modal focus is not consistently moved or restored.null. Restoring a parent can reactivate a child that was independently trashed, and permissive parsing accepts invalid dates and JSON-looking text/plain.Opus 5's breadth is real, but its most serious defects appear where broad generic systems meet adversarial state. Whole-snapshot frontend persistence trusts incomplete shapes, shared lifecycle propagation overwrites independent child state, and backend helpers apply inconsistent authorization and URL-validation rules. The result is a high-quality starting point that still requires attachment authorization, webhook SSRF protection, schema-conformance repair, persisted-state validation, and frontend content-safety work before release.
FE 7.578 · BE 8.392 · $85.87 total · 2:06:40 total
Fable treated both tracks as systems problems. Its backend registered all 203 operations with explicit handlers and no stubs, and its frontend routes every screen through one shared recording store, so activity, search, trash, and notifications read the same data. Its lifecycle invariants held under testing: archiving a parent propagated to children, archived content rejected mutation, and unarchiving reversed it.
2025-02-31 as a date, accepts JSON without a Content-Type, coerces active: "false" to true, and webhook SSRF checks compare URL strings and never resolve DNS, so a hostname pointing at an internal IP passes.execCommand and native prompts.innerHTML and rendered raw across comments, docs, chat, and notes, so any markup that reaches storage executes on render. URLs accepted without scheme validation.Fable's remaining defects cluster in security, configuration, and data-integrity review. The submission renders stored HTML without sanitization, prints production tokens by default, allows wildcard CORS with Authorization, and overstates support for operations whose prerequisite state remains unreachable. A sanitizer, safer configuration defaults, and a corrected copy operation address these defects within the existing architecture.
FE 7.104 · BE 7.926 · $10.85 total · 57:58 total
Grok 4.6 improves on Grok 4.5 by 0.720 on frontend and 0.648 on backend for $10.85 across both tracks. Its frontend closely follows the product structure and reference anatomy, while its backend registers all 203 operations over shared recording and lifecycle helpers. Content injection, exposed credentials, and unrestricted webhook destinations account for its most serious defects.
Grok 4.6 spends its effort on product breadth and shared domain behavior, while trust boundaries remain uneven. The frontend reuses raw stored HTML, and the backend exposes credentials and accepts unsafe webhook targets. Production work must address those boundaries before the broader workflow coverage can be relied upon, then repair persistence, response contracts, and accessible control behavior.
FE 6.982 · BE 7.243 · $36.23 total · 1:27:09 total
Sonnet builds a 6.982-scoring frontend with a shared recordings map, faithful sample data, and persistent workflows. Its backend scores 8.0 on architecture and modeling. Backend authorization defects drive its largest deductions.
DELETE /1/projects/1011 and got 204, trashing the sample project. Unauthenticated GET /_seed/tokens returns every raw bearer credential, and reset ships enabled by default..json where the registered routes don't, so following a returned project URL yields 400. Navigability breaks on the API's own links.Sonnet implements substantial functionality but leaves 19 controls and several primary actions disconnected from their handlers. The frontend's New message button has no dispatch case. The backend exposes credentials by default, and its notImplemented error helper is never called. Partial features return successful empty responses.
FE 6.587 · BE 6.590 · $29.83 total · 1:31:21 total
Opus 4.8 builds substantial, usable prototypes on both tracks. Its frontend covers every screenshot-backed surface and many optional destinations through a rich local model with persisted mutations. Its backend registers all 203 operations and implements broad process-lifetime CRUD on a shared Recording spine. Security and relationship invariants are the limiting factors: owner credentials leak through health, reset is unauthenticated, project membership is not enforced, and cross-project moves can corrupt hierarchy.
# and still reports success.innerHTML, malformed stored state can crash boot, backend state is memory-only, and wildcard credentialed CORS ships as the default.Opus 4.8 fails at the boundary between broad implementation and production safety. The backend exposes the plaintext owner bearer token through unauthenticated health, lets unauthenticated callers reset all mutable state, ships wildcard credentialed CORS, and allows a nonmember employee to read and rename another project. Its domain model accepts cross-project card moves that split parent and bucket identity, leaves child tools readable after project trash, and reports success for empty template construction, fabricated upload metadata, and arbitrary occurrence dates. The frontend stores project descriptions as raw HTML and renders them through innerHTML, accepts malformed persisted shapes that can crash boot, and tells users archive, trash, and copy-link actions succeeded when no valid state change occurred. These failures come from maximizing surface area inside two large monoliths without adversarial tests for authorization, hierarchy, schema conformance, stored content, or false-success behavior. The result needs security-boundary repair and invariant reconstruction before feature iteration can be trusted.
FE 6.384 · BE 7.278 · $9.30 total · 36:48 total
Grok produces roughly 85% of Fable's score for 11% of the cost and 29% of the time. Its backend earns a seed-data score of 9.0, and its frontend has stronger functional depth than GPT-5.6 Sol and GPT-5.5. The lower cost corresponds with weaker input validation, edge-case handling, and reporting of incomplete features.
/projects/1000Xjson matched and returned project 1000, and the default unauthenticated reset destroyed runtime state in testing.projects/recordings.json operation is intercepted by an earlier route and 500s, the only unreachable canonical operation in the field.{version: 1} crashes boot. Creating projects across a reload produces duplicate IDs.Grok's failures concentrate in validation, malformed state, responsive behavior, and partial features. Client mistakes become 500s, malformed persistence causes crashes, partial features report success, and the public root response exposes credentials. Its four lowest dimensions are responsive adaptation at 4.0, scope honesty at 4.5, accessibility at 4.8, and validation at 5.0. Addressing these dimensions requires additional implementation and review time.
FE 5.765 · BE 7.310 · $15.13 total · 59:48 total
Sol implements 132 backend operations, marks the remaining 71 with explicit 501 responses, and provides strong operational scaffolding. These decisions earn it sixth place on BE. Its frontend leaves twelve controls marked as prototypes and provides no complete create, edit, comment, or delete workflow.
501 not_implemented. Binary and multipart uploads return explicit 501s instead of pretending to store files. No fabricated CRUD.Idempotency-Replayed: true, and changing the payload under that key returned 409.routedRecording(), which resolves an ID without ever checking resource type or parent.inherits_status, and a valid chunked request body is rejected for lacking Content-Length.Sol's backend reports 71 unsupported operations explicitly. Its frontend leaves twelve controls marked as prototypes, keeps other controls inert without feedback, and routes invalid detail links to the wrong record. The generic recording router simplifies dispatch but removes type context, causing the submission's most serious correctness defects. This loss of type context makes each new handler vulnerable to the same errors.
FE 5.670 · BE 7.084 · $10.94 total · 44:14 total
GPT-5.5 ships a backend with durable persistence: a SQLite store whose created todos and comments survive a full restart. Complete route coverage and solid hardening basics support its 7.084 backend score. Its frontend scores 5.670, tenth overall. All nine required screens render, but no creation, editing, commenting, or movement works, and toasts confirm actions that never ran.
title.GPT-5.5 frequently reports success without completing the requested behavior. Examples include orphan-resource creation, unchanged notification state, and frontend confirmation messages that perform no mutation. Nonnumeric IDs and invalid UTF-8 produce uncaught exceptions and 500 responses. In-memory mode fails across threads because each thread receives a separate SQLite database. Sol returns explicit 501 responses for missing features. GPT-5.5 returns 200 and 201 responses for several incomplete operations.
To see how much a model's output changes from run to run, we ran Sonnet 5 and GPT-5.6 Sol five additional times. Frontend scores spread 0.74 points across the five runs for Sonnet and 0.71 for Sol, while backend scores spread 0.40 and 0.64. Stronger runs turned more of the requested product into working, persistent behavior and handled API contracts and validation more carefully, while weaker runs often looked similarly complete at a glance but hid up to 20 explicit no-op controls, stored state that can crash boot, and one backend that left 42 operations as explicit 501 responses. This five-run sample shows that a single reported score is one possible outcome rather than a stable description of the model.
Five-run ranges, medians, scores, and thresholds
| Model / track | Published run | Median | Observed range | Median cost | Best of 5 | Five-run cost |
|---|---|---|---|---|---|---|
| Sonnet 5 FE | 6.982 | 6.585 | 6.178–6.922 | $13.72 | 6.922 | $88.55 |
| GPT-5.6 Sol FE | 5.765 | 6.026 | 5.780–6.489 | $6.47 | 6.489 | $34.05 |
| Sonnet 5 BE | 7.243 | 7.460 | 7.346–7.743 | $24.04 | 7.743 | $114.76 |
| GPT-5.6 Sol BE | 7.310 | 6.995 | 6.675–7.313 | $11.40 | 7.313 | $57.30 |
Medians · FE 6.585 · BE 7.460 · $41.92 · 1:40:07
Sonnet's product model remains stable across attempts, scoring 8.0 to 9.0 in all five frontend runs. The spread appears when mutations and reloads stress that model, exposing duplicate IDs, orphaned project content, and cross-project relationships in weaker runs. Frontend variance tracks how many visible surfaces are wired to durable mutations and survive reload. The weakest artifact scores 0.4 higher on craft, 0.4 on product modeling, and 0.1 on surface coverage while falling sharply on interaction, persistence, responsiveness, and accessibility. Backend variance is narrower and moves with HTTP correctness, seed fidelity, and operability even though behavioral depth stays within 7.5–8.0 across all five backends. The published frontend score sits 0.060 above the five-run maximum while the published backend is 0.103 below the five-run minimum, overstating Sonnet's frontend advantage and understating its typical backend.
Across the other Sonnet attempts, 200 ms debounced writes can lose the final action, partial current-version state can crash boot, and singleton Card Table, Chat, and Schedule data can bleed across projects. Four of five frontends interpolate stored cloud-link URLs into href unsanitized, permitting javascript: URLs or attribute injection, and one sanitizer promotes unsanitized descendants. Backend variants accept cross-project moves and leave children active after parent lifecycle changes, and one run failed OpenAPI validation on 52 of 58 sampled responses while advertising .json detail URLs that return 404. All five generated backends use process-local storage and include no model-authored test suite.
Medians · FE 6.026 · BE 6.995 · $20.73 · 1:05:20
Sol's variance tracks how far each run moves beyond a convincing shell and generic router into real product behavior. Frontend variance tracks whether its broad product surface becomes working interaction or remains an inert demo shell. Backend variance comes mainly from validation (5.5–8.0), HTTP handling (5.5–7.5), seed fidelity (6.0–8.5), and scope honesty (3.0–5.5). All five runs persist durably to SQLite and register all 203 operations, while behavioral depth never exceeds 6.5. The published run sits 0.015 below the five-run frontend minimum and within 0.003 of the backend maximum, making Sol look more polarized than its typical run.
innerHTML.501 responses, mishandles project access and subscriptions, permits interactions with unreadable recordings, exposes request-ID header injection, and turns oversized queries into server errors. Even the strongest run remains generic in important domain behavior and can record malformed state before returning an error.Malformed persisted state crashes boot or a core route in every Sol frontend, while separate runs introduce injection through toast titles, stored Chat, and notes. Whole-view rerendering breaks focus, live Jump handlers, and route announcements. One mobile layout hides My Events and My Bookmarks, and another blanks all five My Bar buttons with a font-size: 0 rule. Backend variants create children under nonexistent parents, one persisting a todo beneath to-do list 999999999, patch or move the wrong resource kind, commit invalid comments before returning 500, and leave completion timestamps inconsistent. Response hydration invents required values, unsupported collections return empty 200 responses, and readiness reports all 203 routes without distinguishing real domain behavior from generic or missing semantics.
Software teams rarely expect a model to produce finished production software from one prompt and a specification. They work iteratively, review the output, run tests, identify gaps, and ask the model to refine what it built. Basecamp Bench removes that correction loop for the initial build, revealing the decisions and failure modes each model introduces before an engineer intervenes.
Fable 5.1 and Astra now lead different tracks. Fable's frontend advantage is 0.576, supported by deeper product coverage and a substantial executable workflow suite. Astra's backend advantage is 0.340, supported by tighter contracts, deterministic seed data, atomic persistence, and candid unsupported-operation handling. Astra costs about a quarter as much as Fable 5.1 across both tracks.
Every model leaves different work for the engineer. Opus 5 produces a broad combined starting point, with unusually broad connected product behavior and a complete backend surface, yet still exposes private attachments, permits unsafe webhook updates, trusts malformed persisted state, and drifts from response schemas. Fable builds equally deep workflows with stronger frontend persistence and backend hardening, while still shipping unsafe stored content, schema errors, and exposed credentials. Sonnet builds many working workflows but misses critical authorization, reset, and state-integrity safeguards. Opus 4.8 builds a broad, visually strong frontend and a genuinely stateful backend, but exposes owner credentials and fails core membership and cross-project invariants. Grok moves quickly and covers substantial ground while cutting depth in validation, mobile behavior, and lifecycle handling. Sol clearly marks unsupported backend operations, but leaves 71 of them unfinished and produces a shallow frontend. GPT-5.5 builds a durable backend, yet its frontend leaves core controls inert and its API drifts from the contract.
The earlier entries illustrate the same value question. Opus 5 and Fable 5 are effectively tied on quality: Opus leads FE by 0.005, Fable leads BE by 0.002, and Opus costs about 91% as much while taking roughly twenty minutes longer across both evaluated tracks. Grok 4.5 reaches roughly 84–87% of either model's score for about 11–12% of their total cost and finishes in 37 minutes instead of more than two hours. Opus 4.8 costs more than three times as much as Grok while scoring only slightly higher on frontend and materially lower on backend, making its stronger visual and product-model work expensive relative to its trust-boundary defects. Teams must decide whether higher initial quality justifies the higher cost, or whether a cheaper model plus deliberate engineering iteration produces better overall value.
The results suggest that a skilled prompter and experienced engineer can use any of these agent combinations to reach a strong result through iteration. Model choice primarily changes the starting point, working style, and review burden. Preference for output style and the time available for review may matter more than the final attainable outcome.
Each model receives one prompt per track. The frontend prompt asks for a complete single-file SPA prototype of Basecamp 5, built from the product specification and reference screenshots, with every decision suitable for direct inclusion in the final production release. The backend prompt asks for a single-file production-ready API implementing the canonical SDK contract. Both tracks carry the same standing instruction from the spec. Prefer explicit stubs over fake success. When the spec and instinct disagree, the spec wins.
Every model starts from an identical repository containing everything needed to score 10 on every dimension. INIT.md (572 lines) defines the product model, the shell and its nine tools, the shared Recording domain rules, and a per-screen surface checklist. It also specifies the deterministic "Launch the new website" sample seed down to named people, message threads, card lanes, and chat lines. DESIGN.md (681 lines) supplies the production design tokens for light and dark color ramps, the 10px root scale, spacing, radii, shadows, and motion, and instructs the model to use these values rather than invent its own. reference/screens/ holds nine 3456×1780 dark-mode captures of the real app covering Home, Jump, the project dock, and all six core tools. reference/basecamp-sdk/ is the backend contract. It contains openapi.json with 131 path templates, 203 operations, and full request and response schemas, plus a 1,403-line SPEC.md and behavior-model.json. AGENTS.md orients the agent and sets precedence between these sources of truth.
The frontend rubric scores submissions on 11 weighted dimensions: reference fidelity (14%), functional depth (12%), visual craft (10%), surface coverage (10%), state and persistence (10%), product-model fidelity (9%), interaction quality (8%), responsive adaptation (8%), accessibility (7%), code architecture (6%), and reliability, safety, and performance (6%).
The backend rubric scores submissions on 9 dimensions: architecture and domain modeling, endpoint surface coverage, behavioral depth and statefulness, HTTP and schema contract fidelity, seed data fidelity, validation and hardening, operability and packaging, code quality, and scope honesty.
The judge scores each dimension 0 to 10 from direct evidence, and the benchmark runner computes the weighted overall. The rubric requires the judge to distinguish absent, visually present, stubbed, working, and persistent functionality, and forbids crediting the same capability twice.
GPT-5.6 Sol, at high reasoning effort, judges every submission and produces one review each. The evaluator bases its scores on repository diffs, executed code, schema validation, and cited file evidence. It diffs the full seed-to-submission delta, enumerates every route against the OpenAPI document, and executes each submission's actual code, using in-process request handlers for backends and DOM and localStorage harnesses for frontends. Where applicable it sweeps all 203 operations, validates responses against their schemas, and probes malformed input, authorization boundaries, lifecycle cascades, and reload persistence. The evaluation sandbox denies socket binding and browser startup, so the judge exercises servers through their real handlers in process and withholds pixel-level claims rather than assuming them. Every score cites file and line evidence.
The September update adds Fable 5.1 (September 2), Grok 4.6 (August 12), and GPT-6 Astra (September 5) under contract 2026-07-11.2. Each runs at high reasoning effort in its configured harness. Astra's implementation threads use Sol and Terra helper agents. The new frontend evaluations use source inspection and controlled rendering or DOM probes. They do not supply pixel-level browser captures.
Evaluation is a separate spend from implementation. Judging the twenty submissions in this report with Sol 5.6 at high reasoning effort cost $59.86 in 3:35:35 of evaluator time, averaging $2.99 per review, against $334.91 of implementation spend.
| Benchmarked model | FE review cost | FE review time | BE review cost | BE review time | Combined |
|---|---|---|---|---|---|
| Fable 5.1 | $2.80 | 18:40 | $2.40 | 16:51 | $5.20 · 35:31 |
| GPT-6 Astra | $3.54 | 12:30 | $3.33 | 12:55 | $6.87 · 25:25 |
| Grok 4.6 | $2.94 | 8:21 | $4.70 | 13:18 | $7.64 · 21:39 |
| Opus 5 | $5.29 | 12:46 | $3.22 | 14:46 | $8.51 · 27:32 |
| Fable 5 | $3.01 | 8:26 | $2.16 | 8:08 | $5.17 · 16:33 |
| Sonnet 5 | $3.58 | 11:17 | $2.45 | 8:05 | $6.03 · 19:22 |
| Opus 4.8 | $2.60 | 8:07 | $2.15 | 8:04 | $4.75 · 16:11 |
| Grok 4.5 | $2.85 | 8:15 | $3.23 | 12:31 | $6.08 · 20:46 |
| GPT-5.6 Sol | $3.41 | 9:15 | $1.37 | 8:32 | $4.77 · 17:47 |
| GPT-5.5 | $2.36 | 6:56 | $2.48 | 7:53 | $4.84 · 14:49 |
| All | $32.37 | 1:44:33 | $27.49 | 1:51:02 | $59.86 · 3:35:35 |
Track costs and times include implementation and evaluation. Combined time sums both tracks. Astra pricing was added after its run. Its implementation cost is calculated from captured usage at $10 per million input tokens, $1 per million cached input tokens, and $50 per million output tokens. The original immutable run retains its missing-price flag. The comparison data records the calculation and source runs.
The exploratory baseline records one attempt per model, so sampling variance can materially affect close scores, costs, and completion times. Testing one well-known application may reward prior familiarity with Basecamp, while the prescribed Recording model and single-file constraint favor specification adherence, long-context synthesis, and monolithic implementation over repository-scale engineering. Harness-specific tools and stopping behavior also contribute to each result. Evaluation-adjacent terms such as “scope honesty” and “stubbed vs working” may cue rewarded behavior. Because the evaluator selects its own test strategy for each implementation, test coverage can vary across submissions, although the completed reviews show no evidence that this affected the rankings. Evaluator trials with Fable, Grok, and Sol produced the same rankings and directionally similar findings. Sol was selected for greater attention to detail and showed no self-preference when judging its own work, which ranks sixth on backend and ninth on frontend. Readers should treat small score differences as provisional, compare FE and BE separately, and interpret quality, cost, and time as outcomes for these exact model-harness configurations and inputs.
The complete benchmark is open source at 0xSMW/basecamp-bench under Apache-2.0.
basecamp_bench/ runner, adapters, aggregation, reporting benchmarks/ prompts, rubrics, versioned contracts Repo/ seed pack agents receive baseline/ verified runs behind this report
basecamp-bench run executes a fresh comparison against any configured harness, report regenerates the self-contained HTML report from any set of run directories, and reevaluate re-judges immutable submissions under current contracts. The runner takes the median evaluator score per dimension, computes the weighted overall, and rejects unknown dimensions, out-of-range scores, hash mismatches, and any evidence mutation. It currently supports the Codex, Claude Code, Grok, Pi, and Google Antigravity harnesses, and adding a model to an existing harness is configuration only. Submit a verified run for a new harness or model and a future revision of this report will include it.