Models
Can AI agents build an entire app? Replit's ViBench 2.0 puts them to the test
Replit's ViBench 2.0 tests whether AI agents can build products feature by feature. We examine the scores, costs, failures and limits.
An AI agent can create a convincing app demo in minutes. Keeping that app working as new features are added is a harder challenge. Replit's new ViBench 2.0 tests that problem by asking agents to build products from an empty project, then extend their own code feature by feature.
Published on October 28, the benchmark covers 17 products, 241 feature requests and 544 test plans. In the official runs reported by Replit, the three leading models passed every test plan in 49% to 65% of builds. Most individual plans passed, but interactions between features remained a common source of failure (Replit).
These are results from a benchmark created by Replit, a company that also sells AI development tools. Its partner Fireworks ran the official comparisons. They are useful evidence about this particular test, not a guarantee that one model will build your product successfully.
How ViBench 2.0 works
An agent starts with a short specification and an empty project. It builds an initial version, then adds up to 18 features without knowing the full sequence in advance. After the final feature, a separate grading agent runs the test plans in a real browser, clicking and typing as a user would (Replit).
The products imitate categories such as document editing, code hosting and ride-hailing. Tests check details like whether a fresh database starts correctly, whether an update overwrites another user's work, or whether a split fare adds up.
Eight products and their plans are public; nine are held out. Official comparisons use four builds per product and model, with the same OpenHands-based agent harness and medium reasoning effort. One shared harness helps comparison, but a model optimized for another setup may perform differently (public repository).
Results: passing every test is harder than passing most
Replit reports these official Fireworks results across all 17 products. Pass@1 here means the share of builds that passed every test plan, averaged across four builds per product.
| Model | Full-build pass@1 | Approximate cost per build |
|---|---|---|
| GPT-6.1 Sol | 65% | $12 |
| GPT-6 Astra | 60% | $90 |
| Claude Opus 5.5 | 49% | Not stated in Replit's text |
Source: Replit's ViBench 2.0 report, October 28, 2026. These costs describe the benchmark runs, not the price of building a production app or a subscription.
All three models passed more than 90% of individual test plans. That is a different measure from passing all plans in a build: an application can do most things correctly and still fail the full-build score because one important interaction breaks.
DeepSeek V4.1 Flash passed 77% of individual plans at about $2.50 per build. Replit reports that no open-weight model in the official comparison passed a complete build. The 77% plan rate must not be compared directly with the full-build pass@1 percentages above.
Where the builds break
Replit's examples include a draft pull request that does not display a merge conflict, a document editor that cannot comment on text inside a table cell, and an undo action that reverses a collaborator's change. Nearly three-quarters of failed builds from the leading models failed only at intersections between features, according to Replit (report).
That is a useful distinction for anyone using an agent to build a product. Each feature may appear to work in isolation while the combined workflow fails. A good-looking interface also does not demonstrate that permissions, saved data or concurrent edits are correct. ViBench does not score visual design.
A separate comparison uses only eight products
Replit also compared GPT generations on the eight public products. It reports full-build pass@1 of 31% for GPT-6 Sol, 53% for GPT-6.1 Sol and 47% for GPT-6.1 Terra. Replit ran the earlier GPT models separately, while the 6.1 Sol result came from Fireworks' official runs (Replit).
These numbers use a different subset from the 17-product table. For example, 53% for GPT-6.1 Sol on the public subset and 65% across all products are not contradictory, but should not be treated as the same measurement.
How reliable is the grading?
A grading agent using Claude Opus 5.5 evaluates the builds, so scoring errors remain possible. Replit says a second model confirmed 99.2% of a sample of 477 passing plans; the public repository notes that a grader can still pass a plan that should fail. A detailed methodology write-up was still forthcoming when Replit published its report (Replit; public repository).
The test also uses one harness, specific model settings and a finite set of products. It does not establish production security, scalability, maintainability, legal compliance or performance under real users. Replit's commercial interest and Fireworks' role as its partner are relevant when interpreting the report.
Our take
ViBench 2.0 asks a more realistic question than whether an agent can produce an attractive first screen: can it add features without breaking what came before? Its results suggest that leading models can complete much of the test work, while the last few failures can still matter to a user.
For a real project, use an agent to move quickly, but define acceptance criteria, test feature interactions, inspect database behavior and review security before release. A benchmark score can help select a starting point; your own tests determine whether the result is ready.
Sources
- Replit, ViBench 2.0 report, October 28, 2026
- ViBench public repository and evaluation limitations, checked October 29, 2026
Spotted an error in this article? Report it · Our corrections policy
