Analysis
RoboColosseum: no policy completes a task on the Unitree G1 humanoid, and the benchmark offers 10,000 free evaluation hours
Eight models tested on four robots; on the G1 humanoid, none of five VLA policies finished a task. Academic labs can apply for 10,000 free hours.
On November 5, 2026, RoboColosseum, an open robot-evaluation initiative launched by Hugging Face, BitRobot Network, the Singapore Embodied AI Centre and researcher Jiafei Duan, published its first paper and opened applications for 10,000 free evaluation hours. A precision on the headline: the study tested eight models, not five (five vision-language-action policies plus three frontier AI agents) on four robot types. "No policy completes a task" describes the five VLA policies on the Unitree G1 humanoid in the fine-tune track.
The essentials
- Who was tested: five VLA policies and three frontier AI agents, on four embodiments: SO-101, Franka, Bimanual-YAM and the Unitree G1 humanoid.
- The humanoid result: on the G1, none of the five policies completed any of the five tasks, yet average progress ranged from 13.3% to 23.2%; fewer than 40% of trials met even a quarter of the criteria.
- The wider finding: policy rankings do not transfer from one robot to another (Kendall W of 0.24 across suites).
- The offer: 10,000 free evaluation hours shared among academic labs and partners, allocated after the RoboColosseum team reviews each application.
- A snapshot: the authors describe the study as an initial snapshot, not an exhaustive evaluation of robot foundation models.
What was tested?
The benchmark has two tracks. In the Open Track, models are compared without task-specific adaptation, in blinded pairwise trials under matched local conditions: 5,699 comparisons across 124 sites. In the Fine-tune Track, each VLA policy is adapted to 20 tasks (five per embodiment) using the same 100 demonstrations per task, then makes 25 attempts per task; full successes and milestone progress are reported. The project says it runs hundreds of robots in more than 140 scenes.
What happened on the G1 humanoid?
The paper says no policy completed any of the five G1 tasks. Partial progress still differed between policies, from 13.3% to 23.2% on average, which is why the authors argue for graded scoring and not success alone. The failure is specific to this setup: five tasks, 100 demonstrations each, one humanoid model. It does not say humanoids cannot do these tasks, nor that the policies fail on other robots.
What do the rankings say?
After fine-tuning, GR00T N1.7 led on SO-101 while pi0.5 led on Franka and Bimanual-YAM, so the winner depends on the body. In the Open Track, frontier agents without adaptation rivaled specialized VLA policies, but all systems struggled to finish open-world tasks reliably. The summaries we read do not name the three frontier agents or give per-agent scores.
What are the 10,000 free hours?
According to the project site, teams with a policy for one of its robots can apply; the RoboColosseum team reviews every application and follows up. The hours are shared across all academic partners, not guaranteed to each applicant. The site lists Unitree G1 humanoids, bimanual YAM stations, Franka arms in the DROID setup and SO-ARM101 arms. A relay of the announcement says rollout data, including failures, will be public on Hugging Face. Unitree publicly said it supports the project and that the G1 is the primary humanoid embodiment.
What is not established?
- How the policies would do with more demonstrations, other tasks or another humanoid.
- Which three frontier agents were tested and how each scored.
- Whether the rollout data is already published; we saw it announced, not the dataset.
- How applications are ranked, or how many hours any lab will receive.
- Independent replication of the results.
Summary table
| Point | What is stated as of November 6, 2026 | Source |
|---|---|---|
| Models | 5 VLA policies + 3 frontier agents | arXiv, Hugging Face |
| Embodiments | SO-101, Franka, Bimanual-YAM, Unitree G1 | arXiv |
| G1 result | No task completed; progress 13.3%–23.2% | arXiv, AI Weekly |
| Open Track | 5,699 comparisons, 124 sites | Hugging Face, AI Weekly |
| Rank transfer | Kendall W 0.24 | AI Weekly |
| Free hours | 10,000, by application review | RoboColosseum site |
FAQ
Did five models fail on a humanoid?
Five VLA policies did not complete any G1 task after fine-tuning on 100 demonstrations; three other agents were tested in a different track.
Are the 10,000 hours open to everyone?
They are aimed at academic labs and partners, after an application review.
Does this show humanoids are far from useful?
No: it is one benchmark, with a limited task set and training data.
Method and limits
Sources checked on November 6, 2026. We read search extracts of the paper (arXiv, Hugging Face page), the project site, AI Weekly and two news relays; we did not read the full paper or the dataset. Figures are the authors'.
Sources
- arXiv: RoboColosseum, Evaluating Physical Intelligence at Scale In-The-Wild (verified 2026-11-06)
- Hugging Face papers: RoboColosseum (verified 2026-11-06)
- RoboColosseum project site (verified 2026-11-06)
- AI Weekly: RoboColosseum, policy rankings don't transfer across robots (verified 2026-11-06)
- TechFlow: RoboColosseum launches robot evaluation program (verified 2026-11-06)
- Unitree on X: support for RoboColosseum (verified 2026-11-06)
Spotted an error in this article? Report it · Our corrections policy
