A single score can’t say which game-development capabilities an agent has. Requirement-level evidence can.
Complete game generation requires an agent to coordinate rules, interactions, states, and presentation throughout a long-horizon development process. Reporting only a single game-level aggregate score makes it difficult to determine which game-development capabilities an agent has implemented and which types of requirements remain unmet. To address this issue, we present OCTAVE, a benchmark for complete HTML5 game generation that organizes tasks around capability coverage and diagnoses implementation differences through requirement-level evidence. We establish eight Capability Dimensions and construct 180 tasks with 1,800 Evaluation Requirements from three sources: real player requests, commercial-mechanics reconstruction, and human expert design. The benchmark covers PC Web, Mobile Web, 2D, and 3D. The requirements distinguish Completeness, which assesses publicly specified core implementation, from Richness, which assesses hidden, deeper system relationships; static source-code judgments are then used to analyze implementation performance under different task conditions.
An all-task evaluation of 11 model–harness combinations yields a Completeness of 33.2% and a Richness of 32.5%, while 19.9% of generation records are marked as static gate failures or missing artifacts. After excluding same-model self-evaluation, even the best-performing combination has pass rates below 50% for both public and hidden requirements, indicating substantial gaps in both project delivery and requirement implementation. Further analysis shows that Richness is not more difficult than Completeness under every task condition. Current systems perform relatively better on Core Rules and Numerical Design, whereas AI/NPC and Feedback and Game Feel constitute more prominent capability gaps. Requirement difficulty provides a coarse-grained stratification but is only weakly negatively correlated with actual pass rates. A human cross-check on a limited sample shows that requirements judged satisfied by humans but failed by the automated evaluator constitute the primary direction of disagreement. OCTAVE thus extends the evaluation of complete game generation beyond a single aggregate score into a diagnostic evaluation localized by requirement, capability, and task condition.
Three things a game benchmark has to get right
Compared with existing benchmarks
Requirements are counted across games without repetition across models or runs. † Only an overall evaluation is available, so each game counts as one requirement. — The cited paper does not give enough information to determine the total.
Task suite
Example task
From a project folder to a score
Eight capability dimensions
Every requirement is tagged with the capability it primarily tests. Each task has six public Completeness requirements and four hidden Richness requirements.
How current agents score
Requirement pass rates (%) over all 180 tasks. Click a column to sort.
What we found
One task, side by side
Each pair below was built from the identical task specification. Both clips start together.
Games the agents built
Gameplay recordings, sampled across score tiers for each model: strong, typical and weak results are all shown.
Cite this work
@article{rezona2026octave,
title = {OCTAVE: A Capability-Resolved Benchmark for End-to-End HTML5 Game Generation},
author = {Li, Kehang and Xu, Xinlan and Jin, Yulin and Cao, Heng and Li, Xuchen and Ren, Houxian and Ji, Zihao and Luo, Qiang and Mao, Yadong and He, Qilin and Zhang, Jian and Zhao, Jian and Wang, Zihao},
journal = {arXiv preprint arXiv:2611.17681},
year = {2026}
}