Rezona · Preprint 2026

OCTAVE

A Capability-Resolved Benchmark for End-to-End HTML5 Game Generation

Kehang Li1,4,*, Xinlan Xu3,*, Yulin Jin2, Heng Cao2, Xuchen Li4, Houxian Ren3, Zihao Ji2, Qiang Luo2, Yadong Mao2, Qilin He2, Jian Zhang2, Jian Zhao4, Zihao Wang3,c

1Shanghai Jiao Tong University 2Rezona Lab 3Shandong University 4Zhongguancun Academy *Equal contribution   cCorresponding author

loading…
Abstract

A single score can’t say which game-development capabilities an agent has. Requirement-level evidence can.

Complete game generation requires an agent to coordinate rules, interactions, states, and presentation throughout a long-horizon development process. Reporting only a single game-level aggregate score makes it difficult to determine which game-development capabilities an agent has implemented and which types of requirements remain unmet. To address this issue, we present OCTAVE, a benchmark for complete HTML5 game generation that organizes tasks around capability coverage and diagnoses implementation differences through requirement-level evidence. We establish eight Capability Dimensions and construct 180 tasks with 1,800 Evaluation Requirements from three sources: real player requests, commercial-mechanics reconstruction, and human expert design. The benchmark covers PC Web, Mobile Web, 2D, and 3D. The requirements distinguish Completeness, which assesses publicly specified core implementation, from Richness, which assesses hidden, deeper system relationships; static source-code judgments are then used to analyze implementation performance under different task conditions.

An all-task evaluation of 11 model–harness combinations yields a Completeness of 33.2% and a Richness of 32.5%, while 19.9% of generation records are marked as static gate failures or missing artifacts. After excluding same-model self-evaluation, even the best-performing combination has pass rates below 50% for both public and hidden requirements, indicating substantial gaps in both project delivery and requirement implementation. Further analysis shows that Richness is not more difficult than Completeness under every task condition. Current systems perform relatively better on Core Rules and Numerical Design, whereas AI/NPC and Feedback and Game Feel constitute more prominent capability gaps. Requirement difficulty provides a coarse-grained stratification but is only weakly negatively correlated with actual pass rates. A human cross-check on a limited sample shows that requirements judged satisfied by humans but failed by the automated evaluator constitute the primary direction of disagreement. OCTAVE thus extends the evaluation of complete game generation beyond a single aggregate score into a diagnostic evaluation localized by requirement, capability, and task condition.

A grid of nine games built by agents: real-time strategy, side-scrolling shooters, an isometric puzzle, a platformer, an ecosystem sandbox, a tower defense game and a bookshop sim
Figure 1. HTML5 games built end to end by Fable 5.1 and GPT-6-astra on OCTAVE tasks, spanning strategy, action, puzzle, sandbox and tabletop categories.
Benchmark design

Three things a game benchmark has to get right

Construction: theory, eight capabilities and three task sources are annotated, filtered by platform and specified as goal, mechanics and behavior, then turned into six public and four hidden requirements with a difficulty level and reviewed. Evaluation: eleven model–harness combinations (7 Claude Code, 4 Codex CLI) generate in 120 minutes from a blank start, a static gate checks entry files, imports and execution path (a failure zeroes every requirement), and a source judge marks each requirement pass or fail, with a human cross-check. Results are reported by capability, source and platform.
Figure 2. Benchmark construction (left) and evaluation (right). Tasks are selected for capability coverage, processed into platform-specific specifications, and constructed with public Completeness and hidden Richness requirements tagged C1–C8. Click to enlarge.

Compared with existing benchmarks

Requirements are counted across games without repetition across models or runs. † Only an overall evaluation is available, so each game counts as one requirement. — The cited paper does not give enough information to determine the total.

Task suite

Example task

task.md

            
Evaluation protocol

From a project folder to a score

    Bar charts of requirements per primary capability C1–C8, tasks covering each capability, and requirements per difficulty stratum (450 each).
    Figure 3. (a) Requirements per primary capability (the first capability label of each requirement, n = 1,800); (b) tasks covering each capability (n = 180); (c) requirements per difficulty stratum, four equal-count strata. C7 Feedback and Game Feel is the smallest group.

    Eight capability dimensions

    Every requirement is tagged with the capability it primarily tests. Each task has six public Completeness requirements and four hidden Richness requirements.

    Leaderboard

    How current agents score

    Requirement pass rates (%) over all 180 tasks. Click a column to sort.

    Analysis

    What we found

    Same prompt, two agents

    One task, side by side

    Each pair below was built from the identical task specification. Both clips start together.

    Demos

    Games the agents built

    Gameplay recordings, sampled across score tiers for each model: strong, typical and weak results are all shown.

    Citation

    Cite this work

    bibtex
    @article{rezona2026octave,
      title   = {OCTAVE: A Capability-Resolved Benchmark for End-to-End HTML5 Game Generation},
      author  = {Li, Kehang and Xu, Xinlan and Jin, Yulin and Cao, Heng and Li, Xuchen and Ren, Houxian and Ji, Zihao and Luo, Qiang and Mao, Yadong and He, Qilin and Zhang, Jian and Zhao, Jian and Wang, Zihao},
      journal = {arXiv preprint arXiv:2611.17681},
      year    = {2026}
    }