Open-ended agents in Craftax
Craftax
A long-horizon, open-ended world combining survival, crafting, exploration, combat, and dungeon progression. We evaluate whether language-model agents can turn its achievement ladder into durable capability.
Call budget is the binding cap here — more calls simply means more of the world seen — so a ranking read across budgets measures the budget, not the model. Everything below obeys this filter.
Cost vs. Performance
Average distinct achievements against average API cost per terminal Craftax rollout. Lines connect each model's measured reasoning efforts in low → medium → high order; larger markers mean more effort.
Hover a point for its cohort detail. Terra and Luna retain their Jul 9 launch-price points and add Jul 30 price-cut snapshots using the same measured token usage.
Per-Achievement Breakdown
All 66 possible achievements, ordered by Craftax progression and annotated by capability family and world region. Each cell is based on 15 terminal rollouts.
Trajectories
Select a model and rollout, then inspect the complete Craftax world replay, policy reasoning, actions, survival state, and achievement timeline.
Craftax · sealed V5
gpt-5.6-terra / low / seed 9
trajectory evidence
Achievements over policy turns
The Craftax analogue of Voyager's capability curve: each stepped line is the mean number of distinct achievements across the selected real rollouts. A policy turn equals one in-game action. Crafting icons mark where each cohort first hits a ≥50% unlock rate; one callout per achievement lands on the earliest such unlock.
Achievement capability tree
Direct engine gates and tier prerequisites across all 66 Craftax achievements. Choose a model to color each node by the share of real terminal runs that unlocked it.
Seed-adjusted comparison
Paired achievement effects
Pick two policies. For every achievement, we compare whether each policy unlocked it on the exact same starting seed. This shows where changing the policy alters the capability ladder—not merely which model has the higher aggregate score.
Achievement score
0.00 per run
95% paired interval [0.00, 0.00]
Survival
0 percentage points
candidate minus reference survival rate
Inference cost
+$0.000
mean cost difference per rollout
These are paired treatment effects of policy choice on achievement outcomes over the observed seeds. They are not CATEs: credible conditional effects require more shared seeds and pre-rollout world features such as biome, nearby resources, and threat density.
What sets the best rollouts apart
The top 23 rollouts by achievement score (score ≥ 14.0) against the remaining 70, under the current filters. Both groups played the same engine, so a gap here is a difference in how the policy spent its turns.
What accompanies a high-scoring run
A direct comparison of the top 23 runs with the other 70. Because the groups are defined by achievement score, score itself is intentionally omitted. These are associated outcomes, not independent causes. Orange is the high-scoring group; gray is everyone else. Values are group averages, except survival, which is the share that finished without dying.
Survived to a cap
70%vs33%
+37 percentage points
Tool types crafted
4.30vs2.69
+1.62 per run
Resources gathered
13.7vs8.37
+5.28 per run
Engine steps taken
how long the run stayed active
Resources gathered
resource events reached
Tool types crafted
distinct crafting unlocks
Enemy types beaten
distinct combat unlocks
Survived to a cap
finished without dying
What stronger runs spent
Output tokens: 122,423 vs 57,633
Inference cost: $0.41 vs $0.20
These are costs of longer, more capable runs—not capabilities by themselves.
“Tool” and “enemy” counts are distinct achievement types reached, not repeated crafts or kills; the trace records unlocks, not lifetime action counters.
The best runs
The ten highest-scoring rollouts under the current filters.
Dungeon
Craftax level 0 is the overworld; every level above it is dungeon. This section counts only what happens below ground — descents, floor gates, and the achievements that cannot be earned on the surface.
Descent rate by cohort
Share of each cohort’s terminal rollouts that entered the dungeon at least once. Cohorts are listed with their call budget because a policy with more calls has more turns in which to find a way down.
Floor gates
The dungeon spine in descent order. Each gate must be passed before the next is reachable, so the first empty bar is where every policy stops.
Below-ground achievements
All 30 achievements that cannot be earned on the surface. Unreached entries are kept visible: the shape of what is missing is the result.
Dungeon entries
Every terminal rollout that got below the overworld, with the turn it descended.
Score design
The headline is achievement-ladder score: each distinct capability milestone counts once. The Cost vs. Performance chart can isolate one family or region at a time without changing the canonical score used elsewhere.
Policy-visible observations and evaluator-only evidence are kept separate. Engine-rendered frames are derived from the Rust gold lane and every rollout links to its SHA-256-backed artifact.
Optional Workshop path
Inspect a bounded Craftax comparison with Workshop
Workshop can help clone the public code, inspect rollout evidence, and plan a small comparison. The public download does not currently launch this eval automatically.
- Start
- Published evidence
- Run policy
- Approval before compute
- Expected result
- Score + trace + next step
Help me reproduce a bounded Craftax comparison from the public Synth eval. First inspect the published contract and evidence, then identify the exact container, policy variants, seeds, provider, time, and maximum cost. Do not run paid compute or replace a policy without my explicit approval. Finish by showing the score comparison, one trace, the evidence path, and one proposed next experiment.
Workshop is optional. Downloading it does not start a run, install a starter, or incur provider charges.