@WOB_WorldOracle robot evaluation platform

by Polly Spring: AgiBot

Models that spit out clean, successful grasps can still be hallucinating outcomes the hardware never touched. That mismatch is exactly what WOB_WorldOracle exists to surface. The platform was built by AGIBOT and NVIDIA alongside Princeton, UT Austin and the University of Toronto specifically to grade robot behavior models against what they actually did in the real world. Instead of relying on idealized simulations, it replays recorded real-world actions through a model and compares the predicted future against the actual execution. This matters because errors in a model's physical predictions propagate to any systems built upon it. The paper points to data generation, scalable policy evaluation and closed-loop policy learning as potential uses for action-conditioned world models (ACWMs). But how do you grade a world model? Against what the robot really did. The authors define the property being tested as execution coherence, which measures if a model follows provided actions across different embodiments and over long rollouts. It asks does the model follow the actions it is given, even the bad ones, across different embodiments and action sources, and over long rollouts? What makes this setup unusual is that no existing benchmark combines all listed evaluation criteria. Existing embodied world-model benchmarks lack the combination of action-conditioned evaluation, real-world data, failed executions, long episodes, multiple embodiments, multi-view outputs, and policy-generated actions. The evaluation set contains 335 episodes: 100 on AGIBOT G2, 75 on YAM, 100 with a human hand, and 60 rollouts driven by a learned policy on a real AGIBOT G2 robot. Approximately 40% of the human-operated segments are failures. Models cannot rely solely on success stories due to the high failure rate in the dataset. That is the moment WOB_WorldOracle was built for. #AgiBot