SimForge-Bench case studies

Selected successes and failures from the frozen-v2 first-half runs. Each case page replays every tool call with the images the model saw; the steps listed on each card are the ones highlighted in our case-study notes. Built from saved runs only.

Repair

Find the one object with a placement error and fix it.

SuccessGPT-6 Astra

GPT-6 looks at the tilted side table from the side, at floor level, and rolls it back in five calls

A furnished living room with a couch, two rattan chairs, side tables, pictures and a TV. The side table (2) by the chairs is tilted sideways (a roll fault). GPT-6 takes a close look at it from the south-east, then a close, nearly level view from the east, the side from which a sideways tilt shows plainly. It rotates the table back 30 degrees about the roll axis, checks it from the south and finishes.

这是一个好的对比:下面repair的失败案例是kimi做的,kimi并没有很好的找对观测视角,而在这个gpt6的成功案例中,gpt6 找到了柜子侧面的视角,很好地捕捉到了柜子的问题。
Highlighted steps
Open the full replay →
FailureKimi K3.1 Pro

Kimi looks at the tilted side table only from the front and back, where the tilt does not show, and turns it around

The same living room: the side table (2) is tilted sideways (a roll fault). Kimi spends 13 views looking around the room. It looks at the side table twice, both times from 20 degrees up and straight from the front (south) or the back (north). The table is tilted sideways, so from the front or back it looks upright. Kimi never looks from the side or at floor level. It then turns the table 180 degrees on the spot (the wrong axis), checks from the same front angle, sees nothing wrong, and finishes. The table is still tilted. DeepSeek and GPT 5.6 Sol made the same 180-degree turn in this scene.

Highlighted steps
Open the full replay →

Relations

Move an object so that a spatial relation to another object holds.

SuccessGemini 4 Argon

Gemini checks 'left' from View 1's own side and fixes the depth

The task: move the small picture frame (1) so it is to the left of the potted plant (2) as seen in View 1. View 1 looks from the south-south-west, so its 'left' points west-north-west, not due west. Gemini picked its views from bearing 180, the same side as View 1. It moved the frame 0.57 m west, and the re-rendered low view showed the frame now left of the plant but closer to the camera. An overhead view from the south confirmed this, so Gemini pushed the frame 0.15 m north to bring it level with the plant, checked once more from the south and finished (22 degrees off View-1 left; the limit is 45). GPT-6, Claude, GPT-5.6 and Kimi all made the same west move of about 0.55 m. They checked from the north or not at all and stopped, ending 52-54 degrees off, so they failed.

Highlighted steps
Open the full replay →
FailureKimi K3.1 Pro

Kimi takes one overhead look, slides the clock beside the candle and stops

The task: move the alarm clock (1) so it is in front of the candle (2) as seen in View 1, i.e. between the candle and the View-1 camera. The View-1 camera stands to the east of the dresser, but Kimi read the compass as putting it to the north-east and never took a view from that side to check. Its only look was an overhead view from the north. It then made one move, 0.55 m south along the dresser, which put the clock next to the candle in the same row instead of toward the camera. Kimi finished right away. In the final View 1 the clock stands beside the candle, 58.6 degrees off the 'front' direction. GPT-6 solved the same episode in 7 steps: it slid the clock level with the candle, then moved it 0.12 m east toward the camera (1.1 degrees off).

Highlighted steps
Open the full replay →

Arrangement

Place a set of small objects so that stacking, containment and spacing rules hold.

SuccessGPT-6 Astra

Plate service: three plates stacked with a cup on top, and a candle in a bowl

Three plates stacked directly on each other with a drinking vessel on top, plus a candle inside a bowl and spacing rules between the groups. GPT-6 builds the stack bottom up, looking level and close at each rim. When a lower plate sits at the wrong height, it takes off the layers above it (steps 11-12, 31-32 and 40), re-seats the lower plate a few millimetres up or down, and rebuilds the stack. Every edit returns two fresh images, so many corrections follow straight from the picture an edit returned, without a separate look.

Highlighted steps
Open the full replay →
FailureDeepSeek V4.1 Flash

Failure (DeepSeek V4.1 Flash): a candle and a spice mill sunk through the bowl

The rules ask for three plates stacked with a cup on top, an open bowl holding a spice mill and a candle, and a second cup. DeepSeek stacks the plates and checks them, then puts the mill and the candle into the bowl at a height of 25 mm each. After that it looks only from 45° above or from a distance, which shows the tops of the mill and candle sitting in the bowl. It never looks at the bowl level or close, the view that would show their bottoms coming out underneath it.

正确分析视角:从这个视角可以看出杯子是穿模的,但deepseek从来没有用过这个视角。
Highlighted steps
Open the full replay →