Skip to content

fix(s3): reuse grounded coordinates within a screen - #238

Open
xiao115255 wants to merge 1 commit into
simular-ai:mainfrom
xiao115255:fix/s3-reuse-grounded-coordinates
Open

xiao115255 wants to merge 1 commit into
simular-ai:mainfrom
xiao115255:fix/s3-reuse-grounded-coordinates

Conversation

@xiao115255

Copy link
Copy Markdown

Every action grounds the same point twice. Worker.generate_next_action checks the model's
response with CODE_VALID_FORMATTER, which calls create_pyautogui_code and therefore evaluates
the action: agent.click("...") reaches OSWorldACI.generate_coords and calls the grounding model.
The worker then evaluates the same code again to run it, so generate_coords runs a second time
with the same screenshot and the same referring expression. The model is asked for the same point
twice and only the second answer is used.

generate_coords now keeps the coordinates it has read for the current screenshot, keyed by
referring expression, and drops them whenever the screenshot changes. Coordinates can't outlive
the screen they were read from, and the second call is free. An equal-but-distinct screenshot just
misses the cache, so a miss can never return a stale point.

This is s3 only: CODE_VALID_FORMATTER exists nowhere else, so the s2/s2_5 copies of
generate_coords don't need it.

Fixes #178

How I tested it: tests/test_grounding_coords_cache.py patches the grounding call and counts it
for the two-call sequence above — one call with this change, two without it — plus a new screen and
a second element on the same screen, to check the cache doesn't cross wires. On the unpatched
grounding.py those two cache assertions fail and the rest pass, so the test does fail without the
change. pytest tests → 9 passed; black --check gui_agents → unchanged. Run on Windows with
Python 3.13.

An action grounds its observation twice: CODE_VALID_FORMATTER evaluates the
generated code to check it, and the worker evaluates the same code again to run
it. Both calls reach OSWorldACI.generate_coords with the same screenshot and
referring expression, so the grounding model is asked for the same point twice
and only the second answer is used.

Cache the coordinates per referring expression and drop the cache whenever the
screenshot changes, so the second call is free and coordinates can never
outlive the screen they were read from.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reduce grounding model call frequency

1 participant