Repository navigation
fix(s3): reuse grounded coordinates within a screen - #238
Open
xiao115255 wants to merge 1 commit into
Open
xiao115255 wants to merge 1 commit into
xiao115255 wants to merge 1 commit into
Conversation
An action grounds its observation twice: CODE_VALID_FORMATTER evaluates the generated code to check it, and the worker evaluates the same code again to run it. Both calls reach OSWorldACI.generate_coords with the same screenshot and referring expression, so the grounding model is asked for the same point twice and only the second answer is used. Cache the coordinates per referring expression and drop the cache whenever the screenshot changes, so the second call is free and coordinates can never outlive the screen they were read from.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every action grounds the same point twice.
Worker.generate_next_actionchecks the model'sresponse with
CODE_VALID_FORMATTER, which callscreate_pyautogui_codeand therefore evaluatesthe action:
agent.click("...")reachesOSWorldACI.generate_coordsand calls the grounding model.The worker then evaluates the same code again to run it, so
generate_coordsruns a second timewith the same screenshot and the same referring expression. The model is asked for the same point
twice and only the second answer is used.
generate_coordsnow keeps the coordinates it has read for the current screenshot, keyed byreferring expression, and drops them whenever the screenshot changes. Coordinates can't outlive
the screen they were read from, and the second call is free. An equal-but-distinct screenshot just
misses the cache, so a miss can never return a stale point.
This is s3 only:
CODE_VALID_FORMATTERexists nowhere else, so the s2/s2_5 copies ofgenerate_coordsdon't need it.Fixes #178
How I tested it:
tests/test_grounding_coords_cache.pypatches the grounding call and counts itfor the two-call sequence above — one call with this change, two without it — plus a new screen and
a second element on the same screen, to check the cache doesn't cross wires. On the unpatched
grounding.pythose two cache assertions fail and the rest pass, so the test does fail without thechange.
pytest tests→ 9 passed;black --check gui_agents→ unchanged. Run on Windows withPython 3.13.