[sahiljagtap08]

Fixes #175

What was going wrong

Agent S3 asks the grounding model where an element is, then converts the answer into screen pixels. That conversion assumes the model always answers in a fixed grounding_width x grounding_height box, which the README says to set to 1920x1080 for UI-TARS-1.5.

UI-TARS-1.5 doesn't work that way. Like every model built on Qwen2.5-VL, it answers in pixels of the image it was actually shown, after the model's own resizing step. ByteDance documents this and ships the exact resize rule in their coordinate processing guide.

So the conversion is only right when the screenshot happens to be 1920x1080. That is true on OSWorld, which is why benchmarks never showed it. On a laptop or a high-resolution monitor the screenshot is a different size, and every click is scaled by the wrong factor. The error grows toward the bottom right of the screen, which is where most "OK", "Save" and dock icons live.

Here is what happens today versus with this change, for an element at 90% across and 95% down the screen, with the coordinates UI-TARS-1.5 would return for each case. "Before" and "after" are produced by the real OSWorldACI code:

Screen	Screenshot sent	Target	Before	After
OSWorld 1080p	1920x1080	(1728, 1026)	(1739, 1037)	(1728, 1026)
MacBook Air 13"	1470x956	(1323, 908)	(1003, 800)	(1323, 908)
MacBook Pro 16"	1728x1117	(1555, 1061)	(1406, 1100)	(1555, 1061)
1440p monitor	2400x1350	(2304, 1368)	(2889, 1703)	(2304, 1368)
4K monitor	2400x1350	(3456, 2052)	(4334, 2554)	(3456, 2052)

On a 1440p or 4K display the click is off the screen entirely. On a MacBook it is a few hundred pixels off. Even on 1080p it is 11 pixels off, because the model rounds the image to multiples of 28 pixels before answering and that was never accounted for. That small drift is enough to miss a small icon, which is exactly what the issue describes.

The fix

The grounding agent now remembers the size of the screenshot it sent, and converts the model's answer back from that image's space using the same resize rule the model uses. Nothing is assumed about the screen.

Not every grounding model answers in image pixels. UI-TARS-72B answers in a fixed 0 to 1000 space no matter the image. So there are two modes, and the default picks one from the shape of the grounding box you already pass:

• a square box like 1000x1000 can't be a pixel space for a non-square screen, so it is read as a fixed space, exactly as before
• any other box means "pixels of the screenshot", and the box itself is no longer trusted for scaling

That means every existing command keeps working with no flag changes. For anything unusual, --grounding_coordinate_space image|fixed forces a mode.

I also fixed highlight_text_span while I was here. It used OCR box positions straight from the screenshot as screen positions, so it was wrong whenever the CLI downsized the screenshot.

Changes

• smart_resize ported from the UI-TARS reference into gui_agents/s3/utils/common_utils.py
• OSWorldACI records the screenshot size in generate_coords, and resize_coordinates maps from the right space. New grounding_output_size and screenshot_to_screen helpers
• --grounding_coordinate_space CLI option, default auto
• README section explaining what the grounding box means now

Tests

tests/test_grounding_coordinates.py covers the resize rule against the reference values, each screen size in the table above, the square-box detection, both explicit modes, the fallback before any grounding call, and the OCR scaling. 19 tests pass with the existing suite and black --check gui_agents is clean.

Things worth knowing

• OSWorld runs will see coordinates shift by up to about 11 pixels near the right and bottom edges. That is the 28-pixel rounding being handled correctly for the first time.
• The reference rule uses the default max_pixels of the Qwen2.5-VL processor. If a deployment overrides it to something smaller, the screenshot would need to be capped to match. The CLI's existing 2400px cap stays well under the default.