Skip to content

fix(caching): robust trajectory auto-detection & resume; cache format 0.3 #310

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
philipph-askui wants to merge 1 commit into from
Closed

fix(caching): robust trajectory auto-detection & resume; cache format 0.3 #310

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
167 changes: 31 additions & 136 deletions docs/06_caching.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,7 @@ from askui.models.shared.settings import (
caching_settings = CachingSettings(
strategy="record", # One of: "execute", "record", "auto", or None
cache_dir=".askui_cache", # Directory to store cache files
writing_settings=CacheWritingSettings(
filename="my_test.json" # Filename for the cache file (optional)
),
filename="my_test.json", # Name of the trajectory for this test case
execution_settings=CacheExecutionSettings(
delay_time_between_actions=1.0 # Delay in seconds between each cached action
),
Expand All @@ -42,6 +40,7 @@ caching_settings = CachingSettings(

- **`strategy`**: The caching strategy to use (`"execute"`, `"record"`, `"auto"`, or `None`).
- **`cache_dir`**: Directory where cache files are stored. Defaults to `".askui_cache"`.
- **`filename`**: Name of the trajectory/cache file for this test case (the `.json` suffix is optional). It is the lookup key in `"execute"`/`"auto"` modes and the target filename in `"record"`/`"auto"` modes. If empty, no trajectory is auto-detected and recordings receive an auto-generated filename.
- **`writing_settings`**: Configuration for cache recording (optional). See [Writing Settings](#writing-settings) below.
- **`execution_settings`**: Configuration for cache playback (optional). See [Execution Settings](#execution-settings) below.

Expand All @@ -59,7 +58,7 @@ writing_settings = CacheWritingSettings(

#### Parameters

- **`filename`**: Name of the cache file to write. If not specified, a timestamped filename will be generated automatically (format: `cached_trajectory_YYYYMMDDHHMMSSffffff.json`).
- **`filename`**: Name of the cache file to write. Prefer setting `filename` directly on `CachingSettings` (the top-level `filename` takes precedence and is also used for trajectory lookup in `execute`/`auto` modes). If neither is specified, a timestamped filename will be generated automatically (format: `cached_trajectory_YYYYMMDDHHMMSSffffff.json`).

### Execution Settings

Expand Down Expand Up @@ -89,16 +88,14 @@ Record agent actions to a cache file for later replay:

```python
from askui import ComputerAgent
from askui.models.shared.settings import CachingSettings, CacheWritingSettings
from askui.models.shared.settings import CachingSettings

with ComputerAgent() as agent:
agent.act(
goal="Fill out the login form with username 'admin' and password 'secret123'",
caching_settings=CachingSettings(
strategy="record", # you could also use "auto" here
writing_settings=CacheWritingSettings(
filename="login_test.json"
),
filename="login_test.json",
)
)
```
Expand All @@ -107,7 +104,7 @@ After execution, a cache file will be created at `.askui_cache/login_test.json`

### Executing from Cache (Replaying)

Provide the agent with access to previously recorded trajectories:
Set `strategy="execute"` (or `"auto"`) and give the trajectory's `filename`. The SDK automatically looks up `<cache_dir>/<filename>`:

```python
from askui import ComputerAgent
Expand All @@ -118,116 +115,24 @@ with ComputerAgent() as agent:
goal="Fill out the login form",
caching_settings=CachingSettings(
strategy="execute", # you could also use "auto" here
filename="login_test.json",
)
)
```

When using `strategy="execute"`, the agent receives two additional tools:

1. **`retrieve_available_trajectories_tool`**: Lists all available cache files in the cache directory
2. **`execute_cached_executions_tool`**: Executes a specific cached trajectory

The agent will automatically check if a relevant cached trajectory exists and use it if appropriate. After executing a cached trajectory, the agent will verify the results and make corrections if needed.
If a usable trajectory with that name exists, the SDK surfaces its details (path
and required parameters) to the agent automatically in the first message, and the
agent replays it via the `CacheExecutor` before doing anything else — you no
longer need to describe available cache files in your goal prompt, and there is
no separate "list trajectories" tool. After replay, the agent verifies the
results (via the `verify_cache_execution` tool) and makes corrections if needed.

### Referencing Cache Files in Goal Prompts

When using `strategy="execute"` or `strategy="auto"`, **you need to inform the agent about which cache files are available and when to use them**. This is done by including cache file information directly in your goal prompt.

#### Explicit Cache File References

For specific tasks, mention the cache file name and what it accomplishes:

```python
from askui import ComputerAgent
from askui.models.shared.settings import CachingSettings

with ComputerAgent() as agent:
agent.act(
goal="""Open the website in Google Chrome.

If the cache file "open_website_in_chrome.json" is available, please use it
for this execution. It will open a new window in Chrome and navigate to the website.""",
caching_settings=CachingSettings(
strategy="execute",
cache_dir=".cache"
)
)
```

#### Pattern-Based Cache File References

For test suites or repetitive workflows, you can establish naming conventions:

```python
from askui import ComputerAgent
from askui.models.shared.settings import CachingSettings

test_id = "TEST_001"

with ComputerAgent() as agent:
agent.act(
goal=f"""Execute test {test_id} according to the test definition.

Check if a cache file named "{test_id}.json" exists. If it does, use it to
replay the test actions, then verify the results.""",
caching_settings=CachingSettings(
strategy="execute",
cache_dir="test_cache"
)
)
```
Behavior when a trajectory is **not** found:

#### General Rules for Cache Selection

You can also provide general instructions for the agent to identify applicable cache files:

```python
from askui import ComputerAgent
from askui.models.shared.settings import CachingSettings

with ComputerAgent() as agent:
agent.act(
goal="""Fill out the user registration form.

Look for cache files that match the pattern "user_registration_*.json".
Choose the most recent one if multiple are available, as it likely contains
the most up-to-date interaction sequence.""",
caching_settings=CachingSettings(
strategy="execute",
cache_dir=".cache"
)
)
```

#### Multiple Cache Files

For complex workflows, you can reference multiple cache files:

```python
from askui import ComputerAgent
from askui.models.shared.settings import CachingSettings

with ComputerAgent() as agent:
agent.act(
goal="""Complete the full checkout process:

1. If "login.json" exists, use it to log in
2. If "add_to_cart.json" exists, use it to add items to cart
3. If "checkout.json" exists, use it to complete the checkout

After each cached execution, verify the step completed successfully before proceeding.""",
caching_settings=CachingSettings(
strategy="execute",
cache_dir=".cache"
)
)
```

**Best Practices:**
- Be specific about what the cache file does to help the agent decide if it's applicable
- Include verification instructions after cached execution
- Use consistent naming conventions for easier cache file management
- Mention any prerequisites or expected UI state for the cached trajectory
- `strategy="execute"`: the agent performs the task normally (nothing is recorded).
- `strategy="auto"`: the agent is told no cache exists and performs the task
normally, while recording the run to `filename` for next time. An existing but
invalidated cache is re-recorded (self-healing) rather than replayed.

### Using Custom Execution Settings

Expand Down Expand Up @@ -260,24 +165,21 @@ Enable both reading and writing simultaneously:

```python
from askui import ComputerAgent
from askui.models.shared.settings import CachingSettings, CacheWritingSettings
from askui.models.shared.settings import CachingSettings

with ComputerAgent() as agent:
agent.act(
goal="Complete the checkout process",
caching_settings=CachingSettings(
strategy="auto",
writing_settings=CacheWritingSettings(
filename="checkout_test.json"
),
filename="checkout_test.json",
)
)
```

In this mode:
- The agent can use existing cached trajectories to speed up execution
- New actions will be recorded to the specified cache file
- If a cached execution is used, no new cache file will be written (to avoid duplicates)
- If a usable trajectory named `checkout_test.json` exists, it is replayed and no new cache file is written (to avoid overwriting the existing one)
- Otherwise, the agent performs the task normally and records the run to `checkout_test.json`

## Cache File Format

Expand Down Expand Up @@ -330,13 +232,12 @@ In write mode, the `CacheManager`:

In read mode:

1. Two caching tools are added to the agent's toolbox
2. A special system prompt (`CACHE_USE_PROMPT`) is appended to instruct the agent on how to use trajectories
3. The agent can call `retrieve_available_trajectories_tool` to see available cache files
4. The agent can call `execute_cached_executions_tool` with a trajectory file path to replay it
5. During replay, each tool use block is executed sequentially with a configurable delay between actions (default: 1.0 seconds)
6. Screenshot and trajectory retrieval tools are skipped during replay
7. The agent is instructed to verify results after replay and make corrections if needed
1. The SDK checks whether a trajectory named `filename` exists in `cache_dir`
2. If a usable trajectory is found, its details (path and required parameters) are injected into the first user message, a special system prompt (`CACHE_USE_PROMPT`) is appended, and the `CacheExecutor` speaker plus the `verify_cache_execution` tool are wired up
3. The agent hands off to the `CacheExecutor` via the `switch_speaker` tool to replay the trajectory
4. During replay, each tool use block is executed sequentially with a configurable delay between actions (default: 1.0 seconds)
5. Screenshot and non-cacheable tools are skipped/paused during replay; if a non-cacheable step is encountered the agent executes it manually and (unless it was the last step) resumes replay
6. The agent is instructed to verify results after replay (via `verify_cache_execution`) and make corrections if needed; reporting failure invalidates the cache

The delay between actions can be customized using `CacheExecutionSettings` to accommodate different application response times.

Expand All @@ -354,7 +255,6 @@ from askui import ComputerAgent
from askui.models.shared.settings import (
CachingSettings,
CacheExecutionSettings,
CacheWritingSettings,
)

# Step 1: Record a successful login flow
Expand All @@ -365,24 +265,19 @@ with ComputerAgent() as agent:
caching_settings=CachingSettings(
strategy="record",
cache_dir="test_cache",
writing_settings=CacheWritingSettings(
filename="user_login.json"
),
filename="user_login.json",
)
)

# Step 2: Later, replay the login flow for regression testing
print("\nReplaying login flow for regression test...")
with ComputerAgent() as agent:
agent.act(
goal="""Log in to the application.

If the cache file "user_login.json" is available, please use it to replay
the login sequence. It contains the steps to navigate to the login page and
authenticate with the test credentials.""",
goal="Log in to the application.",
caching_settings=CachingSettings(
strategy="execute",
cache_dir="test_cache",
filename="user_login.json",
execution_settings=CacheExecutionSettings(
delay_time_between_actions=2.0
),
Expand Down
Loading
Loading