Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
226 changes: 226 additions & 0 deletions docs/breadcrumbs.md

Large diffs are not rendered by default.

170 changes: 170 additions & 0 deletions docs/choices.md

Large diffs are not rendered by default.

43 changes: 43 additions & 0 deletions docs/specs/2026-02-09-apple-intelligence-descriptions-toggle.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Apple Intelligence Descriptions (Toggle + Generation + Editor UX)

Date: 2026-02-09

## Goal

- Provide a global opt-in toggle for Apple Intelligence-powered step descriptions.
- Generate concise per-step descriptions on-device using the Foundation Models framework.
- Communicate requirements + privacy clearly (on-device; no step upload).

## Non-Goals

- Screenshot understanding (OCR/Vision / image model prompts).
- Tool-calling to external data sources (Contacts/Calendar/etc).
- Multi-language UI for descriptions (v1 uses concise English prompts).

## UX

- Settings: `Apple Intelligence -> Use for step descriptions` (global toggle, default OFF).
- Toggle UI: custom Apple-style toggle (button + CSS) to match macOS System Settings look reliably inside Tauri/WKWebView.
- Eligibility hint (runtime via Foundation Models availability):
- Device must support Apple Intelligence.
- Apple Intelligence must be enabled in macOS System Settings.
- Model may be temporarily not ready (download/initialization).
- Provide a button to open macOS settings (deep link + fallback):
- Primary deep link: `x-apple.systempreferences:com.apple.Siri-Settings.extension`
- Fallback: `x-apple.systempreferences:com.apple.preference.siri`
- Recorder: after stopping a recording, if toggle is ON, auto-generate descriptions for steps that are missing one.
- Step Editor:
- Toolbar button `Enhance Steps` (uses missing-only when missing exists, otherwise regenerates non-manual steps).
- Per-step sparkle button to regenerate that step.
- Clicking the description text allows inline manual editing (stored as manual source).
- Quick Start (welcome banner): one bullet mentioning the optional toggle.

## Storage

- `localStorage.appleIntelligenceDescriptions` = `"true" | "false"`

## Privacy Copy (must be precise)

- For Apple Intelligence descriptions: runs on-device and works offline.
- StepCast does not upload recorded steps or screenshots for this feature (no third-party LLMs).
- (Do not make broader claims about analytics/update checks.)
35 changes: 35 additions & 0 deletions docs/specs/2026-02-12-ai-grounding-debug-telemetry.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Spec: AI Grounding + Debug Telemetry Hardening

Date: 2026-02-12
Status: implemented

## Problem
Recent sessions show capture correctness but unstable AI labels:
- wrong target label from OCR drift (`vat` instead of `Pepper`)
- context-menu contamination in right-click labels (`@ öffnen`)
- OCR typo in Finder sidebar (`Schreptisch`)
- icon-font glyph leakage in labels (``)
- low observability for why baseline/candidate was chosen

## Goals
1. Keep high-specificity labels when reliable.
2. Fall back deterministically when uncertain.
3. Expose per-step decision data for audits.

## Non-Goals
- app-specific rule engines
- complex ML confidence calibration

## Changes
1. Strengthen label normalization/cleanup (context-menu suffixes, icon glyph filtering, typo canonicalization).
2. Keep `window_title` label for list/sidebar contexts with list container signals.
3. Make quality gate return decision reason, not only text.
4. Include per-step debug payload in AI helper response and recorder logs.

## Acceptance
- replay latest problematic trace yields:
- `step-005` no `vat`
- `step-030` no `Schreptisch`
- `step-031` no `@ öffnen`
- `step-008/009` no icon glyph
- `cargo check` passes.
40 changes: 40 additions & 0 deletions docs/specs/2026-02-12-preclick-screencapturekit-buffer.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Pre-Click ScreenCaptureKit Buffer

Date: 2026-02-12

## Problem

Volatile UI (browser/Telegram menus, popovers) closes before current capture path finishes.
Result: screenshot often misses the clicked menu state; AI label can still be correct from AX, but image context is wrong.

## Goal

Use a short ScreenCaptureKit frame ringbuffer and prefer a pre-click frame for volatile interactions (`AXMenu`, `AXMenuItem`, transient popup/group roles).

## Scope

- Add macOS-only `pre_click_buffer` recorder module.
- Start buffer when recording starts; stop on stop/discard.
- In pipeline transient capture branch, attempt pre-click frame first.
- Keep existing CG/window/region capture as fallback.

## Approach

1. ScreenCaptureKit stream for active display, low-latency settings (BGRA, limited queue depth, capped fps).
2. Maintain small in-memory ringbuffer with recent frames + capture timestamps.
3. On volatile click:
- Select latest frame at or before click timestamp (fallback: latest frame).
- Write frame to current `screenshot_path`.
- Use display bounds as capture bounds for click percentages.
4. If unavailable/failed, continue existing capture logic unchanged.

## Non-Goals

- No OCR/AI prompt changes in this patch.
- No replacement of existing capture pipeline; only an additional robust fast-path.

## Verification

- Unit tests for frame selection policy (before-click preference + fallback).
- `cargo test` targeted recorder tests.
- `cargo check` for full crate compile.
28 changes: 28 additions & 0 deletions docs/specs/2026-02-13-editor-crop-drag-performance.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Spec: Editor Crop Drag Performance & UX Hardening

Date: 2026-02-13
Owner: codex
Status: Implemented

## Goal
- Make drag-based crop repositioning feel immediate and stable.
- Eliminate native image drag artifacts (ghost image + plus cursor).
- Keep render cost low for larger recordings.

## Problems Observed
- Browser native image drag could trigger while user drags crop preview.
- Crop updates on every pointer event can cause unnecessary re-render churn.
- Full-size screenshots in long lists increase initial rendering and decode cost.

## Decisions
- Disable native image drag (`draggable=false`, `onDragStart preventDefault`, `-webkit-user-drag: none`).
- Keep pointer events on crop frame, not image (`pointer-events: none` on cropped image).
- Batch pointer-move crop preview with `requestAnimationFrame`.
- Persist crop only on pointer release and only when movement is meaningful.
- Use `loading="lazy"` and `decoding="async"` for preview and thumbnail images.

## Validation
- Unit tests for editor crop interactions remain green.
- Full frontend test suite remains green.
- Production build passes.

40 changes: 40 additions & 0 deletions docs/specs/2026-02-13-editor-crop-visibility-fix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Spec: Editor Crop Apply Keeps Screenshot Visible

Date: 2026-02-13
Owner: codex
Status: Implemented

## Problem
- After applying a crop in the step editor, the screenshot can disappear.
- User impact: recorded instructions become unusable because the key visual context is gone.

## Root Cause
- Cropped mode uses absolute image positioning.
- Wrapper/frame sizing could collapse (0-size layout path) after crop apply.
- Crop rendering also depended on loaded natural image size; that introduced a fragile state boundary.

## Requirements
- Applying a crop must never hide the screenshot.
- Crop view must render immediately, even if natural image size is not yet known.
- No destructive image rewrite; UI-only crop state stays in `crop_region`.

## Implementation
- `EditorStepCard`:
- Enable cropped rendering whenever a valid non-full crop exists (independent of natural-size availability).
- Add stable frame style with fallback `aspect-ratio` (`16/9`) + `width: 100%`.
- Keep natural-size update on regular image load and on crop-modal image load.
- `editor.css`:
- Make editor image wrapper block-level with deterministic width.
- Ensure crop frame is `position: relative` and width-constrained.
- Prevent collapse when cropped image is absolutely positioned.
- Regression test:
- Added test ensuring cropped frame is rendered with a non-empty aspect ratio style when `crop_region` exists.

## Validation
- Run frontend tests for `EditorStepCard`.
- Run full test suite + build.
- Manual replay:
- Open editor.
- Apply crop on a step with screenshot.
- Verify image remains visible and marker position remains coherent.

48 changes: 48 additions & 0 deletions docs/specs/2026-02-13-step-crop-focus-system.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Spec: Step Focus Crop System

Date: 2026-02-13
Status: Done

## Goal
Provide reliable, non-destructive focus crops for recorded step screenshots so guides remain readable and actionable, especially when source captures are full-screen.

## Requirements
1. Non-destructive: always keep original screenshot file untouched.
2. Persisted crop metadata per step as percentages (`x/y/width/height`) so crops are resolution-independent.
3. Recorder should set a sensible default focus crop for very large captures (not only exact full-screen) to keep dialogs/overlays readable.
4. Editor must allow manual crop adjust + reset-to-full.
5. Click marker must remain accurate after crop in panel/editor/export.
6. Export (HTML/Markdown/PDF) must apply crop to image output.
7. Backward compatibility: steps without crop render/export as full image.

## Data Model
- Add optional `crop_region: BoundsPercent` to step model (frontend + backend).
- Validation constraints: clamp values to [0..100], enforce min width/height.

## Recorder Default Focus Heuristic
Apply when capture bounds are large display-like frames (coverage heuristic against clicked display).
- Base around click location.
- If AX element bounds exist, expand around element with margin.
- Clamp to display bounds in percent.
- Leave `None` for auth placeholders and already-tight captures.

## Editor UX
- Add "Crop" action on step card.
- Open modal cropper with live preview and click marker context.
- Save => persist `crop_region`.
- Reset => clear `crop_region` (full image).

## Export Rules
- If `crop_region` exists, crop image bytes during export pipeline before encoding.
- Marker coordinates remapped to cropped coordinate space for HTML/PDF overlays.
- Markdown image output uses cropped bitmap.

## Edge Cases
- Invalid crop metadata -> ignore crop (fallback full image).
- Marker outside crop -> hide marker (avoid misleading point).
- Missing screenshot -> no image/crop rendering.

## Verification
- Rust unit tests for crop normalization + exported marker remap.
- Frontend tests for marker remap and crop UI command calls.
- Existing tests must remain green.
29 changes: 29 additions & 0 deletions docs/specs/2026-02-14-pre-push-coverage-gate.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# 2026-02-14 pre-push coverage gate hardening

## Context
Pre-push hook blocks branch push due per-file 80% coverage thresholds on:
- `src/components/EditorWindow.tsx`
- `src/components/EditorStepCard.tsx`
- `src/components/SettingsSheet.tsx`
- `src/components/StepItem.tsx`
- `src/utils/stepCrop.ts`

## Goal
Raise coverage via behavior tests (no threshold relaxation, no ignore hints), keep runtime logic unchanged.

## Scope
- Add targeted tests for unexecuted branches:
- AI toggle sync/fallback and generation branching
- crop modal controls + retry branches
- settings fallback/openUrl/toggle branches
- thumbnail retry and sortable branches
- crop helper utility branches
- Validate with exact pre-push chain:
- `cargo clippy -- -D warnings`
- `npm run test:coverage`
- `npx tsc --noEmit`

## Non-goals
- No feature/UI behavior change
- No runtime code refactor
- No threshold config changes
Loading