Skip to content

Add muse2api media provider for muse.ai rendering - #70

Merged
crisng95 merged 3 commits into
mainfrom
claude/gracious-lovelace-1567a3
Sep 30, 2026
Merged

crisng95 merged 3 commits into
mainfrom
claude/gracious-lovelace-1567a3

Conversation

@crisng95

@crisng95 crisng95 commented Sep 30, 2026 •

Copy link
Copy Markdown
Owner

Adds a new media provider that renders images and videos through a muse2api gateway, which exposes the muse.ai web app as an OpenAI-compatible HTTP API.

Summary

This PR introduces Muse2APIProvider, a new media backend that complements the existing flow and assistant providers. It allows Flow Kit to render media through a separate muse2api service that manages muse.ai accounts, failover, and browser automation.

Key Changes

  • New HTTP client (agent/services/muse2api_client.py): Thin async wrapper around muse2api's OpenAI-compatible endpoints

    • Handles image generation (POST /v1/images/generations)
    • Manages video task creation and polling (POST /v1/videos, GET /v1/videos/{id})
    • Downloads generated media (GET /v1/media/{name})
    • Converts local/remote images to data URLs for inline transmission (so the gateway can run on another host)
    • Parses OpenAI error bodies and surfaces error codes (no_account_available, upstream_quota_exhausted, etc.)
  • New provider adapter (agent/sdk/services/muse2api_provider.py): Implements MediaProvider interface

    • Supports image generation and image-to-video (i2v)
    • Degrades chained video and reference-to-video (r2v) to plain i2v when MUSE2API_ALLOW_DEGRADED=1
    • Saves output to output/_shared/muse2api/ as file:// URLs with minted UUIDs
    • Logs reference images as dropped (muse.ai takes no image refs)
  • Configuration (agent/config.py): New env vars for gateway URL, API key, models, timeouts, and concurrency limits

  • Registry integration (agent/sdk/services/registry.py): Registers muse2api provider automatically; becomes available once MUSE2API_URL is set

  • Comprehensive tests (tests/unit/test_muse2api_provider.py): 200 lines covering

    • Image generation with composition hints
    • Video task creation with inline first frames and polling
    • Failed task error reporting
    • OpenAI error body parsing
    • Degraded mode for chained/r2v
    • Missing start frame validation
    • Capability checks and configuration validation
    • Existing image registration
  • Documentation: Updated README, provider guide, doctor guide, and skill docs to explain setup, capabilities, error codes, and troubleshooting. The README spells out that muse2api is a separate service that must be cloned from its own repo and started alongside Flow Kit.

  • Unrelated flaky-test fix (agent/db/schema.py, tests/unit/test_db_connection.py): get_db() raced on its lazy connect — two first callers both opened a connection, the second overwrote the global, and the first ran its PRAGMAs on the second's connection mid-statement. That produced the intermittent sqlite3.OperationalError: database table is locked in test_assistant_provider.py on Python 3.10 (also seen on main, run Closed: created in wrong repository #33) and leaked a connection. The connect now happens under a per-event-loop init lock and the connection is published only once set up. Regression test counts connect() calls under concurrent first callers and fails without the fix.

Implementation Details

  • Input images are always sent as data URLs (base64-encoded inline) rather than URLs, so the gateway can run on a different host than Flow Kit
  • Video polling includes transient error tolerance (up to 3 retries) to handle gateway blips without failing the render
  • Media is downloaded into a local directory and stored as file:// URLs, so /fk-refresh-urls is never needed
  • The provider degrades gracefully: chained video and r2v fail with a clear message unless MUSE2API_ALLOW_DEGRADED=1, which renders them as plain i2v
  • Error codes from the gateway's OpenAI error body are preserved and surfaced to the worker for proper retry/failure classification

https://claude.ai/code/session_01MfvQsdxJhWw15u6Zm51YJM

Flow Kit -> Muse2APIProvider -> muse2api -> muse.ai. A new `muse2api`
MediaProvider renders scene/reference images and i2v video through a
muse2api gateway (https://github.com/crisng95/muse2api):

- agent/services/muse2api_client.py: async HTTP client for
  /v1/images/generations, /v1/videos (+ task polling) and /v1/media,
  Bearer auth, OpenAI error bodies surfaced as "muse2api <code>".
  Input frames are sent inline as data URLs so the gateway can run on
  another host.
- agent/sdk/services/muse2api_provider.py: translates ProviderJobs,
  downloads results into output/_shared/muse2api/ as file:// URLs with
  minted UUID media_ids (Flow-shaped results, pipeline unchanged).
  Chained end frames and r2v fail loudly unless MUSE2API_ALLOW_DEGRADED=1;
  edit/upscale/audio are not offered.
- Registered in ProviderRegistry; available once MUSE2API_URL is set.
- Config, README, fk-provider/fk-doctor and generation skills updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MfvQsdxJhWw15u6Zm51YJM
Spell out that the muse2api gateway is a separate service Flow Kit does
not ship or start: clone crisng95/muse2api, configure the browser driver,
start it, import a muse.ai account, verify /readyz, then point Flow Kit
at it with MUSE2API_URL / MUSE2API_KEY. Adds a pointer from Quick Start
and the env-var table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MfvQsdxJhWw15u6Zm51YJM
…locked")

Two coroutines hitting get_db() before the connection existed both saw
None and both connected; the second overwrote the global, so the first
ran its PRAGMAs (including wal_checkpoint) on the second's connection
while a statement was open on it. That failed intermittently with
"sqlite3.OperationalError: database table is locked" in
test_assistant_provider on CI (Python 3.10, also on main in run #33) and
leaked a connection whose worker thread can keep the interpreter alive.

Connect under a per-event-loop init lock (separate from _db_lock, which
crud holds while calling get_db), set up a local, and publish it only
when ready. Adds a regression test counting connect() calls under
concurrent first callers; it fails without this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MfvQsdxJhWw15u6Zm51YJM
@crisng95
crisng95 merged commit fe40dec into main Sep 30, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants