Coordinate abandoned effect recovery

[cardmagic]

An actor can now retain the exact effect returned by emit and opt into library-coordinated recovery. A stale processing owner is rechecked under the claim-locking protocol; retirement and the originating actor's recovery callback commit together. Fresh owners, pending retries, completed results, and dead effects retain distinct outcomes.

Closes #67. JavaScript counterpart: cardmagic/solid-objects-js#51. Prepares version 0.15.0 with matching gem/lockfile metadata and a dated changelog.

Before, a watchdog could create overlapping external work:

emit(:build_report, revision: self.revision += 1)
# A later watchdog has no supported ownership/reclamation transaction.
start

After, on_recovery enables automatic recovery through the existing effect workers. No actor watchdog or explicit recovery request is required:

self.export_effect = emit(:build_report,
  revision: self.revision += 1,
  on_success: :export_finished, on_failure: :export_failed,
  on_recovery: :recover_export,
  recovery_timeout: 120)

The library checks owner heartbeats during effect polling and stale-process cleanup. The timeout measures heartbeat freshness, not elapsed effect duration. Only on_recovery owns replacement.

Effect workers maintain their owner heartbeat throughout handler execution and success/failure handling. Real Rails consumer QA exposed that the previous between-job-only heartbeat could retire a healthy handler waiting on HTTP. A dedicated heartbeat thread now releases its connection after each update and is stopped/joined on every exit.

An application that also wants explicit status observations can add on_status: :inspect_export to the same emit call, then optionally stage a check from an actor operation:

def check_export
  request_effect_recovery(export_effect) if export_effect
end

on_status reports that requested check; it does not create a status subscription. Completed-status repair shares the guarded helper used by success, so duplicate notifications cannot apply the result twice. The full optional watchdog example uses native Ruby keyword operations in docs/effect-recovery.md.

The lock order is origin instance → effects → recovery bindings → current owner processes, with deterministic ordering within each group and database wall time sampled after locks. Actor-staged checks reuse the fenced transaction. Automatic recovery uses one transaction per candidate. A durable retirement binding plus the existing terminal completed storage state prevents reclamation and fences late completion/failure; callback insertion failure rolls back retirement. Bindings survive effect/message pruning until their originating instance is removed.

Compatibility: emit intentionally changes from nil to its handle; wrappers must return it and operations retaining their old nil result must return nil explicitly. Install the additive migration and upgrade all effect/cleanup roles before opting in. Public frozen constants and precise RBS records describe envelopes. Strict consumer tests document that Steep 2.0 does not narrow the string-keyed record union from the outcome comparison; records are not weakened to untyped hashes.

Retirement is not cancellation or exactly-once external execution. External actions still need idempotency across retries and replacement generations.

Validation (Ruby 3.3.9):

• bundle exec rake: 656 tests / 2303 assertions, no failures, Standard, RuboCop, generated RBS validation, Steep, and Brakeman clean; adapter skips are tested separately.

• Real PostgreSQL 17 recovery suite: 23 tests / 101 assertions, no failures or skips, with deterministic database-lock barriers for both claim/check race orderings, changed claimants, refreshed heartbeats, simultaneous recovery, callback rollback, and lookup uncertainty.

• Strict built/installed-gem consumer tests: 35 assertions, including invalid constants, required fields, constructor mutations, and the verified Steep narrowing boundary.

• Test-first handle regression observed nil instead of the persisted ID; retirement regression expected one callback and received zero. Removing instance-first completion locks reproduced ActiveRecord::Deadlocked instead of LostActivation; restoring the fix passes.

• Full PostgreSQL suite and CI adapter/compatibility matrices cover existing retries and failure callbacks.

• A fresh Ruby process retires an effect and exits with SIGKILL before delivery; a restarted worker delivers one durable recovery callback. The standalone at-least-once proof also passes.

• Automatic scans prefilter freshness using database time and respect claim_scan_limit; a barrier regression verifies fresh candidates do not block unrelated pending claims.

• A fresh Rails consumer installed the built gem, ran the install generator/migrations/doctor, and passed 14 end-to-end HTTP scenarios with separate worker processes. Coverage includes automatic recovery without an actor watchdog, SIGKILL, SIGSTOP/resume with late success/failure, explicit checks, retry/failure compatibility, pruning, foreign handles, and timeout rollback. The documented solid_objects start launcher also passes a long-running HTTP effect test.

• The in-handler heartbeat regression and fix reversal both fail waiting for a heartbeat while the handler is blocked. The fixed test observes eight heartbeat updates, preserves the processing effect, and verifies that the heartbeat thread stops after completion; failure cleanup has separate coverage.

• Transient heartbeat query failures now emit solid_objects.process.heartbeat_failed and retry on the next configured interval without consuming effect attempts. A failing-query regression passes after the fix and fails when it is removed. Both installed consumers also pass a real PostgreSQL stop/start while their HTTP effects remain held: the original owners resume heartbeats, explicit checks defer, and no duplicate execution occurs.

[cardmagic]

@greptileai Please review the current head for recovery/claim races, transaction boundaries, callback idempotency, adapter parity, and public type safety.

[greptile-apps]

Greptile Summary

This PR adds coordinated recovery for abandoned effects, including durable recovery bindings, stable effect handles, transactional retirement/status callbacks, stale-owner scanning, and continuous owner heartbeats.

• Adds the effect-recovery schema, coordinator, payloads, and public RBS contracts.
• Coordinates retirement with actor-instance, effect, recovery-binding, and owner-process locks.
• Keeps process heartbeats active during effect execution and retries heartbeat updates after transient failures.
• Documents migration, compatibility, timeout, idempotency, and external-work limitations.

Confidence Score: 5/5

The PR appears safe to merge; the transient heartbeat failure has been addressed and no new actionable issues remain.

The heartbeat loop now catches database failures, isolates observability failures, waits for the configured interval before retrying, and remains stoppable through its condition variable. The earlier timeout-validation and recovery-scan threads were manually resolved, and their current implementations use flat guards and bounded stale-candidate prefiltering.

Important Files Changed

Filename	Overview
lib/solid_objects/effect_recovery_coordinator.rb	Coordinates stale-owner checks, transactional effect retirement, and recovery/status callback insertion.
lib/solid_objects/process_heartbeat.rb	Maintains effect-owner heartbeats and now reports transient failures while retrying on later intervals.
lib/solid_objects/effect_executor.rb	Runs heartbeat maintenance throughout effect delivery and adopts instance-first completion/failure locking.
lib/solid_objects/actor.rb	Returns stable effect handles and stages validated recovery requests and recovery metadata.
db/migrate/20260915000000_add_solid_objects_effect_recoveries.rb	Adds durable per-effect recovery bindings, timeout configuration, and retirement state.

Sequence Diagram

sequenceDiagram
  participant Worker
  participant Coordinator
  participant Instance
  participant Effect
  participant Recovery
  participant Owner
  participant Mailbox

  Worker->>Coordinator: poll for stale recoverable effects
  Coordinator->>Instance: lock originating actor instance
  Coordinator->>Effect: lock processing effect
  Coordinator->>Recovery: lock recovery binding
  Coordinator->>Owner: lock and recheck heartbeat
  alt Owner remains fresh
    Coordinator-->>Worker: defer recovery
  else Owner is stale or missing
    Coordinator->>Effect: mark completed and clear claim
    Coordinator->>Recovery: persist retired_at
    Coordinator->>Mailbox: enqueue durable recovery callback
    Coordinator-->>Worker: announce committed callback
  end

Loading

_{Reviews (5): Last reviewed commit: "fix: Resume heartbeats after database er..." | Re-trigger Greptile}

[cardmagic]

@greptileai Please review current head 09e05f9. Addressed both findings with flat timeout validation and a bounded database-time freshness prefilter that preserves the locked recheck. Added deterministic PostgreSQL claim/check races, rollback and uncertainty tests, and a fresh-process crash-after-retirement proof. PostgreSQL recovery suite: 20 tests, 85 assertions, no skips; full rake clean. Fixed standalone example and benchmark migration/model loading.

[cardmagic]

@greptileai Please review current head 2fdf5d0. The previous implementation received 5/5; this commit prepares version 0.15.0 with a regenerated lockfile and dated changelog. Packaged type contracts pass (35 assertions). The PR example now leads with automatic on_recovery and separates optional explicit status checks.

[cardmagic]

@greptileai Please review current head 15d2237. Real Rails consumer QA found a healthy HTTP effect being retired because EffectExecutor heartbeated only between jobs. This fix maintains process heartbeats through handler execution and completion/failure, releases each heartbeat connection, and joins the heartbeat thread on every exit. The observed failing regression and reversal time out waiting for an in-handler heartbeat; the fixed tests pass. Full rake: 655 tests, no failures; PostgreSQL recovery: 22 tests/97 assertions, no skips; 14 packaged-consumer HTTP/crash/resume scenarios pass. JS already maintains equivalent heartbeats and passed the same 14 scenarios.

[cardmagic]

@greptileai Please review current head db97952. The transient-heartbeat-error finding is fixed with red/green/reversal regression coverage and a real PostgreSQL restart in the installed Rails consumer. Full rake passes 656 tests / 2303 assertions; PostgreSQL recovery passes 23 tests / 101 assertions. The healthy held HTTP effect resumes heartbeating and is not retired after the outage.