cacheflight

cacheflight deduplicates concurrent async work for the same cache key and stores the result in a user-provided cache backend. It supports flat TTL, stale-while-revalidate (SWR), and probabilistic early expiration (XFetch) — all without forcing a specific cache store or serialization format.

Features

• Cold-miss deduplication — when N callers race for the same missing key, only one recomputes; the rest share the result.
• Stale-while-revalidate — serve a stale value immediately while a background refresh runs.
• Probabilistic early expiration (XFetch) — trigger background refreshes before the TTL expires, proportional to compute time, smoothing out thundering herds.
• Type-state API — invalid configurations are caught at compile time (e.g. calling stale_for() on a flat-TTL instance won't compile).
• Plugable backends — implement CacheBackend for any store, or use the built-in MemoryCache or RedisCache.
• Observability hooks — MetricsHooks trait with no-op defaults, re-implement for production metrics.
• Safe fallback — cache read failures degrade to recompute instead of failing the caller.
• Cache invalidation — invalidate(key) and invalidate_prefix(prefix) to evict entries programmatically.

Installation

[dependencies]
cacheflight = "0.2"

Redis support is included by default. To use only the in-memory backend:

[dependencies]
cacheflight = { version = "0.2", default-features = false }

Quick start

use cacheflight::{CacheFlight, MemoryCache, LookupState};
use std::time::Duration;

#[tokio::main]
async fn main() -> cacheflight::Result<()> {
    let cf = CacheFlight::new(MemoryCache::new())
        .ttl(Duration::from_secs(30));

    // Five concurrent callers — only one recomputes.
    let mut tasks = Vec::new();
    for _ in 0..5 {
        let cf = cf.clone();
        tasks.push(tokio::spawn(async move {
            cf.run("user:42", || async {
                // This runs only once.
                Ok(br#"{"id":42,"name":"Ada"}"#.to_vec())
            }).await
        }));
    }

    for task in tasks {
        let result = task.await??;
        println!("state={:?}, value={}", result.state(), String::from_utf8_lossy(result.value()));
    }

    // Subsequent reads hit the cache.
    let cached = cf.run("user:42", || async {
        unreachable!("should not recompute")
    }).await?;
    assert_eq!(cached.state(), LookupState::CacheHit);

    // Invalidate when the underlying data changes.
    cf.invalidate("user:42").await?;

    Ok(())
}

Backends

MemoryCache (built-in, no extra dependencies)

An in-process cache backed by DashMap. Supports failure injection for testing:

use cacheflight::MemoryCache;

let cache = MemoryCache::new();
cache.fail_one_get();   // next get() fails with a cache read error
cache.fail_one_set();   // next set() fails with a cache write error
cache.insert_raw("k", b"raw".to_vec(), ttl); // bypasses wire format

RedisCache (behind the redis feature, enabled by default)

use cacheflight::RedisCache;
use redis::aio::ConnectionManager;

let conn = ConnectionManager::new(client).await?;
let cache = RedisCache::new(conn);

Custom backend

Implement CacheBackend for any store:

use cacheflight::{CacheBackend, Result, async_trait};
use std::time::Duration;

#[async_trait]
impl CacheBackend for MyStore {
    async fn get(&self, key: &str) -> Result<Option<Vec<u8>>> { /* ... */ }
    async fn set(&self, key: &str, value: Vec<u8>, ttl: Duration) -> Result<()> { /* ... */ }
    async fn delete(&self, key: &str) -> Result<()> { /* ... */ }
    async fn delete_by_prefix(&self, prefix: &str) -> Result<u64> { /* ... */ }
}

Expiry strategies

The API uses Rust's type system to guide configuration:

Flat TTL

let cf = CacheFlight::new(cache)
    .ttl(Duration::from_secs(30));

Once the TTL expires, the entry is treated as expired and the next caller recomputes.

Stale-while-revalidate

let cf = CacheFlight::new(cache)
    .stale_while_revalidate(
        Duration::from_secs(30),   // fresh window
        Duration::from_secs(120),  // stale window
    );

Callers within the fresh window get a CacheHit. During the stale window, callers get the stale value immediately while a single background refresh runs. If multiple callers arrive during the stale window, they all share the same in-flight refresh.

Probabilistic early expiration (XFetch)

let cf = CacheFlight::new(cache)
    .ttl(Duration::from_secs(30))
    .probabilistic_expiry(2.0);

XFetch may trigger a background refresh before the TTL expires, based on the smoothed compute duration (delta EMA) and the beta parameter. Higher beta makes early refreshes more likely. Callers see CacheHit while the refresh runs in the background.

XFetch also works on top of SWR:

let cf = CacheFlight::new(cache)
    .stale_while_revalidate(Duration::from_secs(30), Duration::from_secs(120))
    .probabilistic_expiry(2.0);

Per-call overrides

The run() method returns a RunBuilder that supports per-call overrides:

cf.run("key", work)
    .ttl(Duration::from_secs(10))          // flat TTL only
    .await?;

cf.run("key", work)
    .fresh_for(Duration::from_secs(5))     // SWR only
    .stale_for(Duration::from_secs(60))    // SWR only
    .await?;

cf.run("key", work)
    .beta(4.0)                              // XFetch only
    .await?;

Cache invalidation

// Invalidate a single key.
cf.invalidate("user:42").await?;

// Invalidate all keys matching a prefix.
let count = cf.invalidate_prefix("session:").await?;
println!("invalidated {count} sessions");

Metrics

Implement MetricsHooks to observe cache behavior. All methods have no-op defaults — override only what you need. See docs.rs for the full trait definition.

use cacheflight::{MetricsHooks, CacheMissReason, RecomputeReason};

struct MyMetrics;

impl MetricsHooks for MyMetrics {
    fn on_cache_hit(&self, key: &str) { /* ... */ }
    fn on_cache_miss(&self, key: &str, reason: CacheMissReason) { /* ... */ }
}

let cf = CacheFlight::with_metrics(cache, MyMetrics).ttl(Duration::from_secs(30));

License

Licensed under either of MIT or Apache 2.0 at your option.