Skip to content

README.md

Latest commit

 

History

History
439 lines (324 loc) · 26 KB

README.md

File metadata and controls

439 lines (324 loc) · 26 KB
Raw

KeyLoad, the AI-native database for AI agents

KeyLoad

The AI-native database. One database for AI agents.
Documents, typed tables, graphs, vectors/search, blobs, queues, events and time series in one replicated .NET cluster.
Everything links to everything else, and agents reach all of it through SQL, a built-in MCP server and a typed .NET SDK.

CI MIT license .NET 10 Built on Orleans ZoneTree storage MCP server built in Development preview

Website · Quick start · Capabilities · Why .NET, Orleans and ZoneTree · Docs · Status

Why an AI-native database?

Agents shouldn't need a dozen databases. Yet an agent that needs memory and the ability to act can easily end up wired to seven: Postgres for records, a vector database for embeddings, Redis or RabbitMQ for tasks, S3 for files, Kafka for events, a graph database for relationships and a time-series database for metrics. Every one of them brings its own client, credentials, permission model, backups and failure modes. The agent, and your team, spend their time keeping seven copies of the truth in sync.

flowchart TB
    subgraph Today["Today: one agent, seven systems"]
        direction TB
        A1["AI agent"] --> PG[("Postgres<br/>records")]
        A1 --> VDB[("Vector DB<br/>embeddings")]
        A1 --> MQ[("Redis / RabbitMQ<br/>tasks")]
        A1 --> S3[("S3<br/>files")]
        A1 --> KF[("Kafka<br/>events")]
        A1 --> GDB[("Graph DB<br/>relationships")]
        A1 --> TSDB[("Time-series DB<br/>metrics")]
    end
    subgraph One["With KeyLoad: one database"]
        direction TB
        A2["AI agent"] --> KL[("KeyLoad<br/>every model, one reference,<br/>one permission check")]
    end
    Today ~~~ One
Loading

KeyLoad is an AI-native database: it's designed around how agents work, not bolted onto a database built for something else.

  • One reference for everything. All models share canonical entity references. A queue message can point to a document, that document can be a node in a knowledge graph, and its files, embeddings and history stay attached to it.
  • One language. SQL is the familiar shared language for querying and combining the models. Agents get the same operations as MCP tools, and applications get them through a typed .NET SDK.
  • One permission model. API keys and access policies live in the database itself. Every call is checked, down to rows and fields, and neither clients nor prompts can grant themselves extra roles.
  • One atomic write. A document, its event, a graph edge and the follow-up task in the same partition commit together, or not at all.
  • Limits on every operation. Full scans must be requested explicitly, and every request has caps on work and memory, so a runaway request is rejected at its limit instead of running unbounded.
  • Replicated from day one. The default server is a three-node cluster that keeps three copies of your data (RF3), on Orleans with ZoneTree storage on each node.

Why we think this is the future

Agents don't think in tables, queues or buckets. Picture a support agent handling a refund: it takes the ticket off a queue, loads the customer and the order, finds similar past cases with vector search, links the case into a knowledge graph, attaches the receipt and schedules a follow-up. Spread across separate systems, that one step turns into a distributed saga held together by glue code and stale copies, and the agent needs a key to every one of them.

When everything lives in one database, that step should become one authorized operation. Today its writes within one partition already commit atomically, while leased receives, blob uploads and cross-partition work are still separate steps. Our goal is that building context becomes one query instead of a join written in prompt code, and that access is one policy you can read. That's why we're building KeyLoad: so that nobody has to stand up and run a stack of databases just to let agents get work done.

KeyLoad is a development preview today: the status section lists what works and what's still in progress. If you'd rather run one database than seven, star the repo to follow along, try the quick start and open an issue describing the agent workload you want it to handle.

What you can build

You want to… KeyLoad gives you
Give an agent long-term memory and retrieval (RAG) Documents, files and embeddings in one place, with vector, full-text and hybrid search
Build a knowledge graph Relationships between any entities, including typed rows, and graph traversal
Run reliable agent workflows Queues with scheduling, leases, retries and dead letters, stored next to the data they change
Make event-driven apps Ordered event history, durable subscriptions, change feeds and projections
Analyze time series Timestamped samples, range reads, aggregates, time windows and retention
Store files for agents Chunked uploads and partial reads of large blobs

One database, connected models

Capability matrix

Model In one atomic commit Read and query SQL MCP tools
📄 Documents ✅ Put, patch, delete with revision checks Get by key, indexed queries, live queries SELECT keyload_documents_*, keyload_query_*
🧾 Typed tables ✅ Rows with a schema, primary keys and unique constraints Indexed queries SELECT keyload_documents_commit, keyload_sql_execute
🕸️ Graphs ✅ Add and remove edges Graph traversal CALL keyload_graph_traverse
🧭 Vectors and search ✅ Store a vector with its document Exact vector, full-text and hybrid search CALL keyload_search_execute
📬 Queues and topics ✅ Enqueue, publish Receive with leases, ACK, retries, dead letters, peek SELECT … FROM QUEUE_MESSAGES(…) keyload_messages_*, keyload_subscriptions_* for topics
📜 Events ✅ Append with an expected revision Read, replay, durable subscriptions SELECT … FROM EVENTS(…) keyload_streams_*, keyload_subscriptions_*
📈 Time series ✅ Append samples Ranges, latest value, aggregates, windows, retention CALL keyload_series_*
📦 Files and blobs Separate upload and publish steps Chunked uploads, byte-range reads CALL keyload_blobs_*
🔁 Change feeds Document changes recorded in the same commit Resumable change feed, projections CALL keyload_changes_read, keyload_projections_*

SELECT reads data. CALL runs any operation from the same catalog the MCP server exposes, so every model is reachable from SQL.

Models that work together

These aren't separate silos. Collections, tables and queues live in the same database and reference each other, so one request can work across models:

  • Queue to knowledge graph (QueueToGraph). Reads ready messages from a queue, resolves their linked entities and writes the new knowledge-graph relationships.
  • Graph to queued actions (GraphToQueueMutation). Follows graph relationships and can enqueue actions for each entity it finds.
flowchart LR
    Q[["Queue: inbox"]] -->|"QueueToGraph"| E["Linked entities<br/>documents and rows"]
    E --> G(("Knowledge graph"))
    G -->|"GraphToQueueMutation"| A[["Queue: actions"]]
Loading

What the first version does. The current composition API runs through .NET CommitAsync, SQL CALL keyload_documents_commit(@arguments) or the official MCP server:

  • All data in one request must live in the same atomic partition and transaction domain, which is the PartitionRef you pass in. The whole batch succeeds or rolls back together.
  • Reading a queue into the graph does not lease or ACK the messages, so they stay in the queue for their regular consumers.
  • Blobs remain part of the same database, but use their separate upload and publication operations.
  • KeyLoad is a development preview. Full declarative SQL, the native SQL-client protocol and cross-partition composition are still in development, and production guarantees remain under qualification.

The composition guide and its transaction contract cover the details.

How it works

flowchart LR
    subgraph Callers
        SDK[".NET SDK"]
        MCP["AI agent over MCP"]
        SQL["SQL over HTTP"]
        CLI["CLI and admin console"]
    end
    subgraph Cluster["KeyLoad cluster (3 nodes)"]
        REQ["Request grain<br/>one per request"]
        PART["Partition grains"]
        N1[("Node 1<br/>ZoneTree")]
        N2[("Node 2<br/>ZoneTree")]
        N3[("Node 3<br/>ZoneTree")]
    end
    SDK --> REQ
    MCP --> REQ
    SQL --> REQ
    CLI --> REQ
    REQ --> PART
    PART --> N1
    PART --> N2
    PART --> N3
Loading
  1. A caller sends an operation to any node, through the SDK, MCP, SQL or plain HTTP.
  2. KeyLoad gives that request its own Orleans grain, which calls only the database grains it needs.
  3. Database grains route the work to the storage owner on each node. Storage stays on its node, even when Orleans moves grains around the cluster.
  4. A write is acknowledged only after a majority of the three replicas has persisted it. If a node fails, the other two keep the acknowledged data.

Here's one agent tool call, end to end:

sequenceDiagram
    participant Agent as AI agent
    participant Node as Any KeyLoad node
    participant Req as Request grain
    participant Part as Partition grain
    participant Reps as Three ZoneTree replicas
    Agent->>Node: MCP tool call with an API key
    Note over Node,Req: Key and permissions are checked against the database
    Node->>Req: A new grain for this request
    Req->>Part: One atomic command
    Part->>Reps: Replicate
    Reps-->>Part: A majority persisted it
    Part-->>Agent: One result, all or nothing
Loading

You'll find the full picture in the architecture map.

Why .NET, Orleans and ZoneTree

We picked a stack where the database, its cluster and its storage all run in one runtime, inside one process per node, with no glue in between.

.NET 10: a fast, memory-efficient runtime.

  • Span<T>, pooled buffers and hardware intrinsics let hot paths avoid allocations and use SIMD. Vectorized paths are a priority workstream, and we'll publish measurements before claiming any speedup.
  • One language end to end. The server, SDK, CLI, analyzers and tests are all C#.
  • Aspire orchestrates the whole three-node cluster, locally and in CI, from one AppHost.
  • Open source, cross-platform and at home in Linux containers.

Orleans: a cluster runtime that's proven in production.

  • Virtual actors (grains) that the cluster places, activates and moves for you. Orleans came out of Microsoft Research and has powered Halo's cloud services, among others.
  • KeyLoad gives each request its own grain, so isolation, cancellation and backpressure work per request.
  • Membership and failure detection are built in. KeyLoad also turns on Orleans' experimental distributed grain directory and activation repartitioning, which we're still qualifying.
  • Fast generated binary serialization between nodes.

ZoneTree: storage that lives inside the node.

  • An embedded, persistent, ordered key-value engine written in C#. Data lives in each node's own process, with no extra network hop and no native interop.
  • A write-ahead log on each node, and ordered keys for range scans and indexes.
  • One storage engine under every model: documents, rows, graphs, vectors, queues, events, time series and blobs.
  • ZoneTree.FullTextSearch adds the full-text index on the same foundation.

Together: Orleans moves the routing, and ZoneTree keeps the data in place. Storage never travels with a grain, so the cluster can rebalance work without copying files between nodes.

Quick start

You need: the .NET SDK version pinned in global.json, and Docker.

1. Build the solution

dotnet restore KeyLoad.slnx
dotnet build KeyLoad.slnx --no-restore --configuration Release

2. Build the server image

The cluster runs the server image pinned by its digest, so build it from source and push it to a local registry:

docker run -d --name keyload-registry -p 5050:5000 registry:3
docker build -t localhost:5050/keyload/server:dev . && docker push localhost:5050/keyload/server:dev
export KeyLoad__ContainerImages__Server="localhost:5050/keyload/server:dev@$(docker inspect --format '{{index .RepoDigests 0}}' localhost:5050/keyload/server:dev | cut -d@ -f2)"

3. Start the three-node cluster

dotnet run --project src/KeyLoad.AppHost --configuration Release --no-build

Aspire starts three nodes at http://localhost:5101, :5102 and :5103. Node data and your local development credentials are stored in data/cluster/, which git ignores. Keep that folder between restarts.

4. Look around

dotnet run --project src/KeyLoad.Cli --configuration Release --no-build -- status http://localhost:5101 data/cluster/local-profile.json

Your local admin API key is the AdminKey value in data/cluster/local-profile.json. Export it as KEYLOAD_API_KEY for the examples below.

Use it

From .NET

The client SDK gives you typed database operations. This creates a collection, writes an order and reads it back:

using KeyLoad;
using KeyLoad.Client;

var apiKey = Environment.GetEnvironmentVariable("KEYLOAD_API_KEY")
    ?? throw new InvalidOperationException("Set KEYLOAD_API_KEY.");
using var http = new HttpClient { BaseAddress = new Uri("http://localhost:5101") };
var client = new KeyLoadClient(http, apiKey);

// tenant, database, transaction domain, partition key
var partition = new PartitionRef("acme", "shop", "orders", "customer-42");

var collection = await client.ConfigureResourceAsync(Guid.NewGuid(), new("acme", "shop",
    new ResourceDefinition("orders", ResourceKind.Collection, "orders")));
collection.ThrowIfFail();

var commandId = Guid.NewGuid(); // Reuse this ID if you retry the same write.
var result = await client.CommitAsync(new CommandRequest(commandId, partition,
[
    new PutDocument("orders", "order-1", "{\"status\":\"new\"}", ExpectedRevision: 0)
]));
result.ThrowIfFail();

var order = await client.GetAsync(new EntityRef(partition, "orders", "order-1"));
order.ThrowIfFail();

With SQL

Use SQL to read data and to call any database operation:

using System.Text.Json;

var rows = await client.ExecuteSqlAsync(new SqlOperationRequest(partition,
    "SELECT * FROM orders WHERE status = @status LIMIT 20",
    new() { ["status"] = JsonSerializer.SerializeToElement("new") },
    AllowFullScan: true));
rows.ThrowIfFail();
Console.WriteLine(rows.Value);

Here's what works today:

SELECT * FROM orders WHERE number BETWEEN 1 AND 9 ORDER BY id  -- filter and sort documents
SELECT * FROM QUEUE_MESSAGES('jobs')                           -- peek at a queue without consuming it
SELECT e.eventType FROM EVENTS('events', 'stream-a', 3) AS e   -- read event history
EXPLAIN SELECT * FROM orders                                   -- see the query plan
CALL keyload_documents_commit(@arguments)                      -- run any database operation

Model views (QUEUE_MESSAGES, EVENTS) and unindexed filters require AllowFullScan: true, and model views don't support cursors. Full SQL, including joins, is still in progress. The query guide lists the supported syntax, and the compatibility inventory tracks the rest.

From an AI agent (MCP)

Every node has a built-in MCP server at /mcp. Database operations are available as tools, such as keyload_sql_execute, keyload_documents_commit, keyload_search_execute, keyload_graph_traverse, keyload_messages_receive and keyload_blobs_read_range. Add it to any MCP client. In Claude Code's .mcp.json, for example:

{
  "mcpServers": {
    "keyload": {
      "type": "http",
      "url": "http://localhost:5101/mcp",
      "headers": { "Authorization": "Bearer ${KEYLOAD_API_KEY}" }
    }
  }
}

The agent sees exactly what its API key allows. Permissions are stored in the database, so a prompt can't escalate them. The MCP and API guide describes the full operation catalog, along with the plain HTTP routes under /v1/.

Project status

KeyLoad is a development preview. Try it, build with it and tell us what breaks, but don't trust it with production data yet.

Ready to try (in source, covered by tests) Still in progress
Documents, typed rows, graphs, queues, events, time series, blobs and search behind one permission model Full SQL (joins, foreign keys) and a native SQL client protocol
SQL SELECT, model views and CALL, plus the .NET SDK, MCP server and HTTP API Combining data across partitions in one request
Queue ↔ graph composition within one partition Approximate vector search (ANN); vector search is exact for now
Three-node Orleans cluster, crash recovery, local backup and restore Production, endurance and power-loss qualification
Admin console and CLI Storage upgrades from older versions, which still have an open cold-cluster failure
The full performance comparison with other databases (current runs are on the website)

Full-text search comes from ZoneTree.FullTextSearch. KeyLoad then ranks the results it finds and checks permissions on each one.

For detailed status, see the implementation tracker and the qualification records. We publish performance numbers only from real GitHub Actions runs, on the website.

FAQ

What is an AI-native database?

A database designed around how AI agents work: every kind of agent data in one place, linked by shared references, reachable through MCP and SQL, with permissions an agent can't talk its way around. That's what KeyLoad is built to be.

Is KeyLoad a vector database for .NET?

It includes vector search, but vectors don't sit in a separate store. They live next to the documents, graph, queues and files they belong to, and one request can use all of them. Vector search is exact today, and approximate (ANN) search is in progress.

Can KeyLoad be the memory layer for my agents?

That's the main use case. Documents, embeddings, files, a knowledge graph and event history live together, so long-term memory, retrieval (RAG) and the task queue share one store and one permission model.

Does KeyLoad replace Postgres, Redis, Kafka or S3?

For agent workloads, that's the goal: one database instead of a stack of them. KeyLoad is still a development preview, so check the project status before you move anything.

How do AI agents connect?

Every node runs a built-in MCP server at /mcp. Point any MCP client at it with an API key, and the agent can use exactly the operations that key allows.

Why build a database on Orleans?

Orleans already handles cluster membership, failure detection and request routing for .NET. KeyLoad gives every request its own grain, while ZoneTree keeps each node's data on that node. See Why .NET, Orleans and ZoneTree.

Can I use KeyLoad from Python, TypeScript or another language?

Yes, through MCP or the HTTP API under /v1/. The typed SDK is .NET for now.

Is KeyLoad open source?

Yes. KeyLoad is MIT-licensed and developed in the open on GitHub.

Can I run it in production?

Not yet. Endurance, fault and power-loss qualification are still in progress.

Repository map

Path What's inside
src/KeyLoad.Server The database server: HTTP API, MCP server and admin console
src/KeyLoad.Client The .NET SDK
src/KeyLoad.Cli Command-line tool for operators and workers
src/KeyLoad.AppHost Aspire host for the local cluster and every test suite
src/KeyLoad.Abstractions Shared public contracts
src/KeyLoad.Core, Query, Orleans, Replication, Security, Storage.* Database engine, SQL, cluster, replication, permissions and storage
tests/ Unit, crash-recovery, three-node and website tests (TUnit)
benchmarks/ Comparisons with other databases, plus microbenchmarks
site/ Source for keyload.cloud
docs/ Architecture, feature specifications and design decisions

Contributing

All test suites run through the Aspire AppHost. After building, pick a suite: unit, unit-scalar, recovery, rf3, analyzers, comparison or site.

dotnet run --project src/KeyLoad.AppHost --no-build --no-restore --configuration Release -- --KeyLoadTests:Suite=unit
dotnet format KeyLoad.slnx --verify-no-changes --no-restore

The rf3 suite starts a real three-node cluster in Docker and tests it through the .NET SDK and the official MCP client. Before your first change, read the architecture map and the feature spec in docs/Features/. If you work with AI coding agents, AGENTS.md holds the repository rules.

Credits

KeyLoad is developed by Managed Code and stands on the shoulders of these projects. Thank you to their authors and contributors.

Project How KeyLoad uses it
.NET and ASP.NET Core Server, client SDK and HTTP hosting
Orleans Distributed request execution, cluster routing and binary serialization
ZoneTree Persistent ordered storage for every database model
ZoneTree.FullTextSearch Full-text search index
MCP C# SDK Built-in MCP server and real MCP clients in tests
ManagedCode.Communication Typed operation results and ASP.NET Core/Orleans integration
ManagedCode.Storage File-system storage and backup transfer support
ManagedCode.TimeSeries Time-series aggregation
ManagedCode.Orleans.Graph Grain call relationship policies
Cartograph Segmented backup archives and catalogs
OpenTelemetry .NET Logs, metrics and traces

Development and presentation also use Aspire for Docker orchestration, TUnit for tests, Roslyn for the repository's own code analyzers, BenchmarkDotNet for microbenchmarks, and Three.js for the website's cluster illustration. The benchmark suite compares KeyLoad with the free community editions of PostgreSQL with pgvector, TimescaleDB, MongoDB, Redis, RabbitMQ, KurrentDB, Neo4j, Qdrant and OpenSearch, connecting through the official clients Npgsql, MongoDB .NET Driver, StackExchange.Redis, RabbitMQ .NET Client and KurrentDB .NET Client.

License

KeyLoad is licensed under MIT.

Developed by Managed Code