GitHub - openpeeps/datengine: DatEngine 🚀 Modular AI Agentic Framework written in Nim lang

DatEngine 🚀 Modular AI Agentic Framework written in Nim lang

nimble install datengine

About

DatEngine is an app-agnostic agentic engine written in Nim. Made to build self-hosted, AI agents via command line and browsers. Designed as a pure library with no HTTP server or UI. A CLI, REST, or WebSocket adapter drives it.

The engine orchestrates an LLM agent loop, manages sessions, executes tools with a strict safety envelope, and automates browsers via CDP (Chrome DevTools). Every wire format (config, sessions, tool schemas) flows through openparser as typed Nim objects.

😍 Key Features

Any OpenAI-compatible endpoints
Streaming SSE: token-by-token deltas via ChaChaChat's async HTTP client
Tool calling: agentic function calling with typed parameter schemas and automatic argument decoding
Session persistence: Boogie RDBMS with indexed columns, conversation history survives restarts
Truncation: configurable history window with system-message preservation via replaceMessages
Cancellation: cooperative cancellation via agent flag (checked between tool iterations)
Skills: markdown files with YAML frontmatter, fuzzy-matched per turn via floof and injected into the system message (see below)
Two opt-in sources: global (globalFs disk skills at ~/.<myagent>/skills) and per-session workspace (<skillsDir> relative to Workspace.root); workspace overrides global when names collide. Both empty = skills disabled.
floof fuzzy matching: SIMD-accelerated subsequence search of user input against keywords and names; skillMinScore threshold (default 0.5), top-N cap via skillMaxPerTurn (default 3).
flysystem loading: skills are read through flysystem drivers (globalFs.disk("skills") + Workspace.fs.disk("workspace")); newSkillRegistryFromDrivers does driver-level listing, gitignore rules apply only to workspace skills.
Model-facing tools: skill_list and skill_read let the LLM browse skill content explicitly.

Quick Start

Phase 1 — global init (once at startup): all dirty wiring inside initDatEngine, providers auto-synced:

import datengine

# single call with large param set; everything dirty happens inside:
# newGlobalFs (skills/config/providers at ~/.myagent) + newProviderStore
# + auto syncFromGlobalFs (YAML/JSON at ~/.myagent/providers/*.yml)
# + newSessionStore + baseDir derivation + ensure dirs
var engee = initDatEngine(
  globalHome = getHomeDir() / ".myagent",
  baseDir = "./storage",
  mode = amBuild,              # ask (default, readOnly) / plan / build
  skillsDir = "skills",
  maxIterations = 10,       # max tool-loop iterations per turn (chachachat runToolLoop)
  truncateTo = 50           # keep last N non-system messages in history; 0 = keep all
)
# or from YAML: let cfg = loadEngineConfig("engine.yml"); var engee = initDatEngine(cfg)

# providers via global single source at ~/.myagent/providers (add via API, not init):
engee.addProvider("openai", "https://api.openai.com/v1", "gpt-4o", apiKeyEnv="OPENAI_API_KEY")
# name unique: openai -> openai-1/-2 on collision
# users can also add ~/.myagent/providers/ollama.json manually:
# {"name":"ollama","baseUrl":"http://localhost:11434/v1","model":"llama3"}
echo engee.listProviders().len

# model discovery (async, cached in ProviderStore under models:<name>):
import std/asyncdispatch
let models = waitFor engee.fetchProviderModels("openai")          # uses stored baseUrl/apiKeyEnv
# or before provider exists: let models = waitFor engee.fetchProviderModelsForUrl("https://opencode.ai/zen/go/v1", "", "")
# pick and update: var cfg = engee.getProvider("openai").get; cfg.model = models[0].id; engee.upsertProvider(cfg)
# cached access: let cached = engee.getProviderModels("openai")
echo models.len

Phase 2 — per workspace / per agent (per session / per request):

# isolated Workspace at ./storage/workspaces/<sessionId> + artifacts, gitignore-filtered
let agent = engee.newAgent(sessionId) # or engee.newAgentForUser(sessionId, userId)
# mode-aware tools + skills are auto-wired inside (Ask registers no fs_tools)
let resp = waitFor agent.run("Analyze the files in this workspace")
echo resp.text

# runtime mode switch
engee.setMode(amPlan)              # affects next workspaces
agent.setMode(amPlan)              # affects this agent's workspace (re-applies PolicyRules)
let ws = engee.getWorkspaceForSession(sessionId)
echo ws.root # ./storage/workspaces/<id>

Tool System

Allowlisted CLI Binary allowlist, no shell metacharacters, cwd confinement (Workspace.root), per-tool output caps and timeouts
rtk proxy Token-optimized output for the model (ls, tree, read, grep, find, diff, wc, json)
Document extraction: pdftotext, pdfinfo, pdftoppm, pdftohtml, vips, sips, convert, ffmpeg; renders/screenshots land on the per-session artifacts disk (Workspace.artifactWrite)
Per-session gitignore-aware workspace Workspace owns a per-session Filesystem with disks workspace (filtered via pkg/gitignore IgnoreStack; .env, .git, node_modules never reach the model) + artifacts (unfiltered sibling for downloads/renders). Every path is LocalDriver.resolvePath traversal-proof and atomic. Global state lives on a separate host-wide globalFs: Filesystem with named disks skills/config at ~/.myagent. AgentMode (ask/plan/build) enforces flysystem PolicyRules (ask/plan = readOnly=true, build = writable with 10 MB caps).
Browser automation chopchop CDP: goto, waitForNavigation(NetworkIdle), querySelector, evaluate, screenshot, click, typeText
Safety envelope per-tool byte/line caps, timeouts (30/60/120s), truncation markers, process kill on timeout, plus PolicyError on AgentMode violations
Per-session todos Session.todos: seq[TodoItem] (id, content, status: pending|in_progress|completed|cancelled, priority: high|medium|low) persisted via SessionStore (sessions.todosJson). LLM tools todo_create(content, priority?) → id, todo_update(id, content?, status?, priority?) (single-item patch by id), todo_delete(id), todo_read (explicit, not auto-injected). Enforced “plan before build”: in amPlan/amBuild any non-todo tool is blocked until at least one todo exists (ask exempt). Managed per-session, visible across newAgent(sessionId) reloads.

Providers

OpenAPI-compatible: ProviderConfig(name, baseUrl, model, apiKey, apiKeyEnv) — name globally unique, collisions auto-suffixed blabla → blabla-1/blabla-2.
Global single source: globalFs disk providers at ~/.myagent/providers (flysystem, YAML/JSON via openparser), backed by boogie DocumentStore at ~/.myagent/providers.ddb/.wal. Available everywhere, all sessions/workspaces.
API: newProviderStore(home, globalFs), syncFromGlobalFs(), listProviders(), getProvider(name), upsertProvider / upsertProviderUnique, deleteProvider, resolveApiKey. YAML example at providers/openai.yml: name: openai + baseUrl: "https://api.openai.com/v1" (quote URLs) + apiKeyEnv.
Model discovery: OpenAI-compatible GET {baseUrl}/models → {"object":"list","data":[{"id","created","owned_by"}]} mapped via openparser fromJson into LLModel (models.nim). Async fetcher fetchProviderModels(baseUrl, apiKey) / fetchProviderModels(cfg) / fetchProviderModelsForUrl(baseUrl, apiKey, apiKeyEnv) — no auto-discover on addProvider; caller does let models = await engee.fetchProviderModels("openai") or await engee.fetchProviderModelsForUrl(baseUrl, apiKey, apiKeyEnv) once when adding a new provider, then picks model for addProvider. Fetched list is cached in ProviderStore under models:<name> (ModelListResponse wrapper) via setProviderModels/getProviderModels/hasProviderModels; refresh requires explicit fetchProviderModels call. DatEngine wrappers: fetchProviderModels(providerName), fetchProviderModels(cfg), fetchProviderModelsForUrl, getProviderModels, hasProviderModels. Live example: https://opencode.ai/zen/go/v1/models returns mimo-v2.5, kimi-k2.5, glm-5.3, etc.

Plugins

Global, tools-only, app-controlled: ~/.myagent/plugins/*.so|.dylib|.dll (host-wide globalFs disk plugins at newGlobalFs). When enabled, DatEngine owns a PluginManager (pluginkit ABI 1, semver, NimVersion gate). Plugins are dynamic libraries built via nim c --app:lib --mm:orc --threads:on myplugin.nim.
Contract: plugin exports plugin_datengine_tools_json*(): cstring {.exportc, cdecl, dynlib.} → JSON array [{"name","description","schema":{…}}] and per-tool handler plugin_tool_<name>*(argsJson: cstring): cstring {.exportc, cdecl, dynlib.} (sync, args is JSON object string). Host at src/datengine/plugins.nim:60 attachPluginTools(manager, registry) per-session (in engine.bindAgent) discovers plugin_datengine_tools_json via dynlib.symAddr, parses JSON, and registers each as Tool with a chachachat.ToolHandler wrapper that forwards args: JsonNode as $args cstring and returns handler result. Duplicate names skipped (first wins).
Lifecycle: app drives via DatEngine public API — getPluginsDir, listInstalledPlugins (walk home/plugins), loadPlugin(path) → id, activatePlugin(id), unloadPlugin(id), installPlugin(srcPath, destName?) → dest (copy to home/plugins), uninstallPlugin(id) (unload then removeFile), listLoadedPlugins, hasPlugin, getPluginManager. DatEngine.close unloads all. No auto-scan at initDatEngine, no watcher, no permission enforcement (manifest permissions at pluginkit.nim:94 ignored for now). Example at packages/supranim-packages/pluginkit/example/helloworld.nim style plus plugin_datengine_tools_json/plugin_tool_my_echo.
Config: AgentConfig.pluginsDir*: string (global, empty → home/plugins), YAML-friendly, default via home/plugins at engine.initDatEngine and ensurePluginsDir.
Security: ABI PluginAbiVersion=1 check at pluginkit.nim:540, version/ NimVersion gates. Tools run with same safety envelope as host tools; ask/plan read-only policies still apply via AgentMode.

DatEngine native plugins

Plugins are global dynamic libraries that extend the LLM tool surface at runtime. The host owns the lifecycle; there is no auto-scan or watcher. Build with nim c --app:lib --mm:orc --threads:on.

Plugin side (myplugin.nim):

import pkg/pluginkit
import std/json

plugin myplugin, {
  name: "MyPlugin",
  author: "Example",
  description: "Echo text",
  license: "MIT",
  url: "https://example.com",
  version: "0.1.0"
}:
  discard

proc plugin_datengine_tools_json*(): cstring {.exportc, cdecl, dynlib.} =
  ## JSON array of tools: host at src/datengine/plugins.nim:60 discovers this symbol
  """[{"name":"my_echo","description":"Echo text","schema":{"type":"object","properties":{"text":{"type":"string"}},"required":["text"]}}]"""

proc plugin_tool_my_echo*(argsJson: cstring): cstring {.exportc, cdecl, dynlib.} =
  ## Per-tool handler: `plugin_tool_<name>` — argsJson is the tool args as JSON object string
  let args = parseJson($argsJson)
  let text = args{"text"}.getStr("")
  cstring("echo:" & text)

Compile:

nim c --app:lib --mm:orc --threads:on -o:myplugin.dylib myplugin.nim
# Linux: .so, macOS: .dylib, Windows: .dll

Host side (datengine):

import datengine
import std/asyncdispatch

let engee = initDatEngine(
  globalHome = getHomeDir() / ".myagent",
  baseDir = "./storage",
  mode = amBuild
)
discard engee.addProvider("openai", "https://api.openai.com/v1", "gpt-4o", apiKeyEnv="OPENAI_API_KEY")

# install persists to ~/.myagent/plugins/ (or custom AgentConfig.pluginsDir)
let dest = engee.installPlugin("/path/to/myplugin.dylib")
# or: let dest = engee.installPlugin("/path/to/myplugin.dylib", "myplugin.dylib")

# load + activate (app-controlled, no auto-load)
let id = engee.loadPlugin(dest)       # → hash id, checks ABI/semver/NimVersion at pluginkit.nim:515
engee.activatePlugin(id)              # calls plugin_init (NimMain), status → pluginStatusActive

# discovery
echo engee.getPluginsDir()            # ~/.myagent/plugins or custom
echo engee.listInstalledPlugins()     # ["…/plugins/myplugin.dylib"]
echo engee.listLoadedPlugins().len    # 1
echo engee.hasPlugin(id)              # true

# per-session attach: engine.bindAgent attaches currently loaded plugins to each newAgent
let agent = engee.newAgent("sess-1")
assert agent.registry.hasTool("my_echo")
assert agent.registry.hasTool("todo_create") # built-ins remain
let res = waitFor agent.registry.getTool("my_echo").get.handler("my_echo", %*{"text":"hello"})
echo res # echo:hello

# unload / uninstall (app-controlled)
engee.unloadPlugin(id)
assert not engee.hasPlugin(id)
engee.uninstallPlugin(id) # not needed if already unloaded: no-op; otherwise unloads then removeFile
# or after re-load:
# let id2 = engee.loadPlugin(dest); engee.activatePlugin(id2); engee.uninstallPlugin(id2)
# engee.close() unloads all remaining plugins

See src/datengine/tools/plugin.nim:1 shim and src/datengine/plugins.nim:60 attachPluginTools.

Storage

Boogie RDBMS https://github.com/openpeeps/boogie
Indexed relational store for sessions (indexed columns, structured queries)
Boogie DocumentStore https://github.com/openpeeps/boogie
Schemaless JSON store for providers (providers docstore, putObj/getObj, pairs), single global file.
Flysystem https://github.com/openpeeps/flysystem
Multi-disk sandbox: per-session Workspace.fs (workspace + artifacts disks, traversal-proof, atomic writes) + host-wide globalFs (skills + config + providers + plugins disks at ~/.myagent). All reads/writes go through StorageDriver; no raw readFile paths escape the engine.
OpenParser https://github.com/openpeeps/openparser
Collection parsers/dumpers: Full QR family/JSON/TOML/YAML/FBE/DotEnv/iCal/Regex/SQL/Gettext (po/mo) and more

Skills

Markdown + YAML frontmatter; the engine injects matching raw skill bodies into the system message each turn:

---
name: pdf-analysis
description: How to extract and analyze PDF documents
keywords: [pdf, extract, document, ocr]
---
# PDF Analysis
(instructions for the LLM...)

Testing

Mock LLM: mock OpenAI-compatible server with streaming SSE and tool_call responses
~155 tests: core types, agent lifecycle, tool safety, session round-trip, config parsing, truncation, persistence, boogie storage, skills, providers (DocumentStore, YAML/JSON sync, unique suffix, globalFs)

With Skills

Skills are opt-in via flysystem disks: host-wide globalFs (~/.myagent/skills) and per-session Workspace (<workspace>/skills). Loaded through drivers, not raw paths.

~/.myagent/                    # host globalFs (newGlobalFs)
├── skills/
│   └── pdf-analysis.md       # available in every session
└── config/
    └── engine.yml

<workspace>/                   # per-session Workspace.root via newWorkspace/forSession
└── skills/
    └── project-specific/     # <name>/SKILL.md layout also works
        └── SKILL.md

Legacy newSkillRegistry(fs, skillsDir, globalPath) and newFsTool(root) shims remain for single-workspace scripts, but web apps should use globalFs + Workspace + newSkillRegistryFromDrivers.

On each run, user input is fuzzy-matched against skill keywords and names (floof); matching raw markdown bodies are injected into the system message for that turn. The model can also call skill_list / skill_read explicitly. Agent.setMode and Workspace.setMode can be used to gate plan vs build at runtime.

Architecture

src/datengine/
├── agent.nim          # Agent loop: chachachat Conversation + tools + hooks, holds Workspace, setMode
├── config.nim         # ProviderConfig(name, baseUrl, model, apiKeyEnv) — name globally unique; EngineConfig (agent only, no provider; providers via ProviderStore)
├── models.nim         # Model discovery: LLModel/ModelListResponse via openparser fromJson, async GET {baseUrl}/models (e.g. https://opencode.ai/zen/go/v1/models) + caching
├── mockllm.nim        # OpenAI-compatible mock server for testing
├── prompt.nim         # System prompt builder from tool schemas
├── providers.nim      # Global providers (DocumentStore at ~/.myagent/providers.ddb + globalFs disk providers, OpenAPI-compatible, YAML/JSON via openparser, unique suffix, syncFromGlobalFs) + model cache models:<name>
├── serialization.nim  # openparser glue: fromJsonArgs, jsonOrEmpty, helpers
├── session.nim        # Indexed relational store for sessions (indexed columns, structured queries) + per-session todos (persisted todosJson, LLM-managed via todo_*)
├── skills.nim         # Skill loading via flysystem drivers (globalFs + workspace) + floof matching
├── workspace.nim      # Per-session Workspace (flysystem Filesystem: workspace + artifacts) + globalFs (skills/config/providers at ~/.myagent), AgentMode PolicyRules, gitignore stack, per-session forSession helper
├── engine.nim         # High-level DatEngine (initDatEngine large params, auto sync providers, newAgent factory, getters/setters, fetchProviderModels async wrappers, PluginManager (global plugins at ~/.myagent/plugins), no global agent)
├── plugins.nim        # Host plugin manager (global, tools-only, app-controlled load/activate/install/uninstall, per-session attachPluginTools via plugin_datengine_tools_json → plugin_tool_<name>)
├── tools.nim          # Tool, ToolRegistry, ToolResult, JSON Schema helpers
└── tools/
    ├── cli.nim        # Allowlisted subprocess (threadpool, caps, timeouts): workdir = Workspace.root
    ├── rtk.nim        # rtk output proxy
    ├── document.nim   # poppler/vips/sips/ffmpeg wrappers: outputs to artifacts disk
    ├── fs.nim         # FsTool adapter over Workspace disk (shares LocalDriver + IgnoreStack)
    ├── todo.nim       # Per-session todos (persisted via SessionStore, id-based todo_create/update/delete/read, plan-before-build enforcement)
    ├── plugin.nim     # Thin shim for plugin authors (re-exports pluginkit, documents plugin_datengine_tools_json/plugin_tool_<name> contract)
    └── browser.nim    # chopchop CDP browser automation: screenshots to artifacts disk

Dependencies

Package	Version	Role
chachachat	>= 0.1.0	LLM client, SSE streaming, agent loop, tool calling
openparser	>= 0.1.9	JSON/YAML direct-to-object serialization
flysystem	>= 0.1.0	Multi-disk filesystem sandbox
boogie	>= 0.1.2	A suite of WAL-based embedded data stores. RDBMS, KV Store, GraphStore, VectorStore, Columnar and more
gitignore	>= 0.1.0	Spec-compliant ignore stack for workspace sandbox
chopchop	>= 0.1.0	CDP browser automation (goto, evaluate, screenshot, click)
powpow	>= 0.1.9	Event loop, file watcher, HTTP/WS server
marvdown	>= 0.1.4	Markdown parser: YAML frontmatter for skills, HTML output, JSON AST
sweetsyntax	>= 0.2.0	YAML-driven syntax highlighter & AST explorer: ANSI, HTML, JSON renderers, code folds
pluginkit	>= 0.1.1	Plugin manager: macro DSL for dylibs, semantic versioning, permission system, lifecycle hooks
floof	>= 1.0.0	SIMD-accelerated fuzzy search: skill keyword matching against user input

Planned: BU CLI tools

c-blake/bu: ~70 Nim-native CLI tools (pipe-oriented, zero-config, faster than GNU coreutils). Tier 1 integration planned for agent toolchains:

Tool	Purpose
`dups`	Find duplicate-content files
`topn`	Top-N rows by any column, single-pass
`ndelta`	Numeric diff between two reports
`cols`	Extract columns from delimited text
`noc`	Strip ANSI escape sequences
`ft`	Batch file type test
`newest`	Find N newest/oldest files by timestamp
`since`	Find files newer than a reference
`cstats`	Summary stats for numeric columns
`catz`	Universal decompressor (auto-detect format)
`ru`	High-precision resource usage measurement
`oft`	Most-frequent items (count-min sketch)
`tails`	Unified head+tail with both-ends support

Roadmap

Contributions & Support

🐛 Found a bug? Create a new Issue
👋 Wanna help? Fork it!

Name		Name	Last commit message	Last commit date
.github		.github
src		src
tests		tests
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
datengine.nimble		datengine.nimble

Navigation Menu

Uh oh!

Repository files navigation

About

😍 Key Features

Quick Start

Tool System

Providers

Plugins

DatEngine native plugins

Storage

Skills

Testing

With Skills

Architecture

Dependencies

Planned: BU CLI tools

Roadmap

Contributions & Support

License

About

Releases

Packages

Contributors

Languages

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

About

😍 Key Features

Quick Start

Tool System

Providers

Plugins

DatEngine native plugins

Storage

Skills

Testing

With Skills

Architecture

Dependencies

Planned: BU CLI tools

Roadmap

Contributions & Support

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages