devino

A Rust OpenVINO CPU backend for explicitly verified local model bundles.

Devino owns native library loading, device observation and bounded generation. zixcel-local-inference owns model identity, catalog/acquisition and runtime lifecycle. The caller owns the worker process and authorization; Devino creates no second store or daemon.

Rust interface

1. Admit the complete native distribution and model identity independently.
2. Spawn a dedicated Linux worker and attach BackendProcessContext before native loading. Keep a hard deadline and cleanup in the external owner.
3. Supply exact NativeLibraries paths/hashes to OpenVinoRuntime::load.
4. Verify a complete VerifiedModelBundle, then call OpenVinoCpu::load.
5. Generate with finite GenerationOptions and a GenerationControl.

Input, output and token budgets are bounded. CPU uses one inference thread and one stream. Errors have stable codes; cancellation does not grant a new budget. The native backend remains native code: Rust does not make model parsing or in-process cancellation a sandbox. See the interface.

Development and acceptance

Requires Rust 1.97 and Linux. Build/default tests use runtime linking, so no native SDK, model download or GPU is required:

CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo test --locked -- --test-threads=1

Development tests and explicit native acceptance are separate. The latter exercises Intel's real CPU runtime with a small, synthetic model: deterministic generation, actual token usage, output limits, pre-cancellation and reuse after cancellation. It does not establish useful language quality, arbitrary models, GPU compatibility or long-term leak freedom. No model weights or native runtime are bundled in the Rust package.

Model preparation

Use a reviewed preconverted OpenVINO GenAI bundle for text generation. The historical Python/PyTorch exporter, model downloads, system-install recipes and empty server scaffolding have been retired. Devino does not execute remote model code or force incompatible Optimum/Transformers dependencies.

For compatible conventional models, OpenVINO can read ONNX directly through Intel's Rust bindings, avoiding an intermediate IR export. This is distinct from converting arbitrary PyTorch/LLM models. See conversion alternatives for the evaluated APIs and limits. GPU activation and an HTTP model server are outside the supported Rust interface.

Getting started · source contract · security · Apache-2.0 · notices