Core module definitions for the Log10x engine, the brain that powers log cost analysis, filtering, and optimization. Start with the MCP Server, it guides you through installing and configuring the apps below based on k8s discovery of your environment.
Full Documentation: doc.log10x.com
┌───────────────────────────────────────────────────────────────────────────┐
│ Log10x Engine │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Applications │ │
│ │ │ │
│ │ ┌─────────────────────────┐ ┌─────────────────────────────────┐ │ │
│ │ │ Runtime Apps │ │ Setup / Tools │ │ │
│ │ │ │ │ │ │ │
│ │ │ ├─ Reporter (DaemonSet)│ │ ├─ Dev (local CLI) │ │ │
│ │ │ ├─ Receiver (sidecar) │ │ ├─ Compiler (symbol generation)│ │ │
│ │ │ │ Filter + Compact │ │ └─ MCP scaffold (validation) │ │ │
│ │ │ └─ Retriever (S3 query)│ │ │ │ │
│ │ └─────────────────────────┘ └─────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Pipelines │ │
│ │ │ │
│ │ ┌───────────────────────────────────────────────────────────────┐ │ │
│ │ │ COMPILE: Pull ──► Scan ──► Link ──► Push │ │ │
│ │ │ (GitHub, Docker, Helm) → (AST, Bytecode) → Symbols │ │ │
│ │ └───────────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ │ ┌───────────────────────────────────────────────────────────────┐ │ │
│ │ │ RUN: Input ──► Transform ──► Aggregate ──► Output │ │ │
│ │ │ (Forwarders, Analyzers) → (Script) → (Events, Metrics) │ │ │
│ │ └───────────────────────────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────────────────────────┘
modules/
├── apps/ # Pre-built applications (flat topology)
│ ├── compiler/ # Symbol compilation app
│ ├── dev/ # Local preview CLI
│ ├── mcp/ # stdin/stdout scaffold used by the MCP server
│ ├── reporter/ # DaemonSet pre-SIEM cost insight
│ ├── receiver/ # Sidecar: Filter + Compact modes
│ └── retriever/ # S3 data lake indexing + on-demand query
│
├── pipelines/ # Core pipeline definitions
│ ├── compile/ # Symbol compilation pipeline
│ │ └── modules/ # pull/, scanner/, link/
│ ├── run/ # Event processing pipeline
│ │ └── modules/ # input/, transform/, aggregate/, output/
│ ├── doc/ # Documentation generation pipeline
│ └── shared/ # Shared utilities
│
└── lib/ # Shared libraries
└── script/ # JavaScript libraries
Deployed alongside log forwarders (Fluentd, Fluent Bit, Filebeat, Logstash).
| App | Purpose | Documentation | Run Guide |
|---|---|---|---|
| Reporter | Cost attribution metrics (DaemonSet, pre-SIEM, not in log path) | Overview | Run |
| Receiver | Two modes: Filter (lossy, budget sampling, mute files) and Compact (lossless volume reduction via SIEM-side expand plugin, where the destination supports it) | Overview | Run |
┌─────────────────────────────────────────────────────────────────┐
│ In-Cluster Deployment │
│ │
│ Reporter (DaemonSet) ─── tails pre-SIEM ────► Metrics │
│ │
│ Application ──► Log Forwarder ──► Receiver ──► Analyzer │
│ (logs) (Fluentd, (sidecar, (Splunk, │
│ Filebeat) Filter or Elastic) │
│ Compact) │
│ │
│ Metrics: ───────────────────────► Prometheus ────► Grafana │
└─────────────────────────────────────────────────────────────────┘
| App | Purpose | Documentation | Run Guide |
|---|---|---|---|
| Retriever | S3 data lake indexing & queries | Overview | Run |
| MCP | Agent control plane, reads Reporter metrics, commands Receiver/Retriever via GitOps | Overview | Run |
For agentless SIEM-side cost analysis, use the log10x-mcp server's log10x_poc_from_siem_submit tool.
| App | Purpose | Documentation | Run Guide |
|---|---|---|---|
| Dev | Local CLI to preview savings on your logs | Overview | Run |
| Compile | Generate symbol libraries from source code (optional, default library covers 150+ frameworks) | Overview | Test / Deploy |
Scans source code and generates symbol files for log template extraction.
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ PULL │───►│ SCAN │───►│ LINK │───►│ PUSH │
│ │ │ │ │ │ │ │
│ GitHub │ │ javaParser│ │ Cross-ref│ │ S3 │
│ Docker │ │ pythonAST │ │ analysis │ │ Local │
│ Helm │ │ ANTLR │ │ │ │ │
│ Local │ │ bytecode │ │ │ │ │
└──────────┘ └──────────┘ └──────────┘ └──────────┘
Documentation: Compile Pipeline
| Module Type | Examples | Docs |
|---|---|---|
| Pull | github, docker, helm, local | Pull Modules |
| Scanner | javaParser, pythonAST, scalameta, antlr, bytecode | Scanner Modules |
Processes log events in real-time for optimization, filtering, and reporting.
┌──────────┐ ┌───────────┐ ┌───────────┐ ┌──────────┐
│ INPUT │───►│ TRANSFORM │───►│ AGGREGATE │───►│ OUTPUT │
│ │ │ │ │ │ │ │
│ Forwarder│ │ Script │ │ Group by │ │ Events │
│ Analyzer │ │ Fields │ │ template │ │ Metrics │
│ S3 │ │ Timestamp │ │ app │ │ Stream │
│ Stdin │ │ Structure │ │ level │ │ │
└──────────┘ └───────────┘ └───────────┘ └──────────┘
Documentation: Run Pipeline
| Module Type | Examples | Docs |
|---|---|---|
| Input/Forwarder | fluentd, fluentbit, filebeat, logstash, otel | Forwarders |
| Input/Analyzer | splunk, elastic, datadog, cloudwatchLogs | Analyzers |
| Transform | script, fields, timestamp, structure | Transform |
| Output/Event | stdout, file, http, unix | Event Output |
| Output/Metric | prometheus, datadog, cloudwatch | Metric Output |
All configuration formats and specifications are documented at doc.log10x.com/config.
| Topic | Description | Documentation |
|---|---|---|
| Module Configuration | module.yaml structure and settings | Module Config |
| App Configuration | app.yaml structure and configFolders | App Config |
| YAML Format | YAML syntax and +include directives | YAML Reference |
| JavaScript | Scripting API and functions | JavaScript |
| Symbol Files | Symbol file format and usage | Symbols |
| Folder Structure | Configuration folder conventions | Folders |
| Matching Rules | Pattern matching syntax | Match |
| CLI Options | Command-line interface reference | CLI |
| Topic | Description | Documentation |
|---|---|---|
| Input API | TenXObject structure and input handling | Input API |
| Output API | TenXSummary structure and output handling | Output API |
| JavaScript API | Scripting functions and objects | JavaScript API |
| Compile API | Compiler programmatic interface | Compile API |
For detailed architecture documentation:
| Topic | Documentation |
|---|---|
| Engine Architecture | Architecture Overview |
| Pipeline Architecture | Pipeline Design |
| Module System | Module Architecture |
| Launcher Options | Launcher (sidecar, job, function) |
| GitOps Integration | GitOps Workflows |
| Category | Count |
|---|---|
| Total Directories | 180+ |
| YAML Configuration Files | 275+ |
| Applications | 10 |
| Pipeline Types | 4 |
| Input Modules | 15+ |
| Output Modules | 20+ |
| Scanner Modules | 12+ |
| Supported Forwarders | 5 |
| Supported Analyzers | 6 |
| Resource | URL |
|---|---|
| Documentation Hub | doc.log10x.com |
| Getting Started | doc.log10x.com/home |
| Installation | doc.log10x.com/install |
| Applications | doc.log10x.com/apps |
| Configuration | doc.log10x.com/config |
| API Reference | doc.log10x.com/api |
This repository is licensed under the MIT License.
This repository contains module definitions for the Log10x engine. While these definitions are open source, running Log10x requires a commercial license.
| What's Open Source | What Requires License |
|---|---|
| Module YAML definitions | Log10x engine/runtime |
| Documentation files | Log10x apps (Reporter, Receiver, Retriever) |
| Schema definitions | Executing pipelines |
Get a Log10x License: