ElasticsFake

Lightweight DIY log collector that emulates Elasticsearch for Beats agents

Perfect for home labs, IoT setups, or personal experiments — collect logs reliably without an always-on Elasticsearch cluster.

• Accept logs from Beats agents (Elastic Agent, Filebeat, etc.).
• Save logs locally or forward them to your preferred log storage solution.
• Acts as a shim for Elasticsearch: agents think they are talking to a real ES cluster.
• Lightweight and easy to deploy on Raspberry Pi 3/4 or any mini-PC.

Key points / highlights

• ✅ Elasticsearch-compatible shim for _bulk requests.
• ✅ Local storage – no backend required.
• ✅ Runs on minimal hardware – Raspberry Pi or mini-PC.
• ✅ HTTP Basic Auth – optional, toggle via config.
• ✅ Gzip decompression – enabled by default.
• ✅ Native TLS – optional, toggle via config, no reverse proxy required.
• ✅ Prometheus metrics – exposed at /metrics.
• ✅ DIY / open-source – you can tweak it to your needs.

Usage/Installation

Local app run

cd elasticsfake
pip install -r requirements.txt
CONFIG_FILE_PATH=config.yaml python main.py

# Example output:
# INFO:     Started server process [32111]
# INFO:     Waiting for application startup.
# INFO:     Application startup complete.
# INFO:     Uvicorn running on http://0.0.0.0:9200 (Press CTRL+C to quit)

Ansible Installation

To install in a raspberry-pi or other Debian based compatible distribution you can run the ansible playbook located in this repository.

Create an inventory file under ansible/ folder. Example of mine:

[logs_box]
192.168.1.9 ansible_user=snakecase

Run the ansible playbook.

pip install ansible
ansible-playbook -i inventory playbook.yml

Service configuration examples

Authentication, gzip compression and TLS are all supported natively — no nginx required.

Elasticsfake config (config.yaml)

app:
  auth:
    enabled: true        # set false to disable
    username: elastic
    password: changeme

  gzip:
    enabled: true        # enabled by default

  tls:
    enabled: true        # set false to disable, or front it with your own reverse proxy
    certfile: /path/to/cert.pem
    keyfile: /path/to/key.pem

Elastic Agent configuration

outputs:
  default:
    type: elasticsearch
    hosts: ["http://192.168.1.9:9200"]
    username: elastic
    password: changeme
    compression_level: 1

Logstash

output {
  elasticsearch {
    hosts => ["http://192.168.1.9:9200"]
    user => "elastic"
    password => "changeme"
    manage_template => false
    http_compression => true
  }
}

What Happens After Data Collection

Data handling after collection is beyond the scope of this project.

In my current setup, logs are collected by Elasticsfake on a Raspberry Pi®, and when I start my laptop, Logstash retrieves them from MQTT and injects them into Elastics. This avoids running Elastics continuously while still allowing analysis of activity from all my devices.

Why did I developed this?

• I want to have my own family&friends SIEM but I don't want to have a big resource hungry machine constantly connected.
• I considered using Logstash with an Elasticsearch input but also consumes significant resources.
• I know it might feel like reinventing the wheel, but I couldn’t find a lightweight, extendable Elasticsearch shim for Beats agents that fit my needs.

Why you may want to use this project?

• Easy to tweak and extend — make it fit your own setup.
• Maybe my use case clicks with yours, and you also want a lightweight Elasticsearch buffer for your logs.
• Maybe you don’t run Elasticsearch, but you’ve got an agent that speaks Elasticsearch and has the exact data you need — this project can act as a tiny adapter for your own datastore.
• Or maybe… you just don’t need it. I’m not selling it to anyone, so no hard pitch here. Use it if it fits your project or curiosity. Skip it if not. 😂

About the Author

Hey! I’m Samuel — cybersecurity enthusiast, developer, magician and ham radio hobbyist.

If you want to connect or see what I’m up to, check out my LinkedIn