Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -36,5 +36,15 @@ SMTP_PASSWORD=your-smtp-password/api key value
# No credentials needed in env — syslog is unauthenticated (UDP/TCP).
# Configure host, port, protocol, and facility entirely in watchdog-config.yaml.

# ── Docker socket access (non-root hardening) ────────────────────────────────
#GID = Group ID — a number Linux uses to identify a user group (the group equivalent of a user's UID).
# GID of the group that owns /var/run/docker.sock on this host. The container
# runs as a non-root user and joins this group (via group_add in
# docker-compose.yaml) to read the socket. install.sh auto-detects this; for
# manual setups find it with: stat -c '%g' /var/run/docker.sock
# Replace the placeholder below with that number — docker compose refuses to
# start the container while DOCKER_GID is unset or left as this placeholder.
DOCKER_GID=REPLACE_WITH_DOCKER_SOCKET_GID

# ── Tuning ────────────────────────────────────────────────────────────────────
LOG_LEVEL=INFO
10 changes: 9 additions & 1 deletion DEPLOYMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,8 @@ This guide covers initial deployment, alert channel configuration, and ongoing o
| Git | Required to clone the repository |
| Host permissions | Root, or membership in the `docker` group (to read `/var/run/docker.sock`) |

> The watchdog container itself runs as a non-root user and joins the host's `docker` group at runtime (via the `DOCKER_GID` value in `.env`, auto-detected by `install.sh`) to read the socket — see [2.2 Configure Environment Variables (.env)](#22-configure-environment-variables-env).

```bash
docker compose version
```
Expand Down Expand Up @@ -115,6 +117,12 @@ SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T.../B.../...
SMTP_USERNAME=your-smtp-username/login email
SMTP_PASSWORD=your-smtp-password/api key value

# Docker socket access (non-root container) — GID of the group that owns
# /var/run/docker.sock on this host. Find it with: stat -c '%g' /var/run/docker.sock
# Replace the placeholder below with that number — docker compose refuses to
# start the container while DOCKER_GID is unset or left as this placeholder.
DOCKER_GID=REPLACE_WITH_DOCKER_SOCKET_GID

# Tuning (optional — defaults shown)
LOG_LEVEL=INFO
```
Expand Down Expand Up @@ -723,7 +731,7 @@ docker compose logs docker-container-watchdog
```

Common causes:
- `/var/run/docker.sock` is not accessible — ensure the host socket exists and the container has read access
- `/var/run/docker.sock` is not accessible — ensure the host socket exists and the container has read access. The container runs as a non-root user and needs `DOCKER_GID` in `.env` to match the socket's actual group (`stat -c '%g' /var/run/docker.sock`) — see [2.2 Configure Environment Variables (.env)](#22-configure-environment-variables-env)
- Missing `.env` file — run `cp .env.example .env` and fill in values

### Alert Notifications Not Received
Expand Down
6 changes: 6 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,10 @@ RUN pip install --no-cache-dir \
# Copy agent
COPY watchdog.py .

# Non-root — the docker.sock group membership needed to read the socket is
# granted at runtime via `group_add` in docker-compose.yaml (GID varies per host).
RUN useradd --uid 1000 --create-home --shell /usr/sbin/nologin watchdog \
&& chown -R watchdog:watchdog /app
USER watchdog

CMD ["python3", "-u", "watchdog.py"]
9 changes: 7 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,10 +90,14 @@ Two additional deduplication rules suppress redundant alerts at the event level:
| Item | Size |
|------|------|
| Docker image (`watchdog:latest`) | ~180 MB (python:3.11-slim base + dependencies) |
| Running container (memory) | ~50–80 MB |
| Running container (memory) | ~50–80 MB baseline; capped at `mem_limit: 256m` in `docker-compose.yaml` |
| `watchdog.tar` export | ~170 MB |

> **Note:** The values above are approximate and may vary depending on the host operating system, Docker version, and installed dependencies.
> **Note:** The values above are approximate and may vary depending on the host operating system, Docker version, and installed dependencies. If usage approaches the 256 MB limit (check with `docker stats docker-container-watchdog`), raise `mem_limit`/`mem_reservation` in `docker-compose.yaml` rather than removing the limit — and raise `memswap_limit` to at least the new `mem_limit` in **both** `docker-compose.yaml` and `docker-compose.build.yaml`, since Docker rejects a config where `memswap_limit` is lower than `mem_limit`.

### Container Hardening

The container runs as a non-root user (UID 1000) with a read-only root filesystem, no extra Linux capabilities (`cap_drop: ALL`), and `no-new-privileges` set. These controls, together with the CPU/process caps (`cpus: "0.50"`, `pids_limit: 200`), contain a leak or runaway condition in the watchdog process to this container instead of the host — but they do **not** sandbox the Docker API. Access to `/var/run/docker.sock` is granted by adding the container's user to the host's `docker` group via `group_add` (GID auto-detected by `install.sh`, stored as `DOCKER_GID` in `.env` — see [DEPLOYMENT.md](DEPLOYMENT.md#22-configure-environment-variables-env)), and that socket is effectively host-root-equivalent: anything with access to it can create privileged containers or bind-mount the host filesystem. A compromise of the watchdog process itself is therefore **not** contained by `cap_drop`, `no-new-privileges`, or the read-only filesystem — treat `docker.sock` access as the primary trust boundary when deciding who/what can reach this host.


### Python Dependencies
Expand Down Expand Up @@ -556,6 +560,7 @@ Probe selection is automatic: containers with a Docker `HEALTHCHECK` are monitor

| Version | Date | Author | Changes |
|---------|------------|--------|---------|
| 1.5.3 | 2026-09-16 | Rahul Kumar | Resource Limits Enforced |
| 1.5.2 | 2026-08-31 | Rahul Kumar | Updated error message"Suppressing expected Cyber Controller SQL dump syntax-check container termination (exit 137) alert" |
| 1.5.1 | 2026-08-31 | Rahul Kumar | fixed ignore Dynamic container crash alert |
| 1.5.0 | 2026-08-27 | Rahul Kumar | Added ignore Dynamic container crash alert |
Expand Down
21 changes: 21 additions & 0 deletions docker-compose.build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,27 @@ services:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./watchdog-config.yaml:/etc/watchdog/watchdog-config.yaml:ro
- ./watchdog:/var/log/watchdog
# ── Hardening ────────────────────────────────────────────────────────────
user: "1000:1000" # non-root
group_add:
# host docker.sock group GID — auto-detected into .env by install.sh.
# Required (no fallback): a wrong/guessed GID silently breaks socket access.
- "${DOCKER_GID:?Set DOCKER_GID in .env to the docker.sock GID - run install.sh or see DEPLOYMENT.md}"
read_only: true # immutable root filesystem
tmpfs:
- /tmp
security_opt:
- no-new-privileges:true
cap_drop:
- ALL
# ── Resource limits — contain a leak/runaway to this container, not the host ─
mem_limit: 256m
mem_reservation: 128m
memswap_limit: 256m # no swap beyond mem_limit — hits the limit and restarts instead of degrading the host
# must stay >= mem_limit if you raise it, or Docker rejects this config
mem_swappiness: 0 # avoid swapping this container's pages — needed when host kernel lacks swap accounting (memswap_limit is then unenforceable)
cpus: "0.50"
pids_limit: 200
healthcheck:
test: ["CMD", "python3", "-c", "import docker; docker.from_env().ping()"]
interval: 30s
Expand Down
21 changes: 21 additions & 0 deletions docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,27 @@ services:
- ./watchdog.py:/app/watchdog.py:ro
# Persistent log storage — written directly to host folder
- ./watchdog:/var/log/watchdog
# ── Hardening ────────────────────────────────────────────────────────────
user: "1000:1000" # non-root
group_add:
# host docker.sock group GID — auto-detected into .env by install.sh.
# Required (no fallback): a wrong/guessed GID silently breaks socket access.
- "${DOCKER_GID:?Set DOCKER_GID in .env to the docker.sock GID - run install.sh or see DEPLOYMENT.md}"
read_only: true # immutable root filesystem
tmpfs:
- /tmp
security_opt:
- no-new-privileges:true
cap_drop:
- ALL
# ── Resource limits — contain a leak/runaway to this container, not the host ─
mem_limit: 256m
mem_reservation: 128m
memswap_limit: 256m # no swap beyond mem_limit — hits the limit and restarts instead of degrading the host
# must stay >= mem_limit if you raise it, or Docker rejects this config
mem_swappiness: 0 # avoid swapping this container's pages — needed when host kernel lacks swap accounting (memswap_limit is then unenforceable)
cpus: "0.50"
pids_limit: 200
healthcheck:
test: ["CMD", "python3", "-c", "import docker; docker.from_env().ping()"]
interval: 30s
Expand Down
50 changes: 50 additions & 0 deletions install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -118,9 +118,44 @@ setup_log_directory() {
else
print_info "Log directory already exists: ${LOG_DIR}"
fi
# Container runs as non-root (UID 1000:GID 1000) — must own the dir and any
# pre-existing files (e.g. log/rotated files left by an older root-run
# version) directly; group-writable permissions don't help unless the
# container's GID is actually a member of that group.
if chown -R 1000:1000 "$LOG_DIR" 2>/dev/null; then
chmod 750 "$LOG_DIR"
else
print_error "Could not chown ${LOG_DIR} to UID 1000:GID 1000 (not running as root?)"
echo ""
echo " Re-run this script with sudo, or fix ownership manually:"
echo " sudo chown -R 1000:1000 ${LOG_DIR}"
exit 1
fi
print_info "Logs will be written to: ${LOG_DIR}/watchdog.log"
}

################################################################################
# Docker Socket Group Detection
################################################################################

detect_docker_gid() {
# GID that owns /var/run/docker.sock — the non-root container joins this
# group (via group_add in docker-compose.yaml) to read the socket.
# No silent fallback: a wrong guessed GID breaks socket access at container
# start with a confusing error, so treat detection failure as fatal here.
if [ ! -S /var/run/docker.sock ]; then
print_error "/var/run/docker.sock not found — cannot determine its group GID"
return 1
fi
local gid
gid=$(stat -c '%g' /var/run/docker.sock 2>/dev/null || stat -f '%g' /var/run/docker.sock 2>/dev/null)
if [ -z "$gid" ]; then
print_error "Could not determine the GID that owns /var/run/docker.sock"
return 1
fi
echo "$gid"
}

################################################################################
# Docker Image
################################################################################
Expand Down Expand Up @@ -220,6 +255,15 @@ configure_all() {
IFS= read -r RECONFIG
if [[ ! "$RECONFIG" =~ ^[Yy]$ ]]; then
print_info "Keeping existing configuration"
# Upgrade path: older installs may predate DOCKER_GID — add it now
# rather than silently starting with the unset/wrong-GID fallback.
if [ -f "$ENV_FILE" ] && ! grep -q '^DOCKER_GID=' "$ENV_FILE"; then
print_warning "Existing .env is missing DOCKER_GID — detecting and adding it"
local MIGRATED_GID
MIGRATED_GID=$(detect_docker_gid) || exit 1
printf "\n# Docker socket access (non-root container)\nDOCKER_GID=%s\n" "$MIGRATED_GID" >> "$ENV_FILE"
print_success "Added DOCKER_GID=${MIGRATED_GID} to ${ENV_FILE}"
fi
return 0
fi
echo ""
Expand Down Expand Up @@ -334,6 +378,11 @@ configure_all() {
WATCHDOG_HOST=$(_prompt "Hostname to display in alerts" "$HOST_DEFAULT")
echo ""

# ── Docker socket GID (container runs non-root; needs group access to the socket) ──
local DOCKER_GID
DOCKER_GID=$(detect_docker_gid) || exit 1
print_info "Docker socket GID detected: ${DOCKER_GID}"

# ── Write .env ────────────────────────────────────────────────────────────
{
printf "# .env — generated by install.sh on %s\n" "$(date '+%Y-%m-%d %H:%M:%S')"
Expand All @@ -349,6 +398,7 @@ configure_all() {
"$SNMP_V3_AUTH_KEY" "$SNMP_V3_PRIV_KEY"
fi
printf "# Alert identity\nWATCHDOG_HOST=%s\n\n" "$WATCHDOG_HOST"
printf "# Docker socket access (non-root container)\nDOCKER_GID=%s\n\n" "$DOCKER_GID"
printf "# Tuning\nLOG_LEVEL=INFO\n"
} > "$ENV_FILE"
chmod 600 "$ENV_FILE"
Expand Down