Beta

.env.example

@@ -36,5 +36,20 @@ SMTP_PASSWORD=your-smtp-password/api key value
 # No credentials needed in env — syslog is unauthenticated (UDP/TCP).
 # Configure host, port, protocol, and facility entirely in watchdog-config.yaml.
 
+# ── Non-root hardening (optional, advanced) ──────────────────────────────────
+# By default the container runs as root so `docker compose up` works with zero
+# configuration — install.sh sets these three automatically instead, running
+# the container as UID/GID 1000 joined to the docker.sock group. To opt into
+# the same hardening for a manual install, uncomment and fill in DOCKER_GID
+# (find it with: stat -c '%g' /var/run/docker.sock):
+# WATCHDOG_UID=1000
+# WATCHDOG_GID=1000
+# DOCKER_GID=
+
+# ── Image version ─────────────────────────────────────────────────────────────
+# Release tag the container runs (see the repository VERSION file). install.sh
+# sets this to the version it loaded from watchdog.tar; unset means watchdog:latest.
+# WATCHDOG_VERSION=
+
 # ── Tuning ────────────────────────────────────────────────────────────────────
 LOG_LEVEL=INFO

.github/workflows/ci.yml

@@ -0,0 +1,53 @@
+name: CI
+
+on:
+  pull_request:
+    branches:
+      - main
+
+jobs:
+  validate:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Validate shell scripts
+        run: |
+          bash -n install.sh
+          bash -n uninstall.sh
+
+      - name: Validate Python syntax
+        run: |
+          python3 -m py_compile watchdog.py
+
+      - name: Install test dependencies
+        run: |
+          python3 -m pip install docker==7.1.0 requests==2.32.3 PyYAML==6.0.2
+
+      - name: Run regression tests
+        run: |
+          python3 -m unittest discover -s tests -v
+
+      - name: Validate VERSION file
+        run: |
+          test -f VERSION
+          VERSION=$(tr -d '[:space:]' < VERSION)
+          echo "Repository version: ${VERSION}"
+          echo "${VERSION}" | grep -Eq '^[0-9]+\.[0-9]+\.[0-9]+$'
+          grep -q "| ${VERSION} |" README.md
+
+      - name: Prepare CI environment
+        run: |
+          cp .env.example .env
+          echo "WATCHDOG_VERSION=$(cat VERSION)" >> .env
+
+      - name: Validate Docker Compose
+        run: |
+          docker compose -f docker-compose.yaml config -q
+          WATCHDOG_VERSION="$(cat VERSION)" docker compose -f docker-compose.build.yaml config -q
+
+      - name: Build watchdog image
+        run: |
+          WATCHDOG_VERSION="$(cat VERSION)" docker compose -f docker-compose.build.yaml build

DEPLOYMENT.md

@@ -49,6 +49,8 @@ This guide covers initial deployment, alert channel configuration, and ongoing o
 | Git | Required to clone the repository |
 | Host permissions | Root, or membership in the `docker` group (to read `/var/run/docker.sock`) |
 
+> By default the container runs as root so a manual install needs no host-specific group setup. The automated installer ([3.1 Option A](#31-option-a-automated-installation-installsh-recommended)) instead hardens it to run non-root (UID/GID 1000, joined to the docker.sock group) automatically, with no input required. Either way it's still contained by a read-only root filesystem, dropped Linux capabilities (`cap_drop: ALL`), `no-new-privileges`, and CPU/memory/PID limits (see [Container Hardening in the README](README.md#container-hardening)).
+
 ```bash
 docker compose version
 ```
@@ -121,6 +123,8 @@ LOG_LEVEL=INFO
 
 Only credentials and secrets belong in `.env`. Hosts, ports, recipients, and thresholds are configured in `watchdog-config.yaml`, never hardcoded in this guide or in scripts.
 
+The container runs as root by default for a zero-configuration manual install. If you want the non-root hardening that [Option A](#31-option-a-automated-installation-installsh-recommended) applies automatically, uncomment the `WATCHDOG_UID`/`WATCHDOG_GID`/`DOCKER_GID` lines in `.env.example` and fill in `DOCKER_GID` (`stat -c '%g' /var/run/docker.sock`).
+
 ---
 
 ### 2.3 Configure Watchdog Global Settings
@@ -320,14 +324,14 @@ The script walks through the following stages in order:
 |---|---|
 | **Prerequisites** | Verifies Docker and Docker Compose are installed and running |
 | **Log directory** | Creates `./watchdog/` for persistent log storage |
-| **Load image** | Loads `watchdog.tar` into Docker (`watchdog:latest`); builds from source if archive is absent |
-| **Host identification** | `WATCHDOG_HOST` is hardcoded to `CyberController-Server` in `docker-compose.yaml` |
+| **Load image** | Loads `watchdog.tar` and requires it to contain `watchdog:<VERSION>` (from the root `VERSION` file); aborts on a version mismatch, and builds from source only when the archive is absent |
+| **Host identification** | `WATCHDOG_HOST` is taken from `.env` (the installer prompts for it), defaulting to `CyberController-Server` |
 | **Configuration wizard** | Selects channels and prompts for Slack/SMTP/SNMP/Syslog values interactively |
-| **Credentials + config write** | Generates/updates `.env` and `watchdog-config.yaml` from wizard answers |
-| **Start** | Runs `docker compose up -d` in the background |
-| **Verify** | Checks the container is running and prints a summary with common commands |
+| **Credentials + config write** | Generates/updates `.env` (including `WATCHDOG_VERSION`) and `watchdog-config.yaml` from wizard answers |
+| **Start** | Runs `docker compose up -d` in the background, aborting if Compose reports a failure |
+| **Verify** | Fails the installation unless the container is running **and** uses `watchdog:<VERSION>` |
 
-> **Re-running `install.sh` on an existing installation is safe.** If configuration files already exist, the installer asks whether to reconfigure from scratch. Choose `N` to keep existing files unchanged.
+> **Re-running `install.sh` on an existing installation is safe.** If both configuration files exist, the installer asks whether to reconfigure from scratch. Choose `N` to keep them unchanged — the packaged image is still loaded and the deployment is still moved to the new version. If only one of the two files is present, the wizard regenerates both, since a half-present configuration cannot start.
 
 Continue to [4. Verification](#4-verification).
 
@@ -342,7 +346,7 @@ Use this path when you need full control over configuration files before startin
 *Note* - If internet access is available, build the image and skip to [Start the container](#start-the-container) below.
 
 ```bash
-docker compose -f docker-compose.build.yaml build
+WATCHDOG_VERSION="$(cat VERSION)" docker compose -f docker-compose.build.yaml build
 ```
 
 #### Offline Installation
@@ -426,17 +430,18 @@ Re-run [4.1 Confirm Container Status and Health](#41-confirm-container-status-an
 
 ### 5.2 Roll Back an Image Upgrade
 
-The runtime image is always tagged `watchdog:latest`, so loading or pulling a new image overwrites the previous one. Before upgrading the image (see [6.3 Upgrade Guide](#63-upgrade-guide)), tag the current working image so it can be restored:
+Each release is tagged `watchdog:<VERSION>` and the deployed tag is pinned by `WATCHDOG_VERSION` in `.env`. As long as the previous image is still on the host, rolling back is a version change:
 
 ```bash
-docker tag watchdog:latest watchdog:rollback
+docker images watchdog                       # list the versions available locally
+sed -i 's/^WATCHDOG_VERSION=.*/WATCHDOG_VERSION=1.5.3/' .env
+docker compose up -d
 ```
 
-If the new image misbehaves after `docker load -i watchdog.tar` or `docker compose pull`, restore the previous image and redeploy:
+If the previous image was only ever tagged `watchdog:latest`, preserve it under a rollback tag before loading a new archive:
 
 ```bash
-docker tag watchdog:rollback watchdog:latest
-docker compose up -d
+docker tag watchdog:latest watchdog:rollback
 ```
 
 ### 5.3 Roll Back an Application Upgrade (Git)
@@ -486,7 +491,7 @@ docker compose up -d
 
 ```bash
 # Rebuild the image (requires internet)
-docker compose -f docker-compose.build.yaml build
+WATCHDOG_VERSION="$(cat VERSION)" docker compose -f docker-compose.build.yaml build
 
 # Redeploy using the production runtime file
 docker compose up -d
@@ -518,11 +523,10 @@ docker compose up -d
 
 #### Apply docker-compose.yaml changes (container settings)
 
-For example, edit the `WATCHDOG_HOST` value in `docker-compose.yaml`:
+For example, change the hostname shown in alerts by editing `WATCHDOG_HOST` in `.env`:
 
-```yaml
-environment:
-  WATCHDOG_HOST: my-new-server-name
+```bash
+WATCHDOG_HOST=my-new-server-name
 ```
 
 Then redeploy to take effect:
@@ -642,13 +646,16 @@ docker compose up -d
 ##### Option B: Offline Image Upgrade
 
 1. Request the Radware RE team to create an updated pre-built Docker image for offline installation. See [8.4 Support Contacts](#84-support-contacts).
-2. Load the image. Make sure the new image uses the same name, `watchdog:latest`.
+2. Load the image. The archive carries the release tag `watchdog:<VERSION>` matching the package's `VERSION` file.
 
    ```bash
    docker load -i watchdog.tar
+   sed -i "s/^WATCHDOG_VERSION=.*/WATCHDOG_VERSION=$(cat VERSION)/" .env
    docker compose up -d
    ```
 
+   Running `bash install.sh` performs all three steps (and verifies the result) automatically.
+
 ---
 
 #### Checking the Installed Version
@@ -692,7 +699,7 @@ Shows a removal plan (container name, image size, log directory size), asks for
 bash uninstall.sh --keep-logs
 ```
 
-Stops and removes the `watchdog` container and the `watchdog:latest` Docker image. The `./watchdog/` log directory is left intact so you can review historical logs later.
+Stops and removes the `watchdog` container and the `watchdog:<VERSION>` and `watchdog:latest` Docker image tags. The `./watchdog/` log directory is left intact so you can review historical logs later.
 
 #### Remove everything including logs
 
@@ -723,7 +730,7 @@ docker compose logs docker-container-watchdog
 ```
 
 Common causes:
-- `/var/run/docker.sock` is not accessible — ensure the host socket exists and the container has read access
+- `/var/run/docker.sock` is not accessible — ensure the host socket exists and Docker is running. Manual installs run the container as root, so no host-side group configuration is required; if you opted into non-root hardening (`WATCHDOG_UID`/`WATCHDOG_GID`/`DOCKER_GID` in `.env`), verify `DOCKER_GID` matches `stat -c '%g' /var/run/docker.sock`.
 - Missing `.env` file — run `cp .env.example .env` and fill in values
 
 ### Alert Notifications Not Received
@@ -797,13 +804,14 @@ docker compose restart docker-container-watchdog
 
 #### Host Identification in Alerts
 
-Set `WATCHDOG_HOST` in `docker-compose.yaml` under the watchdog service environment:
+Set `WATCHDOG_HOST` in `.env` (written by `install.sh` from the hostname prompt), then run `docker compose up -d`:
 
-```yaml
-environment:
-  WATCHDOG_HOST: my-server-name
+```bash
+WATCHDOG_HOST=my-server-name
 ```
 
+When it is unset, `docker-compose.yaml` falls back to `CyberController-Server`.
+
 ---
 
 ### 8.3 SNMP Trap Var-Binds

Dockerfile

@@ -13,4 +13,13 @@ RUN pip install --no-cache-dir \
 # Copy agent
 COPY watchdog.py .
 
+# UID 1000 account for install.sh's non-root hardening path (docker-compose.yaml
+# defaults to root; install.sh instead sets WATCHDOG_UID/GID to run as this user).
+RUN useradd --uid 1000 --create-home --shell /usr/sbin/nologin watchdog \
+    && chown -R watchdog:watchdog /app
+
+# Non-root by default so a bare `docker run` is restricted. Compose always sets
+# `user:` explicitly, so manual installs still get their root default.
+USER watchdog
+
 CMD ["python3", "-u", "watchdog.py"]

README.md

@@ -79,8 +79,9 @@ Two additional deduplication rules suppress redundant alerts at the event level:
 ├── requirements-watchdog.txt    # Python dependencies (reference; Dockerfile pip-installs inline)
 ├── install.sh                   # Offline install script
 ├── uninstall.sh                 # Stop + remove script
-├── watchdog.tar                 # Pre-built Docker image (provided, no internet needed)
-└── .env                         # Secrets — NOT committed to git
+├── VERSION                       # Release version — drives the image tag and installer
+├── watchdog.tar                  # Pre-built Docker image (provided, no internet needed)
+└── .env                          # Secrets — NOT committed to git
 ```
 
 ---
@@ -89,11 +90,15 @@ Two additional deduplication rules suppress redundant alerts at the event level:
 
 | Item | Size |
 |------|------|
-| Docker image (`watchdog:latest`) | ~180 MB (python:3.11-slim base + dependencies) |
-| Running container (memory) | ~50–80 MB |
+| Docker image (`watchdog:<VERSION>`) | ~180 MB (python:3.11-slim base + dependencies) |
+| Running container (memory) | ~50–80 MB baseline; capped at `mem_limit: 256m` in `docker-compose.yaml` |
 | `watchdog.tar` export | ~170 MB |
 
-> **Note:** The values above are approximate and may vary depending on the host operating system, Docker version, and installed dependencies.
+> **Note:** The values above are approximate and may vary depending on the host operating system, Docker version, and installed dependencies. If usage approaches the 256 MB limit (check with `docker stats docker-container-watchdog`), raise `mem_limit`/`mem_reservation` in `docker-compose.yaml` rather than removing the limit — and raise `memswap_limit` to at least the new `mem_limit` in **both** `docker-compose.yaml` and `docker-compose.build.yaml`, since Docker rejects a config where `memswap_limit` is lower than `mem_limit`.
+
+### Container Hardening
+
+By default the container runs as **root** so `docker compose up` works with zero configuration — no host-specific group setup needed to read `/var/run/docker.sock`. [install.sh](install.sh) instead auto-detects the socket's GID and opts into running the container as **non-root UID/GID 1000** (joined to that group via `group_add`), with no manual input required; manual installs can do the same by setting `WATCHDOG_UID`/`WATCHDOG_GID`/`DOCKER_GID` in `.env` (see [DEPLOYMENT.md](DEPLOYMENT.md#22-configure-environment-variables-env)). Either way, the container is constrained by a read-only root filesystem, no extra Linux capabilities (`cap_drop: ALL`), `no-new-privileges`, and CPU/memory/PID caps (`cpus: "0.50"`, `pids_limit: 200`, `mem_limit: 256m`), which contain a leak or runaway condition to this container instead of the host. Note that access to `/var/run/docker.sock` is effectively host-root-equivalent for **any** user (root or not): anything that can reach the socket can create privileged containers or bind-mount the host filesystem, so a compromise of the watchdog process is **not** contained by these controls — treat `docker.sock` access as the primary trust boundary when deciding who/what can reach this host.
 
 
 ### Python Dependencies
@@ -129,8 +134,12 @@ All non-secret settings live in `watchdog-config.yaml`. Secrets (webhook URLs, p
 | `alert_on_recovery` | `true` | Send an INFO "recovered" alert once a previously-alarmed container returns to normal |
 | `excluded_containers` | `[]` | Container names to never alert on |
 | `ignored_container_events` | MariaDB syntax-check label rule | Label-based event suppressions for intentional temporary containers |
+| `auto_health_check` | enabled | Probe containers reporting `health=none` by discovering an HTTP/TCP endpoint |
+| `container_health_checks` | `{}` | Per-container probe overrides (`type: http` or `type: exec`) for `health=none` containers; takes precedence over `auto_health_check` |
 | `log_level` | `INFO` | `DEBUG` / `INFO` / `WARNING` / `ERROR` |
 | `log_file` | `/var/log/watchdog/watchdog.log` | Bind-mounted to `./watchdog/watchdog.log` on host. Rotates at 10 MB, 5 backups. Set to `null` to disable |
+| `state_file` | `/var/log/watchdog/state.json` | Persists open alerts and the host boot time so recovery and reboot reporting survive a restart. Set to `""` to disable |
+| `boot_grace_seconds` | `180` | After a detected host reboot, hold alerts this long while services start, then send one `reboot-summary`. `0` alerts immediately and sends the summary on the first poll |
 | `runbook_base_url` | — | URL included in every alert |
 
 Preferred suppression: Temporary MariaDB HA syntax-check containers should be created with Docker label `com.radware.cybercontroller.role=mariadb-ha-syntax-check`. The watchdog ignores only configured events for containers carrying that explicit label; normal service containers still alert.
@@ -248,6 +257,7 @@ bash install.sh
 | `unhealthy` | HIGH | Health probe failing for N consecutive cycles |
 | `restart-loop` | HIGH | Container restarted ≥ threshold times within window |
 | `recovered` | INFO | Previously-alarmed container (`crashed`/`oom`/`unhealthy`/`restart-loop`) is healthy/running again. Fires once per incident; controlled by `alert_on_recovery` (default `true`) |
+| `reboot-summary` | INFO / HIGH | Sent once after a host reboot is detected, when the `boot_grace_seconds` window closes. Reports boot time, how many containers are running, which recovered, and which are still failing. Always sent after a reboot — an INFO summary confirms the node came back cleanly. HIGH if any container is still failing |
 
 OOM alerts include **memory stats** (usage / limit / peak) prepended to the log snippet.
 
@@ -428,7 +438,7 @@ docker compose logs docker-container-watchdog
 Common causes:
 - **Missing `.env` file** — run `cp .env.example .env` and fill in credentials
 - **Docker socket not accessible** — ensure `/var/run/docker.sock` exists and the container has read access
-- **Image not loaded** — run `docker images watchdog`; if empty, build with `docker compose -f docker-compose.build.yaml build` (or `docker build -t watchdog:latest .` directly — see [DEPLOYMENT.md](DEPLOYMENT.md) for details)
+- **Image not loaded** — run `docker images watchdog`; if empty, set `WATCHDOG_VERSION=$(cat VERSION)` and build with `WATCHDOG_VERSION="$WATCHDOG_VERSION" docker compose -f docker-compose.build.yaml build` (see [DEPLOYMENT.md](DEPLOYMENT.md) for details)
 
 ### Alert Notifications Not Received
 
@@ -556,6 +566,8 @@ Probe selection is automatic: containers with a Docker `HEALTHCHECK` are monitor
 
 | Version | Date | Author | Changes |
 |---------|------------|--------|---------|
+| 1.5.4 | 2026-09-23 | Rahul Kumar | Versioned Docker image (`watchdog:<VERSION>`) driven by the root `VERSION` file; installer loads the packaged `watchdog.tar`, rejects an archive that does not carry the release tag, aborts on Compose failures, and verifies the running image; restart-loop detection now uses Docker `RestartCount` instead of counting poll observations; `container_health_checks` overrides implemented; fixed `.env` migration on root-owned files; `watchdog-config.yaml` made readable by the non-root container user |
+| 1.5.3 | 2026-09-16 | Rahul Kumar | Resource Limits Enforced |
 | 1.5.2 | 2026-08-31 | Rahul Kumar | Updated error message"Suppressing expected Cyber Controller SQL dump syntax-check container termination (exit 137) alert"  | 
 | 1.5.1 | 2026-08-31 | Rahul Kumar | fixed ignore Dynamic container crash alert |
 | 1.5.0 | 2026-08-27 | Rahul Kumar | Added ignore Dynamic container crash alert |

VERSION

@@ -0,0 +1 @@
+1.5.4