[noels]

GIT-18. environments/default/addons/longhorn/README.md: operations docs for the Longhorn add-on, written from the procedure actually run on ivylen on 2026-09-25. A 50 MiB random blob was backed up to S3, restored into a new volume and the sha256 matched; all test artefacts were removed, including from S3.

Contents:

• Safety: CRDs hold every volume (preserveResourcesOnDeletion), never helm uninstall (delete release Secrets), the PreDelete uninstall hook versus deletingConfirmationFlag, replica count ≤ nodes (plus how to lower it on existing volumes), 1.12.1 NetworkPolicies.
• Adopting a hand-installed release (render at the live version, server-side diff, pin, enable, drop Helm records).
• S3 backups: scoped IAM, credentials Secret keys, SOPS / make private-secrets, target values, RecurringJobs/groups.
• Manual backup via Snapshot + Backup CRs. Restore via a Volume CR with fromBackup, plus a static PV/PVC. Two traps hit in practice: the webhook rejects a restore until the BackupVolume has synced (poll interval), and dataLocality must be explicit.
• DR drill, step by step, including cleanup.
• Troubleshooting:
  • tls: handshake failure although the target is available: the upload runs from a replica's node, and a stale DHCP search domain with a DNS wildcard hijacks the S3 name;
  • how to restart an instance-manager without crashing volumes, including that Longhorn won't recreate it on a cordoned node, so uncordon first.

The appset header links to it.

🤖 Generated with Claude Code