FIREFLY-2077: Refactor cleanup.sh to improve configuration and maintainability

[loitly]

Ticket: https://jira.ipac.caltech.edu/browse/FIREFLY-2077
Additional changes here: https://github.com/IPAC-SW/irsa-ife/pull/478

• Fix entrypoint.py to prevent duplicate props added to the command line

Many changes to the cleanup process. Best way to understand it is to read the script's docs.

Tests:
The best way to test this is to remote shell into the pod and observe how cleanup.sh behaves.

kubectl -n firefly-dev exec -ti firefly-2077-cleanup-script-6474fc5d5c-wvm7b -- bash
kubectl -n firefly-dev exec -ti firefly-2077-cleanup-script-pct-full-64bb5c94bf-r6lvx -- bash

Cleanup logs are under /firefly/logs/cleanup/firefly/

AGE mode:
https://firefly-2077-cleanup-script.irsakubedev.ipac.caltech.edu/firefly/

Built with envVars:

CLEANUP_INTERVAL=1m
CLEANUP_AGE_HIPS=5m
CLEANUP_AGE_STAGE=5m

Check every minute:

- Removes temp files older than 1 minute.
- Removes HiPS and stage files older than 5 minutes.
- Leaves upload files at the default retention period of 7 days.

PCT-FULL: mode:
https://firefly-2077-cleanup-script-pct-full.irsakubedev.ipac.caltech.edu/firefly/

Built with envVars:

CLEANUP_INTERVAL=1m
CLEANUP_PCT_FULL=30
CLEANUP_PCT_DELTA=15

The script also checks every minute. It evaluates /firefly/workarea and /firefly/shared-workarea separately.

Because /firefly/workarea/firefly is an emptyDir backed by the pod's local node filesystem, df reports usage for the underlying host filesystem. For example:

----- workarea: /firefly/workarea/firefly
Pct-full mode: 68% used, cleans above 30%, target 15%
Need to free 103071075 KB; removing oldest files first

Removed 0 files, freed 0 KB of the 103071075 KB needed
Cleanup: WARNING: /firefly/workarea is still above 15%: nothing left to remove

Used is greater than 30%, so it will attempt to clean all files until it's under 15%.
The results is it deleted all of the files with nothing left to remove.

/firefly/shared-workarea is NFS-mounted filesystem. It's below the configured watermark. Nothing to clean.

----- shared-workarea: /firefly/shared-workarea/firefly
Pct-full mode: 20% used, cleans above 30%, target 15%
Nothing to do -- below 30%

[robyww]

I will look at this PR on Monday. However, a quick comment. Why not use both age mode and percent full mode together? I think that is what Frossie wanted.

The age mode runs and the percent full is a fallback if the disk is still getting too full after the age mode run.

[loitly]

The age mode runs and the percent full is a fallback if the disk is still getting too full after the age mode run.

Ah, I misunderstood. Let me make the change, and I’ll let you know when to review it.