Restore before nginx starts; keep rsync partials out of the cache tree - #7
Merged
Merged
Conversation
… batches nginx's cache loader walks the cache directory once at startup. With the restore running concurrently it found rsync's in-progress partial files (.<hash>.XXXXXX), judged them too small to be cache entries and deleted them, so rsync's rename failed and those entries never landed (seen in 2.0.0 as '[crit] cache file ... is too small' followed by rsync stat and rename errors). rsync now writes partials to <cache root>/.rsync-tmp, outside the loader's path but on the same filesystem, and the backup does the same on the archive. A restore batch that reports errors is retried once at the end and the outcome is reported in /status.
With a redundant instance covering, a fresh container should only take traffic once its cache is fully warm. CACHE_RESTORE_MODE=blocking (new default) runs cache-restore.sh to completion before exec nginx; port 80 stays closed meanwhile, so the orchestrator's health-check grace must cover the restore time. CACHE_RESTORE_MODE=background keeps the 2.0.0 behaviour for single-instance deployments.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
CACHE_RESTORE_MODE=blocking(new default): the archive is copied into the local cache before nginx starts, so a fresh instance only takes traffic once it is fully warm. Port 80 stays closed meanwhile, so the orchestrator's health check must allow for the restore time (Rancher: raise the service's initializing timeout, or rely on the load balancer's check).CACHE_RESTORE_MODE=backgroundkeeps the 2.0.0 behaviour for single-instance deployments.rsync now writes its partial files to
<cache root>/.rsync-tmprather than beside the target, and the backup does the same on the archive. A restore batch that reports errors is retried once at the end, and the result is reported in/status.Why
In 2.0.0 nginx's cache loader, which walks the cache tree once at startup, found rsync's in-progress
.<hash>.XXXXXXpartials during a concurrent restore, judged them "too small" and deleted them from under rsync ([crit] cache file "..." is too smallfollowed by rsync stat/rename errors). Those entries then never landed and were not retried. Blocking mode removes the race by construction; the temp-dir change fixes it for background mode and for anycache-restore.sh --forcerun against a live nginx.How to test
sh test/cache-sync-test.sh(49 checks, adds partial-location assertions); CI startup test now exercises the blocking path end to end. On a deployed 2.0.0 the symptom is the[crit] … is too smallline in the container log during restore.Follow-ups
The Rancher service health check's initializing timeout needs raising before the archive is large.