Repository navigation
Conversation
watsonian
added this pull request to stack #566
October 13, 2026 15:34
The watch stream handles each event in its own goroutine, and the connection retry loop runs separately, but they shared lastUpdateEvent, lastSecretsFetch, watchedValuesMayBeStale and watchRetrySleep without synchronization. These are now guarded by a mutex. The process cleanup goroutine read `c`, `cleanupMount` and terminatedByWatch from the enclosing scope while a restart could reassign them. It now receives the process and cleanup func as arguments, and terminatedByWatch is atomic. The secrets mount's fifoCleanupStarted flag was written by the cleanup func and read by the pipe writer goroutine; it's now atomic.
The watch tests run a doppler binary built with the race detector through concurrent update events during a restart and through a dropped watch connection, and fail if it reports a data race. The binary is built on first use; DOPPLER_TEST_RACE_BINARY uses a prebuilt one. The mount test calls MountSecrets directly and orders the cleanup before the writer's next open, so the writer reads the cleanup flag after it's set. Since CI doesn't run tests with -race, a wrapper test runs it under the race detector.
watsonian
force-pushed
the
watsonian/fix-watch-data-races
branch
from
October 13, 2026 16:37
2d576a0 to
2125630
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Running
doppler run --watchwith a race-instrumented build reports data races. All of them exist onmaster, and none are in the new signal handling code. The watch stream handles each event in its own goroutine, and the connection retry loop runs alongside them. Four variables were shared between those goroutines without synchronization.lastUpdateEventwas written by each update event before it tookwatchMutex.watchedValuesMayBeStalewas set by the retry loop and read by theconnectedevent outside any lock.watchRetrySleepwas reset by event handlers while the retry loop doubled it.lastSecretsFetchwas already protected bywatchMutex, but it now shares the new lock so that one rule covers all four. That lock,stateMutex, is held only briefly for reads and writes and never while taking another lock.Two related accesses are tightened as well.
terminatedByWatchis now an atomic flag, because its write totruewas never covered by the mutex that a comment said protected it. The goroutine that waits for the child now receives the child and its cleanup as arguments, instead of reading variables that the next restart reassigns. The race detector doesn't flag either of these onmaster, probably becauseos/execsynchronizes internally aroundWait. These two changes stop relying on that implicit ordering rather than fixing a reproducible race.--mounthad one more race of its own.cleanupFIFOsets a flag that the goroutine writing to the pipe reads when its nextopenfails, and the flag was a plain bool. It's now atomic.The new tests fail on every run without the fix and pass with it.
TestWatchConcurrentEventsHaveNoDataRacessends updates while a restart is in progress, andTestWatchReconnectHasNoDataRacesmakes the mock API drop the watch stream so doppler reconnects. Both run a race-instrumented doppler built on first use (or one supplied throughDOPPLER_TEST_RACE_BINARY), and fail if it reports a race. The--mountrace depends on goroutine scheduling, and adoppler run-level test caught it only about 75% of the time.TestMountSecretsCleanupWhileWriterIsActiveinstead callsMountSecretsdirectly and forces the order that triggers it. CI doesn't run tests with-race, soTestMountSecretsCleanupHasNoDataRacesruns that test under the race detector in a subprocess. It skips itself when the suite already has the race detector on, or under-short.Stack created with GitHub Stacks CLI • Give Feedback 💬