Skip to content

Fix data races in doppler run --watch and --mount - #565

Open
watsonian wants to merge 2 commits into
watsonian/fix-signal-handlingfrom
watsonian/fix-watch-data-races
Open

watsonian wants to merge 2 commits into
watsonian/fix-signal-handlingfrom
watsonian/fix-watch-data-races

Conversation

@watsonian

Copy link
Copy Markdown
Member

Running doppler run --watch with a race-instrumented build reports data races. All of them exist on master, and none are in the new signal handling code. The watch stream handles each event in its own goroutine, and the connection retry loop runs alongside them. Four variables were shared between those goroutines without synchronization. lastUpdateEvent was written by each update event before it took watchMutex. watchedValuesMayBeStale was set by the retry loop and read by the connected event outside any lock. watchRetrySleep was reset by event handlers while the retry loop doubled it. lastSecretsFetch was already protected by watchMutex, but it now shares the new lock so that one rule covers all four. That lock, stateMutex, is held only briefly for reads and writes and never while taking another lock.

Two related accesses are tightened as well. terminatedByWatch is now an atomic flag, because its write to true was never covered by the mutex that a comment said protected it. The goroutine that waits for the child now receives the child and its cleanup as arguments, instead of reading variables that the next restart reassigns. The race detector doesn't flag either of these on master, probably because os/exec synchronizes internally around Wait. These two changes stop relying on that implicit ordering rather than fixing a reproducible race.

--mount had one more race of its own. cleanupFIFO sets a flag that the goroutine writing to the pipe reads when its next open fails, and the flag was a plain bool. It's now atomic.

The new tests fail on every run without the fix and pass with it. TestWatchConcurrentEventsHaveNoDataRaces sends updates while a restart is in progress, and TestWatchReconnectHasNoDataRaces makes the mock API drop the watch stream so doppler reconnects. Both run a race-instrumented doppler built on first use (or one supplied through DOPPLER_TEST_RACE_BINARY), and fail if it reports a race. The --mount race depends on goroutine scheduling, and a doppler run-level test caught it only about 75% of the time. TestMountSecretsCleanupWhileWriterIsActive instead calls MountSecrets directly and forces the order that triggers it. CI doesn't run tests with -race, so TestMountSecretsCleanupHasNoDataRaces runs that test under the race detector in a subprocess. It skips itself when the suite already has the race detector on, or under -short.


Stack created with GitHub Stacks CLI • Give Feedback 💬

@watsonian
watsonian added this pull request to stack #566 October 13, 2026 15:34
@watsonian
watsonian requested a review from a team as a code owner October 13, 2026 15:34
The watch stream handles each event in its own goroutine, and the
connection retry loop runs separately, but they shared lastUpdateEvent,
lastSecretsFetch, watchedValuesMayBeStale and watchRetrySleep without
synchronization. These are now guarded by a mutex.

The process cleanup goroutine read `c`, `cleanupMount` and
terminatedByWatch from the enclosing scope while a restart could
reassign them. It now receives the process and cleanup func as
arguments, and terminatedByWatch is atomic.

The secrets mount's fifoCleanupStarted flag was written by the cleanup
func and read by the pipe writer goroutine; it's now atomic.
The watch tests run a doppler binary built with the race detector
through concurrent update events during a restart and through a dropped
watch connection, and fail if it reports a data race. The binary is
built on first use; DOPPLER_TEST_RACE_BINARY uses a prebuilt one.

The mount test calls MountSecrets directly and orders the cleanup
before the writer's next open, so the writer reads the cleanup flag
after it's set. Since CI doesn't run tests with -race, a wrapper test
runs it under the race detector.
@watsonian
watsonian force-pushed the watsonian/fix-watch-data-races branch from 2d576a0 to 2125630 Compare October 13, 2026 16:37

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant