Repository navigation
Fix waiting for the data channel send buffer to drain #1267
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
hiroshihorie
wants to merge
12
commits into
hiroshi/close-engine-on-terminal-paths
from
hiroshi/fix-buffer-status-wait
+207
−34
Open
Changes from all commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
eccdb11
Fix waiting for the data channel send buffer to drain
hiroshihorie eaa4688
Announce engine closure on terminal lifecycle paths
hiroshihorie 6f0346b
Signal engine closure before disposal callbacks
hiroshihorie 5c112c3
Merge remote-tracking branch 'origin/hiroshi/close-engine-on-terminal…
hiroshihorie bb1c5b8
Wait on the engine's closed state instead of a closing listener
hiroshihorie dd41b6f
Merge remote-tracking branch 'origin/main' into hiroshi/close-engine-…
hiroshihorie 60bbac9
Reject buffer waits with a LiveKit exception instead of a string
hiroshihorie 44785d8
Merge remote-tracking branch 'origin/hiroshi/close-engine-on-terminal…
hiroshihorie 64e9ccd
Hold buffer waits while the engine reconnects
hiroshihorie daf740d
Wake buffer waits on state changes instead of polling
hiroshihorie 0df11e5
Merge remote-tracking branch 'origin/hiroshi/close-engine-on-terminal…
hiroshihorie 2cb361d
Scope the changeset to data stream writes
hiroshihorie File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| patch type="fixed" "Fix data stream writes that could hang after a full reconnect, carried on after the room disconnected, or leaked a listener while waiting for the send buffer to drain. Writes now also wait for a reconnect to finish" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,155 @@ | ||
| // Copyright 2026 LiveKit, Inc. | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| import 'package:flutter_test/flutter_test.dart'; | ||
|
|
||
| import 'package:livekit_client/livekit_client.dart'; | ||
| import 'package:livekit_client/src/proto/livekit_models.pb.dart' as lk_models; | ||
| import '../mock/e2e_container.dart'; | ||
| import '../mock/peerconnection_mock.dart'; | ||
| import 'signal_client_test.dart' show joinResponse, offerResponse; | ||
|
|
||
| const _full = 10 * 1024 * 1024; | ||
|
|
||
| void main() { | ||
| late E2EContainer container; | ||
|
|
||
| setUp(() async { | ||
| resetMockDataChannels(); | ||
| container = E2EContainer(); | ||
| await container.connectRoom(); | ||
| }); | ||
|
|
||
| tearDown(() async { | ||
| await container.dispose(); | ||
| }); | ||
|
|
||
| void setBufferedAmount(int? amount) { | ||
| for (final dc in mockDataChannels) { | ||
| dc.fakeBufferedAmount = amount; | ||
| } | ||
| } | ||
|
|
||
| /// Starts a full reconnect and answers its join, leaving it to wait for the subscriber offer. | ||
| /// The new, empty publisher channels exist from here on, but the reconnect has not finished. | ||
| Future<void> startFullReconnect() async { | ||
| final ws = container.wsConnector; | ||
| final previousHandlers = ws.handlers; | ||
| container.engine.fullReconnectOnNext = true; | ||
| ws.onDispose(); | ||
| for (var i = 0; i < 200 && identical(ws.handlers, previousHandlers); i++) { | ||
| await Future<void>.delayed(const Duration(milliseconds: 10)); | ||
| } | ||
| await Future<void>.delayed(const Duration(milliseconds: 1)); | ||
| ws.onData(joinResponse.writeToBuffer()); | ||
| } | ||
|
|
||
| void finishFullReconnect() => container.wsConnector.onData(offerResponse.writeToBuffer()); | ||
|
|
||
| /// Fills the reliable channel and sends once, so the engine's cached status also says it is full. | ||
| Future<void> fillReliableChannel() async { | ||
| // The mock loops published data back into this room as if a remote sent it, which the room | ||
| // rejects. These tests only need the send to refresh the engine's cached status. | ||
| for (final dc in mockDataChannels) { | ||
| dc.onMessageSend = null; | ||
| } | ||
| setBufferedAmount(_full); | ||
| await container.room.localParticipant!.publishData([1], reliable: true); | ||
| expect(container.engine.isBufferStatusLow(Reliability.reliable), isFalse); | ||
| } | ||
|
|
||
| test('a wait ends when a full reconnect replaces the full channel', () async { | ||
| await fillReliableChannel(); | ||
| final wait = container.engine.waitForBufferStatusLow(Reliability.reliable); | ||
|
|
||
| // The replaced channel never reports a drain, only the new one is empty. | ||
| await startFullReconnect(); | ||
| finishFullReconnect(); | ||
|
|
||
| await wait.timeout(const Duration(seconds: 1)); | ||
| }); | ||
|
|
||
| test('a wait wakes as soon as the channel reports a drain', () async { | ||
| await fillReliableChannel(); | ||
| var done = false; | ||
| final wait = container.engine.waitForBufferStatusLow(Reliability.reliable).whenComplete(() => done = true); | ||
| await pumpEventQueue(); | ||
| expect(done, isFalse); | ||
|
|
||
| setBufferedAmount(null); | ||
| for (final dc in mockDataChannels) { | ||
| dc.onBufferedAmountLow?.call(0); | ||
| } | ||
|
|
||
| // Only zero-length timers run here, so a wait that polls on a timer would not have woken yet. | ||
| await pumpEventQueue(); | ||
| expect(done, isTrue); | ||
| await wait; | ||
| }); | ||
|
|
||
| test('a wait does not leave an engine listener behind', () async { | ||
| final before = container.engine.events.listeners.length; | ||
| await fillReliableChannel(); | ||
| final wait = container.engine.waitForBufferStatusLow(Reliability.reliable); | ||
|
|
||
| setBufferedAmount(null); | ||
| for (final dc in mockDataChannels) { | ||
| dc.onBufferedAmountLow?.call(0); | ||
| } | ||
| await wait.timeout(const Duration(seconds: 1)); | ||
|
|
||
| expect(container.engine.events.listeners.length, before); | ||
| }); | ||
|
|
||
| test('a wait still fails when the engine closes', () async { | ||
| await fillReliableChannel(); | ||
| final wait = container.engine.waitForBufferStatusLow(Reliability.reliable); | ||
| final outcome = wait.then<Object?>((_) => 'resolved', onError: (Object e) => e); | ||
|
|
||
| container.engine.markClosed(); | ||
|
|
||
| expect(await outcome.timeout(const Duration(seconds: 2)), isNot('resolved')); | ||
| }); | ||
|
|
||
| test('a wait holds until a full reconnect finishes', () async { | ||
| final engine = container.engine; | ||
| await fillReliableChannel(); | ||
|
|
||
| final marker = lk_models.DataPacket( | ||
| kind: lk_models.DataPacket_Kind.RELIABLE, | ||
| user: lk_models.UserPacket(payload: [42]), | ||
| ); | ||
| var markerSent = false; | ||
| onMockReliableChannelCreated = (dc) => dc.onMessageSend = (message) { | ||
| final packet = lk_models.DataPacket.fromBuffer(message.binary); | ||
| if (packet.hasUser() && packet.user.payload.length == 1 && packet.user.payload.first == 42) { | ||
| markerSent = true; | ||
| } | ||
| }; | ||
|
|
||
| // What a stream writer does with each chunk. | ||
| final write = engine | ||
| .waitForBufferStatusLow(Reliability.reliable) | ||
| .then((_) => engine.sendDataPacket(marker, reliability: Reliability.reliable)); | ||
|
|
||
| // The new channel exists and is empty, but the reconnect waits for the subscriber transport. | ||
| await startFullReconnect(); | ||
| await Future<void>.delayed(const Duration(milliseconds: 300)); | ||
| expect(markerSent, isFalse, reason: 'the write went out before the reconnect finished'); | ||
|
|
||
| finishFullReconnect(); | ||
| await write.timeout(const Duration(seconds: 2)); | ||
| expect(markerSent, isTrue, reason: 'the write never reached the new channel'); | ||
| }); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔴 Stream writes hang after silent buffer drain
When a full channel drains without a low-buffer callback,
waitForBufferStatusLownever wakes._sendStateChangedonly fires for explicit status updates, reconnects, or closure, so stream writes stay blocked.Learn more
A stream writer calls waitForBufferStatusLow before each chunk. The wait stores a full buffer status after a send, then relies on a callback to update it. If the channel drains without that callback, neither the cached status nor
_sendStateChangedchanges. The previous timed check also read the live channel, so a silent drain eventually released the write.Example: A reliable channel reports 10 MiB buffered, then drains to zero without invoking
onBufferedAmountLow. The next stream chunk stays pending indefinitely instead of sending on the empty channel.Recommended fix: Preserve a timed fallback that periodically checks
isBufferStatusLow(kind)while retaining notifications for prompt wakeups andmarkClosed()for prompt failure. Cancel the fallback when the wait resolves.Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This isn't a regression. The previous loop only polled the cached
_dcBufferStatusflag, not the live channel, so a drain that never updated the flag hung there too. On iOS, macOS and Android, flutter_webrtc updatesbufferedAmountfrom the buffered amount change event, and that event also callsonBufferedAmountLowwhenever the amount is below the threshold, so a drain can't be silent. Windows and Linux are different: they send no buffered amount events, so the wait can stall once more than 2 MiB is queued. That was already true before this PR, and a timer that re-readsisBufferStatusLowwouldn't help there because the Dart side only learns the new amount fromgetBufferedAmount(). I'll handle desktop in a follow-up.