GH-1234: Release serialization buffers when compression fails - #1290
Open
pj-workspace wants to merge 2 commits into
Open
GH-1234: Release serialization buffers when compression fails#1290pj-workspace wants to merge 2 commits into
pj-workspace wants to merge 2 commits into
Conversation
Generated-by: OpenAI Codex
pj-workspace
requested review from
jbonofre,
laurentgo,
lidavidm and
wgtmac
as code owners
September 10, 2026 03:38
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
Comment on lines
+38
to
+55
| @ParameterizedTest | ||
| @ValueSource(longs = {0, 16}) | ||
| void compressionAllocationFailureReleasesBuffers(long availableBytes) { | ||
| try (BufferAllocator allocator = new RootAllocator(); | ||
| IntVector vector = new IntVector("values", allocator); | ||
| VectorSchemaRoot root = VectorSchemaRoot.of(vector)) { | ||
| vector.allocateNew(1); | ||
| vector.set(0, 42); | ||
| root.setRowCount(1); | ||
| long allocatedBefore = allocator.getAllocatedMemory(); | ||
| int referencesBefore = vector.getDataBuffer().getReferenceManager().getRefCount(); | ||
| allocator.setLimit(allocatedBefore + availableBytes); | ||
| VectorUnloader unloader = new VectorUnloader(root, true, new CopyCodec(), true); | ||
|
|
||
| assertThrows(OutOfMemoryException.class, unloader::getRecordBatch); | ||
| assertEquals(allocatedBefore, allocator.getAllocatedMemory()); | ||
| assertEquals(referencesBefore, vector.getDataBuffer().getReferenceManager().getRefCount()); | ||
| assertEquals(42, vector.get(0)); |
Comment on lines
+145
to
+148
| @Override | ||
| public CompressionUtil.CodecType getCodecType() { | ||
| return CompressionUtil.CodecType.LZ4_FRAME; | ||
| } |
Generated-by: OpenAI Codex
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's Changed
When record batch serialization fails after retaining or compressing buffers,
VectorUnloadercan leave those buffers alive even after the source vectors are closed. Compression allocation failures can also leave an extra reference to the input buffer inAbstractCompressionCodec.Release collected batch buffers if traversal or batch construction throws, while preserving the original exception. Use try-with-resources in
AbstractCompressionCodec.compressso the codec releases the input it owns on both success and failure. Cleanup remains in the component that owns each buffer: the unloader does not release an input already handed to a custom codec.Regression coverage checks allocator exhaustion with zero allocation headroom, runtime exceptions and errors injected on the second compression call, an empty input buffer, and a malformed later vector. The compression-only test stub is explicitly named
SimulatedLz4Codec; it exercises allocation and ownership, not LZ4 round trips. The tests assert memory usage and reference counts return to their previous values and that the source data remains readable. Additional tests exercise allocation failure with the real LZ4 and ZSTD codecs.The memory-leak report and initial outer-cleanup proposal are from #1234; this change also handles the codec's input ownership.
Validation
git diff --checkpassed.mvn -B -ntp -pl compression -am test -Dtest=TestVectorUnloadLoad,TestVectorUnloaderFailure,TestCompressionCodec,TestArrowReaderWriterWithCompression,TestCompressionCodecServiceProvider -Dsurefire.failIfNoSpecifiedTests=falseCloses #1234.
AI assistance: OpenAI Codex was used for implementation and local validation.