#221 - Fix CsvSplitterTask max_lines and infinite loop - #225
Merged
Merged
Conversation
…lines` data lines (instead of `max_lines - 2`), no infinite loop with `max_lines` <= 2 (`max_lines` must now be an integer greater than 0), no header-only file at the end. Update documentation, add tests. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #221.
CsvSplitterTaskdoes not produce files ofmax_lineslines:max_lines - 2data lines: the line counter of the produced file starts at 1 and counts the header line;max_lines≤ 2, the task loops forever: no data line is ever read, so the end of the source file is never reached, and the task keeps producing header-only temporary files (5,833 files in 20 seconds in the reproduction below);max_lineshas no type validation:max_lines: 'abc'fails with aTypeErrorduring the execution.This PR:
max_linesis the number of data lines (headers excluded) of each produced file, the last one may contain fewer lines, as documented ("Maximum number of lines per produced file") and asFileSplitterTaskafter FileSplitterTask produces wrong files: first line lost, line breaks doubled, trailing empty file #205;max_lines:int, greater than 0;csv_splitter_task.md), add tests.The new regression tests fail on
mainand pass with this fix. PHPUnit, PHPStan, PHP-CS-Fixer and Rector pass.Requirements
Breaking changes
Behaviour change: produced files contain 2 more data lines for the same
max_lines(max_linesdata lines instead ofmax_lines - 2). A downstream process relying on the exact size of the chunks is affected: decreasemax_linesby 2 to keep the previous chunks. A non-integer or lower than 1max_linesis now rejected when resolving the options (it failed or looped forever before).🤖 Generated with Claude Code