Skip to content

#221 - Fix CsvSplitterTask max_lines and infinite loop - #225

Merged
njoubert-cleverage merged 2 commits into
mainfrom
221
Sep 28, 2026
Merged

njoubert-cleverage merged 2 commits into
mainfrom
221

Conversation

@njoubert-cleverage

Copy link
Copy Markdown
Member

Description

Fixes #221.

CsvSplitterTask does not produce files of max_lines lines:

  • each produced file contains max_lines - 2 data lines: the line counter of the produced file starts at 1 and counts the header line;
  • with max_lines ≤ 2, the task loops forever: no data line is ever read, so the end of the source file is never reached, and the task keeps producing header-only temporary files (5,833 files in 20 seconds in the reproduction below);
  • when the number of lines is a multiple of the chunk size, a last header-only file is produced;
  • max_lines has no type validation: max_lines: 'abc' fails with a TypeError during the execution.

This PR:

  • max_lines is the number of data lines (headers excluded) of each produced file, the last one may contain fewer lines, as documented ("Maximum number of lines per produced file") and as FileSplitterTask after FileSplitterTask produces wrong files: first line lost, line breaks doubled, trailing empty file #205;
  • validate max_lines: int, greater than 0;
  • do not produce a header-only file: when there is no (more) data line, the temporary file is removed and the task is skipped;
  • update the reference documentation (csv_splitter_task.md), add tests.

The new regression tests fail on main and pass with this fix. PHPUnit, PHPStan, PHP-CS-Fixer and Rector pass.

Requirements

  • Documentation updates
    • Reference
    • Cookbooks
    • Changelog
  • Unit tests

Breaking changes

Behaviour change: produced files contain 2 more data lines for the same max_lines (max_lines data lines instead of max_lines - 2). A downstream process relying on the exact size of the chunks is affected: decrease max_lines by 2 to keep the previous chunks. A non-integer or lower than 1 max_lines is now rejected when resolving the options (it failed or looped forever before).

🤖 Generated with Claude Code

njoubert-cleverage and others added 2 commits September 28, 2026 16:47
…lines` data lines (instead of `max_lines - 2`), no infinite loop with `max_lines` <= 2 (`max_lines` must now be an integer greater than 0), no header-only file at the end. Update documentation, add tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@njoubert-cleverage
njoubert-cleverage merged commit b3a6846 into main Sep 28, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CsvSplitterTask produces files of max_lines - 2 lines, and loops forever with max_lines <= 2

1 participant