feat(operator): read the whole-number parts out of a timestamp column - #8480
feat(operator): read the whole-number parts out of a timestamp column#8480kz930 wants to merge 2 commits into
Conversation
Automated Reviewer SuggestionsBased on the
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 352 | 0.215 | 25,336/74,051/74,051 us | 🔴 +117.6% / 🔴 +359.0% |
| 🔴 | bs=100 sw=10 sl=64 | 783 | 0.478 | 124,131/159,809/159,809 us | 🔴 +16.2% / 🔴 +47.0% |
| ⚪ | bs=1000 sw=10 sl=64 | 907 | 0.553 | 1,098,777/1,152,424/1,152,424 us | ⚪ within ±5% / 🔴 +11.1% |
Baseline details
Latest main 75c85aa from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 352 tuples/sec | 408 tuples/sec | 755.28 tuples/sec | -13.7% | -53.4% |
| bs=10 sw=10 sl=64 | MB/s | 0.215 MB/s | 0.249 MB/s | 0.461 MB/s | -13.7% | -53.4% |
| bs=10 sw=10 sl=64 | p50 | 25,336 us | 23,679 us | 12,957 us | +7.0% | +95.5% |
| bs=10 sw=10 sl=64 | p95 | 74,051 us | 34,032 us | 16,134 us | +117.6% | +359.0% |
| bs=10 sw=10 sl=64 | p99 | 74,051 us | 34,032 us | 20,333 us | +117.6% | +264.2% |
| bs=100 sw=10 sl=64 | throughput | 783 tuples/sec | 824 tuples/sec | 980.1 tuples/sec | -5.0% | -20.1% |
| bs=100 sw=10 sl=64 | MB/s | 0.478 MB/s | 0.503 MB/s | 0.598 MB/s | -5.0% | -20.1% |
| bs=100 sw=10 sl=64 | p50 | 124,131 us | 119,840 us | 101,894 us | +3.6% | +21.8% |
| bs=100 sw=10 sl=64 | p95 | 159,809 us | 137,502 us | 108,718 us | +16.2% | +47.0% |
| bs=100 sw=10 sl=64 | p99 | 159,809 us | 137,502 us | 122,482 us | +16.2% | +30.5% |
| bs=1000 sw=10 sl=64 | throughput | 907 tuples/sec | 915 tuples/sec | 1,011 tuples/sec | -0.9% | -10.3% |
| bs=1000 sw=10 sl=64 | MB/s | 0.553 MB/s | 0.559 MB/s | 0.617 MB/s | -1.1% | -10.4% |
| bs=1000 sw=10 sl=64 | p50 | 1,098,777 us | 1,093,457 us | 996,422 us | +0.5% | +10.3% |
| bs=1000 sw=10 sl=64 | p95 | 1,152,424 us | 1,126,982 us | 1,037,670 us | +2.3% | +11.1% |
| bs=1000 sw=10 sl=64 | p99 | 1,152,424 us | 1,126,982 us | 1,072,152 us | +2.3% | +7.5% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,568.47,200,128000,352,0.215,25336.28,74051.10,74051.10
1,100,10,64,20,2555.13,2000,1280000,783,0.478,124130.63,159809.45,159809.45
2,1000,10,64,20,22061.29,20000,12800000,907,0.553,1098776.74,1152424.07,1152424.07Aggregate groups by a column's values, and a timestamp has a different value in almost every row, so grouping by one puts each row in its own group. Sales by month could not be asked for, though the platform carries a TIMESTAMP type, a Gantt chart and a time-series plot. Nothing here is parsed: the column is already a moment by the time it arrives, and which text became which moment was settled upstream, the way KNIME, Alteryx and Spark all separate parsing from extraction. The parts are read as ISO-8601 states them, since the two runtimes do not agree by default — pandas counts Monday as 0 and java.time counts it as 1. Each part is added under a name derived from the column it came from, so reading two timestamp columns names four distinct results. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
d6f6a34 to
535f06e
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8480 +/- ##
============================================
+ Coverage 93.62% 93.89% +0.26%
+ Complexity 4857 4840 -17
============================================
Files 1212 1210 -2
Lines 50037 49261 -776
Branches 6132 5991 -141
============================================
- Hits 46847 46253 -594
+ Misses 1676 1516 -160
+ Partials 1514 1492 -22
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@carloea2 this adds an operator that reads the year, month, weekday and the rest out of a timestamp column, each as a whole-number column of its own. Would you take a look when you have a moment? |
What changes were proposed in this PR?
An operator that reads the year, quarter, month, day, weekday, day of year, ISO week, hour, minute or second out of a timestamp column, each added as a whole-number column of its own.
Nothing here is parsed. The column is already a moment when it arrives, and which text became which moment was settled upstream, the way KNIME, Alteryx and Spark all separate parsing from extraction. The parts are read as ISO-8601 states them, because the two runtimes disagree by default: pandas counts Monday as 0,
java.timecounts it as 1.Any related issues, documentation, discussions?
Not part of #8325: that set makes the operators Texera already had exportable. This adds a new one, which implements the trait from the start rather than gaining it. The trait is in #8327, which has merged.
Closes #8478, the task this change is the whole of.
Aggregate groups by a column's values, and a timestamp has a different value in almost every row, so grouping by one puts each row in its own group. Sales by month could not be asked for, though the platform carries a TIMESTAMP type, a Gantt chart and a time-series plot.
How was this PR tested?
Twelve unit tests in the operator's own spec, over the schema it declares, the values the executor reads and the Python it emits. The parity harness runs it on 13 configurations, one per field plus a hostile column name and an empty cell, comparing the engine's answer against the exported script's.
The operator in the editor, reading
event_timeand asking for the year, the month and the weekday:The three whole-number columns beside the timestamp they came from. 2024-01-15 is a Monday, 2024-07-04 a Thursday and 2025-02-09 a Sunday, reading 1, 4 and 7 as ISO-8601 states them rather than the 0, 3 and 6 pandas would give:
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)