fix(test): support out-of-nanosecond-range timestamps in unit test comparisons - #6009
Open
mokashang wants to merge 1 commit into
Open
fix(test): support out-of-nanosecond-range timestamps in unit test comparisons#6009mokashang wants to merge 1 commit into
mokashang wants to merge 1 commit into
Conversation
…mparisons When a database engine (e.g. Redshift) returns a TIMESTAMP column as an object-dtype series of python `datetime.datetime` instances, the unit test comparison path parses the YAML-supplied expected values with `pd.to_datetime`, which defaults to nanosecond resolution and overflows outside 1677-09-21..2262-04-11. Values that SQL TIMESTAMP fully supports (e.g. `0001-01-01 00:00:00`) triggered a `Failed to convert expected value into datetime` warning and either a false mismatch or, when the values happened to round-trip cleanly through `str()`, a silent one. Fall back to `datetime64[us]` on `OutOfBoundsDatetime` so the comparison sees equivalent python datetime objects and succeeds. Microsecond resolution covers year 1 through year 294246, matching SQL TIMESTAMP. Fixes SQLMesh#5929 Signed-off-by: mokashang <shangmengjiajiajia@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #5929.
When a database engine returns a TIMESTAMP column as an object-dtype series of python
datetime.datetimeinstances (Redshift is the reporter; any engine whose adapter returns object-dtype behaves the same), the unit-testassert_equalpath parses the YAML expected values throughpd.to_datetime, which defaults to nanosecond resolution. That resolution is bounded to 1677-09-21..2262-04-11 and overflows on values SQL TIMESTAMP fully supports — the reporter's example0001-01-01 00:00:00is a common sentinel value.The overflow was caught by a broad
except ExceptioninsideModelTest.assert_equalthat logged aFailed to convert expected value into datetimewarning and leftexpected[col]as the original strings. Two things happened next:datetime.datetime(1, 1, 1)→"0001-01-01 00:00:00"), the downstream_to_hashablenormalization stringified both sides and the mismatch was silent — the test appeared to pass while emitting a warning.pd.testing.assert_frame_equalraised, producing a confusing failure with the same warning above it.This PR extracts the parsing into
_parse_expected_datetime_columnand falls back todatetime64[us]resolution onpd.errors.OutOfBoundsDatetime. Microsecond resolution covers year 1 through year 294246, matching SQL TIMESTAMP's supported range, and it round-trips cleanly through.dt.date/.dt.time/.dt.to_pydatetime()for all three sentinel value types the caller already dispatches on. The fast path (pd.to_datetime) is preserved verbatim for in-range values, so nothing about existing tests changes.Test Plan
test_out_of_bounds_nanosecond_timestamp_comparisonintests/core/test_test.py. It constructs an object-dtypeactualDataFrame containingdatetime.datetime(1, 1, 1)anddatetime.datetime(9999, 12, 31, 23, 59, 59)— one below and one above pandas' default nanosecond range — and asserts (a)assert_equalsucceeds and (b) noFailed to convert expected valuewarning is emitted. Both conditions are needed because the pre-fix code path silently mismatched on these particular values.main(the assertion onlog_warningfires) and passes on this branch.pytest tests/core/test_test.py— 76 passed, 1 deselected (test_pyspark_python_model; environment lacks a Java runtime, unrelated to this change).ruff checkandruff format --checkclean on both changed files.mypy sqlmesh/core/test/definition.pyclean.Checklist
make styleand fixed any issuesmake fast-test)git commit -s) per the DCO