Skip to content

fix(test): support out-of-nanosecond-range timestamps in unit test comparisons - #6009

Open
mokashang wants to merge 1 commit into
SQLMesh:mainfrom
mokashang:fix/unit-test-timestamp-out-of-bounds
Open

fix(test): support out-of-nanosecond-range timestamps in unit test comparisons#6009
mokashang wants to merge 1 commit into
SQLMesh:mainfrom
mokashang:fix/unit-test-timestamp-out-of-bounds

Conversation

@mokashang

Copy link
Copy Markdown
Contributor

Description

Fixes #5929.

When a database engine returns a TIMESTAMP column as an object-dtype series of python datetime.datetime instances (Redshift is the reporter; any engine whose adapter returns object-dtype behaves the same), the unit-test assert_equal path parses the YAML expected values through pd.to_datetime, which defaults to nanosecond resolution. That resolution is bounded to 1677-09-21..2262-04-11 and overflows on values SQL TIMESTAMP fully supports — the reporter's example 0001-01-01 00:00:00 is a common sentinel value.

The overflow was caught by a broad except Exception inside ModelTest.assert_equal that logged a Failed to convert expected value into datetime warning and left expected[col] as the original strings. Two things happened next:

  • For values whose repr survives str-coercion (e.g. datetime.datetime(1, 1, 1)"0001-01-01 00:00:00"), the downstream _to_hashable normalization stringified both sides and the mismatch was silent — the test appeared to pass while emitting a warning.
  • For values whose repr did not match, pd.testing.assert_frame_equal raised, producing a confusing failure with the same warning above it.

This PR extracts the parsing into _parse_expected_datetime_column and falls back to datetime64[us] resolution on pd.errors.OutOfBoundsDatetime. Microsecond resolution covers year 1 through year 294246, matching SQL TIMESTAMP's supported range, and it round-trips cleanly through .dt.date / .dt.time / .dt.to_pydatetime() for all three sentinel value types the caller already dispatches on. The fast path (pd.to_datetime) is preserved verbatim for in-range values, so nothing about existing tests changes.

Test Plan

  • Added test_out_of_bounds_nanosecond_timestamp_comparison in tests/core/test_test.py. It constructs an object-dtype actual DataFrame containing datetime.datetime(1, 1, 1) and datetime.datetime(9999, 12, 31, 23, 59, 59) — one below and one above pandas' default nanosecond range — and asserts (a) assert_equal succeeds and (b) no Failed to convert expected value warning is emitted. Both conditions are needed because the pre-fix code path silently mismatched on these particular values.
  • Verified the test fails on unpatched main (the assertion on log_warning fires) and passes on this branch.
  • pytest tests/core/test_test.py — 76 passed, 1 deselected (test_pyspark_python_model; environment lacks a Java runtime, unrelated to this change).
  • ruff check and ruff format --check clean on both changed files. mypy sqlmesh/core/test/definition.py clean.

Checklist

  • I have run make style and fixed any issues
  • I have added tests for my changes (if applicable)
  • All existing tests pass (make fast-test)
  • My commits are signed off (git commit -s) per the DCO

…mparisons

When a database engine (e.g. Redshift) returns a TIMESTAMP column as an
object-dtype series of python `datetime.datetime` instances, the unit
test comparison path parses the YAML-supplied expected values with
`pd.to_datetime`, which defaults to nanosecond resolution and overflows
outside 1677-09-21..2262-04-11. Values that SQL TIMESTAMP fully supports
(e.g. `0001-01-01 00:00:00`) triggered a `Failed to convert expected
value into datetime` warning and either a false mismatch or, when the
values happened to round-trip cleanly through `str()`, a silent one.

Fall back to `datetime64[us]` on `OutOfBoundsDatetime` so the comparison
sees equivalent python datetime objects and succeeds. Microsecond
resolution covers year 1 through year 294246, matching SQL TIMESTAMP.

Fixes SQLMesh#5929

Signed-off-by: mokashang <shangmengjiajiajia@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: Failed to convert expected value into datetime for out-of-bounds nanosecond timestamp 0001-01-01 00:00:00 in unit tests

1 participant