Skip to content

Keep missing Event timestamps as None; make Event copyable and picklable - #1247

Merged
jhnwu3 merged 1 commit into
sunlabuiuc:masterfrom
solarsys:fix/event-null-timestamp-and-pickle
Sep 27, 2026
Merged

jhnwu3 merged 1 commit into
sunlabuiuc:masterfrom
solarsys:fix/event-null-timestamp-and-pickle

Conversation

@solarsys

Copy link
Copy Markdown
Collaborator

Summary

This fixes two bugs in pyhealth.data.Event.

  1. A missing timestamp was silently replaced with datetime.now().
  2. Event could not be copied or unpickled (RecursionError).

⚠️ Behaviour change (for the CHANGELOG): Event.timestamp is now None when an event has no time, instead of the current time. That includes Event("x") created without a timestamp. Code that needs a time should pass timestamp= explicitly.

1. Missing timestamps became "now"

Some tables have no time column, for example MIMIC-III/IV patients and all the eICU tables. Their rows have a null timestamp, but Event.__init__ did:

if timestamp is None:
    timestamp = datetime.now()

So every Event that get_events() built from those tables got an invented time. On the bundled MIMIC-III demo, patient 10006:

get_events("patients", return_df=True)["timestamp"]  ->  [None]
get_events("patients")[0].timestamp (1st call)       ->  2026-09-27 10:54:26.852890
get_events("patients")[0].timestamp (2nd call)       ->  2026-09-27 10:54:27.955225
admission timestamp                                  ->  2164-10-23 21:09:00

Why this matters:

  • The value is not reproducible. It changes on every call and every run.
  • The two views of the same row disagree. The DataFrame shows null; the Event shows a date.
  • It silently passes time-window filters. MIMIC dates are shifted into the 2100s, so "now" sorts before every real event, and a filter like e.timestamp <= admission.timestamp wrongly includes demographic rows.
  • Existing code already works around it. The FHIR timeline avoids Event.timestamp for this reason; see the comment in tests/core/test_fhir_dataset.py.

Fix: the timestamp stays None when missing. I found nothing in the library that reads timestamps from these tables (the eICU tasks filter by stay ID). The full test suite passes unchanged, including the FHIR tests.

2. Event could not be copied or unpickled

Event.__getattr__ read self.attr_dict. copy and pickle rebuild the object without calling __init__, then look up special methods such as __setstate__ before attr_dict exists. That lookup lands back in __getattr__, which recurses forever:

pickle.dumps(event)                 -> ok
pickle.loads(pickle.dumps(event))   -> RecursionError
copy.copy(event), copy.deepcopy(e)  -> RecursionError

Because dumps works, a task sample that contains an Event (e.g. in a "raw" field) is written fine and then fails with RecursionError when PyHealth reads it back. I reproduced this with create_sample_dataset.

Fix: __getattr__ now rejects dunder names and reads attr_dict via self.__dict__. Attribute access (event.hadm_id, event["hadm_id"], "hadm_id" in event) is unchanged, and a missing field still raises AttributeError.

Changes

  • pyhealth/data/data.py: both fixes. timestamp is now typed datetime | None, and the docstring has >>> examples.
  • tests/core/test_event.py (new, 8 tests):
    • missing and explicit timestamps
    • from_dict and get_events agree with the DataFrame and are stable across calls
    • pickle, copy and deepcopy round trips
    • attribute access
    • an Event stored inside a sample dataset
  • docs/api/data/pyhealth.data.Event.rst: a new "Events without a time" section, with the migration note and copy/pickle support.
  • examples/event_timestamps_mimic3demo.py: runs on the bundled demo data. It reads a timeless event, filters to timed events, and does a pickle round trip.

Verification

  • New tests: 6 of 8 fail before the fix (3 with the invented timestamp, 3 with RecursionError); all 8 pass after.
  • Full core suite: Ran 1335 tests … OK (skipped=76).
  • tools/check_pr_rules.py: all rules pass.

Out of scope

🤖 Generated with Claude Code

Two bugs in pyhealth.data.Event:

1. Event.__init__ replaced a missing timestamp with datetime.now(). Tables
   without a time column (MIMIC-III/IV `patients`, the eICU tables) have a
   null timestamp, so every Event built from them via get_events() got an
   invented, ever-changing time that disagreed with the DataFrame view
   (`return_df=True` shows null). Because MIMIC dates are shifted to the
   2100s, the invented time also sorts before every real event, silently
   passing time-window filters. The timestamp is now None when missing.
   This is a behaviour change for code that relied on the now() default.

2. Event.__getattr__ read self.attr_dict. copy/pickle create the object
   without __init__ and probe for dunder methods before attr_dict exists,
   so pickle.loads, copy.copy and copy.deepcopy recursed until
   RecursionError. pickle.dumps succeeded, so a task sample holding an
   Event was written fine and crashed when read back. __getattr__ now
   rejects dunder names and reads attr_dict through __dict__.

- tests/core/test_event.py: missing/explicit timestamps, from_dict and
  get_events agreeing with the DataFrame, pickle/copy/deepcopy round
  trips, attribute access, and an Event inside a sample dataset.
- docs: document timeless events and copy/pickle support.
- examples/event_timestamps_mimic3demo.py on the bundled demo data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@jhnwu3 jhnwu3 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good catch. Thanks professor.

LGTM

@jhnwu3
jhnwu3 merged commit 747ccea into sunlabuiuc:master Sep 27, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants