Skip to content

Integrate Synthea with PyHealth - #1244

Open
heraclitus007 wants to merge 1 commit into
sunlabuiuc:masterfrom
heraclitus007:master
Open

heraclitus007 wants to merge 1 commit into
sunlabuiuc:masterfrom
heraclitus007:master

Conversation

@heraclitus007

@heraclitus007 heraclitus007 commented Sep 22, 2026 •

Copy link
Copy Markdown

Summary
Add first-class support for generating synthetic patient populations with Synthea and loading the resulting CSV data through PyHealth’s standard dataset API.

What this adds
This feature provides:

  • A SyntheaGenerator wrapper around the Synthea CLI.
  • Automatic use of the pinned Synthea v4.0.0 release with SHA-256 verification.
  • Support for Synthea CLI arguments and validated synthea.properties overrides.
  • Deterministic, fingerprinted output directories for different generation configurations.
  • Lazy generation and reuse of existing output.
  • SyntheaCSVDataset, which loads generated data through BaseDataset.
  • Schema mappings for Synthea patient, encounter, clinical, payer-transition, and claims tables.
  • Configuration discovery and validation against the active Synthea JAR.
  • An end-to-end notebook demonstrating generation, loading, and schema comparison with the official Synthea source release.

Files added or modified

  • pyhealth/datasets/synthea_generator.py: Synthea process management, configuration handling, JAR resolution, checksum verification, output fingerprinting, and lazy generation.
  • pyhealth/datasets/synthea_csv.py: PyHealth dataset integration for generated Synthea CSV files.
  • pyhealth/datasets/configs/synthea_csv.yaml: Table schemas, patient identifiers, timestamps, attributes, and preprocessing mappings.
  • pyhealth/datasets/configs/synthea_release.yaml: Supported Synthea release metadata and exporter configuration.
  • pyhealth/datasets/init.py: Public exports for SyntheaGenerator and SyntheaCSVDataset.
  • tests/core/test_synthea_generator.py: Unit coverage for generator validation, CLI construction, configuration precedence, output reuse, lazy loading, table selection, and dataset behavior.
  • examples/synthea_end_to_end.ipynb: Complete usage walkthrough and schema-equivalence demonstration.

Design decisions

  • Dataset construction is lightweight; Synthea runs only when data is first loaded.
  • Existing generated output is reused unless regenerate=True is specified.
  • Generation parameters are included in the output fingerprint to keep different populations isolated.
  • CSV export is managed by the wrapper. Other Synthea exporters are disabled for this integration.
  • Thirteen clinical and demographic tables are enabled by default.
  • Claims and claim transactions are supported but remain opt-in because of their size.
  • Legacy procedure exports using date are normalized to the current start field.
  • The integration validates user-provided configuration keys against the selected Synthea JAR.

Out of scope

  • FHIR, JSON, CCDA, text, and other non-CSV Synthea exporters.
  • Loading organizations, payers, and providers as patient event tables.
  • Defining a default downstream prediction task or model.
  • Installing or managing Java.
  • Guaranteeing byte-for-byte reproducibility between the published Synthea JAR and independently compiled source builds.
  • Vendoring Synthea source code or generated patient populations.

Validation

  • Added 26 focused unit tests, including an opt-in live generation test.
  • Added an end-to-end notebook covering the public API.
  • Verified the new files compile successfully.
  • Verified the new source and test files with Ruff.

@heraclitus007 heraclitus007 changed the title synthea push in pyhealth Integrate Synthea with PyHealth Sep 22, 2026

@jhnwu3 jhnwu3 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the idea of a Synthea dataset here, but it feels to me we need a SyntheaGenerator(BaseModel) instead here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants