Skip to content

Fix ddi metric overwriting y_pred in multilabel_metrics_fn - #1243

Open
Om-singhaI wants to merge 1 commit into
sunlabuiuc:masterfrom
Om-singhaI:fix-multilabel-ddi-overwrites-y-pred
Open

Om-singhaI wants to merge 1 commit into
sunlabuiuc:masterfrom
Om-singhaI:fix-multilabel-ddi-overwrites-y-pred

Conversation

@Om-singhaI

Copy link
Copy Markdown

Closes #1242.

multilabel_metrics_fn builds the thresholded (n_samples, n_labels) array y_pred once before the metric loop, and every threshold based metric in the loop reads it. The "ddi" branch needs a different shape, a list of label indices per sample, and it assigned that list back to y_pred. The shared array was gone from that point on, so anything listed after "ddi" got the ragged list instead: f1_*, precision_*, recall_*, jaccard_* and hamming_loss raise ValueError: Classification metrics can't handle a mix of multilabel-indicator and unknown targets, and accuracy raises AttributeError: 'list' object has no attribute 'flatten'.

pyhealth/metrics/regression.py already guards the same situation in its kl_divergence branch, under the comment # Work on copies to avoid mutating x/x_rec for subsequent metrics. The "ddi" branch now keeps its conversion in a local of its own.

What changed:

  • pyhealth/metrics/multilabel.py: the index list goes into y_pred_ddi, so y_pred stays the thresholded matrix. Two lines.
  • tests/core/test_multilabel_ddi_metric_order.py: new unittest module covering "ddi" first, in the middle and last.
  • docs/api/metrics/pyhealth.metrics.multilabel.rst: a short note that the order of the metrics list does not change the result, plus which models write the ddi_adj.npy file that "ddi" reads.
  • examples/drug_recommendation/drug_recommendation_mimic3_gamenet.py: the metric list now starts with "ddi", the ordering that used to crash.

No call that works today changes. With "ddi" last the values are identical before and after.

Testing, on python 3.13.15, scikit-learn 1.7.2 and numpy 2.2.6:

  • python -m unittest tests.core.test_multilabel_ddi_metric_order -v: 3 tests, OK.
  • The same module with pyhealth/metrics/multilabel.py restored from origin/master: Ran 3 tests, FAILED (errors=2), both ValueError: Classification metrics can't handle a mix of multilabel-indicator and unknown targets.
  • python -m unittest tests.core.test_calibration_binary_ece tests.core.test_fairness tests.core.test_scores: 26 tests, OK.
  • ruff check pyhealth/metrics/multilabel.py: 8 findings, all between lines 1 and 17 and all of them already on master. None on the added lines.
  • python tools/check_pr_rules.py --base b96c3f5 --head HEAD: All PR contribution rules passed.
  • The repro from the issue: metrics=["ddi", "jaccard_samples"] raised ValueError on b96c3f5 and now returns {'ddi_score': 0.4, 'jaccard_samples': 0.9166666666666666}, the same numbers as metrics=["jaccard_samples", "ddi"].

Branched from master, since CONTRIBUTING puts hotfixes there.

The ddi branch assigned its per sample index list back to y_pred, so any
threshold based metric listed after ddi got a ragged list instead of the
thresholded matrix and raised. Keep the conversion in its own local name.

@MateehUllah MateehUllah left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the metric-order regression and the surrounding multilabel_metrics_fn flow. Keeping the DDI-specific index representation in y_pred_ddi fixes the underlying issue without changing the shared thresholded y_pred consumed by subsequent threshold-based metrics. The regression coverage exercises DDI first, middle, and last, including hamming_loss, and preserves the existing DDI result when it is last. I also checked the documented model behavior: GAMENet, MICRON, MoleRec, and SafeDrug each persist ddi_adj.npy during model setup, so the added documentation is consistent with the current implementations. I did not find a blocking correctness issue in this patch.

@jhnwu3

jhnwu3 commented Sep 28, 2026

Copy link
Copy Markdown
Collaborator

Thanks will take a deeper look after the CIs finish

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

multilabel_metrics_fn crashes when a thresholded metric comes after "ddi" in the metrics list

3 participants