Fix ddi metric overwriting y_pred in multilabel_metrics_fn - #1243
Om-singhaI wants to merge 1 commit into
Conversation
The ddi branch assigned its per sample index list back to y_pred, so any threshold based metric listed after ddi got a ragged list instead of the thresholded matrix and raised. Keep the conversion in its own local name.
MateehUllah
left a comment
There was a problem hiding this comment.
Reviewed the metric-order regression and the surrounding multilabel_metrics_fn flow. Keeping the DDI-specific index representation in y_pred_ddi fixes the underlying issue without changing the shared thresholded y_pred consumed by subsequent threshold-based metrics. The regression coverage exercises DDI first, middle, and last, including hamming_loss, and preserves the existing DDI result when it is last. I also checked the documented model behavior: GAMENet, MICRON, MoleRec, and SafeDrug each persist ddi_adj.npy during model setup, so the added documentation is consistent with the current implementations. I did not find a blocking correctness issue in this patch.
|
Thanks will take a deeper look after the CIs finish |
Closes #1242.
multilabel_metrics_fnbuilds the thresholded(n_samples, n_labels)arrayy_predonce before the metric loop, and every threshold based metric in the loop reads it. The"ddi"branch needs a different shape, a list of label indices per sample, and it assigned that list back toy_pred. The shared array was gone from that point on, so anything listed after"ddi"got the ragged list instead:f1_*,precision_*,recall_*,jaccard_*andhamming_lossraiseValueError: Classification metrics can't handle a mix of multilabel-indicator and unknown targets, andaccuracyraisesAttributeError: 'list' object has no attribute 'flatten'.pyhealth/metrics/regression.pyalready guards the same situation in itskl_divergencebranch, under the comment# Work on copies to avoid mutating x/x_rec for subsequent metrics. The"ddi"branch now keeps its conversion in a local of its own.What changed:
pyhealth/metrics/multilabel.py: the index list goes intoy_pred_ddi, soy_predstays the thresholded matrix. Two lines.tests/core/test_multilabel_ddi_metric_order.py: new unittest module covering"ddi"first, in the middle and last.docs/api/metrics/pyhealth.metrics.multilabel.rst: a short note that the order of themetricslist does not change the result, plus which models write theddi_adj.npyfile that"ddi"reads.examples/drug_recommendation/drug_recommendation_mimic3_gamenet.py: the metric list now starts with"ddi", the ordering that used to crash.No call that works today changes. With
"ddi"last the values are identical before and after.Testing, on python 3.13.15,
scikit-learn1.7.2 and numpy 2.2.6:python -m unittest tests.core.test_multilabel_ddi_metric_order -v: 3 tests, OK.pyhealth/metrics/multilabel.pyrestored fromorigin/master:Ran 3 tests,FAILED (errors=2), bothValueError: Classification metrics can't handle a mix of multilabel-indicator and unknown targets.python -m unittest tests.core.test_calibration_binary_ece tests.core.test_fairness tests.core.test_scores: 26 tests, OK.ruff check pyhealth/metrics/multilabel.py: 8 findings, all between lines 1 and 17 and all of them already on master. None on the added lines.python tools/check_pr_rules.py --base b96c3f5 --head HEAD:All PR contribution rules passed.metrics=["ddi", "jaccard_samples"]raisedValueErroron b96c3f5 and now returns{'ddi_score': 0.4, 'jaccard_samples': 0.9166666666666666}, the same numbers asmetrics=["jaccard_samples", "ddi"].Branched from
master, since CONTRIBUTING puts hotfixes there.