Repository navigation
fix(drug_discovery): fingerprint readouts leave padding atoms out - #46
Merged
Merged
Conversation
soft_ops.max and soft_ops.argmax reduce over every entry along the axis: the soft argmax standardizes the axis and projects it onto the simplex, so a padded or excluded entry shifts the mean and variance and takes weight. They now take where, a boolean mask of the entries to reduce over, with the semantics of jax.nn.standardize(where=) and jax.nn.softmax(where=): an excluded entry takes no part in the statistics and gets zero weight and zero gradient, so a padded axis gives the unpadded result. standardize_and_squash and proj_simplex take the same mask. Supported for mode="hard" (jnp.max with where and initial) and for method="softsort" in smooth mode, the default; neuralsort, the sorting network, optimal transport and the sparse simplex modes refuse where, since no published implementation of them has a masked form.
The differentiable circular and MACCS fingerprints multiplied the
padding atoms' hidden states by the node mask and then passed them
through biased layers (the hash network and the pattern detectors), so
every padding atom entered the soft max over atoms with nonzero bits.
Ethanol and benzene padded to 13 atoms read up to 0.135 (circular) and
0.058 (MACCS) away from their unpadded fingerprints, so a DTI batch's
drug encodings depended on the largest molecule in the batch.
The readout now reduces over the real atoms only, with
soft_ops.max(..., where=node_mask), the mask sitting at the reduction as
in jax.nn.softmax(where=). Message passing already gives real atoms an
exact zero from padding atoms, which have no bonds. A test makes the
padded-versus-unpadded comparison exact by construction (parameters from
{-1/4, 0, 1/4}, a two-atom molecule) and fails on the old readout in 15
of 16 bits.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two commits:
soft_ops.maxandsoft_ops.argmaxgainwhere, with the semantics ofjax.nn.standardize(where=)andjax.nn.softmax(where=): an excluded entry takes no part in the standardization statistics and gets zero weight and zero gradient. Smooth-modeproj_simplexisjax.nn.softmax(x / softness), so the masked form is JAX's own. It's supported formode="hard"and formethod="softsort"in smooth mode. neuralsort, the sorting network, optimal transport and the sparse simplex modes refusewhere, because no implementation found (softjax, fast-soft-sort, OTT, the SoftSort and NeuralSort code) has a masked form of them.Evidence
Before, padded-and-masked against unpadded output for every drug-discovery graph operator (ethanol and benzene padded to 13 atoms):
The remaining 1e-7 is float32 summation order over a longer axis, not padding.
Tests, written first:
tests/core/test_soft_ops/test_masked_max.py(8). Inputs whose means, variances and softmax denominators are exact under any summation order, so the comparisons are exact: masked soft argmax, soft max, per-row masks, hard mode, zero gradient for excluded entries, and refusal for unsupported methods. Before: 8 failed (nowhere). After: 8 passed.tests/operators/drug_discovery/test_padded_readout.py(2). Padded versus unpadded methanol, exact by construction: parameters from {-1/4, 0, 1/4}, one-hot features, and two heavy atoms, so every sum over atoms has at most two nonzero terms. Against main's code: 15 of 16 bits differ in both operators. After: equal bit for bit.Gate: the soft-op, drug-discovery, DTI and MolNet benchmark tests pass 587 of 587 on CPU. The first commit was gated on its own exported index (511 soft-op and drug-discovery tests).
pre-commit run --all-filespasses all 24 hooks.Merge Danger
Door: two-way.
Blast Radius: padded fingerprints.
dti.pyindexes padded batch rows) and any user code that pads. Unpadded fingerprints don't change.whereis keyword-only, and every existingsoft_ops.maxandargmaxcall works as before.protein/secondary_structure.py:367-371zeroes local H-bonds before a soft max, the same defect class, measured at up to 0.069 on a random map. It's a different operator, and a residue with no non-local partner needs handling, so it gets its own PR.