Repository navigation
feat(browser-inference): run MobileSAM and EfficientViT-SAM-L0 through a per-model definition - #906
Merged
Merged
Conversation
…bleModelDefinition host.ts, client.ts and worker.ts were hardcoded to EfficientSAM-Ti's tensor names and pure functions directly, even though promptable.ts's own runtime contract was already model-agnostic by design. Adds definition.ts's PromptableModelDefinition, bundling a model's tensor names and its requireUsableImage/requireAnswerablePrompt/encoderInput/decoderPrompt/ bestCandidate/binaryMask functions into one value. createModelHost and createModelClient default this parameter to EFFICIENT_SAM_TI_DEFINITION, so the ~40 existing unit-test call sites that construct them with just a factory/channel are unchanged. The worker learns which definition to run a model-load's graphs against from a new modelId field on that protocol message, looked up in a small per-worker registry -- adding a model is one entry in that registry plus a new create<Model>Runtime factory, not a change to any message-handling code. Verified behavior-preserving: typecheck, lint, all 129 unit tests, and all 17 real-Chromium browser tests pass, with the browser's mask SHA-256s still matching the Python parity reference exactly.
…t inputs are per-model host.ts hardcoded [1, 1, maxPoints, 2]/[1, 1, maxPoints] as if every decoder padded points into the same fixed, batched shape EfficientSAM-Ti uses. MobileSAM's official ONNX decoder disagrees on both counts: its point tensors are dynamic-length ([1, N, 2]/[1, N]), and its graph also declares mask_input/has_mask_input, the SAM-family "previous low-res mask" refinement pair that EfficientSAM-Ti's graph does not have. decoderPrompt() now returns its own coords/labels dims alongside the data, and DecoderTensorNames gains optional maskInput/hasMaskInput names plus an emptyMaskInput() definition hook -- host.ts feeds "there is no previous mask" (hasMaskInput zeroed) whenever a decoder declares that pair, rather than omitting an input the graph requires. Verified behavior-preserving: typecheck, lint, all 129 unit tests and all 17 real-Chromium browser tests pass, mask SHA-256s unchanged.
MobileSAM's official ONNX decoder export (SamOnnxModel, upstream's own dummy inputs) declares orig_im_size as float32, while EfficientSAM-Ti's graph declares the same-purpose input int64 -- a real disagreement between two SAM-family graphs. DecoderTensorNames.sizeDtype makes this explicit instead of host.ts hardcoding int64.
Adds mobileSam.ts's PromptableModelDefinition -- including a hand-rolled bilinear resize (align_corners=False, PyTorch's own "half-pixel centers" sampling grid) for the resize-longest-side + normalize + pad step that runs outside the encoder graph -- registers it in worker.ts's model registry, and exposes createMobileSamRuntime from the browser adapter. requireAnswerablePrompt refuses a negative point, matching the measured finding from scripts/browser_models/mobilesam/parity.py: the mechanism is real (a lone label-0 point differs from a lone label-1 point) but not reliably exclusionary, so it is refused rather than answered with an exclusion that was not confirmed. Real-browser proof (mobileSam.spec.ts, 10 tests, all passing): loads the real exported graphs, encodes the fixture image once, refines across three suggest calls on the same held embedding, proves the encoder is not rerun, and reports WebGPU honestly. The masks it produces are bit-identical to the Python parity reference -- exact confidence, exact lit-pixel count, matching SHA-256 -- meaning the TypeScript resize matches torch.nn.functional.interpolate exactly. CI: browser-models job now exports, runs parity for, and browser-tests both models in one job (one shared torch install, one cache step per model keyed on that model's own pin). assert_tests_ran.mjs's REAL_MODEL_SPEC is now REAL_MODEL_SPECS, a list, so each model's minimum-tests floor is independently visible in a diff. Caught and fixed in this change: an unquoted job `name:` containing a colon inside parentheses is invalid YAML (a bare `key:` is parsed as a new mapping key) -- validated with a real YAML parse, not just review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Generalises the browser inference host over a
PromptableModelDefinitionand adds two more promptable segmentation models to@visionset/browser-inference: MobileSAM and EfficientViT-SAM-L0.orig_im_sizedtype are declared per model.mobileSam.tsandefficientVitSam.tsdefine each model's preprocessing, encoder and decoder contract.Scope
Runtime code and its vitest coverage only. The ONNX exporters, parity checks and real-browser specs for these models are not part of this PR, so the new definitions are tested against fakes here.
Verification
@visionset/browser-inference: lint, typecheck and 166 tests pass.@visionset/app(src/data/browserInference): 109 tests pass;@visionset/ui-core(src/inference): 27 tests pass.pnpm -r buildsucceeds.