Skip to content

Wrap the live ICudaEngine after compile instead of serialize/deserialize. - #4698

Open
micwill755 wants to merge 1 commit into
network-constructfrom
convert-finalize
Open

Wrap the live ICudaEngine after compile instead of serialize/deserialize.#4698
micwill755 wants to merge 1 commit into
network-constructfrom
convert-finalize

Conversation

@micwill755

Copy link
Copy Markdown
Collaborator

Summary

  • After build_engine_with_config, in-process convert_module wraps the live ICudaEngine instead of serialize() + C++ deserializeCudaEngine.
  • interpret_module_to_result(..., serialize_engine=False) keeps cuda_engine on SerializedInterpreterResult. Cache insert, convert_exported_program_to_serialized_trt_engine, and get_extra_state / save still serialize from the live engine.
  • Cross-compile-for-Windows still serializes at interpret time. In-process compile uses Python TRTEngine.execute() until save/reload (C++ execute path needs bytes).
  • Klein NVFP4 cold compile: ICudaEngine.serialize 1.15s → 0, setup_engine 2.40s → 0.001s, compile wall 36.95s → 33.17s. First image ~1.12s and steady ~0.43s unchanged.

PR 10 of the compile-startup stack; based on #4693.

Type of change

  • Performance (non-breaking for in-process compile; save/export still produce serialized engines)

Test plan

  • tests/py/dynamo/runtime/test_live_engine_wrap.py (compile tiny Add: serialized_engine is None, live wrap, cosine match, get_extra_state serializes)
  • Klein NVFP4 cold compile (torchtrt-213rc10): serialize 1.15s and setup_engine 2.40s gone; first image and steady unchanged
  • Existing dynamo runtime tests (CI)

Checklist

  • Code follows project style (pre-commit: isort, ruff, black, mypy)
  • Self-review
  • Comment on why the skip is valid
  • Docs update (not needed)
  • New tests
  • Labels

Stack created with GitHub Stacks CLIGive Feedback 💬

@meta-cla meta-cla Bot added the cla signed label Sep 9, 2026
@micwill755
micwill755 added this pull request to stack #4533 September 9, 2026 08:10
@github-actions github-actions Bot added component: tests Issues re: Tests component: conversion Issues re: Conversion stage component: core Issues re: The core compiler component: api [Python] Issues re: Python API component: runtime component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths labels Sep 9, 2026
@github-actions
github-actions Bot requested a review from cehongwang September 9, 2026 08:10
…ize.

In-process convert_module keeps the builder engine and skips the bytes roundtrip. Cache, export, and save still serialize.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla signed component: api [Python] Issues re: Python API component: conversion Issues re: Conversion stage component: core Issues re: The core compiler component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths component: runtime component: tests Issues re: Tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant