Skip to content

Qualcomm AI Engine Direct - Add HTP context graph splitting option - #22595

Open
chenweng-quic wants to merge 1 commit into
pytorch:mainfrom
CodeLinaro:dev1/chenweng/graph_split
Open

Qualcomm AI Engine Direct - Add HTP context graph splitting option#22595
chenweng-quic wants to merge 1 commit into
pytorch:mainfrom
CodeLinaro:dev1/chenweng/graph_split

Conversation

@chenweng-quic

@chenweng-quic chenweng-quic commented Sep 8, 2026 •
edited by pytorch-bot Bot
Loading

Copy link
Copy Markdown
Collaborator

Summary

  • Exposes QNN's HTP context graph-splitting config (QNN_HTP_CONTEXT_CONFIG_OPTION_GRAPH_SPLITTING_CONFIGS / QnnHtpContext_GraphSplit_t) via a new use_graph_splitting option on generate_htp_compiler_spec().
  • Wired up for the offline-prepare (host/x86 export) path only, since this feature is used in finalize.
  • Gated behind QNN_HTP_API_VERSION_MAJOR >= 5 && QNN_HTP_API_VERSION_MINOR >= 49 since this option doesn't exist in older QNN_HTP_API_VERSION

Test plan

python backends/qualcomm/tests/test_qnn_delegate.py -k "test_qnn_backend_graph_splitting" --device ef5e4029 --host localhost --soc_model SM8850 --build_folder build-android --executorch_root .

LLAMA3.2 3B performance:

Graph Graph Preparation Initializing (us) Graph Optimizations (us) Post Graph Optimization (us) Graph Sequencing for Target (us) VTCM Allocation (us) Parallelization Optimization (us) Finalizing Graph Sequence (us) Completion (us) QNN (execute) time (us)
kv_forward1_without_gpe 1146 26843138 433549 6828982 449030 5929118 266695 15171 9552
kv_forward1_with_gpe 496 2014382 26630 427743 27504 246911 22066 1268 9880
kv_forward2_without_gpe 373 27618966 1049136 10700233 313808 7947260 291464 14841 9452
kv_forward2_with_gpe 599 2511473 34099 577162 36742 353654 31248 1754 9787
kv_forward3_without_gpe 405 30262309 541066 8523872 407082 8058194 314898 16692 11136
kv_forward3_with_gpe 512 5575498 75421 1322133 111640 1009982 51157 3371 11437
kv_forward4_without_gpe 1966 10304161 36241 1151113 87287 749097 34706 1944 6703
kv_forward4_with_gpe 342 8387236 41997 1104533 96270 870457 31896 2245 6686
prefill_forward1_without_gpe 403 23425582 521963 20288202 28141820 13415019 534725 19491 23283
prefill_forward1_with_gpe 635 2404414 42852 928676 607167 368336 25194 1790 20824
prefill_forward2_without_gpe 354 22249865 496426 16837420 24738364 9634081 464298 16200 20473
prefill_forward2_with_gpe 557 2521494 41107 929431 765368 426039 28336 1955 20764
prefill_forward3_without_gpe 744 31356868 553758 17889195 17232990 9485483 500217 15025 21125
prefill_forward3_with_gpe 1933 5689220 99188 2404130 2256161 1236475 79918 6054 21835
prefill_forward4_without_gpe 889 10117697 43571 1450046 1323893 1062400 35717 2122 9431
prefill_forward4_with_gpe 331 8484639 50456 1647489 1798310 1306534 45152 2760 9446

This PR was authored with assistance from Claude Code (Anthropic).

cc @cbilgin @psiddh

@chenweng-quic
Qualcomm AI Engine Direct - Add HTP context graph splitting option
0fa0ae7 
Exposes QNN_HTP_CONTEXT_CONFIG_OPTION_GRAPH_SPLITTING_CONFIGS through
generate_htp_compiler_spec(use_graph_splitting=...) for the offline-prepare
(host) export path. Gated behind QNN HTP API >= 5.49 since this option does not
exist in older version.

Co-authored-by with assistance from Claude Code (Anthropic).
@pytorch-bot

pytorch-bot Bot commented Sep 8, 2026 •
edited
Loading

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22595

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 0fa0ae7 with merge base 5cdcc02 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 8, 2026
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@chenweng-quic chenweng-quic added the module: qnn Issues related to Qualcomm's QNN delegate and code under backends/qualcomm/ label Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@abhinaykukkadapu abhinaykukkadapu Awaiting requested review from abhinaykukkadapu abhinaykukkadapu is a code owner
@psiddh psiddh Awaiting requested review from psiddh psiddh is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. module: qnn Issues related to Qualcomm's QNN delegate and code under backends/qualcomm/

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@chenweng-quic