[NO-MERGE] Remove GenAI packages while preserving TorchSharp support - #7738
Open
matouskozak wants to merge 2 commits into
Open
matouskozak wants to merge 2 commits into
matouskozak wants to merge 2 commits into
Conversation
Remove the GenAI libraries, tests, samples, model data, and exclusive dependencies from current main. Preserve the TorchSharp CUDA test setup and document the boundary between source removal and published package support. Related to #7720 and #7719. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #7738 +/- ##
==========================================
+ Coverage 69.89% 70.26% +0.36%
==========================================
Files 1487 1419 -68
Lines 276294 272540 -3754
Branches 28294 27937 -357
==========================================
- Hits 193128 191495 -1633
+ Misses 75676 73633 -2043
+ Partials 7490 7412 -78
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
Remove the package-removal section and its index link after the initial CI runs completed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
One or more issues must be addressed before approval.
Review effort: Lite
Findings: None
What changed in this PR
Removes the GenAI packages and their exclusive support code while retaining TorchSharp, tokenizer packages, CUDA test configuration, and historical release notes.
Changes:
- Deletes GenAI source, tests, samples, configurations, approvals, and design documentation.
- Removes GenAI-only package/version entries and third-party notice.
- Updates documentation and Copilot repository guidance.
| File | Description |
|---|---|
| THIRD-PARTY-NOTICES.TXT | Updated as part of this pull request. |
| test/Microsoft.ML.GenAI.Phi.Tests/SemanticKernelTests.cs | Updated as part of this pull request. |
| test/Microsoft.ML.GenAI.Phi.Tests/Phi2Tests.cs | Updated as part of this pull request. |
| test/Microsoft.ML.GenAI.Phi.Tests/Microsoft.ML.GenAI.Phi.Tests.csproj | Updated as part of this pull request. |
| test/Microsoft.ML.GenAI.Phi.Tests/AutoGenTests.cs | Updated as part of this pull request. |
| test/Microsoft.ML.GenAI.Phi.Tests/Approvals/Phi3Tests.TokenizerTest.approved.txt | Updated as part of this pull request. |
| test/Microsoft.ML.GenAI.Phi.Tests/Approvals/Phi3Tests.Phi3Mini128KLayerSizeTest.approved.txt | Updated as part of this pull request. |
| test/Microsoft.ML.GenAI.Phi.Tests/Approvals/Phi2Tests.TokenizerTest.approved.txt | Updated as part of this pull request. |
| test/Microsoft.ML.GenAI.Phi.Tests/Approvals/Phi2Test.TokenizerTest.approved.txt | Updated as part of this pull request. |
| test/Microsoft.ML.GenAI.Mistral.Tests/Microsoft.ML.GenAI.Mistral.Tests.csproj | Updated as part of this pull request. |
| test/Microsoft.ML.GenAI.Mistral.Tests/Approvals/Mistral_V0_3Tests.TokenizerTest.approved.txt | Updated as part of this pull request. |
| test/Microsoft.ML.GenAI.Mistral.Tests/Approvals/Mistral_7B_Instruct_V0_3Tests.TokenizerTest.approved.txt | Updated as part of this pull request. |
| test/Microsoft.ML.GenAI.Mistral.Tests/Approvals/Mistral_7B_Instruct_V0_3Tests.ItBuildChatTemplateWithToolsFromAutoGenChatHistory.approved.txt | Updated as part of this pull request. |
| test/Microsoft.ML.GenAI.Mistral.Tests/Approvals/Mistral_7B_Instruct_V0_3Tests.ItBuildChatTemplateFromAutoGenChatHistory.approved.txt | Updated as part of this pull request. |
| test/Microsoft.ML.GenAI.LLaMA.Tests/Microsoft.ML.GenAI.LLaMA.Tests.csproj | Updated as part of this pull request. |
| test/Microsoft.ML.GenAI.LLaMA.Tests/LLaMA3_2Tests.cs | Updated as part of this pull request. |
| test/Microsoft.ML.GenAI.LLaMA.Tests/LLaMA3_1Tests.TokenizerTest.received.txt | Updated as part of this pull request. |
| test/Microsoft.ML.GenAI.LLaMA.Tests/LLaMA3_1Tests.cs | Updated as part of this pull request. |
| test/Microsoft.ML.GenAI.LLaMA.Tests/Approvals/LLaMA3_1Tests.TokenizerTest.approved.txt | Updated as part of this pull request. |
| test/Microsoft.ML.GenAI.LLaMA.Tests/Approvals/LLaMA3_1Tests.ItBuildChatTemplateFromSemanticKernelChatHistory.approved.txt | Updated as part of this pull request. |
| test/Microsoft.ML.GenAI.LLaMA.Tests/Approvals/LLaMA3_1Tests.ItBuildChatTemplateFromMEAIChatHistory.approved.txt | Updated as part of this pull request. |
| test/Microsoft.ML.GenAI.LLaMA.Tests/Approvals/LLaMA3_1Tests.ItBuildChatTemplateFromAutoGenChatHistory.approved.txt | Updated as part of this pull request. |
| test/Microsoft.ML.GenAI.Core.Tests/QuantizedLinearTests.cs | Updated as part of this pull request. |
| test/Microsoft.ML.GenAI.Core.Tests/Microsoft.ML.GenAI.Core.Tests.csproj | Updated as part of this pull request. |
| test/Microsoft.ML.GenAI.Core.Tests/CausalLMDatasetTest.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Phi/Utils.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Phi/Resource/README.md | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Phi/Resource/Config/phi-3-mini-4k-instruct-config.json | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Phi/Resource/Config/phi-3-medium-4k-instruct-config.json | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Phi/Resource/Config/phi-2-config.json | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Phi/README.md | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Phi/Phi3/Phi3TokenizerHelper.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Phi/Phi3/Phi3ForCausalLM.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Phi/Phi3/Phi3ChatTemplateBuilder.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Phi/Phi3/Phi3CausalLMTextGenerationService.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Phi/Phi3/Phi3CausalLMChatCompletionService.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Phi/Phi3/Phi3CausalLMChatClient.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Phi/Phi3/Phi3CausalLMAgent.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Phi/Phi2/Phi2TokenizerHelper.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Phi/Phi2/Phi2ForCausalLM.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Phi/Phi2/Config.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Phi/Module/Phi3SuScaledRotaryEmbedding.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Phi/Module/Phi3Model.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Phi/Module/Phi3MLP.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Phi/Module/Phi2RotaryEmbedding.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Phi/Module/Phi2MLP.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Phi/Module/Phi2DecoderLayer.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Phi/Microsoft.ML.GenAI.Phi.csproj | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Phi/Extension/SemanticKernelExtension.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Mistral/Resource/Config/mistral-7B-instruct-v0.3.json | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Mistral/MistralTokenizerHelper.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Mistral/MistralModel.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Mistral/MistralMLP.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Mistral/MistralForCausalLM.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Mistral/MistralDecoderLayer.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Mistral/MistralConfig.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Mistral/Microsoft.ML.GenAI.Mistral.csproj | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.LLaMA/Resource/Config/meta-llama-3.2-3B-Instruct.json | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.LLaMA/Resource/Config/meta-llama-3.2-1B-Instruct.json | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.LLaMA/Resource/Config/meta-llama-3.1-8B-Instruct.json | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.LLaMA/Resource/Config/meta-llama-3.1-70B-Instruct.json | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.LLaMA/Resource/Config/meta-llama-3.1-405B-Instruct.json | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.LLaMA/Module/LlamaModel.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.LLaMA/Module/LlamaMLP.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.LLaMA/Module/LlamaDecoderLayer.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.LLaMA/Microsoft.ML.GenAI.LLaMA.csproj | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.LLaMA/LlamaTokenizerHelper.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.LLaMA/LlamaTextCompletionService.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.LLaMA/LlamaChatCompletionService.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.LLaMA/LlamaCausalLMAgent.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.LLaMA/Llama3CausalLMChatClient.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.LLaMA/Llama3_1ChatTemplateBuilder.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Core/Utility/IChatTemplateBuilder.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Core/Utility/Cache.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Core/Trainer/CausalLMSupervisedFineTuningTrainer.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Core/Trainer/CausalLMDataset.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Core/Pipeline/CausalLMModelOutput.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Core/Pipeline/CausalLMModelInput.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Core/Module/RotaryEmbedding.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Core/Module/RMSNorm.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Core/Module/NewGELUActivation.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Core/Module/IQuantizeModule.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Core/Module/IDynamicLoadModule.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Core/Module/GenAILinear.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Core/Module/DynamicLoadingModule.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Core/Microsoft.ML.GenAI.Core.csproj | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Core/Extension/TensorExtension.cs | Updated as part of this pull request. |
| src/Microsoft.ML.GenAI.Core/CausalLMPipelineChatClient.cs | Updated as part of this pull request. |
| eng/Versions.props | Updated as part of this pull request. |
| docs/samples/Microsoft.ML.GenAI.Samples/Program.cs | Updated as part of this pull request. |
| docs/samples/Microsoft.ML.GenAI.Samples/Phi3Mini/SemanticKernelSample.cs | Updated as part of this pull request. |
| docs/samples/Microsoft.ML.GenAI.Samples/Phi3Mini/AutoGenSample.cs | Updated as part of this pull request. |
| docs/samples/Microsoft.ML.GenAI.Samples/Microsoft.ML.GenAI.Samples.csproj | Updated as part of this pull request. |
| docs/samples/Microsoft.ML.GenAI.Samples/MEAI/Phi3.cs | Updated as part of this pull request. |
| docs/samples/Microsoft.ML.GenAI.Samples/MEAI/Llama3_1.cs | Updated as part of this pull request. |
| docs/samples/Microsoft.ML.GenAI.Samples/Llama/SFT_Llama_3_2_1B.cs | Updated as part of this pull request. |
| docs/samples/Microsoft.ML.GenAI.Samples/Llama/LlamaSample.cs | Updated as part of this pull request. |
| docs/README.md | Updated as part of this pull request. |
| docs/project-docs/components-and-dependencies.md | Updated as part of this pull request. |
| docs/gen-ai/Usage.md | Updated as part of this pull request. |
| docs/gen-ai/Tokenizer.md | Updated as part of this pull request. |
| docs/gen-ai/README.md | Updated as part of this pull request. |
| docs/gen-ai/Package Structure.md | Updated as part of this pull request. |
| docs/gen-ai/DynamicLoading.md | Updated as part of this pull request. |
| docs/gen-ai/CausalLMPipeline.md | Updated as part of this pull request. |
| docs/gen-ai/CausalLanguageModel.md | Updated as part of this pull request. |
| docs/gen-ai/Benchmark && Evaluation.md | Updated as part of this pull request. |
| Directory.Packages.props | Updated as part of this pull request. |
| .github/copilot-instructions.md | Updated as part of this pull request. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
matouskozak
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #7720.
Implement the removal proposed in #7719 on current
main. This is a separate PR; #7719 remains unchanged and open.Microsoft.ML.GenAI.*libraries, four test projects, sample project, model configurations, approval files, and design documents.Microsoft.ML.TorchSharp,Microsoft.ML.Tokenizers.*, shared dependency versions, and the existing CUDA test setup. In particular, retain both theTorchSharp-cuda-windowsversion and its commented test-project reference.