add tool to synchronize add-on repository with latest addonTemplate - #46
add tool to synchronize add-on repository with latest addonTemplate#46abdel792 wants to merge 23 commits into
Conversation
Introduces the syncAddonWithTemplate.py automation tool to streamline synchronizing add-on metadata and infrastructure with upstream template updates. Detailed changes: - Added `syncAddonWithTemplate.py` at repository root to handle AST-aware merging of buildVars.py and pyproject.toml configuration. - Configured `PROTECTED_ELEMENTS` in the sync script to prevent overwriting template-specific files (e.g., `.github/workflows/unitTests.yml` and `tests/`). - Added `.addonmergeignore` support for defining project-specific file exclusion rules during synchronization. - Added dependencies for the sync tool to `pyproject.toml`. - Added `tests/unit/test_syncAddonWithTemplate.py` to validate metadata parsing, AST transformations, TOML formatting, and execution ordering. - Updated `pyproject.toml` to exclude `syncAddonWithTemplate.py` alongside the `tests/` directory from ruff and pyright checks. - Updated `docs/managementFromGit/updatingExistingAddons.md` with full usage instructions, CLI flags, and execution modes for the sync script. - Updated `docs/unitTesting.md` with guidelines for running unit tests locally using unittest and uv.
Update testFormatAuthorList in test_syncAddonWithTemplate.py to assert that empty author email keys are omitted rather than expecting an empty string, matching syncAddonWithTemplate.py behavior.
…umentation Update description of testFormatAuthorList to document that empty email fields are omitted for PEP 621 compliance.
There was a problem hiding this comment.
Pull request overview
Adds a developer-facing synchronization tool to help NVDA add-on authors update their repositories to the latest AddonTemplate structure, including AST-aware metadata handling and TOML merging, with accompanying unit tests and documentation updates.
Changes:
- Introduces
syncAddonWithTemplate.pyto sync template infrastructure while mergingbuildVars.pyandpyproject.toml. - Adds unit tests for key merge/formatting behavior and a small template sanity suite.
- Updates dependency groups/docs to support running the sync tool and the unit test suite.
Reviewed changes
Copilot reviewed 8 out of 10 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
syncAddonWithTemplate.py |
New sync/merge engine (AST + tomlkit) for updating add-on repos from the template. |
tests/unit/test_syncAddonWithTemplate.py |
Unit tests for metadata extraction, buildVars merge, TOML indentation, and dependency merging. |
tests/unit/template/sanity.py |
Minimal sanity tests intended to validate CI test discovery/execution. |
docs/managementFromGit/updatingExistingAddons.md |
Expanded guidance for updating add-ons (automated tool + manual merge). |
docs/unitTesting.md |
Updated instructions for running the unit tests and describing the new suites. |
pyproject.toml |
Adds tomlkit, updates pyright, and excludes the new sync script/tests from ruff/pyright. |
uv.lock |
Locks tomlkit and bumps pyright to match pyproject.toml. |
tests/__init__.py |
Updates package docstring. |
tests/unit/__init__.py |
Updates package docstring. |
.addonmergeignore |
Present at repo root (empty in this PR context). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…ty test - Move test infrastructure docstring from root unit tests to template submodule. - Revert root `tests/__init__.py` docstring to reflect all test types (system and unit). - Remove obsolete `sanity.py` test file.
|
Hi @seanbudd,
|
- Replace underscores with hyphens in `getBasePackageName` and `legacyToolingBases` to handle equivalent Python package name formats during dependency merges. - Bootstrap `.addonmergeignore` from template on first sync if absent locally, allowing it to self-reference and manage its own persistence dynamically.
|
@Copilot,
|
- Replace `methodOrder.index()` with `orderIndex.get(a, defaultOrder)` map. - Prevent `ValueError` when running tests on inherited or dynamic methods.
|
@Copilot,
|
- Fix incorrect path for `test_syncAddonWithTemplate.py` in `unitTesting.md`. - Clarify `tomlkit` installation requirement and fix no-argument command example in `updatingExistingAddons.md`.
|
@Copilot,
|
- Add testSetupAddonMergeIgnore and testAddonMergeIgnore to test_syncAddonWithTemplate.py - Document both new tests in docs/unitTesting.md
|
Hi @seanbudd, @Copilot, Regarding the test coverage for
|
| "ruff==0.14.5", | ||
| "prek==0.4.8", | ||
| "pyright[nodejs]==1.1.407", | ||
| "pyright[nodejs]==1.1.411", |
There was a problem hiding this comment.
do we need to update pyright in this PR?
|
Hi @seanbudd, When running I bumped the lock version to Let me know if you would still prefer me to revert it to keep the PR scope strictly minimal! |
- Fix casing for AddonTemplate repository URL in updatingExistingAddons.md - Remove detailed syncAddonWithTemplate test suite breakdown in unitTesting.md to keep coverage in docstrings
|
Thanks for the review and all these helpful suggestions, @seanbudd! I have addressed the quick fixes and documentation updates:
Regarding your suggestion to split I'm currently refactoring the code into submodules, writing the requested additional test cases, and running all necessary tests. I will address all of your remaining feedback and suggestions throughout this process and post a full update over the weekend. I'll leave the PR in draft mode until everything is ready for review. |
|
Sean, regarding your comment on I'm currently implementing a solution for this and will push it in my next commit over the weekend, as I need to finish up some work commitments first. The previous Replacing it with a dynamic lookup alongside a minimal mapping dictionary for non-supported/replaced dependencies (like The updated logic now dynamically collects all tooling packages directly from the template's Additionally, for deprecated tools that were replaced in the template (such as This maps replaced tools to their modern template equivalents so they aren't accidentally preserved as custom dependencies. Here is a snippet of how the dynamic collection works: # Dynamically collect all template packages (dependencies + dependency-groups)
templateBases: set[str] = {
getBasePackageName(d) for d in tplDeps if isinstance(d, str)
}
if "dependency-groups" in mergedData and isinstance(mergedData["dependency-groups"], MutableMapping):
for grp in mergedData["dependency-groups"].values():
if isinstance(grp, (list, MutableSequence)):
for grpItem in grp:
if isinstance(grpItem, str):
templateBases.add(getBasePackageName(grpItem)) |
- Add syncAddonTool package directory to lighten syncAddonWithTemplate.py. - Import submodules from syncAddonTool into syncAddonWithTemplate.py for improved code readability and structure. - Add test fixtures directory to decouple legacy dictionaries and sample files from test_syncAddonWithTemplate.py. - Add unit tests verifying version priority handling between user dependencies and template dependencies. - Add syncAddonTool.spec at repository root to streamline executable generation with PyInstaller.
…tandalone executable - Update docs/managementFromGit/updatingExistingAddon.md to document the syncAddonTool directory. - Add instructions for building and running the standalone executable using PyInstaller and `uv run --with pyinstaller`.
|
In the last two commits, I made the following updates:
|
|
Hi Sean, Thanks for the feedback! I've noted all your points and will work on applying these updates tomorrow or the day after. |
- Move `syncAddonTool.spec` into the `syncAddonTool/` package directory. - Rename unit test module from `test_syncAddonWithTemplate.py` to `test_syncAddonTool.py` in `tests/unit/template/`. - Update `pyproject.toml` to remove redundant Pyright and Ruff exclusions for the old script and spec file (the spec file is now covered by the existing `syncAddonTool` directory rule). - Update `docs/unittesting.md` to reference `test_syncAddonTool.py`.
|
Hi Sean, I have updated the PR with the following cleanup and refactoring changes:
|
the syncAddonTool workflow for syncing add-ons with AddonTemplate.
|
Hi Sean, I have updated While working on these changes, I also noticed three minor pre-existing typos/formatting issues in the
I deliberately left these untouched to keep the diff strictly focused on introducing However, if you'd like me to clean those up in this PR as well, I'd be glad to push a quick fix! |
Remove instances of `python -m syncAddonTool.py` from updatingExistingAddons.md to keep only valid module and directory execution syntaxes.
|
Hi Sean, Just pushed a quick update to As mentioned earlier, I haven't touched the three pre-existing typos/formatting issues yet—let me know if you'd like me to fix them in this PR or leave them for later. |
- Add missing -v/--verbose flag to CLI options table in docs/managementFromGit/updatingExistingAddons.md. - Suppress redundant dependency decision logs when running with --verbose.
|
Hi Sean, Just pushed a new update to As agreed, I haven't touched the three minor pre-existing typos/formatting issues. |
Prepend full TOML key path (e.g. tool.ruff.include) when logging list merges in pyproject.toml to eliminate ambiguity.
|
While reviewing a verbose debug report, I noticed that section key logging during the I just pushed a small refinement to pass the full TOML context path to the merge logger. This makes the debug output much more explicit and easier to trace:
|
|
Hi @seanbudd, I just merged |
There was a problem hiding this comment.
🔵 Needs a closer look
It adds a large new tool that destructively rewrites/merges files in developer repositories with intricate AST/TOML logic and untested CLI/backup/git paths, alongside documentation that overstates supported behavior, so human review is warranted.
Review details
- Files reviewed: 21/23 changed files
- Comments generated: 7
- Review effort level: Balanced
| - Inside this file, list the names, relative paths, or glob patterns of the files or folders you want the tool to skip during synchronization. | ||
| - The file uses standard `.gitignore` pattern matching syntax (parsed via `pathspec`). |
|
|
||
| * **Add-on Synchronization Tool Tests:** | ||
| ``` bash | ||
| uv run python -m unittest -v tests/unit/template/test_syncAddonTool |
| You can merge the latest template changes into your repository instead of manually copying updated files. | ||
|
|
||
| This document explains the recommended update procedure. | ||
| *This document explains the update procedures, including both the recommended automated method using `syncAddonTool` and the manual Git merge workflow."* |
| uv run python syncAddonTool -ad /path/to/my-nvda-addon -td /path/to/local/AddonTemplate | ||
| ``` | ||
|
|
||
| - **Syntax D (Standalone executable)**: |
| ## Running Tests Locally | ||
|
|
||
| To run the unit test suite locally using `uv`: | ||
| For unit tests to execute successfully, target modules (such as `syncAddonWithTemplate.py`) must be located at the root of the repository as sibling files to the `tests/` directory (at the same hierarchical level). This ensures Python's module discovery properly imports scripts when `unittest` runs from the project root. |
| "__pycache__", | ||
| ".venv", | ||
| "buildVars.py", | ||
| "syncAddonTool", |
| "tomlkit", | ||
| "pathspec", |
|
@nvdaes could we have your assistance in reviewing this please? |
| "ruff==0.14.5", | ||
| "prek==0.4.8", | ||
| "pyright[nodejs]==1.1.407", | ||
| "pyright[nodejs]==1.1.411", |
There was a problem hiding this comment.
I don't know why https://github.com/nvaccess/AddonTemplate/pull/46/changes#r3725285433 was marked as resolved. I have unmarked it. Please update pyright in a separate PR
| ##### Crucial Requirements & Design Constraints | ||
|
|
||
| 1. **Automatic Self-Exclusion:** | ||
| The update tool automatically protects `.addonmergeignore` itself from being overwritten during synchronization. Even if `.addonmergeignore` is present in the template repository, the target add-on's local `.addonmergeignore` file is preserved without needing to explicitly list itself. |
There was a problem hiding this comment.
please put new sentences on new lines in markdown files for diffing and readability
Link to issue number:
Replaces #41.
Summary of the issue:
While upgrading an existing add-on using standard Git commands remains fully supported, some add-on developers prefer an automated and dedicated tool to streamline the process. Manual step-by-step migration of metadata (from
buildVars.pytopyproject.toml) and file merges can require extra care and verification. Providing an automated script offers a convenient alternative for developers who wish to perform these updates quickly and with built-in AST-based metadata handling.Description of developer facing changes:
syncAddonWithTemplate.pyat the repository root as an automated alternative to Git-based manual upgrades for add-on developers..github/workflows/unitTests.ymlandtests/, ensuring they are excluded from add-on synchronization.syncAddonWithTemplate.pyand thetests/directory from static analysis inpyproject.toml(ruffandpyright).tests/unit/test_syncAddonWithTemplate.py).Description of development approach:
syncAddonWithTemplate.pyusing an AST-aware approach to seamlessly migrate legacybuildVars.pymetadata or modernAddonInfosetups intopyproject.toml. It supports timestamped backups, custom exclusions via.addonmergeignore, and includes.github/workflows/unitTests.ymlinPROTECTED_ELEMENTS.pyproject.tomlto excludesyncAddonWithTemplate.pyandtests/fromruffandpyrightchecks.tests/unit/test_syncAddonWithTemplate.py) usingunittestandtempfile, utilizing aload_testshook to guarantee deterministic test execution order.Testing strategy:
Validated the synchronization script and unit test suite locally using
uv:syncAddonWithTemplate.pyacross various CLI modes:-adis optional):<addon>_bak_<timestamp>) were generated,.addonmergeignorerules were respected, and protected elements (including.github/workflows/unitTests.ymlandtests/) were not pushed to target add-on repositories.unittest:Confirmed all tests passed successfully.
4. Confirmed
ruffandpyrightignoresyncAddonWithTemplate.pyandtests/.Known issues with pull request:
None.
Code Review Checklist