You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
TOPS-2851 - add scripts/migrate_v1.py (envars v1 to envars2) - #22
This PR adds scripts/migrate_v1.py. It converts an envars v1 envars.yml to envars2 and encrypts the secrets again with a GCP KMS key. It then checks that v1 and v2 give exactly the same values.
TOPS-2851 moves every live data-app DAG from envars v1 (AWS KMS) to envars2 (tog-data-apps), in 29 draft PRs. The script made the envars.yml in those PRs. With this repo's version, all 29 files give exactly the same values as their v1 file on master, in every environment (one dev value in da-splash-access-dwelltime is an intended fix, documented in that PR). The script is in this repo so that the next migration uses the same tested script, not a copy on one laptop.
What the script does
It decrypts the v1 secrets with envars v1, for the target environments only. This needs AWS credentials for the old AWS KMS key.
It makes the envars2 file with envars init and envars add.
envars2 needs a secret to be scoped. So an inherited v1 secret default gets one encrypted value for each environment (and location) that has no override of its own, and each one uses that scope's own value. A top-level TOKEN: !secret … is handled the same way.
v2 locations are only added for the v1 accounts (master, sandbox) that the file uses. Data-apps use none.
{{ STAGE }} and {{ RELEASE }} become env.get(...), so that they are not circular. {{ RELEASE }} becomes env.get("RELEASE") or env.get("RELEASE_SHA"): v1 read RELEASE_SHA, and that input still works.
The most specific scope is written first (env+location, env, location, default), so that envars add's dependency check does not reject a default before its override exists.
It runs envars validate. Then it compares the v1 values (envars print -y) with the v2 values (envars output --format json) for every environment and location. The comparison is exact: spaces at the start or end, multi-line values, and a missing key versus an explicit null. During the check only RELEASE_SHA is set, so it proves that the v1 input still works. If anything is different, it stops.
Safety
Risk
What the script does
Secret value on the command line (ps, shell history)
Secrets go through envars add --value-from-file with a mode-600 temp file. The file is deleted immediately
Secret value in the output
Commands are printed as VAR=<hidden>. Decrypted output is never printed. A failed comparison prints key names only
Secret value in an error
For commands that handle values, stderr is not printed (an error can quote part of a value). envars subprocesses also get _TYPER_STANDARD_TRACEBACK=1, so a traceback cannot show local variables
Bad or empty secret written
A failed decrypt, or a secret with no value or an empty value, stops the script
Changed value not seen
Exact comparison of structured output, not a text parser, with no exceptions. A missing key is a difference, also against a null
Wrong key
--kms-key is required. There is no default, because the key controls who can decrypt later
KMS SERVICE_DISABLED from the ADC quota project
Optional --quota-project, for envars subprocesses only. The user's gcloud and ADC config do not change
Things the user must do after the file (in the docs, and printed by the script when they apply)
{{ STAGE }} becomes env.get("ENVARS_ENV"). The CLI sets ENVARS_ENV, but get_env() does not, so set os.environ["ENVARS_ENV"] = env before you call it.
If the v2 file has locations, they keep the AWS account IDs. With a GCP KMS key, get_env() cannot find the location by itself, so pass loc=.
Files
scripts/migrate_v1.py: the script.
tests/test_migrate_v1.py: 24 test cases, for example:
secrets go by file, not argv (mode 600, deleted);
hidden stderr;
missing or empty secrets stop the script;
top-level scalar secrets;
per-scope secret defaults, including a location default and a partial location override;
--environments (decryption scope, and spaces);
exact comparison (a padded value fails if it changes) with no RELEASE exception;
both RELEASE forms, and the rendered RELEASE value with the real envars2 CLI for RELEASE_SHA only, RELEASE only, and neither;
overrides are written before defaults; a missing key versus a null;
an end-to-end migration with the real envars2 CLI and a small fake envars v1, which includes a multi-line value and a padded value.
docs/user-guide/07-migrating-from-v1.md, and mkdocs.yml nav: how to use the script.
docs/project/changelog.md: an "Unreleased" entry.
Tests
uv run pytest: 148 passed. They also pass with CI's relative dummy credentials path (the first CI run failed because the end-to-end test changed directory; fixed).
uv run pre-commit run on the changed files: ruff format, ruff check (with bandit) and ty all pass. mkdocs build --strict passes.
A real run of the current script (fa0aee7) against a copy of the v1 envars.yml from da-sighore-to-datalake, in a scratch directory:
6 secret values were decrypted with AWS KMS and encrypted again with tog-data-apps;
the v1 and v2 values matched exactly for dev, prod and staging;
no values were printed.
The exact comparison from this script, run against the envars.yml in all 29 TOPS-2851 data-app PRs: all match their v1 file on master (dwelltime: one intended dev change, see that PR).
AWS CodeBuild: see the check on the latest commit.
Review
Four rounds of automated review (Gemini once, Copilot three times), 19 threads, plus 3 findings in one Copilot review summary. All are fixed or answered, and all threads are resolved.
After merge
Nothing deploys. This PR adds a script, its tests and its docs. The library and the CLI do not change.
The script converts an envars v1 envars.yml to envars2 and encrypts its
secrets again with a GCP KMS key. It checks that v1 and v2 give the same
output for every environment and location. It never prints secret values
and never puts them on the command line. Includes tests and a user-guide page.
JIRA Ticket: TOPS-2851
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request introduces a migration script scripts/migrate_v1.py along with documentation and tests to help users transition from envars v1 to envars2. The script handles re-encrypting secrets using a GCP KMS key and validates that the outputs match. The review feedback highlights three important issues: a critical bug where top-level secrets are skipped during migration, a high-severity issue where multi-line secrets are corrupted by the line-based parser, and a potential IndexError if target_envs is empty.
- Read v1 with `print -y` and v2 with `output --format json`, so that
verification compares exact values (spaces at the start or end,
multi-line values).
- Migrate top-level scalar secrets.
- Write inherited secret defaults for each environment and location with
that scope's own value.
- Decrypt only --environments. Reject empty secrets and an empty
environment list.
- Hide stderr for commands that handle values.
- Note the ENVARS_ENV step for get_env() when {{ STAGE }} is rewritten.
- Fix the end-to-end test for CI's relative credentials path.
JIRA Ticket: TOPS-2851
- Rewrite both RELEASE forms when a value has both.
- Remove the RELEASE exception in verification. Give v1 (RELEASE_SHA)
and v2 (RELEASE) the same value during verification.
- Strip spaces in --environments.
- Note that get_env() needs loc= when the v2 file has locations.
- Docs: give the Git requirement line (the package name is envars).
JIRA Ticket: TOPS-2851
Preserve RELEASE_SHA runtime lookup during migration
scripts/migrate_v1.py:157
These substitutions change the runtime source from RELEASE_SHA to RELEASE, as the comment in verify_migration confirms. A deployment that still sets only RELEASE_SHA gets None or not-set after migration. Verification masks that difference by setting both variables; matching those synthetic inputs does not preserve existing deployment behavior. Preserve the legacy lookup, update the substitution tests, and cover a runtime environment containing only RELEASE_SHA.
Write overrides before defaults to avoid false dependency cycles
scripts/migrate_v1.py:231
Writing defaults before overrides can reject an otherwise valid migration when locations are configured. For example, with only prod and location master, A.default = '{{ B }}', B.default = '{{ A }}', and B.prod = 'fixed' are acyclic once resolved. Adding B's default temporarily creates a cycle, and envars add exits during its per-context dependency check (src/envars/cli.py:387–392) before the override is written. The same problem affects location defaults. Collect additions and write higher-specificity overrides before fallbacks, or assemble the complete configuration before checking dependencies.
Distinguish missing keys from explicit null values
scripts/migrate_v1.py:268
A missing key compares equal to an explicit null because .get() returns None for both: compare({'OPTIONAL': None}, {}) reports no differences. Both readers preserve nulls, so verification can report an exact match after a null-valued variable disappears. Require the key to exist in both mappings before accepting equal values.
…pare
- Collect every value, then write the most specific scope first, so that
envars add's dependency check does not reject a default that is only
circular until its override exists.
- compare: a key that is missing on one side is a difference, even when
the other side has an explicit null.
JIRA Ticket: TOPS-2851
v1 rendered {{ RELEASE }} from RELEASE_SHA. The rewrite is now
env.get("RELEASE") or env.get("RELEASE_SHA", ...), so the v1 input still
works and RELEASE also works. Verification sets only RELEASE_SHA, so it
proves the v1 input. A test renders the template with the real envars2
CLI for RELEASE_SHA only, RELEASE only, and neither.
JIRA Ticket: TOPS-2851
Secret file reading normalizes CRLF and carriage returns
scripts/migrate_v1.py:198
The secret file handoff changes CRLF and bare carriage returns to LF: envars add reads this file with open(value_from_file) at src/envars/cli.py:238–239, which normalizes newlines before encryption. Secrets containing these characters are therefore changed, and the exact verification fails. Use open(value_from_file, newline="") in the CLI reader and add a secret round-trip regression test covering both forms.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
JIRA Ticket: TOPS-2851
Summary
This PR adds
scripts/migrate_v1.py. It converts an envars v1envars.ymlto envars2 and encrypts the secrets again with a GCP KMS key. It then checks that v1 and v2 give exactly the same values.TOPS-2851 moves every live data-app DAG from envars v1 (AWS KMS) to envars2 (
tog-data-apps), in 29 draft PRs. The script made theenvars.ymlin those PRs. With this repo's version, all 29 files give exactly the same values as their v1 file on master, in every environment (one dev value in da-splash-access-dwelltime is an intended fix, documented in that PR). The script is in this repo so that the next migration uses the same tested script, not a copy on one laptop.What the script does
envars initandenvars add.TOKEN: !secret …is handled the same way.master,sandbox) that the file uses. Data-apps use none.{{ STAGE }}and{{ RELEASE }}becomeenv.get(...), so that they are not circular.{{ RELEASE }}becomesenv.get("RELEASE") or env.get("RELEASE_SHA"): v1 readRELEASE_SHA, and that input still works.envars add's dependency check does not reject a default before its override exists.envars validate. Then it compares the v1 values (envars print -y) with the v2 values (envars output --format json) for every environment and location. The comparison is exact: spaces at the start or end, multi-line values, and a missing key versus an explicit null. During the check onlyRELEASE_SHAis set, so it proves that the v1 input still works. If anything is different, it stops.Safety
ps, shell history)envars add --value-from-filewith a mode-600 temp file. The file is deleted immediatelyVAR=<hidden>. Decrypted output is never printed. A failed comparison prints key names only_TYPER_STANDARD_TRACEBACK=1, so a traceback cannot show local variables--kms-keyis required. There is no default, because the key controls who can decrypt laterSERVICE_DISABLEDfrom the ADC quota project--quota-project, for envars subprocesses only. The user's gcloud and ADC config do not changeThings the user must do after the file (in the docs, and printed by the script when they apply)
{{ STAGE }}becomesenv.get("ENVARS_ENV"). The CLI setsENVARS_ENV, butget_env()does not, so setos.environ["ENVARS_ENV"] = envbefore you call it.get_env()cannot find the location by itself, so passloc=.Files
scripts/migrate_v1.py: the script.tests/test_migrate_v1.py: 24 test cases, for example:--environments(decryption scope, and spaces);RELEASEexception;RELEASEforms, and the renderedRELEASEvalue with the real envars2 CLI forRELEASE_SHAonly,RELEASEonly, and neither;docs/user-guide/07-migrating-from-v1.md, andmkdocs.ymlnav: how to use the script.docs/project/changelog.md: an "Unreleased" entry.Tests
uv run pytest: 148 passed. They also pass with CI's relative dummy credentials path (the first CI run failed because the end-to-end test changed directory; fixed).uv run pre-commit runon the changed files: ruff format, ruff check (with bandit) and ty all pass.mkdocs build --strictpasses.fa0aee7) against a copy of the v1envars.ymlfromda-sighore-to-datalake, in a scratch directory:tog-data-apps;envars.ymlin all 29 TOPS-2851 data-app PRs: all match their v1 file on master (dwelltime: one intended dev change, see that PR).Review
Four rounds of automated review (Gemini once, Copilot three times), 19 threads, plus 3 findings in one Copilot review summary. All are fixed or answered, and all threads are resolved.
After merge
Nothing deploys. This PR adds a script, its tests and its docs. The library and the CLI do not change.
Rollback
Revert this PR.