Skip to content

fix(toolkit-lib): stricter, opt-in evaluation modes for EvaluateCloudFormationTemplate - #2074

Draft
iankhou wants to merge 3 commits into
mainfrom
fix/toolkit-lib-evaluator-strict-mode
Draft

iankhou wants to merge 3 commits into
mainfrom
fix/toolkit-lib-evaluator-strict-mode

Conversation

@iankhou

@iankhou iankhou commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Reason for this change

First of three stacked PRs, which eventually add an unstable evaluate action to toolkit-lib (#2066).

Given a CloudFormation template, evaluate resolves that template against deployed resources. This action reuses EvaluateCloudFormationTemplate, which hotswap currently uses. It needs CloudFormation's value semantics, which this PR makes the default; for hotswap these are fixes (see below).

Stack:

  1. This PR: evaluator changes
  2. feat(toolkit-lib): best-effort evaluation of a whole template (internal) #2075: best-effort evaluation of a whole template (internal)
  3. feat(toolkit-lib): unstable evaluate action to resolve a stack's template against its deployed stack #2066: the unstable evaluate action

Description of changes

New opt-in props on EvaluateCloudFormationTemplate. Hotswap doesn't set these props.

  • unavailableParameters: a Ref to one of these parameters (e.g. NoEcho parameters, whose deployed values are masked) fails instead of falling back to the parameter's Default, including where the evaluator uses it internally (Cloud Control identifiers, nested stack parameters).
  • lookups (new SharedLookups class): Cloud Control reads are cached per resource and limited to a number in flight across all evaluators that share it, and nested stack outputs are read from DescribeStacks of the deployed nested stack (by its physical id), instead of evaluated from its template.

Changes that also apply to hotswap:

  • Values are evaluated the way CloudFormation does: an empty string (or 0, false) is a value and only a missing or null value is missing, Fn::Select out of range is an evaluation failure, and Fn::Sub treats ${!Literal} as the literal text ${Literal}. For hotswap, an empty-string parameter no longer falls back to its Default, and Fn::Select out of range no longer becomes an empty string; "", 0 or false attributes and ${!Literal} now hotswap instead of falling back to a full deployment.
  • Unsupported functions throw CfnUnsupportedFunctionException.
  • Failed Cloud Control lookups keep the original error as cause.
  • When the Cloud Control identifier itself can't be evaluated, that evaluation error is thrown as is. Any other error while building the identifier is still wrapped in a CfnEvaluationException, so hotswap still treats it as non-hotswappable rather than failing the deployment.
  • Fn::GetAtt of a nested stack output when the nested template has no Outputs section is an evaluation failure instead of a TypeError (which failed the hotswap deployment).
  • parseIntrinsic is exported, for reuse by the next PR.

Describe any new or updated permissions being added

None.

Description of how you validated changes

Checklist

  • This change contains a major version upgrade for a dependency and I confirm all breaking changes are addressed
    • Release notes for the new version:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

…FormationTemplate

Opt-in props, off for hotswap:
- strictValues: CloudFormation value semantics (empty strings are values, null is missing),
  Fn::Select out of range is an evaluation failure, Fn::Sub handles ${!Literal}
- unavailableParameters: a Ref to one fails instead of falling back to its Default
- lookups (SharedLookups): Cloud Control reads cached per resource and limited across evaluators,
  nested stack outputs read from DescribeStacks

Changes that also apply to hotswap (error handling only):
- unsupported functions throw CfnUnsupportedFunctionException (a CfnEvaluationException, same message)
- failed Cloud Control lookups keep the original error as the cause
- a Cloud Control identifier that can't be evaluated throws its own evaluation error; other errors
  while building it are still wrapped in a CfnEvaluationException
- Fn::GetAtt of a nested stack output when the nested template has no Outputs is an evaluation failure,
  not a TypeError
- parseIntrinsic is exported for reuse
@github-actions

Copy link
Copy Markdown
Contributor

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@iankhou
iankhou deployed to no-approval October 21, 2026 16:36 — with GitHub Actions Active
@codecov-commenter

codecov-commenter commented Oct 21, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.10%. Comparing base (60e0a2d) to head (1af90de).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2074   +/-   ##
=======================================
  Coverage   92.10%   92.10%           
=======================================
  Files          82       82           
  Lines       13186    13186           
  Branches     1878     1878           
=======================================
  Hits        12145    12145           
  Misses       1003     1003           
  Partials       38       38           
Flag Coverage Δ
suite.unit 92.10% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Remove the strictValues option and make its semantics the default:
- an empty string (or 0, false) is a value; only a missing or null value is missing
- Fn::Select with an index out of range is an evaluation failure
- ${!Literal} in Fn::Sub is the literal text ${Literal}

For hotswap, each of these either replaces a wrong value with the correct one (an empty-string parameter
no longer falls back to its Default; Fn::Select out of range no longer becomes an empty string) or turns a
case that fell back to a full deployment into a hotswap with the correct value ("", 0 or false attributes,
${!Literal}). A null attribute is still missing.

This branch had an error being deployed

1 failed (outdated), 1 active, and 1 inactive deployments
run-tests — 1af90ded Deployed Oct 21, 2026 by iankhou via integ_cli (cli-integ-tests, 24.19, 12) #7379
no-approval — 1af90ded Deployed Oct 21, 2026 by iankhou via prepare #7379
automation — a8cf53ca Deployed Oct 21, 2026 by iankhou via Triage Pull Requests #2232
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants