Skip to content

feat(toolkit-lib): best-effort evaluation of a whole template (internal) - #2075

Draft
iankhou wants to merge 1 commit into
fix/toolkit-lib-evaluator-strict-modefrom
feat/toolkit-lib-best-effort-evaluation
Draft

iankhou wants to merge 1 commit into
fix/toolkit-lib-evaluator-strict-modefrom
feat/toolkit-lib-best-effort-evaluation

Conversation

@iankhou

@iankhou iankhou commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Reason for this change

Second of three stacked PRs that add an unstable evaluate action to toolkit-lib (#2066). EvaluateCloudFormationTemplate
evaluates one expression and throws on the first failure. The action needs a whole template evaluated, with
every expression that can be evaluated replaced by its value, and the others reported instead of failing.

Stack:

  1. fix(toolkit-lib): stricter, opt-in evaluation modes for EvaluateCloudFormationTemplate #2074: evaluator changes
  2. This PR: best-effort evaluation of a whole template (internal, not used until feat(toolkit-lib): unstable evaluate action to resolve a stack's template against its deployed stack #2066)
  3. feat(toolkit-lib): unstable evaluate action to resolve a stack's template against its deployed stack #2066: the unstable evaluate action

Description of changes

Adds evaluateTemplateBestEffort(evaluator, template, { unavailableParameters }), an internal function in
lib/api/cloudformation (no public API). It evaluates every expression in Resources and Outputs, one at a
time, and returns the template with the evaluated values and a list of unresolved expressions
({ path, intrinsic, reason }).

  • Evaluates the template's Conditions (Fn::Equals, Fn::And, Fn::Or, Fn::Not, Condition), Fn::If
    and AWS::NoValue, which EvaluateCloudFormationTemplate doesn't support, and drops resources and outputs
    whose condition is false, as CloudFormation does. Fn::Equals compares scalars as strings, as CloudFormation
    does (a deployed '10' equals 10).
  • An expression that can't be evaluated is kept in place, with its arguments evaluated where possible. Only
    the innermost failure is reported, and an unsupported function is reported itself.
  • Expressions that use unavailable (e.g. NoEcho) parameters, and dynamic references ({{resolve:...}}), are
    reported rather than evaluated.
  • An expression that evaluates to no value, or a function whose whole argument is AWS::NoValue, is reported
    instead of dropping the property or leaking the placeholder.
  • Each distinct expression is evaluated once per template, and the attribute lookups the walk will make
    (Fn::GetAtt and ${Resource.Attribute} in Fn::Sub, only in resources, outputs and Fn::If branches that
    CloudFormation creates) are run up front, 8 at a time.
  • Only evaluation failures are reported. A lookup counts as one when its cause is a not-found / unsupported /
    can't-read error (ResourceNotFoundException, TypeNotFoundException, UnsupportedActionException,
    InvalidRequestException, ValidationException, Cloud Control handler failures, AccessDeniedException).
    Anything else (expired credentials, throttling, network errors, bugs) is thrown.

Describe any new or updated permissions being added

None.

Description of how you validated changes

Unit tests with a fake evaluator: resolved and unresolved expressions, innermost-failure reporting, conditions,
Fn::If and AWS::NoValue, NoEcho and dynamic references, error classification (including wrapped causes),
caching, the bounded and reachable-only prefetch, and untouched template sections. The full stack (#2066) was
run against three apps deployed to a test account and to an emulator.

Checklist

  • This change contains a major version upgrade for a dependency and I confirm all breaking changes are addressed
    • Release notes for the new version:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@github-actions

Copy link
Copy Markdown
Contributor

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@iankhou
iankhou deployed to no-approval October 21, 2026 16:36 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:39 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:39 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:39 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:39 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:39 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:39 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:39 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:39 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:39 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:39 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:39 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:39 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:39 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:39 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:39 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:39 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:39 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:39 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:39 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:39 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:39 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:39 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:39 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:39 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:57 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:57 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:57 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:57 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:57 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:57 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:57 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:57 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:57 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:57 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:57 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:57 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:57 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:57 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:57 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:57 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:57 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:57 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:57 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:57 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:57 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:57 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 16:57 — with GitHub Actions Active
evaluateTemplateBestEffort evaluates every expression in a template's Resources and Outputs with
EvaluateCloudFormationTemplate, one expression at a time, and reports the ones it can't evaluate
instead of failing:

- evaluates Conditions (Fn::Equals, And, Or, Not, Condition), Fn::If and AWS::NoValue, and drops
  resources and outputs whose condition is false
- reports only the innermost failure, and unsupported functions themselves
- reports expressions that use unavailable (e.g. NoEcho) parameters, and dynamic references
- evaluates each distinct expression once, and looks up the reachable attributes up front, a few at a time
- only evaluation failures are reported; other errors (expired credentials, throttling) are thrown

Not used yet: the unstable evaluate action builds on it.
@iankhou
iankhou force-pushed the feat/toolkit-lib-best-effort-evaluation branch from ba082b7 to 6442e21 Compare October 21, 2026 17:06
@iankhou
iankhou deployed to no-approval October 21, 2026 17:07 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 17:10 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 17:10 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 17:10 — with GitHub Actions Active
@iankhou
iankhou deployed to run-tests October 21, 2026 17:10 — with GitHub Actions Active

This branch had an error being deployed

1 failed (outdated) and 2 active deployments
run-tests — 6442e21d Deployed Oct 21, 2026 by iankhou via integ_init-templates (init-typescript-lib, 24.19) #7380
no-approval — 6442e21d Deployed Oct 21, 2026 by iankhou via prepare #7380
automation — 2689062a Deployed Oct 21, 2026 by iankhou via Triage Pull Requests #2233
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants