Skip to content

perf: reduce JSON v2 snapshot and normalization overhead - #253

Merged
jinxiao merged 2 commits into
mainfrom
perf/250-json-v2-snapshots
Oct 16, 2026
Merged

jinxiao merged 2 commits into
mainfrom
perf/250-json-v2-snapshots

Conversation

@jinxiao

@jinxiao jinxiao commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Issue

Closes #250
Closes #251
Closes #252

Summary

Correlation extraction and JMESPath queries no longer encode/decode complete ordinary JSON-shaped inputs just to obtain private snapshots. A shared bounded copy preserves JSON v2 float64/default-container semantics; custom types, raw JSON, invalid values and deep/cyclic inputs retain the authoritative JSON v2 round trip. Unrelated invalid fields still fail instead of being bypassed by a direct lookup.

Logger combines root preparation/normalization only when callbacks cannot observe the intermediate root, traverses plain slices directly and avoids unused index strings. Validation directly decodes already serialized snapshots with exact-number options, retaining bytes for diagnostic ordering.

Identical-source local Windows amd64 benchmarks (Go 1.27.1, CGO disabled, GOMAXPROCS=2, one 500ms run) show correlation 4.41→0.75 ms, JMESPath lookup 4.21→0.79 ms, large event logging 5.08→4.31 ms and large string Validation 4.76→2.96 ms. Bound INFO allocations decrease from 53 to 48; its 404-byte output and the direct JSON control remain unchanged. Full results, fixture boundaries and source hashes are in docs/PERFORMANCE_JSON_V2_250.json; these observations do not predict AWS latency.

Testing

  • CGO_ENABLED=0 uv run --no-project python tools/modules.py check --only . --only logger --only jmespath --only validation: four packaged modules pass tests/vet/tidy/isolation and four independent consumers build with GOWORK off.
  • Differential JSON v2 snapshot tests cover integer rounding, float32, nil/empty containers, negative zero, custom marshaler errors/call counts, invalid Unicode/raw numbers, duplicate members, cycles, deep inputs, ownership and concurrent copies. Existing Logger/JMESPath/Validation suites pass, including error stacks, callback keys, ordering and atomic output.
  • License notices pass for all 31 modules; JSON API policy passes for the four changed code areas. A whole-workspace local policy invocation also encountered ignored historical backup files; hosted CI uses a clean checkout.
  • An initial public-proxy consumer reproduced missing Snapshot with the released root dependency. The corrected ff134c1 consumer passes from the public proxy/checksum database with GOWORK off, no replacements and CGO disabled; it executes Logger correlation, JMESPath and the stack Preview API.
  • An initial public-proxy consumer reproduced missing Snapshot with the released root dependency. The corrected ff134c1 consumer passes from the public proxy/checksum database with GOWORK off, no replacements and CGO disabled; it executes Logger correlation, JMESPath and the stack Preview API.
  • A composed native Lambda application exercising Logger, correlation, JMESPath and Validation cross-builds as static Linux amd64/arm64 ELF binaries, CGO disabled.
  • Documentation lock, navigation, guide contracts/regressions and strict site build pass.
  • Benchmarks run once per before/after case with identical new benchmark source. No race detector, AWS test or release action.

Risks and compatibility

No public API, external dependency, parser, unsafe code or release-version change. Logger/JMESPath pin the internal root implementation to v1.2.1-0.20261015225926-e300c9596f35; tidy propagates this internal requirement to affected consumers and go.work maps it locally. No filesystem replacements are added to module files. The fast snapshot path is deliberately restricted and bounded; the existing JSON v2 path remains for unsupported representations and authoritative errors. A late unsupported member can add speculative-copy work before fallback, so the measured gains apply to the ordinary JSON-shaped fixtures rather than every Go representation. Default no-callback Logger root handling is compared against the existing formatter path, including caller-owned root cycles. Snapshot/error normalization work can remain substantial for custom or replacer-heavy inputs; generic callback behavior is preserved.

docs/CHECKLIST.md records scoped local acceptance. Broader resource budgets and unrelated runtime/service gates remain open.

Release notes

  • . | maintenance | Reuse private snapshots for ordinary JSON-shaped inputs with JSON v2 fallback.
  • logger | maintenance | Reduce correlation snapshot and log normalization allocations while preserving callbacks and output.
  • jmespath | maintenance | Avoid serialization round trips when copying ordinary JSON-shaped query inputs.
  • validation | maintenance | Decode existing JSON snapshots without redundant serialization while preserving exact numbers and diagnostics.

Checklist

  • The issue records the scope and acceptance criteria
  • Tests and documentation match the change, or I explained why they are not applicable
  • I kept CGO disabled and did not run cloud AWS tests without authorization
  • I excluded credentials, personal configuration, and generated build artifacts

@jinxiao

jinxiao commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Author review completed against e300c95: checked bounded snapshot fallback, float32/integer conversion and invalid unrelated members; caller/result isolation and concurrent copies; no-callback root cycle equivalence and preserved replacer index keys; exact-number validation decoding and original diagnostic snapshots. Local four-module packaged acceptance, static amd64/arm64 Lambda builds and documentation checks pass. No outstanding defect identified in this review; no independent approval is claimed. Hosted CI is still running.

@jinxiao

jinxiao commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up author review for ff134c1: reproduced a public-proxy build failure in the initial commit because released root v1.2.0 lacks Snapshot, then pinned the implementation commit and synchronized internal consumers/workspace metadata with tools/modules.py tidy. Four packaged modules/consumers pass again. A standalone consumer fetched ff134c1 Logger/JMESPath from the public proxy/checksum database and successfully executed correlation, query and stack Preview APIs with CGO disabled, GOWORK off and no replacements. Performance source hashes are unchanged; no redundant benchmark rerun. Also normalized PR-body line endings after the metadata editor caused the contribution-section parser to fail. Full hosted regression for the updated metadata is running.

@jinxiao

jinxiao commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Final acceptance for ff134c1: full module/independent-consumer and dual-architecture artifact checks, native Lambda/streaming/Valkey simulation, KMS interoperability, DynamoDB Local, documentation, workflow metadata, release tooling, CI gate and Full regression all passed in https://github.com/rambow-cloud/powertools-lambda-go/actions/runs/38822206771 . Latest contribution policy passed after normalizing the body format. Public-proxy/checksum consumers also pass independently with GOWORK off, CGO disabled and no replacements. The implementation and internal dependency fix are ready for review; no merge, AWS test, tag or Go release was performed.

@jinxiao
jinxiao merged commit 717dbae into main Oct 16, 2026
11 of 12 checks passed
@jinxiao
jinxiao deleted the perf/250-json-v2-snapshots branch October 16, 2026 00:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant