fix: avoid duplicate optimizer step with static capture - #1930
fix: avoid duplicate optimizer step with static capture#1930SOlo-travelr wants to merge 2 commits into
Conversation
Signed-off-by: SOlo-travelr <kashyap.60@buckeyemail.osu.edu>
CODEOWNERS review mapCurrent for commit ⏳ @dallasfoster — 2 file(s)
⏳ @laserkelvin — 2 file(s)
Comment |
Greptile SummaryThis PR prevents the active-learning loop from issuing a second optimizer update when static capture already owns backward propagation and optimizer stepping.
Important Files Changed
Reviews (1): Last reviewed commit: "fix: avoid duplicate optimizer step with..." | Re-trigger Greptile |
| if not self.enable_static_capture: | ||
| loss.backward() | ||
| optimizer.step() | ||
| optimizer.step() |
There was a problem hiding this comment.
Scheduler advances past skipped updates
When CUDA float16 static capture encounters non-finite gradients, GradScaler.step suppresses the optimizer update but the outer loop still advances lr_scheduler, causing subsequent successful updates to use a learning-rate schedule that is ahead of the actual parameter updates.
Knowledge Base Used: Active Learning Module
PhysicsNeMo Pull Request
Description
Checklist
Dependencies
Review Process
All PRs are reviewed by the PhysicsNeMo team before merging.
Depending on which files are changed, GitHub may automatically assign a maintainer for review.
We are also testing AI-based code review tools (e.g., Greptile), which may add automated comments with a confidence score.
This score reflects the AI’s assessment of merge readiness and is not a qualitative judgment of your work, nor is
it an indication that the PR will be accepted / rejected.
AI-generated feedback should be reviewed critically for usefulness.
You are not required to respond to every AI comment, but they are intended to help both authors and reviewers.
Please react to Greptile comments with 👍 or 👎 to provide feedback on their accuracy.