Skip to content

feat(config): add fused and foreach optimizer controls - #613

Open
CharlesXu-HQ wants to merge 2 commits into
agentscope-ai:mainfrom
CharlesXu-HQ:feature/optimizer-backend-controls
Open

feat(config): add fused and foreach optimizer controls#613
CharlesXu-HQ wants to merge 2 commits into
agentscope-ai:mainfrom
CharlesXu-HQ:feature/optimizer-backend-controls

Conversation

@CharlesXu-HQ

Copy link
Copy Markdown

Description

Addresses the remaining optimizer-control request in #273.

Motivation

PyTorch can select the fused and foreach optimizer implementations automatically. Explicit control is useful when tuning optimizer memory usage or working around backend-specific behavior.

Changes

  • Add optional fused and foreach fields to OptimizerConfig.
  • Forward explicitly configured values to veRL's FSDP/FSDP2 override_optimizer_config.
  • Preserve backend defaults when the options are unset.
  • Keep Megatron and critic optimizer behavior unchanged.
  • Document both options in the English and Chinese configuration guides.

Testing

  • Targeted optimizer tests: 3 passed, 4 subtests passed.
  • Configuration suite on Python 3.12 / RTX 5090: 18 passed, 1 multi-node Ray test deselected, 4 subtests passed.
  • Pre-commit checks on all changed files passed.
  • CUDA AdamW smoke test with fused=True, foreach=False passed.

Checklist

Please check the following items before code is ready to be reviewed.

  • Code has passed all tests
  • Docstrings have been added/updated in Google Style (not applicable; no public callable added)
  • Documentation has been updated
  • Code is ready for review

@CharlesXu-HQ
CharlesXu-HQ marked this pull request as ready for review August 15, 2026 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant