[BUG] Improper coupling of paramter list between DeepSpeedZeroOptimizer_Stage3 and SuperOffloadOptimizer_Stage3

[delock]

Describe the bug
DeepSpeedZeroOptimizer_Stage3 and SuperOffloadOptimizer_Stage3 shares same parameter list, which would cause divergence easily

** Details **
In

DeepSpeed/deepspeed/runtime/engine.py

Line 1846 in b7cd78f

Stage3ZeroOptimizer = DeepSpeedZeroOptimizer_Stage3 if not self.super_offload(

, DeepSpeedZeroOptimizer_Stage3 and SuperOffloadOptimizer_Stage3 initializer shares same parameter list. This caused extra maintence if any one of these parameter list needs to change. There are two observations:

1. There is already mismatch (i.e. param_names) and this will break SuperOffload.
2. cpuadam_cores_perc added to DeepSpeedZeroOptimizer_Stage3 as parameter but not used.

** Suggestion **
Seperate calls to DeepSpeedZeroOptimizer_Stage3 and SuperOffloadOptimizer_Stage3.

[xylian86]

@delock Thank you for the suggestions.

In the SuperOffload PR, I tried to align the style with the Zenflow to maintain consistency. In Zenflow, both DeepSpeedZeroOptimizer and ZenFlowZeroOptimizer share the same parameter list in a similar manner — as seen here:

DeepSpeed/deepspeed/runtime/engine.py

Line 1771 in b7cd78f

Stage1And2ZeroOptimizer = DeepSpeedZeroOptimizer if not self.zenflow else ZenFlowZeroOptimizer.create(

That said, I’m totally fine with making separate calls for SuperOffload and Zenflow. I can prepare a PR to implement that.