user avatar
accelerate deepspeed and gradient accumulation integrate (#23236)
Sourab Mangrulkar authored
* mixed precision support via accelerate

* fix issues

* fix for the sharded ddp case

* fix flax and tf failing tests

* `refactor the place to create `Accelerator` object

* move ddp prep to accelerate

* fix :sweat_smile:

* resolving comments

* move fsdp handling to accelerate

* fixex

* fix saving

* shift torch dynamo handling to accelerate

* shift deepspeed integration and save & load utils to accelerate

* fix accelerate launcher support

* oops

* fix :bug:

* save ckpt fix

* Trigger CI

* nasty :bug: :sweat_smile:

* as deepspeed needs grad_acc fixes, transfer grad_acc to accelerate

* make tests happy

* quality :sparkles:

* loss tracked needs to account for grad_acc

* fixing the deepspeed tests

* quality :sparkles:

* :sweat_smile::sweat_smile::sweat_smile:

* tests :rage:

* quality :sparkles:



* Trigger CI

* resolve comments and fix the issue with the previous merge from branch

* Trigger CI

* accelerate took over deepspeed integration

---------

Co-authored-by: default avatarStas Bekman <stas@stason.org>
a73b1d59
Name Last commit Last update
..