Skip to content

Commit

Permalink
Clarify setup of optimizer when using empty_init=True (#19067)
Browse files Browse the repository at this point in the history
  • Loading branch information
awaelchli authored Nov 26, 2023
1 parent af852ff commit b79b684
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions docs/source-fabric/advanced/model_init.rst
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,10 @@ When training sharded models with :doc:`FSDP <model_parallel/fsdp>` or DeepSpeed
model = fabric.setup(model) # parameters get sharded and initialized at once
# Make sure to create the optimizer only after the model has been set up
optimizer = torch.optim.Adam(model.parameters())
optimizer = fabric.setup_optimizers(optimizer)
.. note::
Empty-init is experimental and the behavior may change in the future.
For FSDP on PyTorch 2.1+, it is required that all user-defined modules that manage parameters implement a ``reset_parameters()`` method (all PyTorch built-in modules have this too).

0 comments on commit b79b684

Please sign in to comment.