From 2afb2e0aacdd59ffa01d30d0da645ea1b5310a11 Mon Sep 17 00:00:00 2001 From: RandomGamingDev <83996185+RandomGamingDev@users.noreply.github.com> Date: Fri, 26 Jul 2024 00:05:52 -0400 Subject: [PATCH] Added `accelerator` based gradient accumulation for basic_example (#8966) added accelerator based gradient accumulation for basic_example Co-authored-by: Sayak Paul --- docs/source/en/tutorials/basic_training.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/en/tutorials/basic_training.md b/docs/source/en/tutorials/basic_training.md index 62b94de3a3..402c8c59b1 100644 --- a/docs/source/en/tutorials/basic_training.md +++ b/docs/source/en/tutorials/basic_training.md @@ -340,8 +340,8 @@ Now you can wrap all these components together in a training loop with 🤗 Acce ... loss = F.mse_loss(noise_pred, noise) ... accelerator.backward(loss) -... if (step + 1) % config.gradient_accumulation_steps == 0: -... accelerator.clip_grad_norm_(model.parameters(), 1.0) +... if accelerator.sync_gradients: +... accelerator.clip_grad_norm_(model.parameters(), 1.0) ... optimizer.step() ... lr_scheduler.step() ... optimizer.zero_grad()