mirror of https://github.com/huggingface/diffusers.git synced 2026-10-24 20:59:27 +03:00

Files

Buffett Liu 360bef8074 Feat -- Stable Audio 3 (#14119 )

* Stable Audio 3 First Version Commit, require model dict loading on GPUs

* Fixing bugs for sa3 VAE

* Make quality fix

* Fix the unit test failure

* Fix SA3 pipeline noise: learned text-padding embedding + no cross-attn mask

The diffusers Stable Audio 3 pipeline produced noise instead of music because
the cross-attention conditioning was built differently from the reference:

- The reference text conditioner uses padding_mode="learned": padded text
  positions (~245 of 256 for a short prompt) are filled with a trained
  `padding_embedding`, and the DiT attends to *all* positions (its
  cross-attention mask is intentionally disabled). Our pipeline instead zeroed
  padded positions and masked them out, wiping ~95% of the conditioning signal.

Changes:
- transformer_stable_audio3: add learned `prompt_padding_embedding`; in forward,
  replace masked text positions with it (in cond_token_dim space, before
  to_cond_embed) then attend to the full context, matching the reference.
- pipeline_stable_audio_3: stop zeroing padded positions (the DiT now handles
  them); default `silence_padding_duration` to 6.0 (reference headroom default).
- convert_..._to_diffusers: convert
  `conditioner.conditioners.prompt.padding_embedding` into the DiT.
- scheduling_ping_pong: pin schedule endpoints (sigmas[0]=1.0, sigmas[-1]=0.0)
  to match the reference LogSNRShift endpoint preservation.
- run_..._inference: coerce float16->float32 on CPU.
- tests: update transformer state-dict expectations (523 tensors, add
  prompt_padding_embedding) and ping-pong sigma endpoint assertions.

Verified against the reference with identical noise + identical conditioning:
full 8-step ping-pong trajectories agree to 4e-6 (final latent). Removed the
dev-only parity scripts (verify_*_dit_parity, verify_*_vae_parity) from the tree.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Style fix

* Autopipeline available

* Adding document description for testing examples.

* Solving fix for serge review bot suggestion

* Design review comments fix

* CI/CD Fix for ping pong scheduler md file

* Fix doc build: remove autodoc for nonexistent StableAudio3EulerSchedulerOutput

* Fix doc build: qualify non-exported autodoc references with module paths

* Fix details in scheduler and some nits

* suggestion fix

* fix-copies: propagate prepare_latents dtype fix to audio2audio/inpaint pipelines

---------

Co-authored-by: Buffett Liu <jengliu@linkedin.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>

2026-08-18 22:09:32 -07:00

1.6 KiB

Raw Permalink Blame History

AutoencoderSAME

The SAME (Semantically-Aligned Music Encoder) autoencoder is used by Stable Audio 3 to compress stereo audio waveforms into a compact latent sequence and reconstruct them.

The encoder stacks [SAMETransformerResamplingBlock] modules, each of which groups a fixed number of audio patch frames and produces one learnable output token via a differential transformer. The decoder inverts this process, expanding each latent token back to a patch of audio frames.

A soft-norm bottleneck (SoftNormBottleneck) normalises latents before and after the diffusion model, providing stable training dynamics.