mirror of
https://github.com/huggingface/diffusers.git
synced 2026-10-24 20:59:27 +03:00
bc5e3bd9c5
* implement Kandinsky 6 - add TI2VA and SR pipelines - additonally implement PiflowScheduler and MMAudioVAE * update docs for Kandinsky 6 * remove redundant bigvgan code and manual checkpoint loading inside model code * add missing _no_split_modules * use attention processor in SR * fix unguarded dependencies - removed: einops, pydantic - lazy: torchvision, av, librosa * remove dead manual weight loading code; flattent MMAudioVAE structure * move attention processors to transformer files * remove duplicate code in K6 latent_upscaler * address 1st batch of comments * patch transformer k6 * patch SR * patch pipeline style * replace TimestepEmbedding and fix batching * refactor ti2va dit and pipeline * refactor piflow scheduler * refactor SR * patch sr transformer autocast * major refactor * untrack notebooks directory Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * patch mmaudio VAE * patch mmaudio VAE * fix inits * fix flex attn * refactoring * remove __all__ * separate MMAudio VAE and Vocoder * fix naming * refactor checkpointing, magcache, transformer, sr vae for K6 * refactor * address apply_scale_shift_norm problem * update docs * apply style chanes * address review feedback on transformer, SR VAE, and flex attention Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * address comments * fix critical bugs * Revert checkpoint-breaking module renames from28c5326daCommit28c5326da(today, 14:03 +0300) renamed Kandinsky6FusedTransformerDecoderBlock's videoT/audioT submodules to video_dec_block/audio_dec_block, and replaced both Kandinsky6TimeEmbeddings and Kandinsky6SRTimeEmbeddings's hand-rolled in_layer/out_layer with diffusers' Timesteps/TimestepEmbedding, purely for naming clarity. No checkpoint conversion was re-run to match, so every published K6 checkpoint (unchanged since Sep 27, confirmed via identical blob SHA256 across recent Hub commits) still ships the old names. Verified against kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers and kandinskylab/Kandinsky-6.0-VSR-5s-Diffusers: this was silently dropping ~65% of the main transformer's weights (every visual_transformer_blocks.*.{video,audio}T.* parameter) and the SR transformer's time embeddings, then crashing with "Cannot copy out of meta tensor" on enable_model_cpu_offload(). After this revert, all checkpoint keys match exactly (4471/4471 for the main transformer, 459/459 for the SR transformer). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Revert checkpoint-breaking latent-upscaler renames from372f29088Same class of regression as9ae71f0d0, this time in Kandinsky6SRLatentUpscalerX2Branch and Kandinsky6SRLatentUpscalerOutputHead: today's "address comments" commit (18:48 +0300) dropped the private_ prefix from private_mid_blocks/private_upsample/private_blocks/ private_output_proj, and converted OutputHead from nn.Sequential to a plain nn.Module with named norm/activation/conv submodules, again without a matching checkpoint re-conversion. Verified against kandinskylab/Kandinsky-6.0-VSR-5s-Diffusers's latent_upscaler: this left every Kandinsky6SRLatentUpscalerX2Branch parameter (and the OutputHead-based output_proj used standalone by the x2/x4 branches) unfilled on the meta device, crashing enable_model_cpu_offload() the same way as the main transformer did before9ae71f0d0. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * move vocoder and LU * add coppied from resolve i2v cond mode * rename variables * fix lu rename bug * Update src/diffusers/pipelines/kandinsky6/pipeline_kandinsky6_ti2va.py Co-authored-by: YiYi Xu <yixu310@gmail.com> * Update src/diffusers/models/transformers/transformer_kandinsky6.py Co-authored-by: YiYi Xu <yixu310@gmail.com> * address yiyixuxu's comments * fix audio channels bug * fix tests * Fix CI failures on the Kandinsky6 PR - tests/others/test_utils.py: assert the caller file path without assuming the checkout directory is named `diffusers`. - .github/workflows/pr_tests.yml: raise PYTEST_TIMEOUT to 300 s for the example and pipeline CPU steps. Both run several workers with four threads each on an 8-vCPU runner, and the checkpointing example tests run several training subprocesses in sequence. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Avoid staging LFS dedup race in the org push test The second push in TestModelPushToHub.test_push_to_hub_in_organization sent the same model bytes as the first push. The staging server can deduplicate those bytes against the first push's LFS object and then reject the commit with "LFS pointer pointed to a file that does not exist". Change the weights before the second push so its bytes differ. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Revert "Avoid staging LFS dedup race in the org push test" This reverts commit35e6cc4ef. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Revert "Fix CI failures on the Kandinsky6 PR" This reverts commit60cad4341. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Apply suggestion from @yiyixuxu * Apply suggestion from @yiyixuxu * Apply batched suggestions from code review Co-authored-by: YiYi Xu <yixu310@gmail.com> * remove review-only files from the Kandinsky6 PR Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HCdbvRpL9fv3h3WwSUPpfS * Apply suggestion from @yiyixuxu * restore .gitignore to match main Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HCdbvRpL9fv3h3WwSUPpfS * simplify the mono audio handling in _write_audio Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HCdbvRpL9fv3h3WwSUPpfS * remove the no-op set_attention_backend calls from the Kandinsky6 docs Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HCdbvRpL9fv3h3WwSUPpfS * give the Kandinsky6 transformer its own output class instead of reusing LTX-2's Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HCdbvRpL9fv3h3WwSUPpfS * fix the stale pipeline exports and add return annotations for the Kandinsky6 pipelines Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HCdbvRpL9fv3h3WwSUPpfS * address feedback from @leffff --------- Co-authored-by: Denis Koposov <denis.koposov@phystech.edu> Co-authored-by: leffff <levnovitskiy@gmail.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Lev Novitskiy <57654885+leffff@users.noreply.github.com> Co-authored-by: YiYi Xu <yixu310@gmail.com>