Patch release: v5.20.2

This patch notably solves an issue where some inputs could loose their original type inside prepare_inputs_for_generation in generate, leading to problems with multimodal models. It also fixes some other issues listed on the below commits.

It contains the following commits:

• Fix inputs dict casting in generate (#49731) by @Cyrilvallez
• Fix zero expert gradients in bfloat16 with expert parallelism and ep_size < fsdp_size (#49688) by @qgallouedec
• Declare the input modalities of the VLMs that report text only (#49677) by @qgallouedec
• Avoid a host-to-device copy in eager_mask (#49684) by @hmellor

Release v5.20.1

Special surprise including the new model Beam from Reflection AI 🥳

Beam is a token-efficient, frontier-scale model focused on coding and agentic use cases. It is built for teams that want to run a powerful model on their own hardware, infrastructure, or cloud, and customize it to their specific needs.

Links: Documentation

• Fix small bug in auto mappings (#49664) by @yonigozlan in #49664

Some real patch fixes

We broke loading models on older torch versions 😢 should be fixed now. But here is the full list of fixes

• Fix HunYuan-VL image assignment during batched generation (#49557) by @Arnav66692
• Fix DeepGEMM Mega MoE under expert parallelism(#49428) by @IlyasMoutawwakil
• Fix from_pretrained on torch 2.7: require torch>=2.8 for distributed checkpointing (#49652) by @naveecode
• Make DtensorShardOperation work on torch 2.6 and 2.7 (#49657) by @hazelxue

Full Changelog: v5.20.0...v5.20.1

Release v5.20.0

New Model additions

Muse Spark Open

Muse Spark Open is a multimodal mixture-of-experts model from Meta that takes text, images, audio, and video as input and generates text. It has 816B total parameters (42B active per token) and a context window of up to 1,048,576 tokens shared across modalities. It is trained for agentic coding tasks such as multi-file editing, debugging, and tool use. Checkpoints are available in BF16 and NVFP4 formats.

Links: Documentation | Blog Post

• Add Muse Spark Open (#49644) by @guarin in #49644

Ling 3.0

Ling 3.0 Flash is a hybrid Mixture-of-Experts language model from InclusionAI. Its decoder groups five Kimi Delta Attention (KDA) layers with one Multi-head Latent Attention (MLA) layer. The MLA blocks use interleaved rotary embeddings and a head-wise output gate, and the feed-forward blocks combine routed and shared experts. It is natively supported in Transformers as BailingHybridForCausalLM, so it loads without trust_remote_code=True.

Links: Documentation

• [Model] Add native Ling 3.0 Flash support (#49446) by @icavan in #49446

Kolibri1

Kolibri 1 is Aleph Alpha's mixture-of-experts reasoning model, focused on German and English. Each of its 50 layers routes every token to 6 of 384 experts using sigmoid routing, and adds one shared expert. Four sliding-window attention layers alternate with one full-attention layer, and RoPE is applied only in the sliding-window layers.

Links: Documentation

• Add Kolibri 1 (#49463) by @lucamotz in #49463

MiniCPM-V 4.7

MiniCPM-V is a series of efficient multimodal large language models developed by OpenBMB. Like MiniCPM-V 4.6, the MiniCPM-V 4.7 architecture pairs a SigLIP vision encoder that has a window-attention merger with a Qwen3.5 language model backbone, and supports both 4x and 16x visual downsampling modes.

The main addition over 4.6 is canvas M-RoPE: instead of numbering visual tokens along a single 1-D sequence, the model lays every image out on a 2-D canvas and assigns each visual token a (temporal, height, width) position, so slices of the same image keep their spatial relationship and video frames keep their temporal order.

• Fix more stuff (#48979) by @zucchini-nlp in #48979

UnlimitedOcr

UnlimitedOcr is a single 3B-parameter OCR model from Baidu with a standard 32,768-token context length. It builds on DeepSeek OCR and replaces all decoder attention layers with Reference Sliding Window Attention (R-SWA). This keeps the KV cache constant during decoding, so the model can transcribe dozens of pages in a single forward pass. It supports "gundam" mode (640x640 tiles with dynamic cropping) and "base" mode (a single 1024x1024 view), and it can output region detections with bounding boxes.

Links: Documentation | Paper

• Add Unlimited OCR (#46836) by @guarin in #46836

UltraBERT

UltraBERT is a family of bidirectional text encoders trained with a combination of masking ratios, pruning, and distillation for classification and retrieval tasks. The released encoders support sequences of up to 32,768 tokens. They use rotary position embeddings, RMSNorm, SwiGLU feed-forward layers, and alternating local and global attention (every third layer is global). Checkpoints include UltraBERT-150M and UltraBERT-380M for masked language modeling and fine-tuning, and UltraBERT-Embed-150M for text embeddings.

Links: Documentation | Paper

• Integrate the UltraBERT architecture (#49596) by @tomaarsen in #49596

MOSS-Transcribe-Diarize

MOSS-Transcribe-Diarize 0.9B is an end-to-end audio understanding model for long-form multi-speaker transcription, speaker diarization, and timestamps. It combines a Whisper-style encoder, a 4× frame merge step, a multi-modal projector, and a Qwen3 language model. It produces joint transcription and diarization in a single pass, with segments formatted as [start][S01]text[end]. It supports long-form audio through Whisper-window chunking in the processor, promptable transcription with hotword biasing, and 50+ languages.

Links: Documentation

• Add MOSS-Transcribe-Diarize model (#48386) by @itazap in #48386

Breaking changes

🚨 General breaking notice that inference with EP has regressions in speed; we are working on fixing this ASAP! 🚨

Deprecated checks and methods have been removed from cache_utils.py (affecting models such as minimax_m3_vl and pi0), so any code still calling those deprecated cache methods must switch to the current cache API.

• 🚨 Remove deprecated cache utils methods (#49605) by @guarin

Accidental imports from other models were removed from several model files (e.g. embedding_gemma2, ijepa, lightglue), and the modular converter now avoids adding such cross-model imports. If you imported these names through those model modules, import them from their original modules instead.

• 🚨 Remove imports from other models (#49473) by @guarin

The minimum kernels version is now 0.17.2, which allows use_fallback=KernelizeFallback.ALL so that a failed kernel load falls back to the original implementation instead of raising an error. Upgrade kernels to 0.17.2 or later.

• 🚨 Bump min kernels to 0.17.2 for better fallback (#49235) by @SunMarc

A new SubConfigSpec centralizes the initialization of sub-configs, so default sub-configs are collected from dataclass fields instead of by instantiating the class, and this touches the configuration utilities and many model configs. Custom or existing nested configs may need to be updated to declare their sub-configs through this mechanism.

• 🚨 SubConfigSpec for centralized init of subconfigs (#48282) by @zucchini-nlp

The LUKE and mLUKE tokenizers now treat entity special tokens such as <ent> as special tokens (not split, honored by skip_special_tokens, normalized) while keeping the same spacing as regular vocabulary tokens when decoding. Tokenization and decoding output involving these tokens may therefore differ from before, so check any code or saved outputs that depend on the old behavior.

• 🚨🚨🚨 luke fixe (#48408) by @itazap

Exporters

Transformers exporters now support quantization directly during export across Dynamo, ONNX, OpenVINO, and ExecuTorch. The new unified quantization API integrates PT2E, ONNX Runtime, and NNCF quantizers, enabling static and dynamic quantization as well as weight-only compression (INT8/INT4), with configurable calibration workflows. This makes it easier to export optimized models for efficient inference across different backends.

For more information see the docs: https://huggingface.co/docs/transformers/exporters_quantization

Parallelization

This release improves distributed training and inference. FSDP × TP model and optimizer checkpoints can now be saved and loaded across different parallel configurations, and expert-parallel dispatch, tensor-parallel plans, static-cache generate(), and the caching allocator's byte count were fixed. Tied LM heads in composite models now get their own TP plan entry, enable_expert_parallel applies the EP plan alone again, TPU loading uses the `tor

• FSDP x TP Saving / Loading, Optimizer (#49256) by @michaelbenayoun in [#49256]
• FSDP x TP Saving / Loading (#48802) by @michaelbenayoun in [#48802]
• Don't set a device per rank for gloo tensor-parallel tests (#49636) by @IlyasMoutawwakil in [#49636]
• fix(distributed): keep enable_expert_parallel as EP alone (#49633) by @dacorvo in [#49633]
• Fix a port allocation race in FSDP tests (#49455) by @icavan in [#49455]
• Expert-parallel dispatch hands the experts the received rows, not the all-to-all's pending result (#49421) by @IlyasMoutawwakil in [#49421]
• Fix static-cache generate() with host-side inputs and under TP (#49317) by @dacorvo in [#49317]
• Composite lm head tp plan (#49080) by @IlyasMoutawwakil in [#49080]
• Fix: use torch.accelerator device APIs so TP loading works again on TPU (#49502) by @tengomucho in [#49502]
• Count ep and fsdp in caching_allocator_warmup byte calculation (#49384) by @3outeille in [#49384]
• Do not wrap the model in FSDP for expert-parallel token dispatch (#49383) by @3outeille in [#49383]

Generation

This release fixes static-cache generate() bugs with host-side inputs and under tensor parallelism. It also speeds up Whisper generation on TPU by cutting device syncs and recompilation, and fixes Whisper batched timestamp decoding for tensor and NumPy inputs. Pipelines get a cleanup of generation-argument handling that removes spurious warnings.

...

Release v5.19.0

New Model additions

EmbeddingGemma2

EmbeddingGemma 2 is a multimodal embedding model from Google built on the Gemma 4 architecture. It encodes text, images, audio, and video, individually or combined in one input, into a shared 768-dimensional vector space for cross-modal retrieval, semantic similarity, clustering, and classification. It uses Matryoshka Representation Learning, so embeddings can be truncated to 512, 256, or 128 dimensions. It also offers configurable visual and video token budgets, and unused vision or audio towers can be disabled at load time to save memory.

Links: Documentation

• Smthn smthn (#49364) by @vasqu in #49364

Breaking changes

All MoE models whose routers compute logits now return them when output_router_logits=True, following the Qwen3-MoE pattern (a router_logits recorder on the base model, MoeModelOutputWithPast from the backbone, and a MoE causal LM output from the head), so code that relied on the previous outputs or their absence should read the router logits from these output classes.

• 🚨 Return router logits from every MoE model that computes them (#48920) by @qgallouedec

Owlv2ForObjectDetection.embed_image_query now selects the query box with the highest objectness score, as in the original OWLv2 notebook, instead of the OWL-ViT heuristic, so image-guided query embeddings and detections may differ from earlier releases.

• 🚨 Select the OWLv2 image query by objectness (#49200) by @qgallouedec

The "paged|" prefix for SDPA and flash attention implementations is deprecated, so users should set the regular attention implementation (e.g. sdpa or flash_attention_2) for continuous batching instead of paged|sdpa or paged|flash_attention_2.

• 🚨 Attention 🚨 Deprecate "paged|" prefix for SDPA and flash (#49112) by @remi-or

The regular flash and SDPA attention functions (flash_attention.py, sdpa_attention.py) now support continuous batching directly, and "paged|..." implementations for these are redirected to them, while eager still requires the "paged|eager" prefix.

• 🚨 Attention 🚨 Make regular attention support CB (#49101) by @remi-or

In continuous batching, the cache update for the index-based and block-table paths is now fused into a single call, which slightly changes the cache update function's behavior and affects any custom code that calls the separate update paths.

• 🚨 [CB] 🚨 Fuse update for index and block table path (#49088) by @remi-or

Continuous batching internals changed in preparation for removing "paged": `max

• 🚨 [CB] 🚨 Little fixes before removing "paged" (#49069) by @remi-or

Parallelization

Expert parallelism gains a token-dispatch implementation, selected via the new ep_dispatch_experts plan rule and now the default for Qwen3 MoE and Mellum, which removes the requirement that EP size equal TP size. The Trainer was also adapted to work with expert parallelism, and the docs now note that PEFT adapters support tensor parallelism. A CI-related fix for pipeline-parallel chart2table inference was also included.

• [distributed]: adapt Trainer to work Expert Parallel (#48873) by @3outeille in [#48873]
• [distributed] Add expert-parallel token dispatch, default for Qwen3 MoE (#48865) by @3outeille in [#48865]
• Fix pp chart2table inference (#49225) by @zucchini-nlp in [#49225]
• [docs] TP for PEFT adapters (#49060) by @stevhliu in [#49060]

Cache

This release fixes quantized cache handling: generate no longer mutates the user's cache_config, and QuantizedLayer.reorder_cache is repaired. It also adds per-layer cache configuration, so DynamicCache and StaticCache initialize each layer from its own config (sliding window, attention chunk size, conv states, and attention head counts) to better support heterogeneous models. Separately, the deprecation cycle on mask and cache

• Fix quantized cache (#48700) by @jiqing-feng in [#48700]
• [CI] check_bad_commit: use EFS cache to avoid Xet FUSE OOM (exit 137) (#49273) by @ydshieh in [#49273]
• Support per-layer cache configuration (#48178) by @eladsegal in [#48178]
• Remove deprecation cycle on mask and cache (#49231) by @Cyrilvallez in [#49231]

Bugfixes and improvements

• Revert "[CI] Temporarily disable AMD scheduled CI trigger" (#49271) (#49362) by @ydshieh in [#49362]
• ALM testx fIxing... (#49355) by @zucchini-nlp in [#49355]
• Fix DataCollatorForLanguageModeling ignoring seed=0 (#49349) by @akanyaani in [#49349]
• Fix explicit kernelization modes for evaluation models (#49339) by @DimensionSTP in [#49339]
• fix(cohere_compass): add video to input_modalities (#49320) (#49323) by @destopianpirate in [#49323]
• GPT-OSS: read the clamped SwiGLU's alpha and limit from the config (#49346) by @IlyasMoutawwakil in [#49346]
• [distributed] Default MoE ep_plans to token dispatch (#49160) by @3outeille in [#49160]
• Move CI to Python 3.11, with the version set in .python-version (#49311) by @tarekziade in [#49311]
• Bump doc-builder main docs workflow pin (#49350) by @paulinebm in [#49350]
• [distributed] decoupled tp_plan and ep_plan (#48859) by @3outeille in [#48859]
• [distributed] Add dense and expert device mesh (#48857) by @3outeille in [#48857]
• Bump doc-builder workflow pin (#49342) by @paulinebm in [#49342]
• Allow registering new processing backends (#48984) by @zucchini-nlp in [#48984]
• Fix WatermarkDetector repeated-ngram counting (#49315) by @LE0-Lin in [#49315]
• Fix minicpm 4.6V video batching and CI (#49259) by @zucchini-nlp in [#49259]
• remove colwise_gather_output for lm_head for DeepseekV4 (#49314) by @3outeille in [#49314]
• Fix the labels docstring of CLIPSegForImageSegmentation (#49001) by @qgallouedec in [#49001]
• [CI] Temporarily disable AMD scheduled CI trigger (#49271) by @ydshieh in [#49271]
• Remove the unused pandas entry from setup.py deps (#49329) by @tarekziade in [#49329]
• Fix run_glue.py label ids becoming strings with pandas>=3 (#49326) by @tarekziade in [#49326]
• [Fix] Remove old test file with two ancient tests (#49280) by @remi-or in [#49280]
• Fix chunked prefill with multi-axis position ids (Qwen3-VL) (#49319) by @dacorvo in [#49319]
• Why override when you can fix in Mixin (#49105) by @zucchini-nlp in [#49105]
• Do RoPE with mul instead of matmul (#49265) by @Rocketknight1 in [#49265]
• Deprecated pipelines should redirect; removed pipelines should raise (#49308) by @LysandreJik in [#49308]
• Enable model test on upcoming torch_tpu backend (#49207) by @tengomucho in [#49207]
• Fix encoder repetition penalty source rows after batch expansion (#49262) by @gitedmond in [#49262]
• Preserve masked EOS scores in exponential decay length penalty (#49260) by @gitedmond in [#49260]
• [CI] Avoid fetching all artifacts when only specific ones are needed (#49282) by @ydshieh in [#49282]
• Revert "[CI] ssh-runner: add optional cache_type input to switch between bucket and EFS runners (#49159)" (#49270) by @ydshieh in [#49270]
• Add Trainer.loss_is_scaled_for_ga to declare whether compute_loss already scales for gradient accumulation (#49240) by @qgallouedec in [#49240]
• fix precomputed flash kwargs clashing (#47928) by @IlyasMoutawwakil in [#47928]
• Reapply modular examples (#49230) by @Cyrilvallez in [#49230]
• Remove some stuff that does not exist (#49232) by @Cyrilvallez in [#49232]
• Remove rotaries deprecation cycle (#49228) by @Cyrilvallez in [#49228]
• [GTE] Add memory mixin (#49234) by @vasqu in [#49234]
• [CB] Make CB more device agnostic and add support for XPU (#49156) by @remi-or in [#49156]
• correct final metrics due to resuming from checkpoint (#49208) by @SunMarc in [#49208]
• [ESM/Persimmon/NllbMoe/ESMFold] use safetensors repos to fix Xet FUSE SIGBUS/IOError on bucket CI runners (#49194) by @ydshieh in [#49194]
• Dev update (#49212) by @vasqu in [#49212]

Significant community contributions

The following contributors have made significant changes to the library over the last release:

• @vasqu
  • Smthn smthn (#49364)
  • [GTE] Add memory mixin (#49234)
  • Dev update (#49212)
• @remi-or
  • [Fix] Remove old test file with two ancient tests (#49280)
  • [CB] Make CB more device agnostic and add support for XPU (#49156)
  • 🚨 Attention 🚨 Deprecate "paged|" prefix for SDPA and flash (#49112)
  • 🚨 Attention 🚨 Make regular attention support CB (#49101)
  • 🚨 [CB] 🚨 Fuse update for index and block table path (#49088)
  • [Refactor] Make some flash-attention utils more readable (#49071)
  • 🚨 [CB] 🚨 Little fixes before removing "paged" (#49069)

New Model additions

Nemotron 3 Diarization

Nemotron 3 Diarization is an open-weight streaming speaker diarization model designed to determine "who spoke when" in real-world audio. It supports both streaming and offline inference, handles up to eight speakers, and orders speaker outputs by each speaker's first arrival in the input audio.

The model uses the Arrival-Order Speaker Cache (AOSC) 1 and FIFO queue introduced for Streaming Sortformer 1, 2. A single checkpoint supports configurable latency profiles, from an 80 ms input buffer to a 30.4 s offline-style buffer, and configurable output frame resolution in multiples of 10 ms. With chunked inference, the maximum audio duration is not limited.

Links: Documentation

• Add Nemotron3Diarization (#49056) by @eustlb in #49056

NemotronH Omni

NemotronH Omni is a multimodal reasoning model from NVIDIA that pairs the NemotronH hybrid
Mamba-Transformer language model with a RADIO vision encoder and an optional Parakeet-based sound encoder.
Image (and video) patches are projected through a RADIO tower and a pixel-shuffle MLP into the language model's
embedding space at the <image> / <video> context-token positions; audio clips are projected in the same way at
<audio> positions. The result is a single autoregressive model that reasons jointly over text, images, video and
sound.

Links: Documentation

• Add support for Nemotron Omni (#46509) by @meatybobby in #46509

HyperCLOVAX Vision V2

HyperCLOVAX Vision V2 is a multimodal vision-language model developed by NAVER. It combines the HyperClovaX language model backbone with a Qwen2.5-VL vision encoder. The model supports text, image, and video inputs and is capable of chain-of-thought reasoning via built-in thinking tokens (<think>...</think>).

Links: Documentation

• add HyperClovaX Vision (#44314) by @jp1924 in #44314

GTE

GTE was proposed in mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval by Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li and Min Zhang.

GTE is a BERT-style bidirectional encoder that replaces absolute position embeddings with RoPE, uses a gated MLP, and applies layer normalization after each residual connection. The same architecture backs Alibaba's gte-*-v1.5, gte-multilingual-* and gte-en-mlm-* checkpoints as well as Snowflake's snowflake-arctic-embed-m-v2.0.

Links: Documentation

• model: Add GTE to Transformers (#48416) by @harshaljanjani in #48416

Breaking changes

• 🚨 [ROCm] gpt-oss: route FA3 to aiter-flash-attn, generate ROCm fixtures (#46837) by @Abdennacer-Badaoui
• 🚨 Speed up detr image processing (#48066) by @guarin
• 🚨 Remap indexers layer_type (#48974) by @Cyrilvallez
• 🚨 [vLLM] Fix video token counting for Transformers backend video inputs (Part 1) (#48894) by @harshaljanjani
• 🚨 🚨Bring some dinos to modern standards (#46266) by @molbap
• 🚨 [Kernels] Bump version (#48714) by @vasqu

Bugfixes and improvements

• fix incorrect hub tokenizer class (#48641) by @itazap
• [CI] Deduplicate Nvidia/AMD CI reply comments and add headers (#48655) by @ydshieh
• Update dev (#48654) by @vasqu
• [tests] Fix NougatModelIntegrationTest: pin artifact revision and update golden values (#48638) by @ydshieh
• [fix] Fix GlmOcr integration tests: wrong token IDs and image token decode bug (#48650) by @ydshieh
• [GLM 5.3 Flash] Preserve original names when saving text checkpoints (#48676) by @Dovis01
• Register activation kernel layers on XPU (#47858) by @jiqing-feng
• Mirror the registered pytree flatten when building automatic dynamic shapes (#48578) by @IlyasMoutawwakil
• Fix AutoImageProcessor requiring torchvision when only Pillow is installed (#48616) by @blipbyte
• Assert cached decode matches recomputing without a cache (#48289) by @IlyasMoutawwakil
• [higgs_audio_v2] Fix: use config.num_codebooks in audio labels tensor (Simple Fix!) (#48560) by @b-re-w
• [MoE] Fix eager EP (#48653) by @vasqu
• [MusicgenMelody] Fix conditioning silently dropped at generation step 0 (#48679) by @ydshieh
• Fix deepspeed ci (#48640) by @SunMarc
• Synthetic test assets (#48589) by @tarekziade
• Skip the expert-parallel sentinel masking when expert parallelism is off (#48201) by @qgallouedec
• [Fix] yolos offload issue (#48688) by @molbap
• [serge] Fix 2 integration tests for model minimax failing with output_mismatch (tensor values differ (2)) (#48515) by @sergereview[bot]
• [serge] Fix 2 integration tests for model mistral failing with other (other (2)) (#48429) by @sergereview[bot]
• Fix Qwen2.5-VL temporal RoPE for fractional video intervals (#48669) by @yeyeyeping
• Fix slow integration tests on XPU (#48611) by @jiqing-feng
• QA: Added a MemoryCleanupMixin class for tests (#48681) by @tarekziade
• [serge] Fix 1 integration test for model flex_olmo failing with other (other (1)) (#48668) by @sergereview[bot]
• Fix D-FINE / RT-DETR main loss being computed over the denoising queries (#48528) by @stefan-it
• [CI] Replace hardcoded username allowlists with author_association check in workflow triggers (#48712) by @ydshieh
• Add flash_attention_4 in attn_implementation AutoModel docstring (#48684) by @3manifold
• Keep the expert-parallel sentinel slots out of the router gradient (#48689) by @qgallouedec
• [DeepseekV3] OOM cascade root-cause investigation (generator ref leak in conversion_mapping) (#48720) by @ydshieh
• Use huggingface_hub httpx export (#48685) by @Wauplin
• unifying device_mesh init to enable PP + TP inference (#48155) by @3outeille
• Enable FSDP2 + expert parallelism via a 2-D (fsdp, tp) device mesh (#48516) by @qgallouedec
• Switch daily CI to torch 2.14 — update expected outputs (#48750) by @ydshieh
• fix videomae load error (#48675) by @sywangyi
• Fix silently random-initializing RTDetrModel/SEWDForCTC loads (wrong base_model_prefix) (#48744) by @
• Fix Glm4vMoeIntegrationTest: offload_folder + MemoryCleanupMixin (#48776) by @ydshieh
• [InternVL] Normalize num_patches before np.cumsum (#48469) by @lorenzozanee
• Kernel api doc, part 2 (#46889) by @michaelbenayoun
• Fix hidden state selection on Gemma4 assistant's first prefill step (#48704) by @glistening
• Fix inkling embedding norm (#48786) by @Cyrilvallez
• Fix TextToAudioPipeline crash for tokenizer-only models (#48505) by @jiqing-feng
• Cast pixel values to the patch embedding dtype in DeepSeek-OCR-2 (#48632) by @jiqing-feng
• add kernel mapping entry for RMSNormGated, KDA, Conv1D on XPU (#48702) by @kaixuanliu
• Bump peft version requirement (#48716) by @shniubobo
• [serge] Fix 12 integration tests for model edgetam failing with import_or_config (other (12)) (#48322) by @sergereview[bot]
• Restore legacy tensor-parallel initialization for compatibility (#48797) by @3outeille
• Fix kosmos flaky test (#48800) by @IlyasMoutawwakil
• Add integration tests for MuseGlimmerAssistantModel (#48796) by @ydshieh
• Fix AutoModel.from_pretrained not restoring modules_to_save weights (#48595) by @shniubobo
• [docs] mrope and axial rope (#48717) by @stevhliu
• Fix decoder only path for older bert variants (#48785) by @Cyrilvallez
• [Fix] Clean-up ternaries in the DeepSeek family (#48447) by @remi-or
• [utils] Add MUSA support for Flash Attention 2 (#48612) by @XiaomingFun233
• [generate] Drop attention mask early without padding (#48814) by @Cyrilvallez
• Pin utf-8 in test_can_init_all_missing_weights source read (Windows non-UTF-8 locale fix) (#48819) by @dltsum
• Relax static-cache tolerance in test_generate_with_static_cache (1e-5 → 5e-5) (#48815) by @ydshieh
• Detect nested rope_parameters without relying on layer_types (#48798) by @hmellor
• Support for MoE in the GGUF integration (#48529) by @SunMarc
• [docs] gguf (#46357) by @stevhliu
• [apply_chat_template] pass sampling_rate to call (#48794) by @eustlb
• [CI] Add link checker (#48160) by @stevhliu
• Qwen3.8 GGUF (#48660) by @SunMarc
• docs: fix broken #combining-with-fsdp2 anchor in expert_parallelism.md (#48854) by @ydshieh
• [GPTNeoXJapanese] Fix RoPE ignoring partial_rotary_factor (#48652) by @blipbyte
• QA: deactivate rule 41 (#48852) by @tarekziade
• Fix reset on the dynamic cache layers (#48809) by @jiqing-feng
• [generate] Make all methods and logits processors agnostic to lm_head output size (#48846) by @Cyrilvallez
• Fix kosmos (#48853) by @Cyrilvallez
• Add test for causal only variant of some encoder-decoder models (#48760) by @nandan2003
• Fix ESMFold2 ligand iPLDDT weighting: mol_type non-polymer code is 3, not 4 (#48831) by @faustomilletari
• Fix flaky RfDetr test_sav...

Release v5.17.0

New Model additions

HYV4

Hy4-Preview is a 780B-parameter mixture-of-experts language model that activates 49B parameters per
token. Each MoE layer holds 256 routed experts plus one always-active shared expert and routes every
token to 8 of them. The context window is 1M tokens.

The architecture combines four features:

• Multi-head Latent Attention (MLA) compresses keys and values into a low-rank latent
  (kv_lora_rank) that kv_b_proj expands back to one key/value per query head.
• DeepSeek Sparse Attention (DSA) selects index_topk keys per query with a lightweight indexer.
  Following IndexShare, only the layers marked "full"
  in indexer_types run an indexer; "shared" layers reuse the previous full layer's selection.
• Gated MLA with learnable attention sinks, where each head owns a sink logit that participates
  in the softmax and contributes no value, as in GPT-OSS.
• Independent Hyper-Connections (iHC) replace the plain residual path with hc_mult parallel
  residual streams that are collapsed before, and redistributed after, every sublayer.

The implementation does not execute the multi-token prediction (MTP) layers. Released checkpoints
keep those weights so that other runtimes can use them for speculative decoding; they are ignored
at load time.

Links: Documentation

• Add h4 (#48473) by @ArthurZucker in #48473

VibeVoice

VibeVoice is a novel framework for synthesizing high-fidelity, long-form speech with multiple speakers by employing a next-token diffusion approach within a Large Language Model (LLM) structure. It's designed to capture the authentic conversational "vibe" and is particularly suited for generating audio content like podcasts and multi-participant audiobooks.

Links: Documentation

• Implement VibeVoice (#40546) by @pengzhiliang in #40546

NeoMME

NeoMME is a family of efficient 260M and 800M parameter multimodal-native multilingual foundation encoders from H Company. It processes multilingual text tokens and raw image patches in a single bidirectional Transformer encoder, without a separately pretrained vision tower or causal language model.

NeoMME-Retriever is a model fine-tuned from the NeoMME backbone for visual document retrieval with joint late-interaction and dense objectives. It takes text queries and documents (text or page screenshots) and produces multi-vector embeddings for MeanMaxSim scoring (late-interaction) and mean-pooled embeddings for cosine similarity (dense).

Links: Documentation

• Add NeoMME and NeoMME-Retriever (#47992) by @tonywu71 in #47992

Fun-ASR-Nano

Fun-ASR-Nano is an 800M-parameter end-to-end speech recognition model developed by Alibaba DAMO Academy's FunAudioLLM team. It achieves state-of-the-art performance on Chinese, English, and Japanese ASR benchmarks while being significantly smaller than comparable models.

Key features are

• Chinese, English, and Japanese, including 7 Chinese dialects and 26 regional accents
• Hotword customization for domain-specific vocabulary
• Native punctuation output (no separate punctuation model needed)

Links: Documentation

• Add Fun-ASR-Nano model (#46180) by @LauraGPT in #46180

KimiLinear

Kimi Linear is a hybrid linear attention architecture from Moonshot AI, introduced in
Kimi Linear: An Expressive, Efficient Attention Architecture.

At its core is Kimi Delta Attention (KDA), a refinement of Gated DeltaNet
that gives each key channel its own forget gate, so the recurrent state decays per channel instead of per head. KDA is
used in most layers; every fourth layer keeps a full-attention block that reuses DeepSeek-V3's Multi-head Latent
Attention (MLA), and the feed-forward blocks are DeepSeek-V3-style MoE with a shared expert.

Links: Documentation

• Kimi linear (#48250) by @remi-or in #48250

Canary

Canary-1B-v2, a fast, robust multilingual model for Automatic Speech Recognition (ASR) and Speech-to-Text Translation (AST):

Canary reuses the Fast Conformer encoder from Parakeet (loaded through [ParakeetEncoder] / [ParakeetEncoderConfig]) and pairs it with a Transformer decoder that uses fixed sinusoidal positional embeddings, cross-attention to the encoder outputs and tied input/output embeddings. The task is selected through a decoder prompt prefix built by [CanaryProcessor] of the form <|startofcontext|> <|startoftranscript|> <|emo:undefined|> <source_lang> <target_lang> <pnc|nopnc> <|noitn|> <|notimestamp|> <|nodiarize|>, where source_lang == target_lang selects transcription and otherwise selects translation.

Links: Documentation

• model: Add NVIDIA Canary-1B-v2 to Transformers (#46825) by @harshaljanjani in #46825

NeuCodec

The NeuCodec model was proposed in Finite Scalar Quantization Enables Redundant and Transmission-Robust Neural Audio Compression at Low Bit-rates.

NeuCodec is a neural audio codec extending on XCodec2. It takes advantage of the following features:

• Finite Scalar Quantization (FSQ) quantisation resulting in a single codebook, making it ideal for downstream modeling with Speech Language Models.
• Trained with CC data such that there are no Non-Commercial data restrictions.
• At 50 tokens/sec and 16 bits per token, the overall bit-rate is 0.8kbps.
• The codec takes in 16kHz input and outputs 24kHz using an upsampling decoder.
• The FSQ encoding scheme allows for bit-level error resistance suitable for unreliable and noisy channels.

Links: Documentation

• Add support for NeuCodec (#47143) by @harryjulian in #47143

Breaking changes

Vision rotary embeddings (2D/3D) have been standardized into a unified RoPE frequency computation module, so users with custom vision models relying on attention-layer-level or model-specific RoPE grid interleaving logic must migrate to the new centralized modeling_rope_utils.py implementation.

• 🚨 Vision (2d/3d) rotary embeddings (#48105) by @zucchini-nlp

Generation

Generation improvements include a performance optimization that avoids unnecessary accelerator synchronization on every decode step (reducing per-step overhead), and a fix to prevent unconditional downloading of remote hub files during generation. Several correctness fixes were also applied, including enforcing auto-compile cache checks for encoder-decoder models, standardizing past_key_values naming in AfMoE, and resolving flaky export and integration test failures.

• [Generate] Avoid unconditionally downloading remote hub file (#48620) by @vasqu in [#48620]
• [generate] stop synchronizing the accelerator on every decode step (#47975) by @SunMarc in [#47975]
• [AfMoE] Standardize past_key_values argument naming across forward and generate (#48430) by @shenhuaqingshi in [#48430]
• Fix MTP generation test regex gate for escaped layer ignore keys (#48003) (#48262) by @Noxtimo in [#48262]
• fix(generation): Enforce the auto-compile cache check for encoder-decoder models (#48364) by @harshaljanjani in [#48364]
• [serge] Fix 4 integration tests for model generation failing with output_mismatch (list output differs (4)) (#48133) by @sergereview[bot] in [#48133]
• [VibeVoice] Skip generate export tests (flaky) (#48396) by @ydshieh in [#48396]

Cache

Fixed several cache-related bugs, including a quantized cache issue in VibeVoice, incorrect rejection of non-static cache implementations in VoxtralRealtime, missing auto-compile cache checks for encoder-decoder models, and a silent failure when paged attention is called without a cache. Documentation was also updated to clarify ContinuousBatchingConfig usage and sliding window model limitations.

• vibevoice: fix bug for quant cache (#48487) by @kaixuanliu in [#48487]
• Fix VoxtralRealtime rejecting non-static cache implementations (#48082) by @jiqing-feng in [#48082]
• Raise when a paged attention forward is called with no cache (#48297) by @qgallouedec in [#48297]
• [docs] Pass ContinuousBatchingConfig and sliding window models (#48381) by @stevhliu in [#48381]
• Retry get_daily_ci_runs on stale GitHub API cache (#48374) by @ydshieh in [#48374]

Kernels

Kernel support was improved with fixes for nested FLA kernel imports when only fla-core is installed, a warning when hub-kernel functions silently fall back to slower pure-PyTorch reference implementations, and the abi...

Release v5.16.1

This is a special release as we include GLM! (and a few small fixes)

GLM-5.3-Flash

GLM-5.3-Flash, the first natively multimodal model in the GLM-5 series. With 320B total parameters and just 18B active parameters, it outperforms GLM-5.2 across benchmarks and real-world workloads at one-tenth the price, while approaching Claude Opus 4.8 on coding and agentic benchmarks.

GLM-5.3-Flash starts from a newly trained base model, with its architecture and training recipe redesigned around capability and efficiency. For the first time in the GLM series, we introduce a hybrid architecture combining sparse and linear attention, sharply reducing long-context serving costs while preserving precise long-context capabilities. The model also adopts Manifold-Constrained Hyper-Connections (mHC) to further improve scaling efficiency. Together with our latest 30T-token multimodal pre-training corpus, these changes enable GLM-5.3-Flash to deliver more intelligence with less compute.

Links: Documentation

• [Glm 5.3 Flash] GLM 5.3 Flash Support (#48342) by @Dovis01 in #48342

Small patch fixes

Mainly BC behavior for TP and pinning a hf kernel for security reasons 🤗

• Restore BC for the tensor-parallel API (#48300) by @ArthurZucker
• Fix kernel commit and repo paths for ESMFold2 (#48186) by @Rocketknight1

Full Changelog: v5.16.0...v5.16.1

Release v5.16.0

New Model additions

Qwen4-Exp

Qwen4-Exp builds on Qwen3.5's hybrid text and multimodal architecture with three key components: GatedResidual (GR), Qwen Sparse Attention (QSA), and Per-Layer Embedding (PLE).

GR is a Qwen-developed residual architecture that combines Hyper-Connection with GatedNorm. It mixes multiple residual streams with fine-grained elementwise gating before each attention and Mixture-of-Experts (MoE) block, then controls how much of the block output is injected back into each stream.

QSA uses multiple query heads to score compressed key blocks, selects the most relevant contiguous token blocks, and keeps the incomplete trailing block uncompressed. This block-level selection reduces indexing overhead and improves memory locality for long sequences. Combined with Gated DeltaNet, QSA makes Qwen4-Exp the first hybrid architecture to integrate linear and sparse attention, substantially improving inference efficiency for long-context workloads.

PLE enriches selected decoder layers with layer-specific lexical features derived from hashed token n-grams and a dilated depthwise convolution.

Links: Documentation

• Add Qwen4Exp model (#48337) by @Cyrilvallez in #48337

GraniteSpeech5

Granite Speech 5.0 Turbo CTC is a lightweight (~470M parameters) conformer encoder for automatic speech recognition, trained with Connectionist Temporal Classification (CTC) on BPE targets. It is a fast, encoder-only member of the Granite Speech family: transcription requires a single forward pass followed by greedy CTC decoding, with no autoregressive decoder.

Architecturally, it extends the Granite Speech conformer CTC encoder with:

1. Frame stacking + block-wise time subsampling: the feature extractor stacks pairs of log-mel(+delta) frames (2x), and the first two conformer blocks each subsample time by 2 through a stride-2 depthwise convolution (with a mean-pooled residual), for a total 8x time reduction at 10 ms mel hop.

2. Block attention with Shaw's relative positional embeddings: attention is computed over fixed-size blocks (the sequence is right-padded to a whole number of blocks, with padded frames masked out), using separate bias-free query/key/value projections.

3. Self-conditioned CTC: the CTC posteriors of the middle layer are projected and fed back into the hidden states, and the CTC head is shared between this mid-layer self-conditioning and the final prediction.

Links: Documentation

• Add Granite Speech 5.0 - (#48288) by @eustlb in #48288

Step3p7

Step-3.7-Flash was proposed in Step 3.7 Flash by StepFun. It is a 198B-parameter sparse Mixture-of-Experts vision-language model, pairing a 196B-parameter MoE language backbone with a 1.8B-parameter vision encoder for native image understanding.

StepFun hasn't published a technical report for Step-3.7-Flash, so the details below are drawn from the released checkpoint's configuration rather than a paper.

• Sparse MoE decoder: all but the first 3 decoder layers route through a MoE block of 288 routed experts (top-8 per token) plus a single shared expert. The router scores experts with a sigmoid and a learned per-expert bias instead of an auxiliary load-balancing loss, the same strategy as DeepSeek-V3.
• Gated attention: each attention layer adds an extra projection whose sigmoid output gates the attention output per head, before the output projection — the same Gated Attention mechanism used in Qwen3-Next. A subset of layers use fewer heads and a sliding window instead of full attention.
• Multi-token prediction: some checkpoints ship extra decoder layers trained for multi-token prediction, which [~GenerationMixin.generate] can use for speculative decoding via use_mtp=True.
• Vision encoder: a SigLIP-style ViT with 2-D rotary position embeddings and a learned per-layer scale on the attention and MLP branches. Its output is downsampled 4x by two stride-2 convolutions before a linear projector maps it into the text model's hidden size.
• Dynamic image tiling: instead of a fixed tile grid, the image processor picks its tiling window from each image's own aspect ratio, producing one downscaled global view plus zero or more local high-resolution crops per image.

Links: Documentation

• [new model] step 3.7 (#46658) by @itazap in #46658

CohereCompass

CohereCompass is the base architecture for small, specialized (vision-)language models trained by Cohere.

Links: Documentation

• Add CohereCompass modeling (#47878) by @calpt in #47878

ESMC and ESMFold2

ESMC and ESMFold2 are new state-of-the-art protein language and folding models from BioHub. ESMC is trained with a masked language modeling objective, and it can be easily transferred to sequence and token classification tasks for proteins. Checkpoints exist in various sizes, from 300M parameters up to 6B parameters. It works as a drop-in replacement for older ESM-2 and ESM-3 models, with significantly higher accuracy.

ESMFold2 is a state-of-the-art protein folding model which produces high accuracy predictions. It uses an iterated diffusion approach that is significantly different from the original ESMFold, offering huge improvements in accuracy for more complex structures.

Links: Documentation ESMC, Documentation ESMFold2

• Port ESMC and ESMFold2 to Transformers (#46419) by @Rocketknight1 in #46419

Breaking changes

The legacy tensor-parallel implementation has been replaced with a DTensor-native backend, so users relying on the previous TP API for inference or training must migrate to the new DTensor-based interface.

• 🚨 TP dtensor API inference + training (#47579) by @3outeille

attn_implementation="sdpa" dispatch is now properly supported for wav2vec2_conformer, wav2vec2-bert, and SeamlessM4T/v2 models, which may change initialization behavior for users who previously worked around this limitation.

• 🚨[wav2vec2] Support attn_implementation=sdpa dispatch (#46196) by @YangKai0616

FuyuProcessor no longer returns the image_patch_indices output, so any code that depends on this field must be updated to remove references to it.

• 🚨 Leftover processors (#47924) by @zucchini-nlp

Cache

Several cache-related bugs were fixed in this release, including an off-by-one error in the sliding window cache, Whisper speculative decoding cache corruption, CpmAnt use-cache failures, Qwen2.5-Omni/Qwen3-Omni-MoE generation with compilable caches, and compressed-tensors loading for KV-cache-only quantized models. Documentation was also added for cache token removal using negative values, and per-layer cache configuration support (allowing models to use different cache settings per layer) was introduced.

• Cpmant fix use cache (#48013) by @jiqing-feng in [#48013]
• [docs] Cache crop (#47950) by @stevhliu in [#47950]
• Revert "Support per-layer cache configuration and attention-mask selection" (#48175) by @Cyrilvallez in [#48175]
• Support per-layer cache configuration and attention-mask selection (#47901) by @eladsegal in [#47901]
• Fix Qwen2.5-Omni / Qwen3-Omni-MoE generation with a compilable cache (#47872) by @jiqing-feng in [#47872]
• Fix sliding window cache index off-by-one on wraparound (#47708) by @hameedibrh in [#47708]
• [Whisper] Fix speculative decoding: UnboundLocalError, cache corruption, and speed regression (#48000) by @ydshieh in [#48000]
• Fix compressed-tensors loading for KV-cache-only quantized models (#47904) by @kylesayrs in [#47904]

Generation

This release fixes several generation bugs across multiple models, including Whisper speculative decoding issues (UnboundLocalError, cache corruption, speed regression, and left-padded batch position IDs), broken image generation in Emu3, garbage output in OLMo/GPTNeoX, and Qwen2.5-Omni/Qwen3-Omni-MoE generation with compilable caches. Additionally, logit distributions for candidate generators using sampling are now aligned by returning logits after applying logit processors.

• [Whisper] Fix speculative decoding: preserve cleared suppress tokens through super().generate() (#48108) by @ydshieh in [#48108]
• [Whisper] Fix decoder position IDs for left-padded batches in longform generation (#48028) by @ydshieh in [#48028]
• [serge] Fix 2 integration tests for model generation failing with import_or_config (other (2)) (#48061) by @sergereview[bot] in [#48061]
• Align logit distributions for CandidateGenerators using sampling (#48007) by @Cyrilvallez in [#48007]
• [GPTNeoX] Fix post_processor not overridden when loading from pretrained (OLMo garbage generation) (#47988) by @ydshieh in [#47988]
• [emu3] 🦮 Black Labrador is back! Fix image generation broken since #37033 (#47948) by @ydshieh in [#47948]

At...

Patch release v5.15.1

This patch most notably solves a few issues with DFlash and MTP candidate generators, as well as an issue where images could sometimes not be processed on accelerator if using Lanczos filter.

It contains the following commits:

• Fix DFlash candidate token device mismatch with device_map="auto" (#47877) by @sywangyi and @Cyrilvallez
• Align logit distributions for CandidateGenerators using sampling (#48007) by @Cyrilvallez
• Fix MTP config when mlp_layer_types is absent (#48015) by @Cyrilvallez
• Fallback from 'lanczos' to 'bicubic' when on cuda (#48026) by @zucchini-nlp
• Fix gemma4 video to device (#47896) by @guarin

Release v5.15.0

New Model additions

Meta Muse Glimmer

Muse Glimmer, released today, is Meta’s new multimodal model, especially designed for agentic use cases. Distilled from Muse to 30B parameters, and released under the Apache 2.0 license, it can be deployed to local setups for privacy-aware applications such as coding, document analysis, personal assistants, Claw- or Hermes-like setups.

Muse Glimmer is a dense 30B parameter model consisting of:

• 2B ViT-style encoder for vision (Perception Encoder)
• 28B parameter text decoder

We're covering it in the following blogpost: http://hf.co/blog/muse-glimmer

GraniteMoeSWA & GraniteSWA

Links: Documentation

• Add Granite-swa and Granitemoe-swa model support (#47179) by @daviswer in #47179

Links: Documentation

• Add Granite-swa and Granitemoe-swa model support (#47179) by @daviswer in #47179

A.X-K1 & A.X-K2

Links: Documentation

• Add AXK2 from SKT (#47528) by @vasqu in #47528

Links: Documentation

• add_axk1 (#46867) by @kmswin1 in #46867

Cosmos3 Edge

Links: Documentation

• Add Cosmos3 Edge model support (#47181) by @atharvajoshi10 in #47181

Breaking changes

Kernels are now opt-in rather than mandatory for linear attention models (Mamba, GDN, Conv-only, etc.), so users who relied on automatic kernel selection must explicitly enable kernels to maintain previous behavior.

• 🚨 [Kernels] Refactor all linear attn models & native kernels fallback (#47630) by @vasqu

The cache cropping API now only accepts negative values (relative offsets) instead of absolute sizes, so users calling crop methods directly must update their code to pass negative values accordingly.

• 🚨 [cache] Cropping can only be done with negative values (#47720) by @Cyrilvallez

T5 and its model family (MT5, LongT5, etc.) now support SDPA and other attention backends via ALL_ATTENTION_FUNCTIONS, meaning the default attention implementation may change and users relying on the previous eager-only path should explicitly set attn_implementation="eager" if needed.

• 🚨 Enable SDPA (and other attention backends) for T5 and propagate to the T5 family (#47014) by @jiqing-feng

Several small private helper functions (e.g., _is_url, _build_image_tokens) have been removed from multimodal processor files, so users or downstream libraries that imported these private functions directly must remove or replace those references.

• 🚨 Processors update the rest (#46556) by @zucchini-nlp

Attention

This release includes several attention fixes and improvements, including correcting Multi-Head Latent Attention (MLA) cache compression, optimizing Flash Attention max sequence length computation in vision models, and fixing bugs in CTRL flex-attention and SDPA prefill with position bias. Additional changes refactor linear attention models for better maintainability, make Gemma 4's heterogeneous attention config explicit, and improve MPS support via metal-flash-sdpa integration.

• [Fix] Fix multi-head latent attention (MLA) (#47761) by @remi-or in [#47761]
• Refactor all linear attention models to latest best standards for convolution (#47452) by @Cyrilvallez in [#47452]
• Allow metal-flash-sdpa for OpenAIPrivacyFilter on MPS (#46740) by @ArthurZucker in [#46740]
• Use new per_layer_config for Gemma 4 so that heterogeneous attention config is explicit (#47384) by @hmellor in [#47384]
• add paged attention tests support for XPU (#47163) by @kaixuanliu in [#47163]
• Move value padding into the attention interfaces that need it (#47451) by @hmellor in [#47451]
• Simplify function dispatch for linear attention (#47450) by @Cyrilvallez in [#47450]
• Optimize flash attention max seqlen computation in vision attention (#47170) by @ShareLer in [#47170]
• Fix BlockMask crash in CTRL flex-attention generation (#46854) by @jiqing-feng in [#46854]
• [CB] Automatically switch attention implementation to flash (#47330) by @remi-or in [#47330]
• Fix sdpa prefill with position_bias (#47359) by @Cyrilvallez in [#47359]

Vision

Vision improvements in this release include performance optimizations such as faster image preprocessing for vision-language models (GLM4V, MiniMaxM3-VL, and others) by eliminating redundant tensor copies, and more efficient Flash Attention variable-length paths by precomputing maximum sequence lengths once per forward pass. Several bug fixes were also applied, including correcting dtype alignment in Kosmos2/Kosmos2_5 embedding merges, fixing a position-embedding initialization fallback in Phi4Multimodal, resolving PIL resize parity in Hunyuan-VL, and patching stop-sequence handling in the image-text-to-text pipeline.

• Modularize qwen-format vision processors (#47573) by @zucchini-nlp in [#47573]
• Update daily CI Docker image to torch 2.13.0 / CUDA 13.0 (#47738) by @ydshieh in [#47738]
• Align image feature dtype in kosmos2 and kosmos2_5 embedding merge (#47691) by @ in [#47691]
• Speed up image preprocessing for vision-language models (#47453) by @labAxiaoming in [#47453]
• Fix vision position-embedding init width fallback in Phi4Multimodal (#47509) by @ in [#47509]
• Fix Hunyuan-VL PIL image resize parity with reference preprocessing (#47233) by @IMvision12 in [#47233]
• Fix image-text-to-text stop_sequence handling (#47032) by @Sunt-ing in [#47032]
• Refactor image loading in tests to use load_test_image helper (#47218) by @LevelVoid in [#47218]

Generation

Several generation improvements and bug fixes were made, including enabling batched audio generation for Qwen2.5/3-Omni, allowing sliding window cache layers to work with speculative decoding, and fixing memory overhead from static cache persistence across generate() calls. Multiple model-specific bugs were also resolved, including crashes in KyutaiSpeechToText, MusicgenForCausalLM, CTRL flex-attention, and assisted decoding for EncoderDecoder cache and OlmoHybrid models.

• Align OlmoHybrid to use a native cache in generate (#47604) by @Cyrilvallez in [#47604]
• [generate] Stop setting the static cache as an attribute to save memory (#47731) by @Cyrilvallez in [#47731]
• Add support for batched Qwen2.5/3-Omni audio generation (#47186) by @IMvision12 in [#47186]
• [cache] Allow sliding window layers to be roll-backed for speculative decoding (#47447) by @Cyrilvallez in [#47447]
• Fix shape mismatch in KyutaiSpeechToText generate() last window (#46952) by @jiqing-feng in [#46952]
• Fix typo in MusicgenForCausalLM.generate() (#46974) by @jiqing-feng in [#46974]
• Fix assisted decoding for models with EncoderDecoder cache & OlmoHybrid (#47361) by @Cyrilvallez in [#47361]

Cache

Several cache-related bugs were fixed, including correcting NemotronH's missing "mlp" layer-type mapping, resolving recurrent-layer padding masks being skipped during chunked prefill and cache continuation for hybrid models, and fixing assisted decoding for models with EncoderDecoderCache and OlmoHybrid. Additional improvements include aligning OlmoHybrid to use a native cache, enabling sliding window layers to support speculative decoding rollback, and stopping the static cache from being stored as a model attribute to reduce unexpected memory overhead.

• [docs] MPS graph cache (#47304) by @stevhliu in [#47304]
• Fix NemotronH: Register "mlp" in the cache layer-type mappings (#47535) by @qgallouedec in [#47535]
• Fix recurrent-layer padding mask being skipped on continued forwards (chunked prefill, cache continuation) (#47087) by @abcgco in [#47087]

Kernels

⚠️ The kernels python package will very likely be a required dependency for transformers[torch] in the near future. This will help us deliver maximum performance to all users; kernels will only be downloaded from trusted publishers manually approved by the HF team. Please let us know of any issues you're facing beforehands so that we may solidify our integration.

Improved robustness of the kernels integration by refactoring function handling to use layer repos, fixing CI EROFS fallback patches for kernel downloads via HfApi, resolving a positional argument collision in causal_conv1d_fn, and bumping the FP8 kernels version to prevent NaNs.

• [conftest] Fix EROFS fallback for kernel downloads (correct interception point) (#47794) by @ydshieh in [#47794]
• [conftest] Fix EROFS fallback for kernel downloads via HfApi (#47791) by @ydshieh in [#47791]
• [Kernels] Refactor function handling (#46883) by @vasqu in [#46883]
• Kernels and loaders robustification (#47334) by @IlyasMoutawwakil in [#47334]
• Fix causal_conv1d_fn positional activation colliding with hub kernel's `seq_...