Understanding and Harnessing Sparsity in Unified Multimodal Models

Large multimodal models have achieved remarkable progress in both
understanding and generation. Recent efforts pursue unified multimodal models
that integrate heterogeneous components to support both capabilities within a
single framework. However, such unification introduces inference
inefficiencies, e.g., specific tasks or samples may not require the full
knowledge or capacity of the unified model. Yet, a systematic understanding of
how these inefficiencies manifest across different components remains limited.
In this work, we first conduct a systematic analysis of unified multimodal
model components using training-free pruning as a probing methodology,
considering both depth pruning and width reduction. Our study reveals that the
understanding component exhibits notable compressibility in both understanding
and generation tasks, which is more pronounced in the latter. In contrast, the
generation components are highly sensitive to compression, with performance
deteriorating sharply even under moderate compression ratios. To address this
limitation, we propose the Mixture-of-Experts (MoE) Adaptation, inspired by the
dynamic activation patterns observed across different samples. This approach
partitions the generation module into multiple experts and enables sparse
activation to restore generation quality. We validate the effectiveness of
sparse activation through expert-frozen tuning and further demonstrate that a
fully trainable adaptation delivers additional gains. As a result, the adapted
BAGEL model achieves performance comparable to the full model while activating
only about half of its parameters. The code is released at
\href{https://github.com/Shwai-He/SparseUnifiedModel}{this link}.