Mixture of States: Routing Token-Level Dynamics for Multimodal Generation

Scaling text-to-image generation isn't just about stacking layers—it's about smarter moves. The MoS team brings fresh tricks to the transformer table. Prior approaches predominantly rely on cross-attention, self-attention, or unified architectures such as Mixture-of-Transformers (MoT) to align textual and visual representations, enabling the model to follow text guidance effectively. In contrast, our design adopts an orthogonal perspective, guided by the following key principles:

• Adaptive Layer Selection. We find that using a single fixed layer—typically the final-layer feature—or enforcing rigid one-to-one layer alignment is suboptimal. This indicates that diffusion models do not consume language features in a strictly sequential or layer-aligned manner, making a flexible selection mechanism essential.

• Input-Dependent Conditional Signals. Modern text-to-image systems often encode the text once and keep it static throughout denoising. We find that this creates an information mismatch with the evolving nature of the diffusion trajectory. Conditional signals should adapt to the noise level and denoising step rather than remain fixed.

• Token-Specific Conditioning. Our findings indicate that it is more effective to allow each token to source its representation adaptively from different layers, rather than using a single, shared layer embedding to represent all tokens uniformly. This supports a {\it more granular, token-level view} of context conditioning.