NEO-unify: A Native End-to-End Architecture for Unified Multimodal Understanding and Generation
The current bottleneck in multimodal intelligence architecture
For a long time, multimodal research has followed a default paradigm: a visual encoder is responsible for perception and understanding, while a variational autoencoder is used for content generation. Recent work has attempted shared encoders, but this compromise often introduces new structural trade-offs.
Returning to first principles, SenseTime and Nanyang Technological University propose NEO-unify, a native, unified, end-to-end multimodal model architecture that directly processes native inputs: pixels and text themselves. It goes beyond the debate around visual representations, avoids the limits of pretraining priors and scaling laws, and most importantly requires neither a visual encoder nor a VAE.
We are scaling and iterating this direction. More models and open-source results will follow.
NEO-unify: a new native integrated architecture
NEO-unify takes the first step toward a truly end-to-end unified framework. It learns directly from near-lossless information inputs and lets the model shape its internal representation space. It introduces a near-lossless visual interface for unified image input and output, adopts a native Mixture-of-Transformer architecture so understanding and generation can collaborate in one system, and uses a unified learning framework for cross-modal training.
Model performance
Quantitative analysis shows the effectiveness of this design, while generation examples demonstrate the model's ability to preserve visual detail and semantic coherence.
Technical finding 1: encoder-free design preserves both abstract semantics and fine-grained representations
Our previous work NEO showed that native end-to-end models can also learn rich semantic representations. On this basis, we observed an interesting phenomenon: even when the understanding branch is frozen, an independent generation branch can still extract and restore fine-grained visual details from the representation.
Based on this finding, we trained NEO-unify (2B). After an initial 90,000 pretraining steps, it achieved 31.56 PSNR and 0.85 SSIM on MS COCO 2017, compared with 32.65 and 0.91 for Flux VAE. This indicates that near-lossless native inputs can support both high-quality semantic understanding and pixel-level detail fidelity without relying on pretrained visual encoders or VAEs.
We further explored image editing by feeding all multimodal condition information into the understanding branch while leaving the generation branch responsible only for new image generation. Even with the understanding branch frozen, NEO-unify (2B) showed strong image-editing capability while significantly reducing the number of input image tokens.
Technical finding 2: encoder-free architecture and MoT backbone reduce internal conflicts
With pretrained understanding and generation branches, NEO-unify uses the same mid-training and supervised fine-tuning data for joint training. Even with lower data ratios and loss weights, understanding remains stable while generation converges quickly. The two capabilities improve together in the MoT backbone with minimal conflict.
Technical finding 3: higher data efficiency
After web-scale pretraining, NEO-unify goes through mid-training and supervised fine-tuning on diverse high-quality data. Compared with Bagel, it demonstrates higher data efficiency and achieves better performance with fewer training tokens.
Outlook
This is not only an exploration of model architecture; it is also a step toward the next form of intelligence: a closed loop of perception and generation, omni-modal reasoning, visual reasoning, spatial intelligence, and world models.
A new roadmap is unfolding. Models no longer need to translate between modalities; they can think natively across modalities. Multimodal AI is no longer just about connecting different systems, but about building a unified intelligence that has never been split apart and lets the required capabilities emerge from within.