25fps Inference and 3.5s First-Frame Latency: SenseTime SekoTalk Brings Real-Time Voice-Driven Digital Humans Closer

Real-time voice-driven digital humans

As digital human technology develops rapidly, generation efficiency remains a core industry challenge. With deep experience in generative AI and multimodal interaction, SenseTime has introduced SekoTalk, a real-time voice-driven digital human technology.

Through multiple technical innovations, SekoTalk significantly improves digital human video generation efficiency. On an eight-GPU server, it reaches 25fps generation speed with first-frame latency as low as 3.5 seconds, achieving real-time generation ahead of the industry. It also supports accurate lip-sync across multiple speakers and languages, as well as stable long-duration generation.

SekoTalk was launched in August and has been applied in products such as SenseTime Seko and SenseAvatar. It has helped users create hundreds of thousands of works, including viral content with more than 20 million views across the internet.

Algorithm-system co-design for extreme cost-performance

Generation efficiency is key to making digital humans practical, and real-time performance is the north star of generation efficiency. Through model distillation, model-structure optimization, and model-system co-design, SekoTalk achieves a leap in inference efficiency while maintaining generation quality.

Open-source models usually take more than ten minutes to generate a five-second video, while commercial closed-source models often require one to ten minutes. In contrast, SekoTalk reaches 25fps on an eight-GPU server, and when combined with a multimodal model, the overall system can still achieve first-frame latency as low as 3.5 seconds.

Phased DMD distillation approaches base-model quality

Prior diffusion-model distillation experience shows that low-step generation quality is limited by effective model capacity. SOTA video generation models have proven the potential of Mixture-of-Experts in diffusion models: they increase effective model capacity without increasing inference cost.

SenseTime researchers found that directly applying Distribution Matching Distillation to MoE models can reduce motion quality and instruction following. To solve this, the team proposed Phased DMD, modeling denoising as a multi-stage MoE process. It natively supports MoE models and can distill non-MoE teacher models into MoE student models.

Phased DMD significantly improves motion dynamics and diversity. SekoTalk reduces inference cost by 25 times while preserving body motion and emotional expressiveness from the teacher model. The technique has also been applied to open-source base models and contributed back to the community.

LightX2V and model co-design for low-resource deployment

LightX2V is SenseTime's open-source inference framework and the first in the industry to reach real-time video generation. From the beginning of model and system design, it incorporates native optimizations such as low-bit quantization-aware training and sparse attention, together with a self-developed efficient attention operator combining SPARSE, NVFP4, and low-bit communication.

Tests show that LightX2V enables efficient SekoTalk inference across different GPU environments, providing flexible support for deployment in different scenarios.

Audio-visual synchronization for multilingual and multi-person scenarios

Traditional digital human technologies often struggle with accurate lip synchronization in multilingual and multi-person scenarios. SekoTalk uses a series of innovations to achieve highly accurate audio-visual synchronization from single-person lip movement to multi-person interaction.

For multilingual synchronization, SenseTime explored multiple speech encoders including wav2vec2, HuBERT, WavLM, and Whisper. The team found that relying on the wav2vec2 series, even multilingual wav2vec2-large-xlsr-53, lagged behind other encoders in English lip driving and multilingual generalization. SekoTalk uses the best-performing audio encoder from this exploration and achieves accurate driving effects across Chinese, English, minority languages, daily speech, and rap scenarios.

SekoTalk decouples video frame rate from speech feature frame rate. Mainstream video generation models use a 1+4N temporal compression mechanism. To strictly synchronize audio and video frames, SekoTalk optimizes the audio-processing branch by decoupling 16-25fps video frames from 50fps speech features, avoiding lip-detail loss caused by downsampling and keeping audio aligned with any frame rate.

In multi-person dialogue, SekoTalk uses strong model generalization and an attention-mask mechanism to independently and accurately control each character's lip movement and body motion, generating natural group-interaction results.

Efficient low-cost speech module

In digital human generation, previous work often used speech-conditioned classifier-free guidance to improve lip-driving accuracy. This adds 50% more computation and can also produce exaggerated, unnatural faces. SenseTime redesigned the speech injection module using Adaptive Layer Normalization and learnable injection parameters instead of linear projection, reducing computation while preserving expressiveness.

With these improvements, SekoTalk achieves accurate mouth-shape driving without speech-conditioned CFG, avoiding the trade-off between lip accuracy and facial naturalness.

Stable long-duration generation

Color drift and character-ID inconsistency are persistent challenges in long-video generation. SekoTalk proposes hybrid reference-image injection and other strategies to balance motion diversity with visual stability.

During training, the model randomly selects reference images from inside and outside a segment and uses indicators to mark their source. This lets the model learn both within-segment stability and cross-segment generalization. High-level and low-level semantic feature injection strengthens character consistency and accelerates convergence. Separate Patchify branches process noisy video, reference images, and previous frames, improving long-video continuity and stability.

For efficiency, SekoTalk introduces previous-frame features in the temporal dimension and directly reuses latent features from the end of the previous generated segment, avoiding redundant decode-reencode steps. Hierarchical KV cache and causal attention ensure stable continuation while improving inference efficiency.

Validated in real products

SekoTalk's technical value has been proven in practice. Its online experience platform is the first free technical demo platform supporting lip synchronization for more than two people and two-minute long-video generation. The model has also been integrated into Seko, SenseAvatar, and other products.

The real-time version of SekoTalk has shown potential in emotional companionship, online education, and professional consultation, pointing toward a more natural, intelligent, and real-time future for digital humans.

Users can try the free platform at https://sekotalk.com/, view creative samples at http://sekotalk.com/showcase/, visit https://seko.sensetime.com, and join technical discussion at https://github.com/OpenSenseNova/SekoTalk. LightX2V is available at https://github.com/ModelTC/LightX2V.