How AI and AR Are Powering an Intelligent Automotive Future

By Prof. Wang Xiaogang, co-founder and chief scientist, SenseTime

The prospect of gridlocked traffic, polluted skies and increasingly cramped living space is not deterring people from around the world to continue to flock to crowded, sprawling megacities. By the end of this year, U.N. figures project urban areas will be home to 60% of people globally. Efficient and sustainable mobility for these millions of city dwellers is today a massive and growing challenge for government and urban planners. There are plenty of visions for the ideal mass public transport system, but much less reality, while predictions for the end of the automobile are certainly premature. Global unit sales of vehicles are expected to grow from 110 million units to 185 million units by 2027, according to the analysts Market Research Future.

Which means, for most of us living in cities, that cars and the automotive industry will remain a big part of our lives for some time. So the critical priority today must be how to make this form of transport better for us not only as individuals, but as a core part of urban transport overall. This means an automotive experience that is safer, smarter, more effective, efficient and sustainable.

The future of transport and urban mobility is all about being connected. Widespread 5G deployment and massive growth in sensors and intelligence-inducing data mean that automobiles will be rife with smart systems. This will improve safety and create a multisensory in-vehicle experience as well as a smarter overall vehicle ecosystem to help realize the smart mobility vision. 

Technologies that are changing the face of mobility are artificial intelligence and augmented reality, which promise a groundbreaking urban transport and connectivity experience.

Elevating driver, passenger and public safety

With approximately 1.3 million deaths occurring from road traffic accidents globally each year, according to the World Health Organization, AI technologies are being implemented today to improve the driving experience and provide alerts to minimize the risk of accidents. Rapid real-time processing of data by advanced AI algorithms and computer vision technology has enabled SenseTime to deliver a completely transformed in-car experience. At SenseTime, we have taken a holistic view of the complete driver and passenger needs to create an all-encompassing platform with our SenseAuto portfolio to deliver a radically improved experience. Consisting of the SenseAuto Cabin system, which features a Driver Monitoring System (DMS) and an Occupant Monitoring System (OMS), the platform will raise standards in safety, comfort and convenience.

The DMS enables drivers to automatically configure the optimum driving setup, from seat position to rearview mirrors. Internal computer vision provides advanced assessment of driver behavior to warn drivers in case of dangerous driving from drowsiness, distraction or phone use. 

The SenseAuto portfolio also provides autonomous driving features to integrate all-scenario visual sensing and multimodal prediction technologies to make accurate decisions and optimize route planning. Sensors are capable of detecting pedestrians within 150 meters and vehicles within 200 meters so as to perform safe and efficient maneuvers. Drivers today can expect fully automated lane following, lane change, ramp merging and more to ease the burden of driving. With light detection and ranging (LiDAR) sensors and high-accuracy 3-D sensing technology, the system can remain high-performance even under adverse weather conditions and environments.

What can passengers expect from connected vehicles?

As automation rises, there will be a time when drivers become passengers themselves. So what will the in-car experience be like for passengers in this connected-mobility future? Today, SenseTime is already providing AI and AR technologies that enable a number of passenger safety measures as well as a more comfortable and engaging experience with rich in-cabin entertainment.

The SenseMARS Agent product is a humanlike virtual assistant that interacts with users through vivid expressions, gestures and language, allowing them to enjoy multiple in-cabin entertainment functions via voice commands. 

We predict that cars will evolve into connected multiservice, multifunction hubs. Potentially increased time spent in cars will raise demand for more in-vehicle applications and content. The SenseAuto platform provides the ideal point for various partners to be integrated into the in-car experience. High-speed 5G connections will see a stream of content and entertainment apps directly fed into the vehicle environment to provide immersive experiences.

The missing link for smart city and urban mobility management 

Another big promise of connected vehicles is the prospect of vehicle-to-everything (V2X) connectivity’s bringing a vital last-mile link for seamless data from cars to the broader smart-city planning system. This stream of real-time communications could dramatically ease congestion and enable a truly centralized citywide traffic management system as well as provide critical response and emergency services with essential information. From an environmental perspective, vehicle data like speed, fuel economy, travel patterns, etc. could provide invaluable insight on emission trends.

The potential for future innovation in this area is significant, as the cloud platform’s V2X communications could bring a surge in intelligent capabilities for transportation, retail and cultural tourism as well as for many other sectors. In essence, the emergence of more powerful AI and AR technologies in conjunction with high-speed 5G connections will create a strong ecosystem of low-latency mission-critical applications and capabilities. This will have profound transformative potential for the whole automotive industry and pave the way for innovation that drives a more connected, safer and more sustainable future for urban mobility.