Shanghai releases the “Key Technology Research Plan for the Metaverse in Shanghai (2023-2025)”

Shanghai releases "Metaverse Technology Plan (2023-2025)"

Source: Shanghai Science and Technology Commission

“Metaverse” is an important platform for the future virtual world to interact with the real world. In order to implement the “14th Five-Year Plan for the Construction of a Science and Technology Innovation Center with Global Influence in Shanghai” and the “Action Plan for Cultivating the “Metaverse” New Track in Shanghai (2022-2025)”, accelerate the breakthrough of key technologies in the “Metaverse” field, this action plan is formulated.

I. Overall Ideology and Goals

Guided by Xi Jinping’s new era of socialism with Chinese characteristics, comprehensively implement the spirit of the 19th National Congress of the Communist Party of China, and implement the decision-making arrangements of the municipal party committee and government on the “Metaverse”, adhere to mission orientation, source-driven, key breakthroughs, and empowerment development, and take immersive technology and Web3 technology as the two main attack directions, and independent innovation and open collaboration as the promotion path. Efforts will be made to enhance the technological self-reliance and self-improvement capabilities in the “Metaverse” field. By 2025, significant breakthroughs will be made in the research of relevant scientific issues in the forefront and advantageous areas, and original innovation achievements will continue to emerge; focusing on technological advantages and key areas, a batch of key technologies will be mastered, and the underlying technology support capabilities will continue to improve; focusing on key scene demands, accelerate the systematic breakthrough of “Metaverse” technology; the level of technology transformation and industrialization will continue to improve, forming an internationally competitive “Metaverse” innovation cluster, ensuring the security and resilience of innovation chains, industrial chains, and supply chains, and providing strong support for accelerating the construction of a science and technology innovation center with global influence.

• Bit Block Summit NYC 2023 successfully concluded: Global blockchain and AI elites gathered in New York to explore the innovation of Web3 technology and new investment opportunities.
• Vitalik: Ethereum needs to go through three technical transitions – L2, wallet security, and privacy.
• Why has blockchain become a key technology? In which industries is it applied?

– Significant progress has been made in original and leading scientific and technological research. Strategic frontiers and key technologies such as artificial intelligence-generated content (AIGC), space computing, intelligent coding, high-performance and low-power consumption micro-displays, multimodal sensing, Web3 network operating systems, cross-scale acquisition and reconstruction, and brain-computer interfaces have continuously made new breakthroughs, forming more than 30 core technologies and products with significant influence around intelligent terminals, content creation, and cloud platforms, and promoting the “Metaverse” innovation chain, industrial chain, and value chain to move towards the high-end.

– The advanced application technology system is accelerating its construction. Form a number of technology innovation and service platforms to effectively promote collaborative innovation and achievement transformation. Facing important application needs such as digital transformation of urban economy, life, and governance, build high-level key technology test and verification scenarios, form a full-chain technology supply capability, and build a “Metaverse” technology system with global competitiveness.

——The innovation ecosystem of the “metaverse” is constantly improving. Various elements such as high-quality incubators, investment and financing services, high-level research and development institutions, technology-based backbone enterprises, and innovative teams are gathering. The supervision technology and governance level of the “metaverse” are constantly improving, and an open, inclusive, dynamic, healthy, and orderly innovation ecosystem is accelerating formation.

Second, the main direction

According to the cross-border and composite technical characteristics of “metaverse” such as immersive, open, sustainable and real-time, and people-centered, focusing on “metaverse” content, storage and calculation, transmission and terminals and other technical aspects, combined with the development of domestic and foreign industries. Based on the research and development foundation of this city, focusing on two major main directions of immersive technology and Web3 technology, it will create new heights in key technology areas such as immersive audio and video, immersive computing, new display, perceptual interaction, and blockchain.

Main direction one: immersive technology

Around the fields of immersive audio and video and immersive computing, focus on key technology research and development such as AI-generated content (AIGC), cross-scale acquisition and reconstruction, spatial computing, intelligent coding, etc., reduce the cost of scene construction, and improve the flexibility of “metaverse” content production, and make new progress in engine technology. Focusing on the field of new display and perceptual interaction, focus on the research and development of technologies and terminal devices such as near-eye display and multimodal sensing, driving the research and development and application of chips, components and algorithm technologies in this city, deepening the linkage between content and hardware, and seizing the key entrance of the “metaverse”.

Main direction two: Web3 technology

Facing the decentralized/multi-centralized organizational rules of “metaverse”, focusing on the blockchain field, focus on the research and development of Web3 network operating system and other technologies, and build a new type of blockchain system architecture that is high-performance, scalable, and secure and controllable to provide “metaverse” applications with a neutral and trustworthy information processing platform.

Third, carry out the layout of immersive audio and video technology

Facing the demand for massive immersive audio and video content production in the “metaverse”, based on the technical route of AI empowering virtual and real integration, focus on key technology research and development such as AI-generated content (AIGC), cross-scale acquisition and reconstruction, digital human generation and driving, and virtual space three-dimensional engine, etc., to achieve the automation and low-cost production of ultra-realistic “metaverse” immersive audio and video content, and support the construction of “metaverse” content creation ecology.

(1) Artificial Intelligence Generated Content (AIGC) Technology

Research on general artificial intelligence generation models, multi-modal collaborative generation, and content-controllable generation technologies to build a low-cost, efficient immersive audio-visual content generation system, achieving controllable and recognizable sensitive generation information. By 2025, the correlation alignment and mutual generation of multi-modal content such as text, images, music, video, and 3D models will be achieved, and the high-fidelity, high-efficiency, and controllable generation of multi-modal immersive audio-visual content will be realized, as well as the ability to avoid sensitive information during content intelligent generation.

(2) Cross-Scale Collection and Reconstruction Technology

Research on cross-scale multisensor fusion collection, large-scale and high-dynamic neural radiation field modeling and generation technologies to form an autonomous and controllable cross-scale collection and reconstruction technology system. By 2025, a multi-modal collaborative collection system including laser radar, light field array, and gradient light array will be built; high-precision, minute-level modeling capabilities of multi-scale scenes such as city (large), indoor (medium), and static objects (small) will be formed, and core indicators such as image resolution, rendering speed, and image accuracy of new perspective will reach the international leading level.

(3) Digital Human Generation and Driving Technology

Research on high-fidelity reconstruction, automatic binding, and real-time driving technologies for faces, expressions, and movements, building an optical motion capture camera system, and integrating facial expressions, movements, and speech into a digital human intelligent generation system. By 2025, a high-precision positioning and intelligent analysis system for human body and rigid body movement will be formed, and the positioning and tracking accuracy will reach the international leading level. High-precision modeling and binding of virtual humans will be achieved, and the reconstruction accuracy will reach sub-millimeter level, supporting intelligent binding. The number of basic facial expressions and the speed of facial 4D data acquisition will both reach the international leading level. Artificial intelligence driving capabilities will be formed to support multi-modal expression and motion driving based on text, audio, video, etc.

(4) Virtual Space 3D Engine Technology

Research on cross-platform, end-cloud collaborative architecture, breakthroughs in technology such as multi-threaded synchronous and asynchronous loading balance, multi-threaded particle calculation, and multi-core CPU resource scheduling, and achieving high-frequency data rendering on low-cost and low-bandwidth devices. By 2025, rendering efficiency and power consumption indicators will reach the advanced level in China.

Section 4: Accelerating Immersive Computing Technology Breakthroughs

Based on the immersive computing technology requirements of the “metaverse” virtual-real fusion, a technology route for immersive computing based on edge-cloud collaboration is established, with a focus on key technology research and development such as space computing, intelligent coding, and distributed rendering and distribution, to improve the efficiency of “metaverse” content computing and distribution.

(1) Space Computing Technology

Research on millimeter-level visual positioning services, high-performance synchronous positioning and mapping creation (SLAM), and large-scale real-time user interaction in multiple scenarios, to promote the seamless real-time embedding of “metaverse” digital interactive information and the real environment, and improve the operational service capabilities of virtual-real fusion digital space. By 2025, high-precision positioning will be achieved in various indoor and outdoor scenarios, with a positioning accuracy of millimeters for small scenes of about 100 square meters and centimeters for large scenes of about 100,000 square meters.

(2) Intelligent Coding Technology

Research on efficient structural expression of three-dimensional irregular mesh data, multi-source media synchronous correlation and network element collaborative processing, storage-computing network integrated processing chips, etc., to form 6 degrees of freedom (6DoF) data efficient compression coding methods, cross-network synchronous encapsulation protocols, and innovative media processor structures. Research on artificial intelligence-assisted visual video feature coding and immersive video compression coding technology to build a new type of encoding and decoding model for virtual-real fusion. By 2025, under the conditions of peak signal-to-noise ratio (PSNR) of a reconstructed geometric quality of 70dB and an attribute quality of 40dB, densely packed 6DoF point clouds will achieve 250 times compression rate, and achieve accurate synchronization and distribution across networks.

(3) Distributed Rendering and Distribution Technology

Research on rendering optimization technology for cloud-native, high-real-time and high-quality scheduling algorithms for audio and video data of cloud rendering content. By 2025, the computing power infrastructure will meet the immersive application needs of the “metaverse”, realizing a cloud-native system that can be elastically scaled and supports mixed rendering, and further reducing the delay of 4K high-definition media stream push and interaction; providing multi-level services to significantly improve service reliability and resource utilization under heterogeneous businesses.

Section 5: Carrying Out the Layout of New Display Key Technologies

Focus on the development trends and demands of high-definition, deeply immersive content presentation for the “metaverse”, with a focus on key technology breakthroughs such as high-performance, low-power micro display devices, high-brightness lightweight optical modules, and holographic and light field displays, seize the high ground of key entry points for the “metaverse”, and promote the high-quality development of new display industries such as near-eye display and naked-eye holography.

(I) High-performance, low-power micro-display technology

Research digital drive and gaze rendering, integrated system architecture of sense-display algorithm, pixel structure design and other technologies. By 2025, silicon-based OLED display modules and self-driving chips, silicon-based micro-LED display modules and self-driving chips, Fast-LCD display devices will reach the international advanced level, high-brightness, wide color gamut, low-power micro-display single-eye resolution will reach above 4K.

(II) High-brightness lightweight optical module

Research geometric optical waveguide technology for near-eye display, break through the molecular bonding process for consumer optical glass, develop two-dimensional pupil expansion geometric optical waveguide lenses and their display modules, and promote the mass production of consumer-grade geometric optical waveguide lenses and their display modules. By 2025, the indicators such as field of view angle, mirror distance, eye movement range, entry brightness, transmittance, optical distortion will reach the international leading level, supporting the mass production of small and lightweight products.

(III) Holographic and light field display technology

Research on large-angle, high-information-volume holographic 3D optical wave modulation, ultra-precision micro-nano manufacturing, real-time rendering of light field, real-time generation algorithm of holographic image data, and develop holographic 3D wavefront modulation devices, 3D video processing chips, and light field naked-eye 3D display modules, to achieve high-immersion holographic display and large-angle multi-viewpoint light field display. By 2025, the fitting precision of optical devices for light field naked-eye 3D will reach sub-micron level, and the indicators such as field of view angle, visual resolution, display frame rate, spatial bandwidth product and speckle noise contrast of holographic images, and the number of viewpoints of light field display will reach the international advanced level.

(VI) Strengthen breakthroughs in perception and interaction technology

Aiming at the long-term, high-response and diversity demands of natural interaction in the “metaverse”, the research and development of key software and hardware technologies such as low-power interaction algorithms, multimodal sensing, gesture and eye movement tracking, and brain-machine interfaces are laid out, to achieve multimodal information perception and interaction such as vision, hearing, touch, and posture in natural environments.

(1) Multi-Modal Sensing Technology

Research on micro distance sensing, depth vision sensing, low-power piezoelectric acoustic/flexible fiber/fabric skin/high-precision micro-electromechanical inertia and other state sensing and lightweight interaction algorithms, to promote domestic chips and domestic parts in terminal applications. By 2025, key technical indicators such as micro-distance calculation, gaze rendering, multi-mode non-blind area human-computer interaction, sensing density, pressure resolution, zero bias accuracy, etc. will reach the international advanced level, supporting high-performance, low-power consumption sensors for distance, vision, sound, pressure, and inertia.

(2) Gesture and Eye Tracking Technology

Research on high-performance, low-power consumption, and high-robustness gesture and eye tracking technologies to promote the natural, situational, and intelligent evolution of “metaverse” human-computer interaction. By 2025, key technical indicators such as the number of recognition types, tracking delay, finger tracking accuracy, and eye tracking accuracy will reach the international advanced level, achieving high-precision, multi-type, and low-latency gesture recognition and eye tracking.

(3) Brain-Computer Interface Technology

Conduct interdisciplinary theoretical and technical research, build a virtual-real fusion brain-computer interface system oriented towards the “metaverse,” research on cutting-edge technologies such as brain-computer interface neural interfaces, neural fusion chips, flexible electrodes, neural signal encoding and decoding, and develop new ultra-low power consumption brain-computer interaction hardware systems and prototype products for the “metaverse.”

(7) Research and Development of Blockchain Technology

Based on heterogeneous multi-chain fusion technology, focusing on the research and development of supporting technologies such as Web3 network operating system, blockchain digital identity, distributed trusted storage, and trusted computing chips and digital supervision technology, constructing a high-performance, scalable, and secure new blockchain system architecture, and forming a basic infrastructure for carrying large-scale “metaverse” innovation applications based on decentralized/multi-centralized organizational rules.

(1) Web3 Network Operating System

Research on blockchain system abstract, resource management, and scheduling models, breakthroughs in underlying blockchain system adaptation and encapsulation, on-chain data resource parsing and positioning, cross-chain interoperability and other technologies, improve blockchain data standards, system application interface standards and cross-chain interoperability standards, and form a Web3 network operating system for cross-chain blockchain systems. Develop application development middleware and tool chains that adapt to a variety of heterogeneous underlying blockchain systems (including mainstream public chains and alliance chains), support bottom-layer heterogeneous blockchain system access, digital identity content management, cross-chain application data migration and other functions through a unified interface.

（2）Digital Identity Management Technology

Research on autonomous and controllable low-cost blockchain digital identity SIM card chips, identity authentication and permission management, virtual-real identity mapping and binding technologies, develop secure and convenient blockchain digital identity management terminals, support national encryption algorithms (SM series) and other mainstream cryptographic algorithms, new anti-quantum/threshold/aggregate digital signature algorithms, BIP32/44 and other account derivation protocols, and protect the security of private key generation, storage, and digital signature through hardware security technology. Support real-name authentication of blockchain accounts and build a digital identity system that binds “metaverse” digital identity with real-world identity.

（3）Distributed Trusted Storage Technology

Research on blockchain dynamic data sharding storage and other technologies, support data access through smart contracts, efficiently verify the validity of data on the chain, and linearly expand the storage space of a single-chain system with the number of consensus nodes.

（4）Trusted Computing Chip Technology

Breakthroughs in key software and hardware technologies such as trusted execution environment, cryptographic algorithms (zero-knowledge proof, homomorphic encryption, etc.) acceleration chips, develop autonomous and controllable trusted computing chips, and achieve computing performance and efficiency that reach international advanced levels.

8. Accelerate Innovation Platform and Scene Construction

Around the innovation, experimentation, and ecosystem construction of “metaverse” technology, build a number of technology innovation and service platforms, create key technology testing and verification scenarios, accelerate domestic technology innovation and application, and promote the incubation and aggregation of “metaverse” enterprises.

（1）Technology Innovation Support Platform

Relying on third-party institutions, industry leaders, and university research institutes, build “metaverse” key technology innovation platforms and innovation public service platforms, continuously optimize the “metaverse” technology breakthroughs, application innovations, and industrial development support environment. Build a “metaverse” trusted technology supervision support platform to support relevant regulatory policy research and governance system construction.

(II) Technical Experimental Verification Scenarios

Relying on the resource advantages and characteristics of this city, and facing important application needs such as digital transformation of urban economy, life, and governance, encourage “metaverse” technology innovation enterprises in this city to collaborate with industry application demanders to create high-level technology experimental verification scenarios in cultural tourism, industrial manufacturing, etc., support technology experimental verification and optimization iteration, promote the formation of a number of competitive solutions, and accelerate the modernization level of urban governance capabilities.

IX. Guarantee Measures

(I) Strengthen Organizational Leadership

Establish a working group for key technology research and development of “metaverse”, strengthen top-level design and overall coordination, promote linkage between the city and the district, and ensure the implementation of key tasks. Establish an expert group, continue to closely follow domestic and foreign trends, organize technical exchanges, analyze and judge strategic trends, and effectively support strategic decision-making.

(II) Implement Organized Scientific Research

Explore government long-term scientific research support methods for major frontier scientific issues, encourage social forces to invest, and further stimulate the vitality of basic research. Facing major scenario application needs, strengthen the innovation subject status of enterprises, create innovation consortia, and conduct major key technology, major strategic products, and major experimental verification research. Explore mechanisms such as “revealing the list and taking the lead”, milestone management, and horse racing system to speed up the application of innovative achievements.

(III) Improve Entrepreneurship Incubation and Financial Services

Encourage parks and high-quality incubators to provide “metaverse” enterprises with high-level technology screening, intellectual property operation, innovation chain, and industry chain resource links, etc., and smooth the “discovery-transformation-incubation-industrialization” chain. Strengthen the linkage between investment and incubation, leverage and guide “patient capital” to flow to high-quality scientific and technological achievements, seed period, start-up period, and high-growth technology enterprises. Increase the supply of scientific and technological financial products, guide banks, insurance, guarantee and other financial institutions to increase cooperation with high-quality incubators, focus on innovative financial products and services for “metaverse” “early, small, and hard” enterprises, and promote a batch of innovative enterprises to land on the science and technology innovation board.

(4) Strengthening Talent Attraction and Cultivation

Based on a global perspective, make good use of the talent support measures in our city, and gather a group of “metaverse” talents. Guide universities, research institutes, enterprises, and others to increase their efforts in cultivating young “metaverse” talents, and support young talents to take on major roles. Support eligible local universities to independently set up related second-level disciplines or interdisciplinary fields in key industrial areas, and build a talent cultivation model that involves joint participation of industry enterprises and universities, covering professional master’s and doctoral degrees, to cultivate urgently needed talents.

We will continue to update Blocking; if you have any questions or suggestions, please contact us!