3D Geometry for Video Compression

3D Geometry for Video Compression

Challenge

Based on Cisco reports, video streaming and download will grow to more than 82% of all consumer internet traffic by 2022. Considering the growing demand for a more powerful video compression standard, the international Joint Video Exploration Team (JVET) was formed in 2015 and recently released the Versatile Video Coding (VVC/H.266), which is up to 50% more efficient than the previous standards. VVC uses state-of-the-art 2D motion models to remove the temporal redundancy between frames. However, most 2D videos are captured initially in a 3D environment; thus, developing 3D motion models based on the scene’s 3D geometry and camera trajectories (Fig. 1) should be beneficial. 

Solution

We introduced a method for extracting 3D geometry data from 2D videos captured by a moving monocular camera and synthesizing 3D-based virtual Reference Pictures (RPs). These novel RPs are offered to VVC to be used in motion compensation. The proposed method can work with any un-calibrated monocular cameras and provides decent coding gains. However, having a calibrated camera or any cameras equipped with depth sensors (e.g., LiDAR) or movement measurement sensors (e.g., inertial measurement unit) would increase the coding gain and decrease encoding/decoding time. We have proposed three coding pipelines to adapt the proposed method to the receiver’s hardware limitations.

Advantages

• Coding gain up to 5% compared to VVC with an acceptable increase in decoding time.

• Applicable to all video sequences captured with a moving camera.

• Adjustable to available computational power at the receiver side.

Status

• A German patent application is filed

• A working prototype (software) is available

RWTH Aachen University is looking for partners for patent Exploitation.

RWTH Technology
#2464
Fields of Application
Video Coding
Keywords
#video coding; #VVC