Based on Cisco reports, video streaming and download will grow to more than 82% of all consumer internet traffic by 2022. Considering the growing demand for a more powerful video compression standard, the international Joint Video Exploration Team (JVET) was formed in 2015 and recently released the Versatile Video Coding (VVC/H.266), which is up to 50% more efficient than the previous standards. VVC uses state-of-the-art 2D motion models to remove the temporal redundancy between frames. However, most 2D videos are captured initially in a 3D environment; thus, developing 3D motion models based on the scene’s 3D geometry and camera trajectories (Fig. 1) should be beneficial.
Solution
We introduced a method for extracting 3D geometry data from 2D videos captured by a moving monocular camera and synthesizing 3D-based virtual Reference Pictures (RPs). These novel RPs are offered to VVC to be used in motion compensation. The proposed method can work with any un-calibrated monocular cameras and provides decent coding gains. However, having a calibrated camera or any cameras equipped with depth sensors (e.g., LiDAR) or movement measurement sensors (e.g., inertial measurement unit) would increase the coding gain and decrease encoding/decoding time. We have proposed three coding pipelines to adapt the proposed method to the receiver’s hardware limitations.
Advantages
Coding gain up to 5% compared to VVC with an acceptable increase in decoding time.
Applicable to all video sequences captured with a moving camera.
Adjustable to available computational power at the receiver side.
Status
An international patent application is filed
A working prototype (software) is available
RWTH Aachen University is looking for partners for patent exploitation
RWTH Technology #2464 Anwendungsgebiete Video Coding Stichworte #video coding #VVC
