Based on industry reports, video streaming and downloads make up a significant portion of overall consumer internet traffic. For example, according to Ericsson's reports, video streaming accounts for 78% of smartphone users' data consumption, up from 61% in 2018. the Considering the growing demand for a more powerful video compression standard, the international Joint Video Exploration Team (JVET) released the Versatile Video Coding (VVC/H.266) in 2020, which is up to 50% more efficient than the previous standards. VVC uses state-of-the-art 2D motion models to remove the temporal redundancy between frames. However, most 2D videos are captured initially in a 3D environment; thus, developing 3D motion models based on the scene’s 3D geometry and camera trajectories (Fig. 1) should be beneficial.
Solution
We introduced a method for extracting 3D geometry data from 2D videos captured by a moving monocular camera and synthesizing 3D-based virtual Reference Pictures (RPs). These novel RPs are offered to the video encoder to be used in motion compensation. The proposed method can work with any un-calibrated monocular cameras and provides decent coding gains. However, having a calibrated camera or any depth sensors (e.g., LiDAR) or movement measurement sensors (e.g., inertial measurement unit) would increase the coding gain and decrease encoding/decoding time. Recent advancements in the area of 'AI on device' have made it possible to estimate precise depth data from a single frame on a common System-on-Chip (SoC) in just a few milliseconds. This will allow us to perceive 3D geometry more quickly and accurately, leading to greater efficiency and easier exploitation. We have proposed three coding pipelines to adapt the proposed method to the receiver’s hardware limitations.
Advantages
Coding gain up to 5% compared to VVC with an acceptable increase in decoding time.
Applicable to all video sequences captured with a moving camera, e.g., smartphones, vehicle ADAS cameras and robot/drone cameras.
Adjustable to available computational power at the receiver side.
Status
The patent procedure is pending in Europe and US
A working prototype (software) is available
RWTH Aachen University is looking for partners for R&D Projects, further development or patent exploitation
RWTH Technology #2464 Anwendungsgebiete Video Coding Stichworte #video coding #VVC
