Abstract
While panoramic videos provide people with a realistic and immersive viewing experience through a 360° field of view (FoV) and adjustable viewports, the wide FoV results in large data volume, which puts great pressure on video storage and transmission. To address this issue, numerous panoramic video compression methods based on eliminating spatial redundancy have been developed. However, none of these methods consider the background redundancy observed in various panoramic videos applications like surveillance and game live streaming. In this paper, we propose a background modeling-based panoramic video compression approach to improve user experience in scenarios where background changes are not obvious. Specifically, we study two background modeling schemes and utilize the constructed background as a long-term reference frame in SVT-HEVC coding framework. Besides, we develop a viewport prediction approach by combining the constructed background with edge information. Experimental results show that the proposed method can achieve a gain of about 8% compared to the original encoder, and enhances the smoothness and clarity of the reconstructed panoramic videos. Compared with existing deep learning-based viewport prediction methods, our method only takes half the time to predict the viewport with essentially the same accuracy.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Disclosure of Interests
The authors have no competing interests to declare that are relevant to the content of this article.
References
Song, J., Yang, F., Zhang, W., Zou, W., Fan, Y., Di, P.: A fast FoV-switching DASH system based on tiling mechanism for practical omnidirectional video services. IEEE Trans. Multimed. 22(9), 2366–2381 (2019)
Corbillon, X., Devlic, A., Simon, G., Chakareski, J.: Viewportadaptive navigable 360-degree video delivery. In: IEEE International Conference on Communications, volume abs/1609.08042 (2017)
Xie, L., Xu, Z., Ban, Y., Zhang, X., Guo, Z.: 360ProbDASH: improving QoE of 360 video streaming using tile-based http adaptive streaming. In: ACM International Conference on Multimedia, pp. 315–323 (2017)
ITUT Recommend. T. 800-iso fcd15444-1: Jpeg2000 image coding system. International Organization for Standardization, ISO/IEC JTC1 SC29/WG1 (2000)
Wiegand, T.: Draft ITU-T recommendation and final draft international standard of joint video specification (ITU-T Rec. H. 264-ISO/IEC 14496–10 AVC). JVT-G050 (2003)
Bross, B.: High efficiency video coding (HEVC) text specification draft 9 (SoDIS). In: 11th JCT-VC Meeting, October 2012 (2012)
AVS Workgroup: Draft of advanced audio video coding-part 2: Video. In 7th AVS Meeting, Beijing, China (2003)
Zhao, T., Lin, J., Song, Y., Wang, X., Niu, Y.: Game theory-driven rate control for 360-degree video coding. In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 3998–4006 (2021)
Ray, B., Jung, J., Larabi, M.-C.: A low-complexity video encoder for equirectangular projected 360 video content. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1723–1727. IEEE (2018)
Liu, Y., Jiang, B., Guo, T., Sitaraman, R.K., Towsley, D., Wang, X.: Grad: learning for overhead-aware adaptive video streaming with scalable video coding. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 349–357 (2020)
Sánchez, Y., Skupin, R., Schierl, T.: Compressed domain video processing for tile based panoramic streaming using HEVC. In: 2015 IEEE International Conference on Image Processing (ICIP), pp. 2244–2248. IEEE (2015)
Liu, B., Chen, Y., Liu, S., Kim, H.-S.: Deep learning in latent space for video prediction and compression. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 701–710 (2021)
Qian, F., Ji, L., Han, B., Gopalakrishnan, V.: Optimizing 360 video delivery over cellular networks. In: Proceedings of the 5th Workshop on All Things Cellular: Operations, Applications and Challenges, pp. 1–6 (2016)
Ban, Y., Xie, L., Xu, Z., Zhang, X., Guo, Z., Wang, Y.: Cub360: exploiting cross-users behaviors for viewport prediction in 360 video adaptive streaming. In: 2018 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6. IEEE (2018)
Zhang, R., et al.: Buffer-aware virtual reality video streaming with personalized and private viewport prediction. IEEE J. Sel. Areas Commun. 40(2), 694–709 (2021)
Heyse, J., Vega, M.T., De Backere, F., De Turck, F.: Contextual bandit learning-based viewport prediction for 360 video. In: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 972–973. IEEE (2019)
Feng, X., Liu, Y., Wei, S.: LiveDeep: online viewport prediction for live virtual reality streaming using lifelong deep learning. In: 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 800–808. IEEE (2020)
Feng, X., Bao, Z., Wei, S.: LiveObj: object semantics-based viewport prediction for live mobile virtual reality streaming. IEEE Trans. Vis. Comput. Graph. 27(5), 2736–2745 (2021)
Hu, K., et al.: Understanding user behavior in volumetric video watching: dataset, analysis and prediction. In: Proceedings of the 31st ACM International Conference on Multimedia, pp. 1108–1116 (2023)
Hoare, C.A.R.: Quicksort. Comput. J. 5(1), 10–16 (1962)
Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 6, 679–698 (1986)
Acknowledgments
This work was supported by the National Key R&D Program of China (2021YFF0900500), and the National Natural Science Foundation of China (NSFC) under grants U22B2035, 62272128.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Wang, C., Wang, X., Wu, K., Fan, X. (2024). Panoramic Video Inter Frame Prediction and Viewport Prediction Based on Background Modeling. In: Huang, DS., Zhang, C., Guo, J. (eds) Advanced Intelligent Computing Technology and Applications. ICIC 2024. Lecture Notes in Computer Science, vol 14871. Springer, Singapore. https://doi.org/10.1007/978-981-97-5609-4_20
Download citation
DOI: https://doi.org/10.1007/978-981-97-5609-4_20
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-5608-7
Online ISBN: 978-981-97-5609-4
eBook Packages: Computer ScienceComputer Science (R0)