CN116193196A - Virtual surround sound rendering method, device, equipment and storage medium - Google Patents

Abstract

The embodiment of the application provides a virtual surround sound rendering method, device, equipment and storage medium. In the embodiment of the application, when the rendering mode of the multimedia data to be rendered is determined, not only the channel number and the video/audio scene of the multimedia data to be rendered can be determined, but also the transformation condition of the head rotation angle of the user to be listened to the multimedia data to be rendered is considered, and finally, based on the channel number and the video/audio scene of the multimedia data to be rendered and the head rotation angle of the user to be listened to, the determined target rendering function is matched with the current environment of the user to be listened to, the channel number of the multimedia data to be rendered and the head rotation angle of the user to be listened to, so that the target virtual surround sound is obtained by rendering the multimedia data to be rendered through the target rendering function, and therefore, better audio experience can be brought to the user to be listened to.

Description

Virtual surround sound rendering method, device, equipment and storage medium

technical field

The present application relates to the technical field of virtual surround sound, and in particular to a virtual surround sound rendering method, device, device and storage medium.

Background technique

In order to meet the needs of most users for portable and high-quality mobile audio-visual experience, virtual surround sound technology, which can realize high-quality audio-visual surround effects with relatively simple equipment, has emerged as the times require. Compared with multi-channel surround sound technology, virtual surround sound technology usually only needs two channels to achieve the effect of surround sound, which makes users hope that they can still enjoy "home theater" with as few playback devices as possible. The wish of audio-visual effect is realized. However, how to improve the existing virtual surround sound technology so as to apply the virtual surround sound technology to a wider range of scenarios and improve the audio experience of listeners still needs to provide further solutions.

Contents of the invention

Various aspects of the present application provide a virtual surround sound rendering method, device, device, and storage medium, which are used to improve the existing virtual surround sound technology so as to apply the virtual surround sound technology to a wider range of scenarios, and at the same time Improve the listener audio experience.

An embodiment of the present application provides a virtual surround sound rendering method, including: determining the number of channels of the multimedia data to be rendered and the audio-visual scene; determining the head rotation angle of the user listening to the multimedia data to be rendered; data, the audio-visual scene, and the head rotation angle, determine a target rendering function, so as to render the multimedia data to be rendered based on the target rendering function, and obtain target virtual surround sound.

The embodiment of the present application also provides a virtual surround sound rendering device, including: a scene determination module, used to determine the number of channels and video and audio scenes of the multimedia data to be rendered; an angle determination module, used to determine the angle of the multimedia data to be rendered Listen to the user's head rotation angle; the resource rendering module is used to determine the target rendering function based on the number of channels, the audio-visual scene and the head rotation angle to render the multimedia data to be rendered to obtain the target Virtual surround sound.

The embodiment of the present application also provides an electronic device, including: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and used to execute the computer program for : determine the number of channels and audiovisual scenes of the multimedia data to be rendered; determine the head rotation angle of the listening user of the multimedia data to be rendered; determine based on the number of channels, the audiovisual scenes and the head rotation angle A target rendering function, for rendering the multimedia data to be rendered based on the target rendering function to obtain target virtual surround sound.

The embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is caused to implement the steps in the virtual surround sound rendering method provided in the embodiment of the present application. .

In the embodiment of the present application, when determining the rendering method of the multimedia data to be rendered, not only the number of channels and the audio-visual scene of the multimedia data to be rendered can be determined, but also the head rotation of the listening user of the multimedia data to be rendered can be considered Angle transformation, finally, based on the number of channels of the multimedia data to be rendered, the audio-visual scene and the head rotation angle of the listening user, the determined target rendering function is compatible with the current environment of the listening user and the number of channels of the multimedia data to be rendered and the head rotation angle of the listening user, so that the target virtual surround sound can be obtained by rendering the multimedia data to be rendered through the target rendering function, which can also bring a better audio experience to the listening user. Moreover, this rendering method fully takes into account Differences in different scenarios can also be applied to more scenarios.

Description of drawings

The drawings described here are used to provide a further understanding of the application and constitute a part of the application. The schematic embodiments and descriptions of the application are used to explain the application and do not constitute an improper limitation to the application. In the attached picture:

FIG. 1 is a schematic flowchart of a system implemented by a virtual surround sound rendering method provided in an embodiment of the present application;

FIG. 2 is a schematic flowchart of a virtual surround sound rendering method provided by an exemplary embodiment of the present application;

3 is a schematic diagram of establishing a three-dimensional coordinate system for the head of the listening user according to the virtual surround sound rendering method provided by the embodiment of the present application;

FIG. 4 is a schematic diagram of a scene where binaural multimedia data is rendered to both ears of a listening user according to a virtual surround sound rendering method provided by an embodiment of the present application;

FIG. 5 is a schematic diagram of a scene where 5.1-channel multimedia data is rendered to both ears of a listening user according to a virtual surround sound rendering method provided by an embodiment of the present application;

FIG. 6 is a schematic diagram of a scene where 7.1-channel multimedia data is rendered to both ears of a listening user according to a virtual surround sound rendering method provided by an embodiment of the present application;

FIG. 7 is a schematic diagram of a scene where the room simulation provided by the exemplary embodiment of the present application is that the left ear and the right ear receive multimedia data played by a 5.1-channel virtual speaker;

FIG. 8 is a waveform diagram of the room-related impulse response function between the 5.1-channel virtual speaker and the left ear and the right ear in the audio-visual scene shown in FIG. 7 provided by an exemplary embodiment of the present application;

FIG. 9 is a schematic diagram of the sound propagation path generated in the target room-related impulse response function according to an embodiment of the present application according to the image source method;

FIG. 10 is a schematic diagram of the propagation path of acoustic energy rays in the target room-related impulse response function generated according to the diffuse rain ray tracing algorithm according to an embodiment of the present application;

Fig. 11 is an energy schematic diagram of an acoustic energy ray received by the receiver in the target room-related impulse response function generated according to the diffuse rain ray tracing algorithm according to an embodiment of the present application;

FIG. 12 is a schematic diagram of a rendering process provided by an exemplary embodiment of the present application, where the multimedia data to be rendered is 5.1-channel multimedia data as an example;

Fig. 13 is a schematic structural diagram of a virtual surround sound rendering device provided by an exemplary embodiment of the present application;

Fig. 14 is a schematic structural diagram of an electronic device provided by an exemplary embodiment of the present application.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with specific embodiments of the present application and corresponding drawings. Apparently, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

First, terms and terms involved in one or more embodiments of the present application are explained.

Virtual surround sound (English name is Virtual Surround or Simulated Surround), is a sound that can process multi-channel signals and play them back in two parallel speakers or headphones, and can make people feel the effect of surround sound . The virtual surround sound system is based on two-channel stereo, without adding channels and speakers, and broadcasting the sound field signal after being processed by the circuit, so that the listener feels that the sound comes from multiple directions and generates a simulated stereo field.

Head Related Transfer Functions (Head Related Transfer Functions, abbreviation: HRTF), also known as ATF (anatomical transfer function), is a sound localization algorithm. HRTF is a set of filters that can use technologies such as ITD (Interaural Time Delay), IAD (Interaural Amplitude Difference) and pinna frequency vibration to produce stereo sound effects, so that when the sound is transmitted to the pinna, ear canal and tympanic membrane in the human ear, The listener will have the feeling of surround sound effect. Through digital signal processing (Digital Signal Processing, DSP), HRTF can process the sound source of the virtual world in real time.

Mono audio is to mix audio signals from different directions and record it uniformly by recording equipment, and then play it by a speaker. Mono audio refers to a signal with only one channel, or multiple identical channels that contain no directional information. The audio in both earphones is the same in mono audio mode.

5.1 channel refers to the center channel, in layman’s terms, five speakers plus a subwoofer, front left and right channels, rear left and right surround channels, a center speaker, and the so-called 0.1 sound channel subwoofer channel. A total of 6 speakers can be connected to a 5.1-channel system.

7.1 channel refers to surround sound, that is, 7 speakers plus a subwoofer. The surround is actually virtual, and there are actually only 5 sound zones (left front surround, right front surround, center surround, left rear surround, right rear surround). The remaining 2 sound zones (left surround, right surround) are allocated from the main sound zone. Specifically, 7.1 channels include 2 front channels, 1 center channel, 2 side surround channels, 2 rear surround channels and 1 subwoofer.

As mentioned in the background technology, with the popularization and application of mobile wearable devices such as personal computers, smart phones, and head-mounted displays, more and more users, such as game lovers in three-dimensional space, hope to enjoy multi-channel surround sound easily and conveniently. At the same time, it is hoped that there will be as few playback devices as possible, but the playback effect of the original multi-channel system can still be maintained, which also puts forward higher requirements for the application range of virtual surround sound. However, most of the existing virtual surround sound technologies are rendered for a single scene, and the rendered scene is usually limited to a static scene.

In view of this, in order to improve the existing virtual surround sound technology so as to apply the virtual surround sound technology to a wider range of scenarios and improve the audio experience of the listener, the method provided in the embodiment of the present application considers determining the multimedia data to be rendered In the rendering method, the number of channels of the multimedia data to be rendered, the video and audio scene, and the change of the head rotation angle of the user listening to the multimedia data to be rendered are considered comprehensively, so that the target virtual surround sound rendered based on the determined rendering method can give Listening users can bring a better audio experience, and at the same time, taking into account the differences of different scenes, this rendering method can also be applied to more scenes. Specifically, in this specification, a virtual surround sound rendering method is provided. This specification also relates to a virtual surround sound rendering device, an electronic device, and a computer-readable storage medium, which will be described in detail in the following embodiments one by one.

Referring to FIG. 1 , FIG. 1 is a schematic flowchart of a system implementing a virtual surround sound rendering method provided by an embodiment of the present application. In one embodiment of the present application, in order to make the rendered virtual surround sound bring more immersive and spatial sound experience to the listening user, the number of channels of the multimedia data to be rendered, the listening user's experience scene, and the listening user's head position to determine the rendering function of the multimedia data to be rendered. Specifically, as shown in FIG. 1 , S1 channel analysis is performed on the multimedia data to be rendered to determine the number of channels of the multimedia data to be rendered. The number of channels may include monophonic, dual-channel, 5.1-channel and 7.1-channel Then execute S2 to determine the audio-visual scene, so as to determine the audio-visual scene of the multimedia data to be rendered from the provided recording studio small room, movie theater and concert hall, etc., and then perform head tracking on the listening user to execute S3 to determine the head position, Then execute S4 based on the number of channels of the multimedia data to be rendered, the audio-visual scene and the head position to determine that the rendering function is a target-related impulse response function, and then execute S5 to render based on the multiple channels of the multimedia data to be rendered based on the target-related impulse response function, The virtual surround sound is obtained, and finally the virtual surround sound execution S6 is input to the listening device for playback.

Referring to FIG. 2 , FIG. 2 shows a schematic flowchart of a virtual surround sound rendering method provided by an exemplary embodiment of the present application. As shown in Figure 2, the method may include:

Step 210, determine the number of audio channels and audiovisual scenes of the multimedia data to be rendered.

Among them, the number of channels of the multimedia data to be rendered can be determined by analyzing the channels of the multimedia data to be rendered. For example, the number of channels of the multimedia data to be rendered can be carried in the audio file of the multimedia data to be rendered. The content in the audio file header of the data is obtained. Generally speaking, the number of channels of the multimedia data to be rendered may include 1, 2, 6 and 8. When the channel number of the multimedia data to be rendered is 1, the channel number corresponds to monophonic. Since the listening user listens with both ears, the rendered multimedia data can be copied to the dual channel, that is, in the initial case The content of the resources arriving in the two channels is the same, and then through stereo rendering to the listening user's ears, the rendering of the multimedia data to be rendered in the two channels is completed. When the number of channels of the multimedia data to be rendered is 2, the multimedia data to be rendered is two-channel multimedia data; when the number of channels of the multimedia data to be rendered is 6, the multimedia data to be rendered can be 5.1 channels For multimedia data, when the number of channels of the multimedia data to be rendered is 8, the multimedia data to be rendered may be 7.1-channel multimedia data.

Optionally, the video and audio scenes to be rendered multimedia data include but not limited to small rooms in recording studios, movie theaters, and concert halls. Among them, one audio-visual scene corresponds to one room impact response function, and different audio-visual scenes correspond to different room impact response functions, so that the multimedia data to be rendered is rendered based on the room impact response function corresponding to a specific audio-visual scene to simulate the The playback effect of the resources to be rendered in the audio-visual scene enhances the sense of space and presence of the listening user listening to the rendered virtual surround sound.

In some exemplary embodiments, the audiovisual scene can be customized by the listening user to determine the audiovisual scene of the multimedia data to be rendered, including:

In response to an access request for multimedia data to be rendered, display a list of preset audio-visual scenes to the listening user. Room shock response function;

In response to listening to the user's selection instruction from the preset audio-visual scene list, the audio-visual scene of the multimedia data to be rendered is determined.

For example, when the listening user chooses to play the multimedia data to be rendered, that is, when the multimedia data to be rendered is accessed, a preset audio-visual scene list may be displayed on the listening device for the listening user to select. The preset audio-visual scene list may include For audio-visual scenes such as small rooms in recording studios, cinemas, and concert halls, users can select any audio-visual scene from the preset audio-visual scene list to simulate the playback scene of multimedia data to be rendered by clicking and other operations.

Step 220, determine the head rotation angle of the user listening to the multimedia data to be rendered.

It should be understood that since the rendering function is related to the relative position of the sound source to the ears, there may be large individual differences. Therefore, for different listening users, the ideal effective listening range is limited, and a slight rotation of the listening user's head may also cause problems such as inversion of the front and rear sound images. The embodiment of the present application is based on this, before determining the rendering function of the multimedia data to be rendered, the head-tracking module of the listening device may also be used to determine the head rotation angle of the listening user. Among them, the head tracking module of the listening device can be realized by modules such as gyroscopes and acceleration sensors in the listening device, and the listening device includes but is not limited to mobile wearable devices with multimedia data playback functions such as mobile phones, earphones, and head-mounted displays.

In some exemplary embodiments, a three-dimensional coordinate system may be established for the listening user's head, and the head rotation angle of the listening user may be determined according to the angle change value of each coordinate plane in the three-dimensional coordinate system. Specifically, determining the head rotation angle of the user listening to the multimedia data to be rendered includes:

Establish a three-dimensional coordinate system of the listening user's head;

Obtain the angle change value of the head of the listening user with respect to each plane of the three-dimensional coordinate system through the head tracking module built in the mobile device worn by the listening user;

Based on the angle change values corresponding to the planes of the three-dimensional coordinate system, the head rotation angle of the listening user is determined.

Wherein, the three-dimensional coordinate system of the listening user's head includes XOZ plane, ZOY plane and XOY plane;

FIG. 3 is a schematic diagram of establishing a three-dimensional coordinate system for a listening user's head according to a virtual surround sound rendering method provided by an embodiment of the present application. In Fig. 3, the origin of the three-dimensional coordinate system of the listening user's head may be the intersection point between the first connecting line and the second connecting line, wherein the first connecting line is the connecting line between the two ear tips of the listening user, and the second connecting line The two connecting lines are the connecting line between the central point between the eyes of the listening user and the point corresponding to the central point on the back of the head. The three-dimensional coordinate system of the head of the listening user is composed of the X coordinate system for pointing to the left and right (the The X coordinate system is parallel to the first connecting line), the Z coordinate system for pointing forward and backward (the Z coordinate system is parallel to the second connecting line), and the Y coordinate system for pointing up and down (the Y coordinate system is related to the listening user The lines perpendicular to the tangential plane of the center point of the top of the head are parallel to each other). Wherein, the deflection angle in the direction of the X coordinate system may be called a pitch angle, the deflection angle in the direction of the Y coordinate system may be called a yaw angle, and the deflection angle in the direction of the Z coordinate system may be called a roll angle. Usually, since the virtual speakers corresponding to each channel are placed on a plane, when determining the head rotation angle of the listening user, it can be determined based on the angle change value of the XOZ plane. However, since the listening device is worn on the listening user, it will rotate with the rotation of the listening user's head. Therefore, it cannot be guaranteed that the virtual speakers corresponding to each channel are always on the same plane. Based on this, the embodiment of the present application can correct the angle change value of the XOZ plane according to the angle change values of the ZOY plane and the XOY plane.

Suppose φ is the azimuth angle rotated counterclockwise from the XOZ plane, and θ is the elevation angle calculated from the XOY plane. Then the conversion between the spherical coordinates (x, y, z) and the azimuth angle and elevation angle can be realized by the following formulas (1)~(3):

Among them, r is the radius of the spherical coordinates, and the default value is 1.

Step 230, based on the number of channels, the audiovisual scene and the head rotation angle of the listening user, determine the target rendering function to render the multimedia data to be rendered to obtain the target virtual surround sound.

Wherein, the target rendering function may include a target head-related impulse response function and a target room impulse response function. The target head-related impulse response function is related to the listening user's head rotation angle, that is, the target head-related impulse response function is determined based on the listening user's head rotation angle, and a target head-related impulse response function corresponds to a time-domain diagram, which can be Based on the waveform in the time domain diagram, the multimedia data to be rendered input into each channel is filtered. Different head rotation angles correspond to different target-related impulse response functions, that is to say, the waveforms in the time-domain diagrams of the target head-related impulse response functions corresponding to different head rotation angles are different.

The target room-related impulse response function is related to the audio-visual scene of the multimedia data to be rendered, that is, the target room-related impulse response function is determined based on the audio-visual scene of the multimedia data to be rendered, and a target room-related impulse response function corresponds to a time-domain graph, which can be Based on the waveform in the time domain diagram, the multimedia data to be rendered input into each channel is filtered. Different audio-visual scenes correspond to different target room impulse response functions, that is to say, the waveforms in the time-domain diagrams of the target room-related impulse response functions corresponding to different audio-visual scenes are different.

In some exemplary embodiments, the target rendering function includes a target head-related impulse response function and a target room-related impulse response function, and the target is determined based on the number of channels of the multimedia data to be rendered, the audio-visual scene and the head rotation angle of the listening user. The head-related impulse response function and the target room-related impulse response function are used to render the multimedia data to be rendered to obtain the target virtual surround sound, including:

Determine the angle between the virtual speaker and the listening user's head based on the number of channels and the audiovisual scene;

updating the angle between the virtual speaker and the head of the listening user based on the head rotation angle of the listening user;

Based on the updated angle between the virtual speaker and the listening user's head and the audio-visual scene, the target head-related impulse response function and the target room-related impulse response function are determined to render the multimedia data to be rendered to obtain the target virtual surround sound.

In some exemplary embodiments, in order to eliminate the head positioning effect of the listening user and improve the problem of sound image position confusion, the embodiments of the present application may change the listening user's head position, that is, the listening user's head angle appears When changing, correct the angle between the virtual speaker corresponding to each channel and the listening user's head, so that the relative position between the virtual speaker corresponding to each channel and the listening user's head does not change with the listening user's head position Changes due to changes, thereby eliminating the head positioning effect of the listening user and avoiding confusion in the sound and image positions. Specifically, based on the head rotation angle of the listening user, the included angles between the plurality of virtual speakers and the listening user's head are updated, including:

Determine the direction and angle values of the listening user's head rotation angle;

Based on the direction and angle value of the listening user's head rotation angle, determine the correction direction and correction value of the included angle between the virtual speaker and the listening user's head;

Based on the correction direction and the correction value, the angle between the virtual speaker and the listening user's head is updated.

For example, if the head of the listening user rotates 90° counterclockwise on the XOZ plane with the origin of the three-dimensional coordinate system of the head as the center point, in order to eliminate the relative position change between the virtual speakers corresponding to each channel and the head of the listening user Due to the head positioning effect and the confusion of the sound image position, the virtual speakers corresponding to each channel can also listen to the user's head with the origin of the three-dimensional coordinate system as the center point and rotate 90° counterclockwise or 270° clockwise, so that each The relative position between the virtual speaker corresponding to the channel and the listening user's head remains unchanged.

In some exemplary embodiments, based on the angle between the updated virtual speaker and the listening user's head and the audiovisual scene, the target head-related impulse response function and the target room-related impulse response function are determined for the multimedia to be rendered The data is rendered to obtain the target virtual surround sound, including:

Based on the angle between the updated virtual speaker and the head of the listening user, determine a target head-related impulse response function and a target room-related impulse response function corresponding to the virtual speaker;

Based on the target head-related impulse response function and the target room-related impulse response function, the multimedia data to be rendered input to the virtual speaker is rendered to obtain the target virtual surround sound.

In some exemplary embodiments, after the target head-related impulse response function and the target room-related impulse response function corresponding to each channel are superimposed, rendering processing is performed on the multimedia data to be rendered to be played by the channel. Specifically, based on the target head-related impulse response function and the target room-related impulse response function, the multimedia data to be rendered input to the virtual speaker is rendered to obtain the target virtual surround sound, including:

The target head-related impulse response function and the target room-related impulse response function are superimposed to obtain the superimposed target impulse response function;

Perform convolution operation on the superimposed target impulse response function and the multimedia data to be rendered corresponding to the virtual speaker;

Based on the result of the convolution operation, the target virtual surround sound is obtained.

The method provided in the embodiment of the present application will be described in detail below by taking the rendering process of multimedia data to be rendered in monophonic, dual-channel, 5.1-channel and 7.1-channel as examples.

For the monophonic multimedia data to be rendered, it is usually necessary to copy the multimedia data content in the monophonic to the dual-channel, and then render to the listening user’s ears through the dual-channel rendering to complete the rendering. Referring to FIG. 4 , FIG. 4 is a schematic diagram of a scene where binaural multimedia data is rendered to both ears of a listening user according to a virtual surround sound rendering method provided by an embodiment of the present application. Assuming that the virtual speakers corresponding to the left and right channels in this scene are L and R respectively, and the target-related impulse response functions of the virtual speakers L and R to the user's ears are H _RL , H _LL , H _LR , H _RR , then listen to The sound pressures _PL and P _R of the user's ears are the sum of the sound pressures of the ears generated by the virtual speakers respectively. Wherein, P _L =H _LL *L+H _RL *R, P _R =H _LR *L+H _RR R, and * represents a convolution operation.

For 5.1-channel multimedia data to be rendered, refer to FIG. 5 , which is a schematic diagram of rendering 5.1-channel multimedia data to the ears of a listening user according to a virtual surround sound rendering method provided in an embodiment of the present application. Assuming that the virtual speakers corresponding to each channel in the 5.1 channel are L, C, R, LR, RS, LFE respectively, the target-related impulse response function of the virtual speakers L, C, R, LS, RS, LFE to the user's ears H _RL , H _LL , H _LR , H _RR , H _CL , H _CR , H _RSR , H _RSL , H _LSR , H _LSL , H _LFER , H _LFEL , then listen to the sound pressure _PL and P _R is the sum of the binaural sound pressures produced by the virtual speakers corresponding to each channel. Among them, P _L ＝H _LL *L+H _LR *R+H _CL *C+H _LSL *LS+H _RSL *RS+H _LFEL *LFE, P _R ＝H _LR *L+H _Rr r+H _CR * C+H _LSR *LS+H _RSR *RS+H _LFER *LFE, * means convolution operation.

For 7.1-channel multimedia data to be rendered, refer to FIG. 6 , which is a schematic diagram of rendering 7.1-channel multimedia data to the ears of a listening user according to a virtual surround sound rendering method provided by an embodiment of the present application. Assume that the virtual speakers corresponding to each channel in the 7.1 channel are L, C, R, LS, RS, LBS, RBS, LFF, and the virtual speakers L, C, R, LS, RS, LBS, RBS, LFE to the listening user The target-related impulse response functions of both ears are H _RL , H _LL , H _LR , H RR , H _CL , H _CR , H _RSR , H _RSL , H _LSR , H _LSL , H _RBSR , H _RBSL , _{H LBSR ,} H _LBSL , H _LFER , H _LFEL , then the binaural sound pressures _PL and _PR are the sum of the binaural sound pressures produced by the virtual speakers corresponding to each channel. Among them, P _L ＝H _LL *L+H _RL *R+H _CL *C+H _LSL *LS+H _RSL *RS+H _LBSL *LBS+H _RBSL *RBS+H _LFEL *LFE, P _R ＝H _LR *L+H _RR *R+H _CR *C+H _LSR *LS+H _RSR *RS+H _LBSR *LBS+H _RBSR *RBS++H _LFER *LFE, * means convolution operation.

It should be understood that when the influence of the listening user's head rotation angle on the virtual surround sound is considered separately, the above-mentioned target rendering function is the target head-related impulse response function; is the target room-related impulse response function. When comprehensively considering the listening user's head rotation angle and the influence of the audio-visual scene on the virtual surround sound, the above-mentioned target rendering function may be a rendering function obtained by superimposing the target head-related impulse response function and the target room-related impulse response function.

When considering the impact of audio-visual scenes on virtual surround sound, the roomsim simulation tool can be used to generate a rectangular room as shown in Figure 7. The length, width and height of the room can be set according to requirements, and the six planes of the room can be adjusted according to reflection and absorption. The different needs of the coefficients are set for different materials. For example, when simulating a small room in a recording studio, materials such as sound-absorbing cotton with better absorption can be selected as the materials for the six planes of the room. The room simulation shown in FIG. 7 is a schematic diagram of a scenario in which the left ear and the right ear receive multimedia data played by a 5.1-channel virtual speaker. Fig. 8 is the oscillogram of the room-related impulse response function between the 5.1-channel virtual loudspeaker and the left ear and the right ear in the audio-visual scene shown in Fig. 7, and the room-related impulse response function mainly includes direct sound, early specular reflection and Reverb tail and several other perceptually relevant components.

As an implementation manner, the above-mentioned target head-related impulse response function and target room-related impulse response function can be generated using an image-source method (image-source method in English) and a diffuse rain ray-tracing algorithm (diffuserain ray-tracing algorithm). In practical applications, waveform diagrams of target room-related impulse response functions under different audio-visual scenes and waveform diagrams of target head-related impulse response functions under different angles can be generated in advance. When performing virtual surround sound rendering, the corresponding audio-visual The waveform diagram of the target room-related impulse response function in the scene and the waveform diagram of the target head-related impulse response function in the corresponding angle can be filtered for the input video and audio signals.

Taking the generation process of the target room-related impulse response function as an example, see FIG. 9 , which is a schematic diagram of the sound propagation path in the generation of the target room-related impulse response function according to the image source method according to an embodiment of the present application. In the image source method, a virtual image source can be created by mirroring the sound source on the wall of the room. As shown in Figure 9, the sound source is S, and the mirror images of the sound source S on the four walls are S1～S4, S1～ S4 is the created virtual image source. The straight line from the virtual image sources S1-S4 to the receiver R corresponds to the sound propagation path from the sound source S reflected on multiple walls to the receiver R in the room. By taking the length of these lines and the absorption and reflection coefficients of the walls it intersects, the contribution of the corresponding sound propagation path to the impulse response of the room can be calculated. The image source method can accurately find all propagation paths in a room, which makes it ideal for simulating direct sound and low-order reflections. However, this method is computationally inefficient for higher-order reflections because the number of virtual sources increases rapidly with the reflection order, so the image source method is usually suitable for simulating early specular reflections.

Continuing to take the generation process of the target room-related impulse response function as an example, see FIG. 10 , which is a schematic diagram of the propagation path of acoustic energy rays in the target room-related impulse response function generated according to the diffuse rain ray tracing algorithm according to an embodiment of the present application. The diffuse rain ray tracing algorithm generates higher order reflections, reverb tails and diffuse reflections. In the diffuse rain ray tracing algorithm, a ray of acoustic energy is emitted from a sound source S and traced throughout the room, as shown in Figure 10. When the sound energy ray hits the surface of the object, the sound energy carried by it can be reduced according to the absorption of the surface of the object. Next, the direct contribution of the acoustic energy ray to the room impulse response can be determined by launching a second ray from the point of impact to the receiver R and recording the angle of incidence, time of arrival, and residual acoustic energy at the receiver for this ray. The original first ray emitted from the sound source S can continue to be emitted in random directions from the point of impact and tracked further until its sound energy falls below a preset energy threshold. All rays of acoustic energy can be treated in this way, and since both absorption and diffuse reflection are frequency-dependent phenomena, the process is repeatable across all frequency bands.

As shown in FIG. 11 , it is an energy schematic diagram of an acoustic energy ray received by a receiver in a target room-related impulse response function generated according to a diffuse rain ray tracing algorithm according to an embodiment of the present application. At receiver R, the energy of a ray with frequency f at arrival time t at angle ψ=(θ, φ) is accumulated in the time-frequency histogram Ei(n,k). For each spherical bin, the time-frequency histogram Ei(n,k) can be transformed into the target room-related impulse response function through the following steps. First, it is a Poisson noise process. The spectrogram of the Poisson noise signal is white and can be shaped according to the time-frequency histogram Ei(n,k). Then, the shaped noise signal is convolved with the impulse response of the receiver R with a ψ angle. Finally, the sum of all the spherical bin signals obtained is an impulse response, and then superimposed on the output of the image source method, the complete target room-related impulse response function can be obtained.

It should be noted that, according to different playback devices (that is, the above-mentioned listening devices), virtual surround sound can be divided into virtual surround sound based on speaker playback and virtual surround sound based on earphone playback. In essence, both of them use binaural signals to realize the effect of virtual surround sound. It is only necessary to select the virtual speakers corresponding to each channel in the above-mentioned binaural sound pressure _PL , _PR formulas and playback equipment. The corresponding virtual surround sound effect can be realized by performing filtering processing on the corresponding target-related impulse response function.

In addition, in order to prevent the filtered signal from popping and other sharp noises, the virtual surround sound after virtual surround processing can also be protected by a preset limiter, that is, no matter how the input level increases, its maximum output level cannot be greater than the maximum preset output level value. Specifically, based on the number of channels, the audio-visual scene and the listening user's head rotation angle, the target rendering function is determined to render the multimedia data to be rendered, and the target virtual surround sound is obtained, including:

Determine the target-related impulse response function based on the number of channels, the audio-visual scene and the head rotation angle of the listening user;

Render the multimedia data to be rendered based on the target rendering function to obtain the rendered virtual surround sound;

Converting the level in the rendered virtual surround sound greater than the maximum preset output level value to the maximum preset output level value to obtain the target virtual surround sound.

Referring to FIG. 12 , it is a schematic diagram of a rendering process provided by an exemplary embodiment of the present application, where the multimedia data to be rendered is 5.1-channel multimedia data as an example. As shown in Figure 12, the process may include: S121, inputting the multimedia data to be rendered; S122, analyzing the multimedia data to be rendered, and determining that the multimedia data to be rendered is 5.1-channel multimedia data; S123, determining to listen to the target audio and video selected by the user Scene; S124, obtain the head rotation angle information of the listening user through the head tracking module of the listening device; S125, determine the room-related impulse response function corresponding to the target audio-visual scene and the head correlation corresponding to the head rotation angle information of the listening user Impulse response function; S126, superimpose the room-related impulse response function and the head-related impulse response function to obtain a rendering function; S127, use the rendering function to render multiple channels of the multimedia data to be rendered, and output to the listening device.

Taking the 5.1 channel as an example, assuming that from a counterclockwise perspective, the angles between the virtual speakers L, C, R, LS, RS, LFe and the listening user's head are 45°, 0°, 315°, 135°, respectively. °, 225°, 22.5°, and they are all on the same plane. At this time, from the head-related impulse response function that determines the corresponding angle position, the input signal of each virtual speaker is rendered using the formulas of _PL and _PR in the 5.1 channel to obtain the current output of each virtual speaker. Assuming that when the head tracking module detects that the head of the listening user rotates counterclockwise by 90°, the relative angles between the virtual speakers of L, C, R, LS, RS, and LFE and the head of the listening user change. For: 315°, 270°, 225°, 45°, 135°, 67.5°. At this time, it is necessary to select the head-related impulse response function at the current angle to render the input signals of each virtual speaker using the formulas of _PL and _PR in the 5.1 channel to obtain the output to the current virtual speakers. That is to say, the head-related impulse response function selected by each virtual speaker needs to be refreshed in real time according to the change of the listening user's head angle, so that the sound source does not rotate with the rotation of the user's head, eliminates the positioning effect in the head, and improves the sound image The problem of location confusion.

In addition, the method provided by this embodiment can be applied to any application scene where there is virtual surround sound rendering, and can determine the audio frequency for rendering in real time according to the number of channels with rendered multimedia data, the audio-visual scene, and the head rotation angle of the listening user. The target rendering function of the multimedia data to be rendered makes the rendered virtual surround sound more suitable for the actual device, the actual scene and the head angle of the listening user, thereby bringing a better audio experience to the user.

In the virtual surround sound rendering method provided by some embodiments of the present application, when determining the rendering method of the multimedia data to be rendered, not only the number of channels of the multimedia data to be rendered and the audio-visual scene can be determined, but also the multimedia data to be rendered can be considered The conversion of the head rotation angle of the user listening to the data, and finally based on the number of channels of the multimedia data to be rendered, the audio-visual scene and the head rotation angle of the listening user, so that the determined target rendering function is consistent with the current environment of the listening user, The number of channels of the multimedia data to be rendered matches the head rotation angle of the listening user, so that the target virtual surround sound can be obtained by rendering the multimedia data to be rendered through the target rendering function, which can also bring a better audio experience to the listening user , Moreover, this rendering method fully takes into account the differences of different scenes, and can also be applied to more scenes.

It should be noted that the subject of execution of each step of the method provided in the foregoing embodiments may be the same device, or the method may also be executed by different devices. For example, the execution subject of steps 210 to 230 may be device A; for another example, the execution subject of steps 210 to 220 may be device A, and the execution subject of step 230 may be device B; and so on.

In addition, in some of the processes described in the above embodiments and accompanying drawings, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear herein or executed in parallel , the sequence numbers of the operations, such as 210, 220, etc., are only used to distinguish different operations, and the sequence numbers themselves do not represent any execution sequence. Additionally, the processes may include more or fewer operations, and the operations may be performed sequentially or in parallel. It should be noted that the descriptions of "first" and "second" in this article are used to distinguish different messages, devices, modules, etc. are different types.

Fig. 13 is a schematic structural diagram of a virtual surround sound rendering device provided by an exemplary embodiment of the present application. As shown in Figure 13, the device includes: a scene determination module 1310, an angle determination module 1320 and a resource rendering module 1330, wherein:

Scene determination module 1310, configured to determine the number of audio channels and audio-visual scenes of the multimedia data to be rendered;

An angle determination module 1320, configured to determine the head rotation angle of the user listening to the multimedia data to be rendered;

The resource rendering module 1330 is configured to determine a target rendering function to render the multimedia data to be rendered based on the number of channels, the audio-visual scene and the head rotation angle of the listening user to obtain target virtual surround sound.

The virtual surround sound rendering device provided by the embodiment of the present application can not only determine the number of channels and audio-visual scenes of the multimedia data to be rendered, but also consider the listening of the multimedia data to be rendered when determining the rendering method of the multimedia data to be rendered. The transformation of the user's head rotation angle, finally based on the number of channels of the multimedia data to be rendered, the audio-visual scene and the listening user's head rotation angle, so that the determined target rendering function is consistent with the listening user's current environment, the multimedia to be rendered The number of channels of the data and the head rotation angle of the listening user match, so that the target virtual surround sound obtained by rendering the multimedia data to be rendered through the target rendering function can bring a better audio experience to the listening user, and , this rendering method fully takes into account the differences of different scenes, and can also be applied to more scenes.

Further optionally, the target rendering function includes a target head-related impulse response function and a target room-related impulse response function, and the resource rendering module 1330 based on the and the audio-visual scene and the head rotation angle of the listening user, Determine the target rendering function to render the multimedia data to be rendered to obtain the target virtual surround sound, specifically for:

Based on the number of channels and the audio-visual scene, determine the angle between the virtual speaker and the head of the listening user;

updating an angle between the virtual speaker and the listening user's head based on the head rotation angle;

Based on the updated angle between the virtual speaker and the listening user's head and the audio-visual scene, determine the target head-related impulse response function and the target room-related impulse response function to treat the target The multimedia data is rendered for rendering to obtain the target virtual surround sound.

Further optionally, when the resource rendering module 1330 updates the angle between the virtual speaker and the listening user's head based on the head rotation angle, it is specifically used to:

determining the direction and angle value of the head rotation angle of the listening user;

determining a correction direction and a correction value of an included angle between the virtual speaker and the listening user's head based on the direction and angle value of the listening user's head rotation angle;

Based on the correction direction and the correction value, the angle between the virtual speaker and the listening user's head is updated.

Further optionally, the resource rendering module 1330 determines the target head-related impulse response function and the When the target room-related impulse response function is used to render the multimedia data to be rendered to obtain the target virtual surround sound, it is specifically used for:

determining a target head-related impulse response function and a target room-related impulse response function corresponding to the virtual speaker based on the updated angle between the virtual speaker and the head of the listening user;

Based on the target head-related impulse response function and the target room-related impulse response function, the multimedia data to be rendered input to the virtual speaker is rendered to obtain the target virtual surround sound.

Further optionally, the resource rendering module 1330 renders the multimedia data input to the virtual speaker to be rendered based on the target head-related impulse response function and the target room-related impulse response function to obtain the target virtual In surround sound, specifically for:

superimposing the target head-related impulse response function and the target room-related impulse response function to obtain a plurality of superimposed target impulse response functions;

performing a convolution operation on the superimposed target impulse response function and the corresponding multimedia data to be rendered of the virtual speaker;

Based on the result of the convolution operation, the target virtual surround sound is obtained.

Further optionally, when the scene determination module 1310 determines the video and audio scene of the multimedia data to be rendered, it is specifically used for:

Responding to the access request of the multimedia data to be rendered, displaying a preset audio-visual scene list to the listening user, the preset audio-visual scene list includes a plurality of preset audio-visual scenes, a plurality of the preset audio-visual scenes The assumed audio-visual scenes correspond to different room shock response functions;

In response to the listening user's instruction to select the audio-visual scene of the multimedia data to be rendered from a list of preset audio-visual scenes, the audio-visual scene of the multimedia data to be rendered is determined.

Further optionally, when the angle determination module 1320 determines the head rotation angle of the user listening to the multimedia data to be rendered, it is specifically used for:

Establishing the three-dimensional coordinate system of the head of the listening user;

Obtain the angle change value of the head of the listening user with respect to each plane of the three-dimensional coordinate system through a head tracking module built in the mobile device worn by the listening user;

Based on the angle change values corresponding to the planes of the three-dimensional coordinate system, the head rotation angle of the listening user is determined.

The virtual surround sound rendering device can implement the methods in the method embodiments shown in FIGS. 1 to 12 . For details, reference may be made to the virtual surround sound rendering method in the embodiments shown in FIGS. 1 to 12 , which will not be repeated here.

Fig. 14 is a schematic structural diagram of an electronic device provided by an exemplary embodiment of the present application, and the electronic device may include mobile wearable devices such as earphones and head-mounted displays. As shown in FIG. 14 , the device includes: a memory 141 and a processor 142 .

The memory 141 is used to store computer programs, and can be configured to store other various data to support operations on the computing device. Examples of such data include instructions for any application or method operating on the computing device, contact data, phonebook data, messages, pictures, videos, etc.

The processor 142 is coupled with the memory 141, and is used to execute the computer program in the memory 141, so as to: determine the number of channels and audio-visual scenes of the multimedia data to be rendered; determine the head rotation of the listening user of the multimedia data to be rendered Angle: based on the number of channels of the multimedia data to be rendered, the audio-visual scene and the head rotation angle of the listening user, a target related impulse response function is determined to render the multimedia data to be rendered to obtain a target virtual surround sound.

Further optionally, the target rendering function includes a target head-related impulse response function and a target room-related impulse response function, and the processor 142 determines The target rendering function is used to render the multimedia data to be rendered to obtain the target virtual surround sound, specifically for:

Based on the number of channels and the audio-visual scene, determine the angle between the virtual speaker and the head of the listening user;

updating an angle between the virtual speaker and the listening user's head based on the head rotation angle;

Based on the updated angle between the virtual speaker and the listening user's head and the audio-visual scene, determine the target head-related impulse response function and the target room-related impulse response function to treat the target Render the multimedia data to obtain the target virtual surround sound.

Further optionally, when the processor 142 updates the angle between the virtual speaker and the listening user's head based on the head rotation angle, it is specifically used to:

determining the direction and angle value of the head rotation angle of the listening user;

determining a correction direction and a correction value of an included angle between the virtual speaker and the listening user's head based on the direction and angle value of the listening user's head rotation angle;

Based on the correction direction and the correction value, the angle between the virtual speaker and the listening user's head is updated.

Further optionally, the processor 142 determines the target head-related impulse response function and the target head-related impulse response function based on the updated angle between the virtual speaker and the listening user's head and the audio-visual scene. When the room-related impulse response function is used to render the multimedia data to be rendered to obtain the target virtual surround sound, it is specifically used for:

determining a target head-related impulse response function and a target room-related impulse response function corresponding to the virtual speaker based on the updated angle between the virtual speaker and the head of the listening user;

Based on the target head-related impulse response function and the target room-related impulse response function, the multimedia data to be rendered input to the virtual speaker is rendered to obtain the target virtual surround sound.

Further optionally, the processor 142 renders the multimedia data input to the virtual speaker to be rendered based on the target head-related impulse response function and the target room-related impulse response function to obtain the target virtual surround When sounding, it is specifically used for:

superimposing the target head-related impulse response function and the target room-related impulse response function to obtain a plurality of superimposed target impulse response functions;

performing a convolution operation on the superimposed target impulse response function and the corresponding multimedia data to be rendered of the virtual speaker;

Based on the result of the convolution operation, the target virtual surround sound is obtained.

Further optionally, when the processor 142 determines the video and audio scene of the multimedia data to be rendered, it is specifically used for:

Responding to the access request of the multimedia data to be rendered, displaying a preset audio-visual scene list to the listening user, the preset audio-visual scene list includes a plurality of preset audio-visual scenes, a plurality of the preset audio-visual scenes The assumed audio-visual scenes correspond to different room shock response functions;

In response to the listening user's instruction to select the audio-visual scene of the multimedia data to be rendered from a list of preset audio-visual scenes, the audio-visual scene of the multimedia data to be rendered is determined.

Further optionally, when the processor 142 determines the head rotation angle of the user listening to the multimedia data to be rendered, it is specifically used for:

Establishing the three-dimensional coordinate system of the head of the listening user;

Obtain the angle change value of the head of the listening user with respect to each plane of the three-dimensional coordinate system through a head tracking module built in the mobile device worn by the listening user;

Based on the angle change values corresponding to the planes of the three-dimensional coordinate system, the head rotation angle of the listening user is determined.

Further, as shown in FIG. 14 , the electronic device further includes: a communication component 143 , a display 144 , a power supply component 145 , an audio component 146 and other components. FIG. 14 only schematically shows some components, which does not mean that the electronic device only includes the components shown in FIG. 14 . In addition, according to different implementation forms of the traffic playback device, the components in the dotted box in FIG. 14 are optional components, but not mandatory components. For example, when an electronic device is implemented as a terminal device such as a smart phone, a tablet computer, or a desktop computer, the components within the dashed box in Figure 14 may be included; when the electronic device is implemented as a service end such as a conventional server, cloud server, data center, or server array, the equipment, the components in the dotted box in Figure 14 may not be included.

Correspondingly, the embodiment of the present application also provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the processor can implement the steps in the above-mentioned embodiment of the virtual surround sound rendering method.

The above-mentioned communication component in FIG. 14 is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on communication standards, such as WiFi, 2G or 3G, or a combination thereof. In one exemplary embodiment, the communication component receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component may further include a near field communication (Near Field Communication, NFC) module, a radio frequency identification (Radio Frequency Identification, RFID) technology, an infrared data association (Infrared Data Association, IrDA) technology, Ultra Wideband (Ultra WideBand, UWB) technology, Bluetooth (Bluetooth, BT) technology, etc.

The above memory in FIG. 14 can be realized by any type of volatile or non-volatile storage device or their combination, such as Static Random-Access Memory (Static Random-Access Memory, SRAM), Electrically Erasable and Programmable Only Electrically Erasable Programmable read only memory (EEPROM), Electrically Erasable Programmable Read Only Memory (EPROM), Programmable read-only memory (PROM), read-only memory ( Read-Only Memory, ROM), magnetic memory, flash memory, magnetic disk or optical disk.

The above-mentioned display in FIG. 14 includes a screen, and the screen may include a liquid crystal display (Liquid Crystal Display, LCD) and a touch panel (Touchpanel, TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may not only sense a boundary of a touch or swipe action, but also detect duration and pressure associated with the touch or swipe action.

The above-mentioned power supply component in FIG. 14 provides power for various components of the device where the power supply component is located. A power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component resides.

The above-mentioned audio components in FIG. 14 may be configured to output and/or input audio signals. For example, the audio component includes a microphone (microphone, MIC). When the device where the audio component is located is in an operation mode, such as a calling mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in a memory or sent via a communication component. In some embodiments, the audio component further includes a speaker for outputting audio signals.

Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowcharts and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

Memory may include non-permanent storage in computer readable media, in the form of random access memory (RAM) and/or nonvolatile memory such as read only memory (ROM) or flash RAM. Memory is an example of computer readable media.

Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can be implemented by any method or technology for storage of information. Information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (Phase-change memory, PRAM), static random access memory (SRAM), dynamic random access memory (Dynamic Random Access Memory, DRAM), other types of random Access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM ROM ( CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic tape cartridge, magnetic tape magnetic disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media excludes transitory computer-readable media, such as modulated data signals and carrier waves.

It should also be noted that the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes Other elements not expressly listed, or elements inherent in the process, method, commodity, or apparatus are also included. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

The above descriptions are only examples of the present application, and are not intended to limit the present application. For those skilled in the art, various modifications and changes may occur in this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims (10)

1. A virtual surround sound rendering method, characterized in that, comprising: Determine the number of channels and audio-visual scenes of the multimedia data to be rendered; determining the head rotation angle of the user listening to the multimedia data to be rendered; Determine a target rendering function based on the number of channels, the video and audio scene, and the head rotation angle, so as to render the multimedia data to be rendered based on the target rendering function to obtain target virtual surround sound. 2. The method according to claim 1, wherein the target rendering function comprises a target head-related impulse response function and a target room-related impulse response function, and the based on the number of channels and the audio-visual scene and the According to the head rotation angle, the target rendering function is determined to render the multimedia data to be rendered to obtain the target virtual surround sound, including: Based on the number of channels and the audio-visual scene, determine the angle between the virtual speaker and the head of the listening user; updating an angle between the virtual speaker and the listening user's head based on the head rotation angle; Based on the updated angle between the virtual speaker and the listening user's head and the audio-visual scene, determine the target head-related impulse response function and the target room-related impulse response function to treat the target The multimedia data is rendered for rendering to obtain the target virtual surround sound. 3. The method according to claim 2, wherein the updating the angle between the virtual speaker and the listening user's head based on the head rotation angle comprises: determining the direction and angle value of the head rotation angle of the listening user; determining a correction direction and a correction value of an included angle between the virtual speaker and the listening user's head based on the direction and angle value of the listening user's head rotation angle; Based on the correction direction and the correction value, the angle between the virtual speaker and the listening user's head is updated. 4. The method according to claim 2 or 3, wherein the target is determined based on the updated angle between the virtual speaker and the listening user's head and the audio-visual scene. The head-related impulse response function and the target room-related impulse response function are used to render the multimedia data to be rendered to obtain the target virtual surround sound, including: determining a target head-related impulse response function and a target room-related impulse response function corresponding to the virtual speaker based on the updated angle between the virtual speaker and the head of the listening user; Based on the target head-related impulse response function and the target room-related impulse response function, the multimedia data to be rendered input to the virtual speaker is rendered to obtain the target virtual surround sound. 5. The method according to claim 4, wherein the multimedia data to be rendered input to the virtual speaker is rendered based on the target head-related impulse response function and the target room-related impulse response function , to get the target virtual surround, including: superimposing the target head-related impulse response function and the target room-related impulse response function to obtain a plurality of superimposed target impulse response functions; performing a convolution operation on the superimposed target impulse response function and the corresponding multimedia data to be rendered of the virtual speaker; Based on the result of the convolution operation, the target virtual surround sound is obtained. 6. The method according to claim 1, wherein the determining the audio-visual scene of the multimedia data to be rendered comprises: Responding to the access request of the multimedia data to be rendered, displaying a preset audio-visual scene list to the listening user, the preset audio-visual scene list includes a plurality of preset audio-visual scenes, a plurality of the preset audio-visual scenes The assumed audio-visual scenes correspond to different room shock response functions; In response to the listening user's instruction to select the audio-visual scene of the multimedia data to be rendered from a list of preset audio-visual scenes, the audio-visual scene of the multimedia data to be rendered is determined. 7. The method according to claim 1, wherein the determining the head rotation angle of the user listening to the multimedia data to be rendered comprises: Establishing the three-dimensional coordinate system of the head of the listening user; Obtain the angle change value of the head of the listening user with respect to each plane of the three-dimensional coordinate system through a head tracking module built in the mobile device worn by the listening user; Based on the angle change values corresponding to the planes of the three-dimensional coordinate system, the head rotation angle of the listening user is determined. 8. A virtual surround sound rendering device, comprising: The scene determination module is used to determine the number of audio channels and the audio-visual scene of the multimedia data to be rendered; An angle determination module, configured to determine the head rotation angle of the user listening to the multimedia data to be rendered; The resource rendering module is configured to determine a target rendering function to render the multimedia data to be rendered based on the number of channels, the audio-visual scene, and the head rotation angle to obtain target virtual surround sound. 9. An electronic device, comprising: a memory and a processor; The memory is used to store computer programs; The processor, coupled to the memory, is configured to execute the computer program for: Determine the number of channels and audio-visual scenes of the multimedia data to be rendered; determining the head rotation angle of the user listening to the multimedia data to be rendered; Determine a target rendering function based on the number of channels, the video and audio scene, and the head rotation angle, so as to render the multimedia data to be rendered based on the target rendering function to obtain target virtual surround sound. 10. A computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, the processor is caused to realize the virtual surround sound described in any one of claims 1-7 Steps in the render method.

Images