CN118413802A - Spatial audio rendering method and device - Google Patents

Abstract

The present application provides a spatial audio rendering method and device. The spatial audio rendering method of the present application includes: obtaining the delay information of the audio content provider; obtaining the predicted head movement position according to the delay information; sending first rendering information to the audio content provider, wherein the first rendering information includes the predicted head movement position. The present application can eliminate the delay, so that the rendering effect exactly matches the user's head movement position, achieving a real and immersive listening experience.

Description

Spatial audio rendering method and device

Technical Field

The present application relates to audio processing technology, and in particular to a spatial audio rendering method and device.

Background technique

In the real three-dimensional space, sound can have two characteristics: sense of direction and sense of space. Spatial audio technology usually refers to the technology that realizes the immersive audio experience of simulating the above two characteristics on head-mounted playback devices such as headphones. People's judgment of the sense of direction of sound is mainly affected by factors such as time difference, sound level difference, human body filtering effect learned during growth, and head shaking. Among them, the three elements of time difference, sound level difference, and human body filtering effect can be comprehensively expressed as head related transfer functions (HRTF); head shaking in all directions is of great help to us in judging the location of the sound source. The indoor sound field can be composed of direct sound, early reflection sound (ER) and late reverberation sound (LR). People's sense of space for sound is mainly based on ER and LR.

Using spatial audio technology, stereo, multi-channel surround sound, specially produced multi-object sound sources and other sound sources in different formats can be rendered into binaural signals, so that you can enjoy a real, immersive listening experience on headphones. Currently, head tracking can be performed through sensors such as gyroscopes in headphones, and the corresponding rotation or displacement changes can be reflected in the spatial rendering of the sound source.

However, spatial audio rendering usually requires a lot of computing power, and head tracking requires low latency. If spatial audio rendering is placed on the audio content provider (for example, mobile phones, tablets, PCs, etc.), although it can meet the computing power requirements, it will introduce higher latency, especially in head movement scenarios, which will affect the overall listening experience.

Summary of the invention

The present application provides a spatial audio rendering method and device to eliminate time delay, so that the rendering effect exactly matches the user's head movement position, achieving a real, immersive listening experience.

In a first aspect, the present application provides a spatial audio rendering method, comprising: obtaining delay information of an audio content provider; obtaining a predicted head movement position based on the delay information; and sending first rendering information to the audio content provider, the first rendering information including the predicted head movement position.

This application predicts the user's head movement position to obtain the predicted head movement position, and after sending the predicted head movement position to the audio content provider, the audio content provider can render the audio frame based on the predicted head movement position, so that the rendered audio frame obtained can provide the listening effect corresponding to the predicted head movement position. Then when the rendered audio frame is sent to the audio signal playback end, the predicted time can offset the delay caused by rendering processing, link transmission, etc., so that when the audio frame is played at the audio signal playback end, it just matches the user's head movement position, achieving the real and immersive listening experience described above.

The delay information is used to indicate the delay of the audio content provider, which can include the following situations:

The first type of delay information includes the delay sent by the audio content provider.

After determining the delay caused by its own transmission link, rendering processing, etc., the audio content provider can encapsulate the delay in the first information sent to the audio signal player and send it to the audio signal player. In this way, the audio signal player can directly extract the delay of the audio content provider from the first information from the audio content provider.

The second type of delay information includes the second historical head movement position sent by the audio content provider.

The audio signal playback end (such as headphones) can periodically detect the current position of the head of the user wearing the headphones (referred to as the measured head movement position in this article) through sensors (such as gyroscopes, gravity sensors, etc.), so that the audio signal playback end can send the measured head movement position to the audio content provider.

After receiving the above-mentioned measured head movement position, the audio content provider does not process it first, and still processes the audio frame according to the established process. When the processing is completed and the first information is to be sent to the audio signal playback end, the above-mentioned measured head movement position is carried in the first information. Since a period of time has passed at this time, the above-mentioned measured head movement position becomes the historical head movement position (referred to as the second historical head movement position in this article). It should be understood that, in a mathematical sense, the above-mentioned measured head movement position and the above-mentioned second historical head movement position are the same.

Based on this, the audio signal playback end can obtain the time difference between the time of receiving the second historical head movement position and the time of sending the actual head movement position (that is, calculating the time difference between the receiving time and the sending time of the same head movement position), thereby indirectly obtaining the delay of the audio content provider.

The third type is that the delay information includes the second historical head movement position and the first uncertainty coefficient sent by the audio content provider. The second historical head movement position can refer to the description of the second situation above, which will not be repeated here.

The first uncertainty coefficient is an important parameter in the Kalman filter, which helps to improve the accuracy of the prediction results.

In order to allow users to enjoy a real, immersive listening experience through the audio signal playback end (such as headphones), spatial audio technology can be used to render sound sources in different formats such as stereo, multi-channel surround sound, and specially produced multi-object sound sources into binaural signals. In the real world, when our head turns or moves, the absolute position of the sound source itself does not change, but the relative direction of the sound source and the head will change. For example, there is a guitar playing in front of you. If you turn to the right, the sound of the guitar will move relatively to your left. For another example, there is a guitar on the left side of the stage and a saxophone on the right. When you move to the side of the stage, the sounds of the guitar and the saxophone will overlap and come from the same direction. It can be seen that the so-called real and immersive listening experience is just like the listening experience of guitar or stage performance described above. As the position of the user's head changes (including but not limited to the up, down, left, right displacement of the head, head rotation, etc.), the sound from the same audio source will be rendered into different listening effects in the user's ears. This rendering is the rendering method with head motion effect, that is, head tracking (Head Tracking) is performed through the sensor in the earphones, and the corresponding rotation or displacement changes are reflected in the spatial rendering of the sound source.

At the same time, although the audio content provider can provide higher computing power to realize the above-mentioned rendering algorithm with head movement effect, due to the characteristics of the audio content provider itself, it will introduce higher latency, resulting in confusion of important spatial information such as direction. For example, the sound and image of an object is designed to be directly in front of the space. If the user's head turns, the sound and image will first move to the front of his face, and then return to the front of the space. There is an obvious delay "damping feeling", which affects the listening experience.

In order to solve the problem caused by the above delay, the audio signal playback end can predict the user's head movement position, and then send the predicted head movement position to the audio content provider, so that the audio content provider can render the audio frame based on the predicted head movement position, so that the rendered audio frame obtained can provide a listening effect corresponding to the predicted head movement position. Then when the rendered audio frame is sent to the audio signal playback end, the predicted time can offset the delay caused by rendering processing, link transmission, etc., so that when the audio frame is played at the audio signal playback end, it just matches the user's head movement position, achieving the real and immersive listening experience described above.

In the present application, the audio signal playback end can first obtain the first historical head movement position corresponding to the delay information; then obtain the measured head movement position through the sensor; and then obtain the predicted head movement position based on the first historical head movement position and the measured head movement position.

According to the three cases of delay information mentioned above, the audio signal playback end can use three methods to obtain the first historical head movement position, namely:

1. When the delay information includes the delay sent by the audio content provider (the first case above), the first historical head movement position corresponding to the delay is extracted from the cache, and the cache pre-stores the correspondence between multiple sets of delays and historical head movement positions.

As described above, the audio signal playback end can periodically detect the measured head movement position of the user wearing the audio signal playback end through the sensor. For the convenience of subsequent use, the audio signal playback end can store the corresponding relationship between the measured head movement position and the detection time. In this way, when the above-mentioned time delay is obtained, the measured head movement position at that time corresponding to the time delay can be obtained by querying the above-mentioned corresponding relationship (hereinafter referred to as the first historical head movement position).

2. When the delay information includes the second historical head movement position sent by the audio content provider (the second case above), the second historical head movement position is used as the first historical head movement position.

Referring to the description of the second case above, it can be known that the audio content provider directly encapsulates the second historical head movement position in the first information and sends it to the audio signal player. In this way, the audio signal player can directly obtain the measured head movement position (the second historical head movement position) corresponding to the time delay, so the audio signal player can directly determine the second historical head movement position as the first historical head movement position.

3. When the delay information includes the second historical head movement position and the first uncertainty coefficient sent by the audio content provider (the third case above), the second historical head movement position is used as the first historical head movement position.

With reference to the description in paragraph 2 above, in the third case of the time delay information, the audio signal playback end may also directly determine the second historical head movement position as the first historical head movement position.

The audio signal playback end may obtain the measured head movement position through a sensor, where the measured head movement position is the most recently detected head movement position of the user.

In the present application, obtaining the predicted head movement position according to the first historical head movement position and the measured head movement position may include the following two algorithms:

The first method is that, based on 1 and 2 above, the audio signal playback end can obtain the difference between the first historical head movement position and the measured head movement position; obtain the head movement change rate, which is obtained based on the N measured head movement positions obtained previously, N>1; and obtain the predicted head movement position based on the difference and the head movement change rate.

The difference between the first historical head movement position and the measured head movement position may refer to the distance between the first historical head movement position and the measured head movement position. For example, in a three-dimensional world, the head movement position may be represented by an xyz coordinate axis, so the difference may be the distance between the coordinate values of the first historical head movement position and the measured head movement position. In addition, the head movement position may also be represented by other methods, which are not specifically limited in the present application.

As described above, the audio signal playback end can periodically detect the measured head movement position of the user wearing the audio signal playback end through a sensor. Therefore, the audio signal playback end can obtain the head movement change rate based on the N measured head movement positions obtained previously. The N measured head movement positions can include N positions counting forward from the current measured head movement position.

The audio signal playback end can predict the head movement change value based on the difference and the head movement change rate by using cubic spline interpolation and other means, and then add the head movement change value to the current measured head movement position to obtain the predicted head movement position.

The second method is that, based on the above 3, the audio signal playback end obtains the head movement change rate, which is obtained based on the N measured head movement positions obtained previously, where N>1; the second uncertainty coefficient is obtained based on the head movement change rate and the first uncertainty coefficient; the second uncertainty coefficient, the first historical head movement position and the measured current head movement position are input into the Kalman filter to obtain the predicted head movement position.

The method for obtaining the head movement change rate can refer to the above description.

The head movement change rate combined with the first uncertainty coefficient can obtain the second uncertainty coefficient. The second uncertainty coefficient, the first historical head movement position and the measured current head movement position are input into the Kalman filter. In the Kalman filter, the Kalman gain coefficient can be determined according to the second uncertainty coefficient and the estimated uncertainty of the previous iteration, and the Kalman gain coefficient is used as the latest gain coefficient. In this way, the Kalman filter can output the predicted head movement position.

It should be noted that, in addition to the above two methods, the present application may also use other algorithms to obtain the predicted head movement position according to the first historical head movement position and the measured head movement position, and there is no specific limitation on this.

The audio signal playback end sends the first rendering information to the audio content provider end, and the first rendering information includes the predicted head movement position, so that the audio content provider end can render the audio frame with the head movement effect according to the predicted head movement position. The rendered audio frame obtained in this way can provide a listening effect corresponding to the predicted head movement position. Then when the rendered audio frame is sent to the audio signal playback end, the predicted time can offset the delay caused by the rendering process, link transmission, etc., so that when the audio frame is played at the audio signal playback end, it just matches the user's head movement position, achieving the real and immersive listening experience described above.

In a possible implementation, when the first rendering mode is adopted, the audio content provider obtains the current frame. When the audio format of the audio source is a non-stereo format, the audio content provider obtains the predicted head movement position. The audio content provider renders the current frame according to the predicted head movement position to obtain a first binaural signal. When the audio format of the audio source is a stereo format, the audio content provider does not perform spatial rendering on the audio source. The audio content provider sends first information to the audio signal player, and the first information includes a first audio stream, an audio format of the audio source, and delay information. When the audio format is indicated as a stereo format, the audio signal player obtains the measured head movement position through a sensor. The audio signal player renders the current frame according to the measured head movement position to obtain a second binaural signal. The audio signal player plays audio according to the target binaural signal, and the target binaural signal includes the first binaural signal or the second binaural signal.

The first rendering mode may be a non-low-latency link mode, in which the audio content provider can render the audio frame with a head movement effect based on the predicted head movement position. The user can choose whether to select the first rendering mode by operating the application (application, APP) installed on the audio content provider. Correspondingly, the present application also includes a second rendering mode, in which the audio content provider renders the audio frame without a head movement effect, and the process can be described below. Similarly, the user can choose whether to select the second rendering mode by operating the APP installed on the audio content provider. The user's operation can refer to the following embodiments.

The current frame may be one of the frames of the audio source, and may generally refer to an audio frame currently being processed by the audio content provider.

The audio format of the audio source includes a stereo format or a non-stereo format, wherein the non-stereo format may include but is not limited to audio in multi-channel, multi-object formats such as 5.1, 7.1, and 3DA.

In this application, the audio content provider can perform high-computing rendering on audio in multi-channel, multi-object formats (non-stereo formats) such as 5.1, 7.1, and 3DA, so as to give full play to the computing power advantage of the audio content provider. Based on this, when the audio format of the audio source is a non-stereo format, the audio content provider can render the audio frame with a head movement effect, so it is necessary to obtain the predicted head movement position.

The audio content provider can obtain the predicted head movement position from the rendering information previously received from the audio signal player. Optionally, the rendering information can be the latest received rendering information, which can improve the accuracy of the predicted head movement position. The acquisition of the predicted head movement position can refer to the embodiment shown in Figure 3, which will not be repeated here.

In the present application, the audio content provider can render the direct sound part, early reflection sound (ER) part and late reverberation sound (LR) part in the current frame with head movement effect according to the predicted head movement position to obtain a first binaural signal.

Optionally, the direct sound with the head movement effect and the reverberation with the head movement effect (obtained by mixing ER and LR) are respectively rendered by an audio effector and mixed, and the mixed output result is rendered again by the audio effector and transmitted in the form of a first binaural signal. The aforementioned audio effector includes but is not limited to an equalizer, a dynamic compression effector, a low-frequency enhancement effector, etc.

In this application, the audio content provider does not need to render the audio source in stereo format, and the audio signal player is responsible for rendering the leading motion effect. Therefore, the processing can perform high-computing power rendering on audio in multi-channel, multi-object formats (non-stereo formats) such as 5.1, 7.1, 3DA, etc., so as to give full play to the computing power advantage of the audio content provider. Based on this, when the audio format of the audio source is a non-stereo format, the audio content provider can render the audio frame with a leading motion effect, so the audio content provider can encapsulate the audio source in the first audio stream.

As described above, when the audio format indicates a monaural format, the first audio stream includes a first binaural signal rendered by the audio content provider; when the audio format indicates a stereo format, the first audio stream includes an audio source.

The audio source in stereo format is rendered with the head movement position by the audio signal player. Since the IMU data transmission link latency of the audio signal player is low, the audio signal player can obtain the current head movement position (referred to as the measured head movement position in this article) through the sensor in real time.

The audio signal playback end can render the direct sound part in the current frame with the head movement effect according to the measured head movement position, and render the ER and LR in the current frame without the head movement effect to obtain a second binaural signal.

Optionally, the direct sound with the head movement effect and the reverberation without the head movement effect (obtained by mixing ER and LR) are respectively rendered by an audio effector and mixed, and the mixed output is rendered again by the audio effector and mixed again to obtain a second binaural signal. The aforementioned audio effector includes but is not limited to an equalizer, a dynamic compression effector, a low-frequency enhancement effector, etc.

After the above steps, the audio signal player obtains a target binaural signal, which may be a first binaural signal (the audio source is in a non-stereo format) or a second binaural signal (the audio source is in a stereo format).

In one possible implementation, when the second rendering mode is adopted, the audio content provider obtains the current frame; renders the current frame without the head movement effect to obtain a second binaural signal (the second binaural signal is different from the second binaural signal mentioned above, and is referred to as the third binaural signal below for the sake of distinction); sends second information to the audio signal player, the second information includes a second audio stream, and the second audio stream includes the third binaural signal. The audio signal player obtains the measured head movement position through a sensor, and renders the current frame (the current frame is the audio frame after the audio content provider performs the aforementioned rendering without the head movement effect) according to the measured head movement position to obtain a fourth binaural signal. The audio signal player plays audio according to the target binaural signal, and the target binaural signal includes the fourth binaural signal.

In the second rendering mode, the audio content provider renders the audio frames without the head motion effect, which can give full play to the computing power advantage of the audio content provider. The lack of head motion effect can shorten the rendering delay, and the audio signal player performs low-computing power rendering with head motion effect instead. Compared with the first rendering mode, the delay is lower, but the rendering effect of the head motion effect is poor.

In a second aspect, the present application provides a spatial audio rendering device, including: an acquisition module for acquiring delay information of an audio content provider; a prediction module for acquiring a predicted head movement position based on the delay information; and a sending module for sending first rendering information to the audio content provider, wherein the first rendering information includes the predicted head movement position.

In a possible implementation, the prediction module is specifically used to obtain a first historical head movement position corresponding to the time delay information; obtain the measured head movement position through a sensor; and obtain the predicted head movement position based on the first historical head movement position and the measured head movement position.

In one possible implementation, the delay information includes the delay sent by the audio content provider; the prediction module is specifically used to extract the first historical head movement position corresponding to the delay from the cache, and the cache pre-stores the correspondence between multiple sets of delays and historical head movement positions.

In one possible implementation, the delay information includes a second historical head movement position sent by the audio content provider, where the second historical head movement position is the actual head movement position when sending the second rendering information, and the second rendering information is sent earlier than the first rendering information; the prediction module is specifically used to use the second historical head movement position as the first historical head movement position.

In a possible implementation, the prediction module is specifically used to obtain the difference between the first historical head movement position and the measured head movement position; obtain the head movement change rate, which is obtained based on the N measured head movement positions obtained previously, N>1; and obtain the predicted head movement position based on the difference and the head movement change rate.

In a possible implementation, the delay information includes a second historical head movement position and a first uncertainty coefficient sent by the audio content provider, wherein the second historical head movement position is the actual head movement position when the second rendering information is sent, and the first uncertainty coefficient comes from the second rendering information, and the second rendering information is sent earlier than the first rendering information; the prediction module is specifically used to use the second historical head movement position as the first historical head movement position.

In one possible implementation, the prediction module is specifically used to obtain the head movement change rate, which is obtained based on N previously measured head movement positions, N>1; obtain a second uncertainty coefficient based on the head movement change rate and the first uncertainty coefficient; and input the second uncertainty coefficient, the first historical head movement position and the measured current head movement position into a Kalman filter to obtain the predicted head movement position.

In a possible implementation, it also includes: a receiving module, which is used to receive first information sent by the audio content provider, the first information including an audio format, a first audio stream and the delay information; wherein the audio format indicates whether the audio source is in a stereo format or a non-stereo format; when the audio format indicates the non-stereo format, the first audio stream includes a first binaural signal rendered by the audio content provider; when the audio format indicates the stereo format, the first audio stream includes the audio source; a rendering module, which is used to obtain a measured head position through a sensor when the audio format indicates the stereo format; a second binaural signal obtained by rendering a current frame according to the measured head position, the current frame being one of the frames of the audio source; and a playing module, which is used to play audio according to a target binaural signal, the target binaural signal including the first binaural signal or the second binaural signal.

In a possible implementation, the rendering module is specifically used to render the direct sound part in the current frame with the head movement effect according to the measured head movement position, and to render the early reflected sound ER and the late reverberation sound LR in the current frame without the head movement effect, so as to obtain the second binaural signal.

In a third aspect, the present application provides a spatial audio rendering device, comprising: an acquisition module, used to acquire a current frame when a first rendering mode is adopted, wherein the current frame is one of the frames of an audio source; when the audio format of the audio source is a non-stereo format, acquire a predicted head movement position; a rendering module, used to render the current frame according to the predicted head movement position to obtain a first binaural signal; and a sending module, used to send first information to an audio signal playback end, wherein the first information includes a first audio stream, the audio format and delay information of the audio source, and the first audio stream includes the first binaural signal.

In a possible implementation, the rendering module is specifically used to render the direct sound part, the early reflection sound ER part and the late reverberation sound LR part in the current frame with head movement effect according to the predicted head movement position, so as to obtain the first binaural signal.

In a possible implementation, the acquisition module is specifically configured to obtain the predicted head movement position from rendering information sent by the audio signal playback end.

In a possible implementation manner, the delay information includes delay.

In a possible implementation, the rendering information further includes the measured head movement position when the audio signal playback end sends the rendering information; correspondingly, the delay information includes the measured head movement position.

In a possible implementation, the rendering information also includes the measured head position and the first uncertainty coefficient when the audio signal playback end sends the rendering information; correspondingly, the delay information includes the measured head position and the first uncertainty coefficient.

In a possible implementation manner, when the audio format of the audio source is a stereo format, the first audio stream includes the audio source.

In a possible implementation, the acquisition module is further used to acquire the current frame when a second rendering mode is adopted; the rendering module is further used to render the current frame without a head motion effect to obtain a second binaural signal; the sending module is further used to send second information to the audio signal playback end, the second information includes a second audio stream, and the second audio stream includes the second binaural signal.

In a fourth aspect, the present application provides an audio signal playback device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method implemented by the audio signal playback end as described in any one of the first aspects above.

In a fifth aspect, the present application provides an audio content providing device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method implemented by the audio content provider as described in any one of the first aspects above.

In a sixth aspect, the present application provides a computer-readable storage medium, comprising a computer program, wherein when the computer program is executed on a computer, the computer executes any one of the methods described in the first aspect.

In a seventh aspect, the present application provides a computer program product, wherein the computer program product comprises a computer program code, and when the computer program code is executed on a computer, the computer executes any one of the methods in the first aspect.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of an application scenario of the present application;

FIG2 is a schematic diagram of the structure of an audio content provider 200 of the present application;

FIG3 is a flow chart of a process 300 of the spatial audio rendering method of the present application;

FIG4 is a flow chart of a process 400 of the spatial audio rendering method of the present application;

FIG5 is a schematic diagram of the overall flow of the spatial audio rendering method of the present application;

FIG6 is a schematic diagram of a specific process of the spatial audio rendering method of the present application;

FIG7 is a flow chart of a prediction algorithm for head movement position;

8a and 8b are schematic diagrams of the flow chart of the prediction algorithm of the head movement position;

9a and 9b are schematic diagrams of the flow chart of the prediction algorithm of the head movement position;

FIG10 is a schematic diagram of the framework flow of the present application;

FIG11 is a schematic diagram of the process of the low latency mode of the present application;

FIG12 is a schematic diagram of the process of the non-low latency mode of the present application;

FIG13 is a schematic diagram of an exemplary structure of a spatial audio rendering device 1300 of the present application;

FIG. 14 is a schematic diagram of an exemplary structure of a spatial audio rendering device 1400 of the present application.

Detailed ways

In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

The terms "first", "second", etc. in the specification embodiments, claims, and drawings of the present application are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, including a series of steps or units. The method, system, product, or device is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And/or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and/or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character "/" generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

In the real three-dimensional space, sound can have two characteristics: sense of direction and sense of space. Spatial audio technology usually refers to the technology that realizes the immersive audio experience of simulating the above two characteristics on head-mounted playback devices such as headphones. People's judgment of the sense of direction of sound is mainly affected by factors such as time difference, sound level difference, human body filtering effect learned during growth, and head shaking. Among them, the three elements of time difference, sound level difference, and human body filtering effect can be comprehensively expressed as head related transfer functions (HRTF); head shaking in all directions is of great help to us in judging the location of the sound source. The indoor sound field can be composed of direct sound, early reflection sound (ER) and late reverberation sound (LR). People's sense of space for sound is mainly based on ER and LR.

Using spatial audio technology, sound sources in different formats, such as stereo, multi-channel surround sound, specially produced multi-object sound sources, etc., can be rendered into binaural signals, so that you can enjoy a real, immersive listening experience on headphones.

When our head rotates or moves, the absolute position of the sound source itself will not change, but the relative direction between the sound source and the head will change. For example, there is a guitar playing in front of you. If you turn to the right, the sound of the guitar will move to your left. For another example, there is a guitar on the left side of the stage and a saxophone on the right side. When you move to the side of the stage, the sounds of the guitar and saxophone will overlap and come from the same direction. Head tracking can currently be performed through sensors such as gyroscopes in headphones, and the corresponding rotation or displacement changes can be reflected in the spatial rendering of the sound source.

However, spatial audio rendering usually requires a lot of computing power, and head tracking requires low latency. If spatial audio rendering is placed on the audio content provider (for example, mobile phones, tablets, PCs, etc.), it can meet the computing power requirements, but it will introduce high latency, especially in head movement scenarios. For audio sources in multi-channel, multi-object formats, it will cause confusion of important spatial information such as orientation, resulting in a decrease in the overall rendering quality of spatial audio, affecting the overall listening experience.

In order to solve the above technical problems, the present application provides a spatial audio rendering method and device. The following embodiments will illustrate the technical solution of the present application.

FIG1 is a schematic diagram of an application scenario of the present application. As shown in FIG1 , the scenario includes an audio content provider and an audio signal player. The interconnection method between the audio content provider and the audio signal player includes but is not limited to Bluetooth technology. The audio content provider may include but is not limited to mobile phones, tablets, laptops, desktop computers, etc., which can provide higher computing power, but may have higher latency; the audio signal player may include but is not limited to true wireless stereo (TWS) headphones, wireless headphones, wireless neckband headphones, etc., which have lower latency but provide lower computing power. In addition, the device serving as the audio signal player is provided with sensors such as gyroscopes to collect user head movement information.

In this application, the rendering algorithm includes two parts. One part of the algorithm is deployed at the audio content provider end, which is used to perform high-computing power rendering of audio in multi-channel, multi-object formats such as 5.1, 7.1, 3DA, etc., so as to give full play to the computing power advantage of the audio content provider end; the other part of the algorithm is deployed at the audio signal playback end, which is used to perform low-computing power rendering of audio in stereo format. Its inertial measurement unit (IMU) data transmission link has a low latency and can also run independently, so it does not rely on a specific audio content provider end, and can meet the user's requirements for connecting to any device of an audio content provider end.

Based on this, the present application also provides a user's head movement position prediction algorithm for predicting the user's future head movement position. The predicted head movement position is used to assist the rendering algorithm of the audio content provider, which can reduce the delay impact of the audio content provider.

It should be noted that the application scenario shown in FIG1 is an example, but this should not constitute any limitation to the present application, and the present application does not make any specific limitation to the application scenario of the spatial audio rendering method.

Fig. 2 is the structural representation of the audio content providing terminal 200 of the present application.It should be understood that the audio content providing terminal 200 shown in Fig. 2 is only an example, and the audio content providing terminal 200 can have more or less parts than shown in the figure, can combine two or more parts, or can have different component configurations.The various parts shown in Fig. 2 can be realized in the combination of hardware, software, or hardware and software including one or more signal processing and/or application specific integrated circuits.

The audio content provider 200 may include: a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc. The sensor module 280 may include a pressure sensor 280A, a gyroscope sensor 280B, an air pressure sensor 280C, a magnetic sensor 280D, an acceleration sensor 280E, a distance sensor 280F, a proximity light sensor 280G, a fingerprint sensor 280H, a temperature sensor 280J, a touch sensor 280K, an ambient light sensor 280L, a bone conduction sensor 280M, etc.

The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and/or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

The controller may be the nerve center and command center of the audio content provider 200. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

In some embodiments, the processor 210 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver/transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input/output (GPIO) interface, a subscriber identity module (SIM) interface, and/or a universal serial bus (USB) interface, etc.

The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 210 may include multiple groups of I2C buses. The processor 210 may be coupled to the touch sensor 280K, the charger, the flash, the camera 293, etc. through different I2C bus interfaces. For example, the processor 210 may be coupled to the touch sensor 280K through the I2C interface, so that the processor 210 communicates with the touch sensor 280K through the I2C bus interface, thereby realizing the touch function of the audio content provider 200.

The I2S interface can be used for audio communication. In some embodiments, the processor 210 can include multiple groups of I2S buses. The processor 210 can be coupled to the audio module 270 via the I2S bus to achieve communication between the processor 210 and the audio module 270. In some embodiments, the audio module 270 can transmit an audio signal to the wireless communication module 260 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 270 and the wireless communication module 260 can be coupled via a PCM bus interface. In some embodiments, the audio module 270 can also transmit audio signals to the wireless communication module 260 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 210 and the wireless communication module 260. For example, the processor 210 communicates with the Bluetooth module in the wireless communication module 260 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 270 can transmit an audio signal to the wireless communication module 260 through the UART interface to implement the function of playing music through a Bluetooth headset.

The MIPI interface can be used to connect the processor 210 with peripheral devices such as the display screen 294 and the camera 293. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 210 and the camera 293 communicate via the CSI interface to implement the shooting function of the audio content provider 200. The processor 210 and the display screen 294 communicate via the DSI interface to implement the display function of the audio content provider 200.

The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 210 with the camera 293, the display screen 294, the wireless communication module 260, the audio module 270, the sensor module 280, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

The USB interface 230 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 230 can be used to connect a charger to charge the audio content provider 200, and can also be used to transmit data between the audio content provider 200 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other user devices, such as AR devices, etc.

It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the audio content provider 200. In other embodiments of the present application, the audio content provider 200 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

The charging management module 240 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 240 may receive charging input from a wired charger through the USB interface 230. In some wireless charging embodiments, the charging management module 240 may receive wireless charging input through a wireless charging coil of the audio content provider 200. While the charging management module 240 is charging the battery 242, it may also power the user device through the power management module 241.

The power management module 241 is used to connect the battery 242, the charging management module 240 and the processor 210. The power management module 241 receives input from the battery 242 and/or the charging management module 240, and supplies power to the processor 210, the internal memory 221, the external memory, the display screen 294, the camera 293, and the wireless communication module 260. The power management module 241 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 241 can also be set in the processor 210. In other embodiments, the power management module 241 and the charging management module 240 can also be set in the same device.

The wireless communication function of the audio content provider 200 can be implemented through the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modem processor and the baseband processor.

Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the audio content provider 200 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

The mobile communication module 250 can provide solutions for wireless communications including 2G/3G/4G/5G applied to the audio content provider 200. The mobile communication module 250 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 250 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 250 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 250 can be set in the processor 210. In some embodiments, at least some of the functional modules of the mobile communication module 250 can be set in the same device as at least some of the modules of the processor 210.

The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 270A, a receiver 270B, etc.), or displays an image or video through a display screen 294. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 210 and be set in the same device as the mobile communication module 250 or other functional modules.

The wireless communication module 260 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) and the like applied to the audio content provider 200. The wireless communication module 260 can be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via the antenna 2, modulates the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 210. The wireless communication module 260 can also receive the signal to be sent from the processor 210, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

In some embodiments, the antenna 1 of the audio content provider 200 is coupled to the mobile communication module 250, and the antenna 2 is coupled to the wireless communication module 260, so that the audio content provider 200 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and/or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and/or a satellite based augmentation system (SBAS).

The audio content provider 200 implements the display function through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the audio content provider 200 may include 1 or N display screens 294, where N is a positive integer greater than 1.

The audio content provider 200 can realize the shooting function through ISP, camera 293, video codec, GPU, display screen 294 and application processor.

The ISP is used to process the data fed back by the camera 293. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 293.

The camera 293 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the audio content provider 200 may include 1 or N cameras 293, where N is a positive integer greater than 1.

The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the audio content provider 200 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

Video codecs are used to compress or decompress digital videos. The audio content provider 200 can support one or more video codecs. In this way, the audio content provider 200 can play or record videos in multiple coding formats, such as moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, intelligent cognition and other applications of the audio content provider 200 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the audio content provider 200. The external memory card communicates with the processor 210 through the external memory interface 220 to implement a data storage function. For example, files such as music and videos are stored in the external memory card.

The internal memory 221 can be used to store computer executable program codes, which include instructions. The processor 210 executes various functional applications and data processing of the audio content provider 200 by running the instructions stored in the internal memory 221. The internal memory 221 may include a storage program area and a storage data area. Among them, the storage program area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The storage data area may store data (such as audio data, a phone book, etc.) created during the use of the audio content provider 200, etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

The audio content provider 200 can implement audio functions such as music playing and recording through an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, and an application processor.

The audio module 270 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 270 can also be used to encode and decode audio signals. In some embodiments, the audio module 270 can be arranged in the processor 210, or some functional modules of the audio module 270 can be arranged in the processor 210.

The speaker 270A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The audio content provider 200 can listen to music or listen to a hands-free call through the speaker 270A.

The receiver 270B, also called a "handset", is used to convert audio electrical signals into sound signals. When the audio content provider 200 receives a call or voice message, the voice can be received by placing the receiver 270B close to the ear.

Microphone 270C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by approaching the microphone 270C with his mouth to input the sound signal into the microphone 270C. The audio content provider 200 can be provided with at least one microphone 270C. In other embodiments, the audio content provider 200 can be provided with two microphones 270C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the audio content provider 200 can also be provided with three, four or more microphones 270C to realize the collection of sound signals, noise reduction, identification of sound sources, realization of directional recording function, etc.

The earphone interface 270D is used to connect a wired earphone and can be a USB interface 230 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

The pressure sensor 280A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 280A can be set on the display screen 294. There are many types of pressure sensors 280A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 280A, the capacitance between the electrodes changes. The audio content provider 200 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 294, the audio content provider 200 detects the touch operation intensity according to the pressure sensor 280A. The audio content provider 200 can also calculate the touch position according to the detection signal of the pressure sensor 280A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to a first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

The gyroscope sensor 280B can be used to determine the motion posture of the audio content provider 200. In some embodiments, the angular velocity of the audio content provider 200 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 280B. The gyroscope sensor 280B can be used for anti-shake shooting. Exemplarily, when the shutter is pressed, the gyroscope sensor 280B detects the angle of the shaking of the audio content provider 200, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the audio content provider 200 through reverse movement to achieve anti-shake. The gyroscope sensor 280B can also be used for navigation and somatosensory game scenes.

The air pressure sensor 280C is used to measure air pressure. In some embodiments, the audio content provider 200 calculates the altitude through the air pressure value measured by the air pressure sensor 280C to assist in positioning and navigation.

The magnetic sensor 280D includes a Hall sensor. The audio content provider 200 can use the magnetic sensor 280D to detect the opening and closing of the flip leather case. In some embodiments, when the audio content provider 200 is a flip phone, the audio content provider 200 can detect the opening and closing of the flip cover according to the magnetic sensor 280D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover automatic unlocking and other features are set.

The acceleration sensor 280E can detect the magnitude of acceleration of the audio content provider 200 in all directions (generally three axes). When the audio content provider 200 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the user device and is applied to applications such as horizontal and vertical screen switching and pedometers.

The distance sensor 280F is used to measure the distance. The audio content provider 200 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the audio content provider 200 can use the distance sensor 280F to measure the distance to achieve fast focusing.

The proximity light sensor 280G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The audio content provider 200 emits infrared light outward through the light emitting diode. The audio content provider 200 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the audio content provider 200. When insufficient reflected light is detected, the audio content provider 200 can determine that there is no object near the audio content provider 200. The audio content provider 200 can use the proximity light sensor 280G to detect that the user holds the audio content provider 200 close to the ear to talk, so as to automatically turn off the screen to achieve the purpose of power saving. The proximity light sensor 280G can also be used in leather case mode, pocket mode automatic unlocking and lock screen.

The ambient light sensor 280L is used to sense the ambient light brightness. The audio content provider 200 can adaptively adjust the brightness of the display screen 294 according to the perceived ambient light brightness. The ambient light sensor 280L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 280L can also cooperate with the proximity light sensor 280G to detect whether the audio content provider 200 is in a pocket to prevent accidental touch.

The fingerprint sensor 280H is used to collect fingerprints. The audio content provider 200 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint answering calls, etc.

Temperature sensor 280J is used to detect temperature. In certain embodiments, audio content provider 200 utilizes the temperature detected by temperature sensor 280J to execute temperature processing strategy. For example, when the temperature reported by temperature sensor 280J exceeds a threshold value, audio content provider 200 performs a performance reduction of a processor near temperature sensor 280J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold value, audio content provider 200 heats battery 242 to avoid abnormal shutdown of audio content provider 200 caused by low temperature. In other embodiments, when the temperature is lower than another threshold value, audio content provider 200 performs a boost on the output voltage of battery 242 to avoid abnormal shutdown caused by low temperature.

The touch sensor 280K is also called a "touch panel". The touch sensor 280K can be set on the display screen 294, and the touch sensor 280K and the display screen 294 form a touch screen, also called a "touch screen". The touch sensor 280K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 294. In other embodiments, the touch sensor 280K can also be set on the surface of the audio content provider 200, which is different from the position of the display screen 294.

The bone conduction sensor 280M can obtain a vibration signal. In some embodiments, the bone conduction sensor 280M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 280M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 280M can also be set in an earphone and combined into a bone conduction earphone. The audio module 270 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 280M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 280M to realize a heart rate detection function.

The key 290 includes a power key, a volume key, etc. The key 290 may be a mechanical key or a touch key. The audio content provider 200 may receive key inputs and generate key signal inputs related to user settings and function controls of the audio content provider 200.

Motor 291 can generate vibration prompts. Motor 291 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 294, motor 291 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

Indicator 292 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

The SIM card interface 295 is used to connect a SIM card. The SIM card can be connected to and separated from the audio content provider 200 by inserting the SIM card interface 295 or pulling it out from the SIM card interface 295. The audio content provider 200 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 295 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 295 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 295 can also be compatible with different types of SIM cards. The SIM card interface 295 can also be compatible with external memory cards. The audio content provider 200 interacts with the network through the SIM card to realize functions such as calls and data communications. In some embodiments, the audio content provider 200 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the audio content provider 200 and cannot be separated from the audio content provider 200.

It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the main control device. In other embodiments of the present application, the main control device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

Based on the above embodiments, FIG3 is a flowchart of process 300 of the spatial audio rendering method of the present application. As shown in FIG3, process 300 can be applied to the application scenario shown in FIG1, and is executed by the audio signal playback end to obtain the predicted head movement position for use by the audio content provider when rendering the audio. Process 300 is described as a series of steps or operations, and it should be understood that process 300 can be executed in various orders and/or occur simultaneously, and is not limited to the execution order shown in FIG3.

Process 300 includes the following steps:

Step 301: Obtain delay information of the audio content provider.

The delay information is used to indicate the delay of the audio content provider, which can include the following situations:

The first type of delay information includes the delay sent by the audio content provider.

After determining the delay caused by its own transmission link, rendering processing, etc., the audio content provider can encapsulate the delay in the first information sent to the audio signal player and send it to the audio signal player. In this way, the audio signal player can directly extract the delay of the audio content provider from the first information from the audio content provider.

The second type of delay information includes the second historical head movement position sent by the audio content provider.

The audio signal playback end (such as headphones) can periodically detect the current position of the head of the user wearing the headphones (referred to as the measured head movement position in this article) through sensors (such as gyroscopes, gravity sensors, etc.), so that the audio signal playback end can send the measured head movement position to the audio content provider.

After receiving the above-mentioned measured head movement position, the audio content provider does not process it first, and still processes the audio frame according to the established process. When the processing is completed and the first information is to be sent to the audio signal playback end, the above-mentioned measured head movement position is carried in the first information. Since a period of time has passed at this time, the above-mentioned measured head movement position becomes the historical head movement position (referred to as the second historical head movement position in this article). It should be understood that, in a mathematical sense, the above-mentioned measured head movement position and the above-mentioned second historical head movement position are the same.

Based on this, the audio signal playback end can obtain the time difference between the time of receiving the second historical head movement position and the time of sending the actual head movement position (that is, calculating the time difference between the receiving time and the sending time of the same head movement position), thereby indirectly obtaining the delay of the audio content provider.

The third type is that the delay information includes the second historical head movement position and the first uncertainty coefficient sent by the audio content provider. The second historical head movement position can refer to the description of the second situation above, which will not be repeated here.

The first uncertainty coefficient is an important parameter in the Kalman filter, which helps to improve the accuracy of the prediction results.

This situation is related to the algorithm for obtaining the predicted head movement position in step 302, and the specific content can be referred to the description below.

Step 302: Obtain the predicted head movement position according to the time delay information.

As mentioned above, in order to allow users to enjoy a real, immersive listening experience through the audio signal playback end (such as headphones), spatial audio technology can be used to render sound sources in different formats such as stereo, multi-channel surround sound, and specially produced multi-object sound sources into binaural signals. In the real world, when our head turns or moves, the absolute position of the sound source itself does not change, but the relative direction of the sound source and the head will change. For example, there is a guitar playing in front of you. If you turn to the right, the sound of the guitar will move relatively to your left. For another example, there is a guitar on the left side of the stage and a saxophone on the right. When you move to the side of the stage, the sounds of the guitar and the saxophone will overlap and come from the same direction. It can be seen that the so-called real and immersive listening experience is just like the listening experience of guitar or stage performance described above. As the position of the user's head changes (including but not limited to the up, down, left, right displacement of the head, head rotation, etc.), the sound from the same audio source will be rendered into different listening effects in the user's ears. This rendering is the rendering method with head motion effect, that is, head tracking (Head Tracking) is performed through the sensor in the earphones, and the corresponding rotation or displacement changes are reflected in the spatial rendering of the sound source.

At the same time, although the audio content provider can provide higher computing power to realize the above-mentioned rendering algorithm with head movement effect, due to the characteristics of the audio content provider itself, it will introduce higher latency, resulting in confusion of important spatial information such as direction. For example, the sound and image of an object is designed to be directly in front of the space. If the user's head turns, the sound and image will first move to the front of his face, and then return to the front of the space. There is an obvious delay "damping feeling", which affects the listening experience.

In order to solve the problem caused by the above delay, the audio signal playback end can predict the user's head movement position, and then send the predicted head movement position to the audio content provider, so that the audio content provider can render the audio frame based on the predicted head movement position, so that the rendered audio frame obtained can provide a listening effect corresponding to the predicted head movement position. Then when the rendered audio frame is sent to the audio signal playback end, the predicted time can offset the delay caused by rendering processing, link transmission, etc., so that when the audio frame is played at the audio signal playback end, it just matches the user's head movement position, achieving the real and immersive listening experience described above.

In the present application, the audio signal playback end can first obtain the first historical head movement position corresponding to the delay information; then obtain the measured head movement position through the sensor; and then obtain the predicted head movement position based on the first historical head movement position and the measured head movement position.

According to the three situations of the time delay information in step 301, the audio signal playback end can use three methods to obtain the first historical head movement position, namely:

1. When the delay information includes the delay sent by the audio content provider (the first case above), the first historical head movement position corresponding to the delay is extracted from the cache, and the cache pre-stores the correspondence between multiple sets of delays and historical head movement positions.

As described above, the audio signal playback end can periodically detect the measured head movement position of the user wearing the audio signal playback end through the sensor. For the convenience of subsequent use, the audio signal playback end can store the corresponding relationship between the measured head movement position and the detection time. In this way, when the above-mentioned time delay is obtained, the measured head movement position at that time corresponding to the time delay can be obtained by querying the above-mentioned corresponding relationship (hereinafter referred to as the first historical head movement position).

2. When the delay information includes the second historical head movement position sent by the audio content provider (the second case above), the second historical head movement position is used as the first historical head movement position.

Referring to the description of the second case above, it can be known that the audio content provider directly encapsulates the second historical head movement position in the first information and sends it to the audio signal player. In this way, the audio signal player can directly obtain the measured head movement position (the second historical head movement position) corresponding to the time delay, so the audio signal player can directly determine the second historical head movement position as the first historical head movement position.

3. When the delay information includes the second historical head movement position and the first uncertainty coefficient sent by the audio content provider (the third case above), the second historical head movement position is used as the first historical head movement position.

With reference to the description in paragraph 2 above, in the third case of the time delay information, the audio signal playback end may also directly determine the second historical head movement position as the first historical head movement position.

The audio signal playback end may obtain the measured head movement position through a sensor, where the measured head movement position is the most recently detected head movement position of the user.

In the present application, obtaining the predicted head movement position according to the first historical head movement position and the measured head movement position may include the following two algorithms:

The first method is that, based on 1 and 2 above, the audio signal playback end can obtain the difference between the first historical head movement position and the measured head movement position; obtain the head movement change rate, which is obtained based on the N measured head movement positions obtained previously, N>1; and obtain the predicted head movement position based on the difference and the head movement change rate.

The difference between the first historical head movement position and the measured head movement position may refer to the distance between the first historical head movement position and the measured head movement position. For example, in a three-dimensional world, the head movement position may be represented by an xyz coordinate axis, so the difference may be the distance between the coordinate values of the first historical head movement position and the measured head movement position. In addition, the head movement position may also be represented by other methods, which are not specifically limited in the present application.

As described above, the audio signal playback end can periodically detect the measured head movement position of the user wearing the audio signal playback end through a sensor. Therefore, the audio signal playback end can obtain the head movement change rate based on the N measured head movement positions obtained previously. The N measured head movement positions can include N positions counting forward from the current measured head movement position.

The audio signal playback end can predict the head movement change value based on the difference and the head movement change rate by using cubic spline interpolation and other means, and then add the head movement change value to the current measured head movement position to obtain the predicted head movement position.

The second method is that, based on the above 3, the audio signal playback end obtains the head movement change rate, which is obtained based on the N measured head movement positions obtained previously, where N>1; the second uncertainty coefficient is obtained based on the head movement change rate and the first uncertainty coefficient; the second uncertainty coefficient, the first historical head movement position and the measured current head movement position are input into the Kalman filter to obtain the predicted head movement position.

The method for obtaining the head movement change rate can refer to the above description.

The head movement change rate combined with the first uncertainty coefficient can obtain the second uncertainty coefficient. The second uncertainty coefficient, the first historical head movement position and the measured current head movement position are input into the Kalman filter. In the Kalman filter, the Kalman gain coefficient can be determined according to the second uncertainty coefficient and the estimated uncertainty of the previous iteration, and the Kalman gain coefficient is used as the latest gain coefficient. In this way, the Kalman filter can output the predicted head movement position.

It should be noted that, in addition to the above two methods, the present application may also use other algorithms to obtain the predicted head movement position according to the first historical head movement position and the measured head movement position, and there is no specific limitation on this.

Step 303: Send first rendering information to the audio content provider, where the first rendering information includes the predicted head movement position.

The audio signal playback end sends the first rendering information to the audio content provider end, and the first rendering information includes the predicted head movement position obtained in step 302, so that the audio content provider end can render the audio frame with the head movement effect according to the predicted head movement position. The rendered audio frame obtained in this way can provide a listening effect corresponding to the predicted head movement position. Then when the rendered audio frame is sent to the audio signal playback end, the predicted time can offset the delay caused by the rendering process, link transmission, etc., so that when the audio frame is played at the audio signal playback end, it just matches the user's head movement position, achieving the real and immersive listening experience described above.

This application predicts the user's head movement position to obtain the predicted head movement position, and after sending the predicted head movement position to the audio content provider, the audio content provider can render the audio frame based on the predicted head movement position, so that the rendered audio frame obtained can provide the listening effect corresponding to the predicted head movement position. Then when the rendered audio frame is sent to the audio signal playback end, the predicted time can offset the delay caused by rendering processing, link transmission, etc., so that when the audio frame is played at the audio signal playback end, it just matches the user's head movement position, achieving the real and immersive listening experience described above.

FIG4 is a flow chart of a process 400 of the spatial audio rendering method of the present application. As shown in FIG4 , the process 400 can be applied to the communication system shown in FIG1 , and is jointly executed by the audio content provider and the audio signal player to complete the rendering of the audio source and play it on the audio signal player for the user to listen to. The process 400 is described as a series of steps or operations. It should be understood that the process 400 can be executed in various orders and/or occur simultaneously, and is not limited to the execution order shown in FIG4 .

Process 400 includes the following steps:

Step 401: When the first rendering mode is adopted, the audio content provider obtains the current frame.

The first rendering mode may be a non-low-latency link mode, in which the audio content provider can render the audio frame with a head movement effect based on the predicted head movement position. The user can choose whether to select the first rendering mode by operating the application (application, APP) installed on the audio content provider. Correspondingly, the present application also includes a second rendering mode, in which the audio content provider renders the audio frame without a head movement effect, and the process can be described below. Similarly, the user can choose whether to select the second rendering mode by operating the APP installed on the audio content provider. The user's operation can refer to the following embodiments.

The current frame may be one of the frames of the audio source, and may generally refer to an audio frame currently being processed by the audio content provider.

Step 402: When the audio format of the audio source is a non-stereo format, the audio content provider obtains a predicted head movement position.

The audio format of the audio source includes a stereo format or a non-stereo format, wherein the non-stereo format may include but is not limited to audio in multi-channel, multi-object formats such as 5.1, 7.1, and 3DA.

In this application, the audio content provider can perform high-computing rendering on audio in multi-channel, multi-object formats (non-stereo formats) such as 5.1, 7.1, and 3DA, so as to give full play to the computing power advantage of the audio content provider. Based on this, when the audio format of the audio source is a non-stereo format, the audio content provider can render the audio frame with a head movement effect, so it is necessary to obtain the predicted head movement position.

The audio content provider can obtain the predicted head movement position from the rendering information previously received from the audio signal player. Optionally, the rendering information can be the latest received rendering information, which can improve the accuracy of the predicted head movement position. The acquisition of the predicted head movement position can refer to the embodiment shown in Figure 3, which will not be repeated here.

Step 403: The audio content provider renders the current frame according to the predicted head movement position to obtain a first binaural signal.

In the present application, the audio content provider can render the direct sound part, early reflection sound (ER) part and late reverberation sound (LR) part in the current frame with head movement effect according to the predicted head movement position to obtain a first binaural signal.

Optionally, the direct sound with the head movement effect and the reverberation with the head movement effect (obtained by mixing ER and LR) are respectively rendered by an audio effector and mixed, and the mixed output result is rendered again by the audio effector and transmitted in the form of a first binaural signal. The aforementioned audio effector includes but is not limited to an equalizer, a dynamic compression effector, a low-frequency enhancement effector, etc.

In the branch of step 403, the first audio stream may include the first binaural signal.

Step 404: When the audio format of the audio source is a stereo format, the audio content provider does not perform spatial rendering on the audio source.

In this application, the audio content provider does not need to render the audio source in stereo format, and the audio signal player is responsible for rendering the leading motion effect. Therefore, the processing can perform high-computing power rendering on audio in multi-channel, multi-object formats (non-stereo formats) such as 5.1, 7.1, 3DA, etc., so as to give full play to the computing power advantage of the audio content provider. Based on this, when the audio format of the audio source is a non-stereo format, the audio content provider can render the audio frame with a leading motion effect, so the audio content provider can encapsulate the audio source in the first audio stream.

Step 405: The audio content provider sends first information to the audio signal player, where the first information includes a first audio stream, an audio format of an audio source, and delay information.

As described above, when the audio format indicates a monaural format, the first audio stream includes a first binaural signal rendered by the audio content provider; when the audio format indicates a stereo format, the first audio stream includes an audio source.

The delay information can be referred to the description of the embodiment shown in FIG3 , which will not be repeated here.

Step 406: When the audio format is indicated as a stereo format, the audio signal playback end obtains the measured head movement position through a sensor.

The audio source in stereo format is rendered with the head movement position by the audio signal player. Since the IMU data transmission link latency of the audio signal player is low, the audio signal player can obtain the current head movement position (referred to as the measured head movement position in this article) through the sensor in real time.

Step 407: The audio signal player renders the current frame according to the measured head movement position to obtain a second binaural signal.

The audio signal playback end can render the direct sound part in the current frame with the head movement effect according to the measured head movement position, and render the ER and LR in the current frame without the head movement effect to obtain a second binaural signal.

Optionally, the direct sound with the head movement effect and the reverberation without the head movement effect (obtained by mixing ER and LR) are respectively rendered by an audio effector and mixed, and the mixed output is rendered again by the audio effector and mixed again to obtain a second binaural signal. The aforementioned audio effector includes but is not limited to an equalizer, a dynamic compression effector, a low-frequency enhancement effector, etc.

Step 408: The audio signal playing end plays audio according to the target binaural signal, where the target binaural signal includes the first binaural signal or the second binaural signal.

After the above steps, the audio signal player obtains a target binaural signal, which may be a first binaural signal (the audio source is in a non-stereo format) or a second binaural signal (the audio source is in a stereo format).

In one possible implementation, when the second rendering mode is adopted, the audio content provider obtains the current frame; renders the current frame without the head movement effect to obtain a second binaural signal (the second binaural signal is different from the second binaural signal mentioned above, and is referred to as the third binaural signal below for the sake of distinction); sends second information to the audio signal player, the second information includes a second audio stream, and the second audio stream includes the third binaural signal. The audio signal player obtains the measured head movement position through a sensor, and renders the current frame (the current frame is the audio frame after the audio content provider performs the aforementioned rendering without the head movement effect) according to the measured head movement position to obtain a fourth binaural signal. The audio signal player plays audio according to the target binaural signal, and the target binaural signal includes the fourth binaural signal.

In the second rendering mode, the audio content provider renders the audio frames without the head motion effect, which can give full play to the computing power advantage of the audio content provider. The lack of head motion effect can shorten the rendering delay, and the audio signal player performs low-computing power rendering with head motion effect instead. Compared with the first rendering mode, the delay is lower, but the rendering effect of the head motion effect is poor.

The technical solution of the present application is described in detail below through several specific embodiments. In the following embodiments, the audio content provider may be a mobile phone, and the audio signal player may be a headset. The audio source may also refer to the sound source or input signal, the rendering may also refer to (binaural) spatial audio rendering, the head movement position may also refer to IMU data, the rendering algorithm deployed on the mobile phone may also refer to the first rendering part of the binaural spatial audio rendering algorithm, the rendering algorithm deployed on the headset may also refer to the second rendering part of the binaural spatial audio rendering algorithm, the audio format may also refer to the format flag, and the delay may also refer to the dynamic Delay of the spatial audio link.

Embodiment 1

Figure 5 is a schematic diagram of the overall process of the spatial audio rendering method of the present application. As shown in Figure 5, the present application is a spatial audio processing scenario in which a mobile phone and headphones are jointly rendered in a first rendering mode. The joint rendering strategy in this embodiment can be designed as follows: the mobile phone determines that if the sound source is stereo, it is sent to the headphones for binaural spatial audio rendering; if the sound source is non-stereo, it is rendered on the mobile phone, and after the rendering is completed, it is sent to the headphones, and the headphones directly output to the speakers.

FIG6 is a schematic diagram of a specific process of the spatial audio rendering method of the present application. As shown in FIG6 , the process of this embodiment includes:

S1. The mobile phone receives the input signal and detects the input signal format. At the same time, the headset sends the predicted IMU data to the first rendering part of the binaural spatial audio rendering algorithm of the mobile phone;

The first rendering part of the binaural spatial audio rendering algorithm includes direct sound (Direct) rendering processing, early reflection (ER) rendering processing, and late reverberation (LR) rendering processing. The reverberation part after the direct sound part is mixed with ER and LR uses the received IMU data to process the head movement effect. The direct sound with head movement effect and the reverberation with head movement effect are respectively rendered by the audio effector and enter the first mixing module (Mixer1). The output result of the first mixing module is rendered again by the audio effector and transmitted to Bluetooth in a dual-channel form. The above-mentioned audio effectors include but are not limited to equalizers, dynamic compression effectors, low-frequency enhancement effectors, etc.

S2. If the input signal format is stereo, the mobile phone directly transmits the input signal and the format flag to the first mixing module;

S3. If the input signal format is not stereo, use the predicted IMU data and render the input signal according to the first part of the binaural spatial audio rendering algorithm, and pass the rendered audio stream and format flag to the first mixing module;

S4, packaging the audio stream, format flag and dynamic Delay of the mobile phone space audio link of the first mixing module, and transmitting them to the headset via Bluetooth or the like;

S5. The headset stores the measured IMU data, and selects the corresponding historical measured IMU data and the measured IMU data of the headset according to the Delay reported by the mobile phone and transmits them to the prediction module. The measured IMU data of the headset is transmitted to the second rendering part of the binaural spatial audio rendering algorithm. The prediction module generates predicted IMU data according to the historical measured IMU data and the measured IMU data of the headset, and sends the predicted IMU data to the mobile phone again with reference to S1.

FIG. 7 is a flow chart of a head movement position prediction algorithm. As shown in FIG. 7 , the flow chart includes the following steps:

S5.1. The headset stores historical IMU data in a cache, retrieves corresponding historical IMU data from the historical cache according to the received mobile phone delay, calculates the difference between the measured IMU data and the historical IMU data, and sends the difference to the prediction unit;

S5.2, calculating the change rate of the past N measured headphone IMU data (for example, including yaw, pitch, roll or four elements), and passing the change coefficient to the prediction unit;

S5.3, predicting the head movement change value based on the change rate and the difference calculated in S5.1 by using cubic spline interpolation or other means;

S5.4, adding the head movement change value to the measured headphone IMU data to generate a predicted head movement position;

S5.5. Send the predicted head movement position to the mobile phone.

S6. The earphone detects the sound source format. If the sound source format is stereo, the measured IMU data of the earphone is used to render the input signal according to the second part of the binaural spatial audio rendering algorithm, and the rendered audio stream is given to the second mixing module;

The second rendering part of the binaural spatial audio rendering algorithm includes direct sound (Direct) rendering processing and basic low-computing power reverberation rendering processing. The direct sound part uses the received IMU data to process the head movement effect. The direct sound with head movement effect and the basic reverberation without head movement effect are mixed after being rendered by the audio effector. The overall output result after mixing is rendered again by the audio effector and enters the second mixing module (Mixer2). The above-mentioned audio effectors include but are not limited to equalizers, dynamic compression effectors, low-frequency enhancement effectors, etc.

S7, if the sound source format is not stereo, the earphone directly transmits the input signal to the second mixing module;

S8, sending the second mixing module to the earphone speaker for playing.

Embodiment 1 takes full advantage of the high computing power of mobile phones and the low latency of headphones, and the headphones can operate independently. To address the problem of long latency in mobile phone links, an end-to-end latency prediction algorithm is designed to significantly reduce head movement latency (estimated to be reduced by more than 50%).

Embodiment 2

Compared with the first embodiment, the differences in the second embodiment mainly focus on the Bluetooth transmission content in S1 and S4 above, and the head movement position prediction algorithm in S5.

In this embodiment, S1 changes to the mobile phone receiving an input signal and detecting the format of the input signal, while the earphone sends earphone IMU data and predicted IMU data to the first rendering part of the mobile phone binaural spatial audio rendering algorithm.

S4 changes to packaging the IMU data, audio stream and format flag of the first mixing module and transmitting them to the headset via Bluetooth or the like.

S5 changes to the headset transmitting the historical IMU data reported by the mobile phone to the prediction module, and at the same time transmitting the measured IMU data to the prediction module, and transmitting the measured IMU data to the second rendering part of the binaural spatial audio rendering algorithm. The prediction module generates predicted IMU data based on the historical IMU data and the measured IMU data, and sends the measured IMU data and predicted IMU data to the mobile phone with reference to S1.

FIG8a and FIG8b are schematic diagrams of a flow chart of a prediction algorithm for head movement position. As shown in FIG8a and FIG8b, the flow chart includes the following steps:

S5.1: Calculate the difference between the measured IMU data and the historical IMU data reported by the mobile phone, and send the difference to the prediction algorithm;

S5.2: Calculate the change rate of the past N measured headphone IMU data (e.g., including yaw, pitch, roll or four elements), and pass the change coefficient to the prediction unit;

S5.3: Predict the head movement change value based on the change rate and the difference calculated in S5.1 using cubic spline interpolation or other methods;

S5.4: Add the head motion change value to the measured headphone IMU data to generate a predicted head motion position;

S5.5: Send the measured head movement position and the predicted head movement position to the mobile phone.

Embodiment 3

Compared with the first embodiment, the differences in the third embodiment mainly focus on the Bluetooth transmission content in S1 and S4, and the prediction algorithm in S5.

In the third embodiment, S1 is changed to the mobile phone receiving the input signal and detecting the format of the input signal, while the earphone sends the uncertainty coefficient Rn and the predicted IMU data to the first rendering part of the mobile phone binaural spatial audio rendering algorithm.

S4 changes to packaging the IMU data, uncertainty coefficient Rn, audio stream and format flag of the first mixing module and transmitting them to the headset via Bluetooth or the like.

S5 changes to the headset transmitting the IMU data and uncertainty coefficient Rn reported by the mobile phone to the prediction module, and at the same time transmitting the measured IMU data to the prediction module, and transmitting the measured IMU data to the second rendering part of the binaural spatial audio rendering algorithm. The prediction module generates an IMU data prediction value based on the IMU data, uncertainty coefficient Rn and the headset IMU data reported by the mobile phone, and sends the measured IMU data and predicted IMU data to the mobile phone with reference to S1.

FIG9a and FIG9b are schematic diagrams of a flow chart of a prediction algorithm for head movement position. As shown in FIG9a and FIG9b, the flow chart includes the following steps:

S5.1. Calculate the change rate of the past N measured headphone IMU data (for example, including yaw, pitch, roll or four elements), combine the historical measurement uncertainty coefficient Rn, determine the measurement uncertainty coefficient Rn+1, and give Rn+1 to the Kalman filter. At the same time, give the historical head movement position reported by the mobile phone and the measured head movement position measured by the headphone sensor to the Kalman filter;

S5.2, determine the Kalman gain coefficient Kn+1 according to the uncertainty coefficient Rn+1 and the estimation uncertainty Pn+1 of the previous iteration, and send the gain coefficient to the current state update module;

S5.3. According to Xn+1=Xn+Kn+1*(Yn-Xn), the predicted head movement position is updated and the predicted head movement position is output. At the same time, the estimated uncertainty is updated according to Pn+1=(1-Kn)*Pn, where Yn is the current actual head movement position and Xn is the historical head movement position.

Embodiment 4

Figure 10 is a schematic diagram of the framework flow of the present application. As shown in Figure 10, this embodiment provides a compatible solution for spatial audio processing scenarios of non-low-latency (first rendering mode) and low-latency (second rendering mode) mobile phones and headphones jointly rendering. In the second rendering mode, the headphones no longer perform reverberation rendering processing and cannot perform spatial audio rendering independently from the mobile phone, but have the characteristics of low computing power requirements and simple structure.

S1. Add a low-latency mode to the user interface (UI) of the APP that supports spatial audio, and send a low-latency mode flag based on the user's choice;

S2: If it is low latency mode, the phone pre-processes all types of audio signals, including downmixing and reverberation processing;

S3, transmitting to the earphone for direct sound rendering in binaural spatial audio rendering algorithm and then outputting to the earphone speaker;

S4. If it is not low-latency mode, the mobile phone will render the dynamic effect, which includes non-stereo direct sound processing and reverberation processing of all format signals. After rendering, the signal will be transmitted to the headphones.

S5. The earphone determines the input format, performs direct sound rendering processing on the stereo sound source and outputs it to the earphone speaker, and directly outputs the non-stereo sound source to the earphone speaker.

In this embodiment, the process of the low-latency mode is shown in Figure 11 (Figure 11 is a schematic diagram of the process of the low-latency mode of the present application). After the user selects the low-latency mode, the mobile phone pre-processes the received stereo, 5.1/7.1, 3DA and other format audio sources. The preprocessing includes down-mixing the audio sources in multi-channel formats such as 5.1/7.1 or multi-object formats such as 3DA, and unifying them into a stereo format. It is then sent to the first rendering part of the binaural spatial audio rendering algorithm for basic reverberation rendering and sound effector processing without head effects. The reverberation part is mixed with the downmix input of the sound effector and transmitted to the headphones via Bluetooth.

The earphone receives stereo sound with reverberation effect from the mobile phone, combines it with the IMU head movement data input by the earphone sensor, and uses the second part of the binaural spatial audio rendering algorithm to render direct sound. The second part of the binaural spatial audio rendering algorithm includes a direct sound azimuth rendering module and an audio effector module.

In this embodiment, the process of the non-low-latency mode is shown in Figure 12 (Figure 12 is a schematic diagram of the process of the non-low-latency mode of this application). After the mobile phone receives input sources of various formats, the first part of the binaural spatial audio rendering algorithm is used to perform direct sound rendering of non-stereo sound sources and reverberation rendering of all formats of sound sources. The direct sound and reverberation rendering of this part both include head movement effects, and the reverberation adopts the implementation scheme of ER and LR rendering separately and then adding them together. The required predicted head movement position can refer to the prediction algorithm in Examples 1 to 3 above to optimize the head movement delay. Figure 12 uses the prediction algorithm in Example 2 as an example for illustration.

The mobile phone processes the direct sound rendering result or the original stereo input and reverberation rendering result through the corresponding sound effector, mixes the result into a stereo format, and transmits the result to the headset via Bluetooth.

The headphones determine the original input format, and the non-stereo format is directly output to the speaker. The stereo format is processed by the second part of the binaural spatial audio rendering algorithm and the IMU head movement data input by the headphone sensor for direct sound rendering and sound effect processing, and then output to the speaker for playback.

In this embodiment, UI design is added, and the joint rendering solution is selected based on the user's concerns about head motion latency and spatial audio rendering quality.

Joint rendering of mobile phones and headphones: The rendering algorithm is divided into two parts: mobile phone and headphones. The mobile phone renders multi-channel/multi-object parts such as 5.1/7.1/3DA, and renders reverberation effects, giving full play to the computing power advantages of the mobile phone; the headphones render direct sound for stereo, giving full play to the advantages of the short latency of the IMU data transmission link.

Head movement position prediction: To address the problem of long head movement delay in mobile phones, a head movement delay prediction algorithm is designed. The prediction algorithm receives the mobile phone link delay, headphone delay and current IMU value on the headset to generate predicted IMU data. The measured IMU data of the headset is passed to the second part of the binaural spatial audio rendering algorithm, and the predicted IMU data is passed to the first part of the binaural spatial audio rendering algorithm. This solution predicts the head movement position and can reduce the impact of link delay by more than 1/2.

FIG13 is a schematic diagram of an exemplary structure of a spatial audio rendering device 1300 of the present application. As shown in FIG13 , the spatial audio rendering device 1300 of the present embodiment can be applied to an audio signal playback end. The spatial audio rendering device 1300 may include: an acquisition module 1301, a prediction module 1302, a sending module 1303, a receiving module 1304, a rendering module 1305, and a playback module 1306.

The acquisition module 1301 is used to obtain the delay information of the audio content provider; the prediction module 1302 is used to obtain the predicted head movement position based on the delay information; the sending module 1303 is used to send the first rendering information to the audio content provider, and the first rendering information includes the predicted head movement position.

In a possible implementation, the prediction module 1302 is specifically used to obtain a first historical head movement position corresponding to the time delay information; obtain the measured head movement position through a sensor; and obtain the predicted head movement position based on the first historical head movement position and the measured head movement position.

In one possible implementation, the delay information includes the delay sent by the audio content provider; the prediction module 1302 is specifically used to extract the first historical head movement position corresponding to the delay from the cache, and the cache pre-stores the correspondence between multiple groups of delays and historical head movement positions.

In one possible implementation, the delay information includes a second historical head movement position sent by the audio content provider, where the second historical head movement position is the actual head movement position when sending the second rendering information, and the second rendering information is sent earlier than the first rendering information; the prediction module 1302 is specifically used to use the second historical head movement position as the first historical head movement position.

In one possible implementation, the prediction module 1302 is specifically used to obtain the difference between the first historical head movement position and the measured head movement position; obtain the head movement change rate, which is obtained based on the N measured head movement positions obtained previously, N>1; and obtain the predicted head movement position based on the difference and the head movement change rate.

In one possible implementation, the delay information includes a second historical head movement position and a first uncertainty coefficient sent by the audio content provider, wherein the second historical head movement position is the actual head movement position when the second rendering information is sent, and the first uncertainty coefficient comes from the second rendering information, and the second rendering information is sent earlier than the first rendering information; the prediction module 1302 is specifically used to use the second historical head movement position as the first historical head movement position.

In one possible implementation, the prediction module 1302 is specifically used to obtain the head movement change rate, which is obtained based on N previously measured head movement positions, N>1; obtain a second uncertainty coefficient based on the head movement change rate and the first uncertainty coefficient; and input the second uncertainty coefficient, the first historical head movement position and the measured current head movement position into a Kalman filter to obtain the predicted head movement position.

In a possible implementation, a receiving module 1304 is used to receive first information sent by the audio content provider, the first information including an audio format, a first audio stream and the delay information; wherein the audio format indicates whether the audio source is in a stereo format or a non-stereo format; when the audio format indicates the non-stereo format, the first audio stream includes a first binaural signal rendered by the audio content provider; when the audio format indicates the stereo format, the first audio stream includes the audio source; a rendering module 1305 is used to obtain a measured head position through a sensor when the audio format indicates the stereo format; a second binaural signal is rendered for a current frame according to the measured head position, the current frame being one of the frames of the audio source; a playing module 1306 is used to play audio according to a target binaural signal, the target binaural signal including the first binaural signal or the second binaural signal.

In one possible implementation, the rendering module 1305 is specifically used to render the direct sound part in the current frame with the head movement effect according to the measured head movement position, and to render the early reflected sound ER and the late reverberation sound LR in the current frame without the head movement effect, so as to obtain the second binaural signal.

The device of this embodiment can be used to execute the technical solution executed by the client in the method embodiment shown in Figure 3 or Figure 4. Its implementation principle and technical effects are similar and will not be repeated here.

FIG14 is a schematic diagram of an exemplary structure of a spatial audio rendering device 1400 of the present application. As shown in FIG14 , the spatial audio rendering device 1400 of the present embodiment can be applied to an audio content provider. The spatial audio rendering device 1400 may include: an acquisition module 1401, a rendering module 1402, and a sending module 1403.

The acquisition module 1401 is used to acquire the current frame when the first rendering mode is adopted, and the current frame is one of the frames of the audio source; when the audio format of the audio source is a non-stereo format, the predicted head movement position is acquired; the rendering module 1402 is used to render the current frame according to the predicted head movement position to obtain a first binaural signal; the sending module 1403 is used to send first information to the audio signal playback end, the first information includes a first audio stream, the audio format and delay information of the audio source, and the first audio stream includes the first binaural signal.

In a possible implementation, the rendering module 1402 is specifically used to render the direct sound part, the early reflection sound ER part and the late reverberation sound LR part in the current frame with head movement effect according to the predicted head movement position, so as to obtain the first binaural signal.

In a possible implementation, the acquisition module 1401 is specifically configured to obtain the predicted head movement position from rendering information sent by the audio signal playback end.

In a possible implementation manner, the delay information includes delay.

In a possible implementation, the rendering information further includes the measured head movement position when the audio signal playback end sends the rendering information; correspondingly, the delay information includes the measured head movement position.

In a possible implementation, the rendering information also includes the measured head position and the first uncertainty coefficient when the audio signal playback end sends the rendering information; correspondingly, the delay information includes the measured head position and the first uncertainty coefficient.

In a possible implementation manner, when the audio format of the audio source is a stereo format, the first audio stream includes the audio source.

In a possible implementation, the acquisition module 1401 is further used to acquire the current frame when a second rendering mode is adopted; the rendering module 1402 is further used to render the current frame without a head motion effect to obtain a second binaural signal; the sending module 1403 is further used to send second information to the audio signal playback end, the second information includes a second audio stream, and the second audio stream includes the second binaural signal.

The device of this embodiment can be used to execute the technical solution executed by the client in the method embodiment shown in Figure 3 or Figure 4. Its implementation principle and technical effects are similar and will not be repeated here.

In the implementation process, each step of the above method embodiment can be completed by the hardware integrated logic circuit or software instruction in the processor. The processor can be a general processor, a digital signal processor (digital signal processor, DSP), an application-specific integrated circuit (application-specific integrated circuit, ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware coding processor to perform, or the hardware and software modules in the coding processor are combined to perform. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, and other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

The memory mentioned in the above embodiments may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims (38)

1. A spatial audio rendering method, comprising: Obtaining the delay information of the audio content provider; Acquire a predicted head movement position according to the time delay information; Sending first rendering information to the audio content provider, where the first rendering information includes the predicted head movement position. 2. The method according to claim 1, characterized in that the step of obtaining the predicted head movement position according to the time delay information comprises: Acquire a first historical head movement position corresponding to the time delay information; The actual head motion position is obtained through the sensor; The predicted head movement position is acquired according to the first historical head movement position and the measured head movement position. 3. The method according to claim 2, characterized in that the delay information includes the delay sent by the audio content provider; The acquiring a first historical head movement position corresponding to the delay information includes: The first historical head movement position corresponding to the time delay is extracted from a cache, wherein the cache pre-stores a plurality of sets of correspondences between time delays and historical head movement positions. 4. The method according to claim 2, characterized in that the delay information includes a second historical head movement position sent by the audio content provider, the second historical head movement position is a measured head movement position when sending the second rendering information, and the second rendering information is sent earlier than the first rendering information; The acquiring a first historical head movement position corresponding to the delay information includes: The second historical head movement position is used as the first historical head movement position. 5. The method according to any one of claims 2 to 4, characterized in that the step of obtaining the predicted head movement position according to the first historical head movement position and the measured head movement position comprises: Obtaining a difference between the first historical head movement position and the actually measured head movement position; Obtaining a head movement change rate, where the head movement change rate is obtained based on N previously obtained measured head movement positions, where N>1; The predicted head movement position is obtained according to the difference and the head movement change rate. 6. The method according to claim 2, characterized in that the delay information includes a second historical head movement position and a first uncertainty coefficient sent by the audio content provider, the second historical head movement position is the measured head movement position when the second rendering information is sent, the first uncertainty coefficient comes from the second rendering information, and the second rendering information is sent earlier than the first rendering information; The acquiring a first historical head movement position corresponding to the delay information includes: The second historical head movement position is used as the first historical head movement position. 7. The method according to claim 6, characterized in that the step of obtaining the predicted head movement position according to the first historical head movement position and the measured head movement position comprises: Obtaining a head movement change rate, where the head movement change rate is obtained based on N previously obtained measured head movement positions, where N>1; Obtaining a second uncertainty coefficient according to the head movement change rate and the first uncertainty coefficient; The second uncertainty coefficient, the first historical head motion position and the measured current head motion position are input into a Kalman filter to obtain the predicted head motion position. 8. The method according to any one of claims 1 to 7, further comprising: Receive first information sent by the audio content provider, the first information including an audio format, a first audio stream, and the delay information; wherein the audio format indicates whether the audio source is in a stereo format or a non-stereo format; when the audio format indicates the non-stereo format, the first audio stream includes a first binaural signal rendered by the audio content provider; when the audio format indicates the stereo format, the first audio stream includes the audio source; When the audio format indication is the stereo format, obtaining the measured head motion position through a sensor; a second binaural signal obtained by rendering a current frame according to the measured head motion position, wherein the current frame is one of the frames of the audio source; The audio is played according to a target binaural signal, where the target binaural signal includes the first binaural signal or the second binaural signal. 9. The method according to claim 8, characterized in that the second binaural signal obtained by rendering the current frame according to the measured head motion position comprises: The direct sound part in the current frame is rendered with the head movement effect according to the measured head movement position, and the early reflection sound ER and the late reverberation sound LR in the current frame are rendered without the head movement effect to obtain the second binaural signal. 10. A spatial audio rendering method, comprising: When the first rendering mode is adopted, a current frame is obtained, where the current frame is one of the frames of the audio source; When the audio format of the audio source is a non-stereo format, obtaining a predicted head movement position; Rendering the current frame according to the predicted head movement position to obtain a first binaural signal; Sending first information to an audio signal playback end, the first information including a first audio stream, an audio format and delay information of the audio source, the first audio stream including the first binaural signal. 11. The method according to claim 10, wherein rendering the current frame according to the predicted head motion position to obtain the first binaural signal comprises: The direct sound part, the early reflection sound ER part and the late reverberation sound LR part in the current frame are respectively rendered with the head movement effect according to the predicted head movement position to obtain the first binaural signal. 12. The method according to claim 10 or 11, characterized in that obtaining the predicted head movement position comprises: The predicted head movement position is obtained from rendering information sent by the audio signal playing end. The method according to claim 12 , wherein the delay information comprises a delay. 14. The method according to claim 12, characterized in that the rendering information further comprises a measured head motion position when the audio signal playback end sends the rendering information; Correspondingly, the time delay information includes the measured head motion position. 15. The method according to claim 12, characterized in that the rendering information further comprises a measured head motion position and a first uncertainty coefficient when the audio signal playback end sends the rendering information; Correspondingly, the time delay information includes the measured head motion position and the first uncertainty coefficient. 16 . The method according to claim 10 , wherein when the audio format of the audio source is a stereo format, the first audio stream includes the audio source. 17. The method according to any one of claims 10 to 16, further comprising: When the second rendering mode is adopted, obtaining the current frame; Rendering the current frame without head movement effect to obtain a second binaural signal; Sending second information to the audio signal playback end, where the second information includes a second audio stream, and the second audio stream includes the second binaural signal. 18. A spatial audio rendering device, comprising: An acquisition module, used to acquire the delay information of the audio content provider; A prediction module, used for obtaining a predicted head movement position according to the time delay information; A sending module is used to send first rendering information to the audio content provider, where the first rendering information includes the predicted head movement position. 19. The device according to claim 18 is characterized in that the prediction module is specifically used to obtain a first historical head movement position corresponding to the time delay information; obtain the measured head movement position through a sensor; and obtain the predicted head movement position based on the first historical head movement position and the measured head movement position. 20. The device according to claim 19, characterized in that the delay information includes the delay sent by the audio content provider; The prediction module is specifically used to extract the first historical head movement position corresponding to the time delay from a cache, and the cache pre-stores a plurality of sets of correspondences between time delays and historical head movement positions. 21. The device according to claim 19, wherein the delay information comprises a second historical head movement position sent by the audio content provider, the second historical head movement position is a measured head movement position when sending the second rendering information, and the second rendering information is sent earlier than the first rendering information; The prediction module is specifically configured to use the second historical head movement position as the first historical head movement position. 22. The device according to any one of claims 19-21 is characterized in that the prediction module is specifically used to obtain the difference between the first historical head movement position and the measured head movement position; obtain the head movement change rate, and the head movement change rate is obtained based on N measured head movement positions obtained previously, N>1; obtain the predicted head movement position based on the difference and the head movement change rate. 23. The device according to claim 19, wherein the delay information comprises a second historical head movement position and a first uncertainty coefficient sent by the audio content provider, the second historical head movement position is a measured head movement position when sending the second rendering information, the first uncertainty coefficient comes from the second rendering information, and the second rendering information is sent earlier than the first rendering information; The prediction module is specifically configured to use the second historical head movement position as the first historical head movement position. 24. The device according to claim 23 is characterized in that the prediction module is specifically used to obtain the head movement change rate, which is obtained based on N previously measured head movement positions, N>1; obtain a second uncertainty coefficient based on the head movement change rate and the first uncertainty coefficient; input the second uncertainty coefficient, the first historical head movement position and the measured current head movement position into a Kalman filter to obtain the predicted head movement position. 25. The device according to any one of claims 18 to 24, further comprising: A receiving module, configured to receive first information sent by the audio content provider, wherein the first information includes an audio format, a first audio stream, and the delay information; wherein the audio format indicates whether the audio source is in a stereo format or a non-stereo format; when the audio format indicates the non-stereo format, the first audio stream includes a first binaural signal rendered by the audio content provider; when the audio format indicates the stereo format, the first audio stream includes the audio source; a rendering module, configured to, when the audio format indication is the stereo format, obtain a measured head motion position through a sensor; and render a second binaural signal obtained by performing rendering on a current frame according to the measured head motion position, wherein the current frame is one of the frames of the audio source; The playing module is used to play audio according to a target binaural signal, where the target binaural signal includes the first binaural signal or the second binaural signal. 26. The device according to claim 25 is characterized in that the rendering module is specifically used to render the direct sound part in the current frame with the head movement effect according to the measured head movement position, and to render the early reflected sound ER and the late reverberation sound LR in the current frame without the head movement effect, so as to obtain the second binaural signal. 27. A spatial audio rendering device, comprising: An acquisition module, configured to acquire a current frame when the first rendering mode is adopted, wherein the current frame is one of the frames of the audio source; and acquire a predicted head movement position when the audio format of the audio source is a non-stereo format; A rendering module, configured to render the current frame according to the predicted head movement position to obtain a first binaural signal; The sending module is used to send first information to the audio signal playing end, wherein the first information includes a first audio stream, an audio format and delay information of the audio source, and the first audio stream includes the first binaural signal. 28. The device according to claim 27 is characterized in that the rendering module is specifically used to render the direct sound part, the early reflection sound ER part and the late reverberation sound LR part in the current frame with head movement effect according to the predicted head movement position, so as to obtain the first binaural signal. 29. The device according to claim 27 or 28, characterized in that the acquisition module is specifically used to obtain the predicted head movement position from the rendering information sent by the audio signal playback end. 30. The apparatus according to claim 29, wherein the delay information comprises a delay. 31. The device according to claim 29, characterized in that the rendering information further includes the measured head movement position when the audio signal playback end sends the rendering information; Correspondingly, the time delay information includes the measured head motion position. 32. The device according to claim 29, characterized in that the rendering information further includes the measured head motion position and the first uncertainty coefficient when the audio signal playback end sends the rendering information; Correspondingly, the time delay information includes the measured head motion position and the first uncertainty coefficient. 33. The device according to any one of claims 27 to 32, characterized in that when the audio format of the audio source is a stereo format, the first audio stream includes the audio source. 34. The device according to any one of claims 27 to 33, characterized in that the acquisition module is further used to acquire the current frame when the second rendering mode is adopted; The rendering module is further used to render the current frame without head motion effect to obtain a second binaural signal; The sending module is further used to send second information to the audio signal playing end, where the second information includes a second audio stream, and the second audio stream includes the second binaural signal. 35. An audio signal playing device, comprising: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9. 36. An audio content providing device, comprising: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 10 to 17. 37. A computer-readable storage medium, characterized in that it comprises a computer program, and when the computer program is executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 17. 38. A computer program product, characterized in that the computer program product comprises computer program code, and when the computer program code is run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 17.