RU2846304C1 - Methods, apparatus and systems for expansion of three degrees of freedom (3dof+) mpeg-h 3d audio - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to means of processing information on the position of an audio object. Obtaining information on orientation of listener, indicating orientation of head of listener, wherein information on orientation of listener contains information on yaw, pitch and roll of listener head. Obtaining information on the displacement of the listener, indicating the displacement of the head of the listener relative to the nominal listening position through the input interface of MPEG-H 3D Audio decoder. Position of the object is modified based on the information on the displacement of the listener by applying translation to the position of the object. When the listener shift information indicates the listener's head shift from the nominal listening position due to a small position shift, distance between the position of the sound object and the listening position after displacement of the head of the listener is equal to the distance between the modified position of the sound object and the nominal listening position.

EFFECT: higher processing efficiency.

9 cl, 5 dwg

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of the following priority applications: U.S. Provisional Application No. 62/654,915 (Ref. D18045USP1), filed April 9, 2018; U.S. Provisional Application No. 62/695,446 (Ref. D18045USP2), filed July 9, 2018; and U.S. Provisional Application No. 62/823,159 (Ref. D18045USP3), filed March 25, 2019, which are incorporated herein by reference.

AREA OF TECHNOLOGY

The present invention relates to methods and apparatus for processing position information indicating the position of a sound object and information indicating a displacement of the position of a listener's head.

BACKGROUND OF THE INVENTION

The first edition (15 October 2015) and revisions 1-4 of the ISO/IEC 23008-3 MPEG-H 3D Audio standard did not provide support for small translational user head movements in a Three Degrees of Freedom (3DoF) environment.

BRIEF DESCRIPTION OF THE ESSENCE OF THE INVENTION

The first edition (15 October 2015) and Editions 1-4 of the ISO/IEC 23008-3 MPEG-H 3D Audio standard introduced functionality to enable a 3DoF environment where the user (listener) performs head-turning actions. However, such functionality at best only supports signaling and rendering of the rotational scene displacement. This means that the sound stage can remain stationary in space while the listener's head orientation changes, which is the 3DoF property. However, there is no way to account for the small translational movement of the user's head within a true MPEG-H 3D Audio ecosystem.

Thus, there is a need for methods and apparatus for processing position information of sound objects that can take into account a small translational movement of the user's head, potentially in combination with a rotational movement of the user's head.

The present invention provides apparatus and systems for processing position information having the features of the corresponding independent and dependent claims of the invention.

According to one aspect of the present invention, a method is described for processing position information indicating the position of an audio object, wherein the processing may be compatible with the MPEG-H 3D Audio standard. The position of the object may be used for rendering the audio object. The audio object may be included in object-based audio content along with the position information. The position information may be (part of) metadata for the audio object. The audio content (e.g., the audio object along with the position information) may be transmitted in an encoded audio bitstream. The method may include receiving the audio content (e.g., the encoded audio bitstream). The method may include obtaining listener orientation information indicating the orientation of the listener's head. The listener may be called a user, such as a decoder performing the method. The listener's head orientation (listener orientation) may be an orientation of the listener's head relative to the nominal orientation. The method may further include obtaining listener offset information indicating the offset of the listener's head. The listener head offset may be an offset relative to the nominal listening position. The nominal listening position (or nominal listener position) may be a default position (e.g., a specified position, an expected position for the listener's head, or a sweet spot in a speaker arrangement). The listener orientation information and the listener offset information may be obtained by means of an input interface of the MPEG-H 3D Audio decoder. The listener orientation information and the listener offset information may be derived based on information from sensors. The combination of orientation information and position information may be referred to as position information. The method may further include determining the position of an object from the position information. For example, the position of an object may be extracted from the position information. Determining (e.g., extracting) the position of an object may further be based on information regarding the geometry of the speaker arrangement of one or more speakers in the listening environment. The position of an object may also be referred to as a channel position of the audio object. The method may further include modifying the position of the object based on the listener offset information by applying a translation to the position of the object. The modification of the object position may relate to adjusting the object position to offset the listener's head from the nominal listening position. In other words, the modification of the object position may relate to applying position offset compensation to the object position. The method may also further include further modifying the modified object position based on listener orientation information, such as by applying a rotational transformation to the modified object position (e.g., rotation relative to the listener's head or the nominal listening position). The further modification of the modified object position for rendering the audio object may include a rotational offset of the sound stage.

The proposed method, adapted as described above, provides a more realistic listening experience, especially for sound objects that are close to the listener's head. In addition to the three (rotational) degrees of freedom typically offered to the listener in a 3DoF environment, the proposed method can also take into account the translational movements of the listener's head. This allows the listener to reach close sound objects from different angles and even sides. For example, the listener can listen to the sound object "mosquito", which is located near the listener's head, from different angles by slightly shifting his head, possibly in addition to turning his head. As a result, the proposed method can provide an improved, more realistic, immersive listening experience for the listener.

In some embodiments, the modification of the object position and the further modification of the modified object position may be performed in such a way that the audio object, after rendering on one or more real or virtual speakers in accordance with the further modified object position, is psychoacoustically perceived by the listener as originating from a fixed position relative to the nominal listening position, regardless of the displacement of the listener's head from the nominal listening position and the orientation of the listener's head relative to the nominal orientation. Accordingly, the audio object may be perceived as moving relative to the listener's head, when the listener's head performs a displacement from the nominal listening position. Similarly, the audio object may be perceived as rotating relative to the listener's head, when the listener's head performs an orientation change from the nominal orientation. The one or more speakers may be, for example, part of a headset or may be part of a speaker composition (for example, a 2.1, 5.1, 7.1 speaker composition, etc.).

In some embodiments, modifying the position of an object based on information about the displacement of a listener may be performed by translating the position of the object using a vector that is positively correlated with the amplitude and negatively correlated with the direction of the vector of displacement of the listener's head from the nominal listening position.

This ensures that nearby sound objects are perceived by the listener as moving in accordance with the movement of his head. This contributes to a more realistic sensation of listening to these sound objects.

In some embodiments, the listener displacement information may indicate a displacement of the listener's head from the nominal listening position due to a small positional displacement. For example, the absolute value of the displacement may be no more than 0.5 m. The displacement may be expressed in Cartesian coordinates (e.g., x, y, z) or spherical coordinates (e.g., azimuth, elevation angle, radius).

In some embodiments, the listener displacement information may indicate a displacement of the listener's head from the nominal listening position, which may be achieved by moving the listener's upper body and/or head. In this way, the listener may perform the displacement without moving the lower body. For example, the listener's head displacement may be performed when the listener is sitting on a chair.

In some embodiments, the position information may comprise an indication of the distance from the audio object to the nominal listening position. The distance (radius) may be less than 0.5 m. For example, the distance may be less than 1 cm. Alternatively, the distance from the audio object to the nominal listening position may be set to a default value by the decoder.

In some embodiments, the listener orientation information may comprise information about the yaw, pitch, and roll of the listener's head. The yaw, pitch, and roll may be provided relative to a nominal orientation (e.g., initial orientation) of the listener's head.

In some embodiments, the listener displacement information may comprise information about the displacement of the listener's head from the nominal listening position, expressed in Cartesian coordinates or spherical coordinates. Thus, the displacement may be expressed in terms of x, y, z coordinates for Cartesian coordinates and in terms of azimuth, elevation angle, radius coordinates for spherical coordinates.

In some embodiments, the method may further include detecting the orientation of the listener's head by means of wearable and/or stationary equipment. Similarly, the method may further include detecting the displacement of the listener's head from the nominal listening position by means of wearable and/or stationary equipment. The wearable equipment may be, correspond to and/or include, for example, a headset or an augmented reality (AR)/virtual reality (VR) headset. The stationary equipment may be, correspond to and/or include, for example, camera sensors. This ensures that accurate information is obtained about the displacement and/or orientation of the listener's head and, thus, ensures realistic processing of nearby sound objects in accordance with the orientation and/or displacement.

In some embodiments, the method may further include rendering the audio object on one or more real or virtual speakers in accordance with the further modified position of the object. For example, rendering the audio object may be performed for the left and right speakers of the headset.

In some embodiments, rendering may be performed taking into account acoustic absorption for short distances from the sound object to the listener's head based on head-related transfer functions (HRTFs) for the listener's head. In this way, rendering of close sound objects will be perceived by the listener as even more realistic.

In some embodiments, the further modified position of the object may be adjusted for the input format used by the MPEG-H 3D Audio rendering module. In some embodiments, rendering may be performed using the MPEG-H 3D Audio rendering module. In some embodiments, processing may be performed using the MPEG-H 3D Audio decoder. In some embodiments, processing may be performed by the scene shifter of the MPEG-H 3D Audio decoder. Accordingly, the proposed method provides for the implementation of a limited sense of six degrees of freedom (Six Degrees of Freedom, 6DoF) (i.e., 3DoF+) within the MPEG-H 3D Audio standard.

According to another aspect of the present invention, an additional method for processing position information indicating the position of an object for an audio object is described. The position of the object can be used to render the audio object. The method can include obtaining listener displacement information indicating the displacement of the listener's head. The method can further include determining the position of the object from the position information. The method can also further include modifying the position of the object based on the listener displacement information by applying a translational movement to the position of the object.

The proposed method, adapted as described above, provides a more realistic listening experience, especially for sound objects that are close to the listener's head. Due to the ability to take into account small translational movements of the listener's head, the proposed method allows the listener to reach close sound objects from different angles and even sides. As a result, the proposed method can provide an improved, more realistic, immersive listening experience for the listener.

In some embodiments, the modification of the object position based on the listener displacement information may be performed in such a way that the sound object, after rendering on one or more real or virtual speakers in accordance with the modified object position, is psychoacoustically perceived by the listener as arising from a fixed position relative to the nominal listening position, regardless of the listener's head displacement from the nominal listening position.

In some embodiments, modifying the position of an object based on information about the displacement of a listener may be performed by translating the position of the object using a vector that is positively correlated with the amplitude and negatively correlated with the direction of the vector of displacement of the listener's head from the nominal listening position.

According to another aspect of the present invention, an additional method for processing position information indicating the position of an object for an audio object is described. The position of the object can be used for rendering the audio object. The method can include obtaining listener orientation information indicating the orientation of the listener's head. The method can further include determining the position of the object from the position information. The method can also further include modifying the position of the object based on the listener orientation information, such as by applying a rotational transformation to the position of the object (e.g., rotation relative to the listener's head or a nominal listening position).

The proposed method, adapted as described above, can take into account the orientation of the listener's head to provide the listener with a more realistic listening experience.

In some embodiments, the modification of the object position based on the listener orientation information may be performed in such a way that the sound object, after rendering on one or more real or virtual speakers in accordance with the modified object position, is psychoacoustically perceived by the listener as arising from a fixed position relative to the nominal listening position, regardless of the orientation of the listener's head relative to the nominal orientation.

According to another aspect of the present invention, an apparatus for processing position information indicating an object position for an audio object is described. The object position can be used for rendering the audio object. The apparatus can comprise a processor and a memory device coupled to the processor. The processor can be adapted to obtain listener orientation information indicating the orientation of the listener's head. The processor can be further adapted to obtain listener displacement information indicating the displacement of the listener's head. The processor can be further adapted to determine the object position from the position information. The processor can be further adapted to modify the object position based on the listener displacement information by applying a translation to the object position. The processor can also be further adapted to further modify the modified object position based on the listener orientation information, for example by applying a rotational transformation to the modified object position (for example, rotation relative to the listener's head or a nominal listening position).

In some embodiments, the processor may be adapted to modify the position of the object and further modify the modified position of the object such that the audio object, after rendering on one or more real or virtual speakers in accordance with the further modified position of the object, is psychoacoustically perceived by the listener as arising from a fixed position relative to the nominal listening position, regardless of the displacement of the listener's head from the nominal listening position and the orientation of the listener's head relative to the nominal orientation.

In some embodiments, the processor may be adapted to modify the position of the object based on the listener displacement information by translating the position of the object using a vector that is positively correlated with the amplitude and negatively correlated with the direction of the listener's head displacement vector from the nominal listening position.

In some embodiments, the listener displacement information may indicate a displacement of the listener's head from the nominal listening position due to a small positional shift.

In some embodiments, the listener displacement information may indicate a displacement of the listener's head from a nominal listening position, which may be achieved by movement of the listener's upper body and/or head.

In some embodiments, the position information may comprise an indication of the distance from the audio object to the nominal listening position.

In some embodiments, the listener orientation information may include information about the yaw, pitch, and roll of the listener's head.

In some embodiments, the listener displacement information may comprise information about the displacement of the listener's head from the nominal listening position, expressed in Cartesian coordinates or spherical coordinates.

In some embodiments, the apparatus may further comprise wearable and/or stationary equipment for detecting the orientation of the listener's head. In some embodiments, the apparatus may further comprise wearable and/or stationary equipment for detecting the displacement of the listener's head from the nominal listening position.

In some embodiments, the processor may be further adapted to render the audio object on one or more real or virtual speakers in accordance with the further modified position of the object.

In some embodiments, the processor may be adapted to perform acoustic absorption-aware rendering for short distances from the sound object to the listener's head based on the HRTF for the listener's head.

In some embodiments, the processor may be adapted to adjust the further modified position of the object for the input format used by the MPEG-H 3D rendering module. In some embodiments, the rendering may be performed using the MPEG-H 3D Audio rendering module. That is, the processor may implement the MPEG-H 3D Audio rendering module. In some embodiments, the processor may be adapted to implement the MPEG-H 3D Audio decoder. In some embodiments, the processor may be adapted to implement the scene shifter of the MPEG-H 3D Audio decoder.

According to another aspect of the present invention, an additional apparatus for processing position information indicating the position of an object for an audio object is described. The position of the object can be used for rendering the audio object. The apparatus can comprise a processor and a memory device connected to the processor. The processor can be adapted to obtain listener displacement information indicating the displacement of the listener's head. The processor can be further adapted to determine the position of the object from the position information. The processor can be further adapted to modify the position of the object based on the listener displacement information by applying a translational movement to the position of the object.

In some embodiments, the processor may be adapted to modify the position of the object based on information about the displacement of the listener such that the audio object, after rendering on one or more real or virtual speakers in accordance with the modified position of the object, is psychoacoustically perceived by the listener as arising from a fixed position relative to the nominal listening position, regardless of the displacement of the listener's head from the nominal listening position.

In some embodiments, the processor may be adapted to modify the position of the object based on the listener displacement information by translating the position of the object using a vector that is positively correlated with the amplitude and negatively correlated with the direction of the listener's head displacement vector from the nominal listening position.

According to another aspect of the present invention, an additional apparatus for processing position information indicating the position of an object for an audio object is described. The position of the object can be used for rendering the audio object. The apparatus can comprise a processor and a memory device connected to the processor. The processor can be adapted to obtain listener orientation information indicating the orientation of the listener's head. The processor can be further adapted to determine the position of the object from the position information. The processor can also be further adapted to modify the position of the object based on the listener orientation information, for example by applying a rotational transformation to the modified position of the object (for example, rotation relative to the listener's head or the nominal listening position).

In some embodiments, the processor may be adapted to modify the position of the object based on information about the orientation of the listener such that the audio object, after rendering on one or more real or virtual speakers in accordance with the modified position of the object, is psychoacoustically perceived by the listener as arising from a fixed position relative to the nominal listening position, regardless of the orientation of the listener's head relative to the nominal orientation.

According to another aspect, a system is described. The system may comprise an apparatus according to any of the above aspects and wearable and/or stationary equipment capable of detecting the orientation of the listener's head and detecting the displacement of the listener's head.

It should be understood that the method steps and the characteristic features of the apparatus can be interchanged in various ways. In particular, the details of the disclosed method can be implemented as an apparatus adapted to perform some or all of the method steps, and vice versa, as will be understood by a person skilled in the art. In particular, it should be understood that the apparatus according to the present invention can refer to an apparatus for implementing or performing the methods according to the above embodiments and variations thereof, and that the corresponding statements made with respect to the methods similarly apply to the corresponding apparatus. Likewise, it should be understood that the methods according to the present invention can refer to methods of operating the apparatus according to the above embodiments and variations thereof, and that the corresponding statements made with respect to the apparatus similarly apply to the corresponding methods.

BRIEF DESCRIPTION OF THE FIGURES

Below, the invention is explained illustratively with reference to the accompanying graphic materials, in which:

Fig. 1 shows a schematic example of an MPEG-H 3D Audio system;

Fig. 2 schematically shows an example of an MPEG-H 3D Audio system according to the present invention;

Fig. 3 schematically shows an example of a sound rendering system according to the present invention;

Fig. 4 shows schematically an exemplary system of Cartesian coordinate axes and their relationship to spherical coordinates; and

Fig. 5 is a block diagram schematically illustrating an example of a method for processing position information for an audio object according to the present invention.

DETAILED DESCRIPTION

In the context of this document, 3DoF is typically a system that can correctly handle the user's head movement, in particular head rotation characterized by three parameters (e.g., yaw, pitch, roll). Such systems are often available in various gaming systems, such as virtual reality (VR)/augmented reality (AR)/mixed reality (MR) systems, or other similar audio environments.

In the context of this document, a user (e.g. an audio decoder or a playback system containing an audio decoder) may also be referred to as a "listener".

In the context of this document, 3DoF+ shall mean that in addition to the user's head movement, which can be correctly processed by a 3DoF system, translational movements shall also be processed.

In the context of this document, the expression "small" shall indicate that the movements are limited to a threshold value, which is typically no more than 0.5 meters. This means that the movements do not exceed 0.5 meters from the initial position of the user's head. For example, the user's movements are limited due to the fact that he or she is sitting on a chair.

In the context of this document, the term "MPEG-H 3D Audio" refers to the technical description specified in ISO/IEC 23008-3 and/or any future revisions, editions or other versions of ISO/IEC 23008-3.

In the context of audio standards provided by the MPEG organization, the difference between 3DoF and 3DoF+ can be defined as follows:

• 3DoF: allows the user to experience yaw, pitch, roll motion (eg of the user's head);

• 3DoF+: Allows the user to experience yaw, pitch, roll, and limited translational motion (e.g. of the user's head), such as while sitting in a chair.

Limited (small) translational head movements may be movements that are limited to a specific radius of motion. For example, movements may be limited due to the user's sitting position, such as without using the lower body. Small translational head movements may refer to or correspond to the displacement of the user's head relative to the nominal listening position. The nominal listening position (or nominal listener position) may be a default position (such as, for example, a target position, an expected position for the listener's head, or the sweet spot of a speaker arrangement).

3DoF+ perception can be compared to limited 6DoF perception, in which translational movements can be described as limited or small head movements. In one example, sound is also rendered based on the position and orientation of the user's head, including possible acoustic absorption. Rendering can be performed taking into account acoustic absorption for small distances from the sound object to the listener's head, for example based on the head-related transfer functions (HRTF) of the listener's head.

With respect to methods, systems, apparatus, and other devices compatible with the functionality established by the MPEG-H 3D Audio standard, which may indicate that 3DoF+ is supported for any future version(s) of the MPEG standards, such as future versions of the Omnidirectional Media Format (e.g., as standardized in future versions of MPEG-I), and/or in any updates to MPEG-H Audio (e.g., revisions or newer standards based on the MPEG-H 3D Audio standard), or any related or reference standards that may require an update (e.g., standards that clarify specific types of metadata and SEI messages).

For example, the functionality of the audio rendering module, which is normative for the audio standard specified in the MPEG-H 3D Audio specification, can be extended to include rendering of the sound stage to accurately account for user interaction with the sound stage, such as when the user moves their head slightly from side to side.

The present invention provides various technical advantages, including the advantage of providing MPEG-H 3D Audio capable of handling 3DoF+ use cases. The present invention extends the MPEG-H 3D Audio standard to support 3DoF+ functionality.

In order to support 3DoF+ functionality, the audio rendering system must take into account limited/small offsets in the user/listener's head position. The position offsets must be determined based on the relative offset from the initial position (i.e. the default/nominal listening position). In one example, the amplitude of this offset (e.g. the radius offset, which can be determined based on r _offset =||P ₀ -P ₁ ||), where P ₀ is the nominal listening position and P ₁ is the offset head position of the listener) is approximately 0.5 m at most. In another example, the offset amplitude is limited to being the offset only obtained when the user is sitting on a chair and does not move his/her lower body (however, his/her head moves relative to his/her body). This (small) offset distance provides very little (perceived) level and panning difference for distant sound objects. However, for close objects, such a small offset distance may become perceptually significant. Of course, the listener's head movement may influence the perception of where the precise localization of a sound object is. This perceptual effect may remain important (i.e., be noticeable to the user/listener's perception) provided that the relationship between (i) the user's head offset (e.g., r _offset =||P ₀ -P ₁ ||)) and the distance to the sound object (e.g., r) trigonometrically yields angles that are within the range of users' psychoacoustic ability to detect the direction of a sound. Such a range may vary for different settings of the sound renderer, the audio material, and the playback configuration. For example, assuming a localization accuracy range of, say, +/- 3° with a side-to-side freedom of the listener's head movement of +/- 0.25m, this would correspond to a distance to the object of ~5m.

For objects that are located close to the listener (e.g. objects less than 1m away from the user), proper handling of listener head position offset is important for 3DoF+ scenarios, as there are significant perceptual effects during both panning and level changes.

One example of processing objects near listeners, for example, is when a sound object (e.g., a mosquito) is located very close to the listener's face. A sound system, such as a sound system that enables VR/AR/MR capabilities, should allow the user to perceive this sound object from all sides and at all angles, even when the user makes small translational movements of the head. For example, the user should be able to accurately perceive the object (e.g., a mosquito), even when the user moves his head without moving his lower body.

However, a system compatible with the current version of MPEG-H 3D Audio cannot currently handle this correctly. Instead, using a system compatible with the MPEG-H 3D Audio system results in the "mosquito" being perceived as being in an incorrect position relative to the user. In scenarios that involve 3DoF+ performance, small translational movements should result in a significant difference in the perception of the sound object (e.g. when moving the head to the left, the "mosquito" sound object should be perceived as being on the right side relative to the user's head, etc.).

The MPEG-H 3D Audio standard contains a bitstream syntax that enables the transmission of object distance information via bitstream syntax, for example via the object_metadata() syntax element (starting at 0.5m).

The prodMetadataConfig() syntax element can be inserted into the bitstream provided by the MPEG-H 3D Audio standard, which can be used to signal that an object is at very close distances to the listener. For example, the prodMetadataConfig() syntax can signal that the distance between the user and the object is less than a specific distance threshold (e.g. less than 1cm).

Fig. 1 and Fig. 2 show the present invention based on headphone rendering (i.e., when the speakers move with the listener's head).

Fig. 1 shows an example of behavior 100 of a system that is compatible with the MPEG-H 3D Audio system. In this example, it is assumed that the listener's head is located at position P ₀ 103 at time t ₀ and moves to position P ₁ 104 at time t ₁ > t ₀ . The dotted circles around positions P0 and P1 indicate the allowed 3DoF+ motion region (e.g., with a radius of 0.5 m). Position A 101 indicates the reported position of the object (at time t ₀ and time t ₁ , i.e., it is assumed that the reported position of the object is constant over some time). Position A also indicates the position of the object rendered by the MPEG-H 3D Audio renderer at time t ₀ . Position B 102 indicates the position of the object rendered by MPEG-H 3D Audio at time t ₁ . The vertical lines extending upward from positions P ₀ and P ₁ indicate the corresponding orientations (e.g., viewing directions) of the listener's head at times t ₀ and t ₁ . The offset of the user's head between position P ₀ and position P ₁ can be represented by r _offset =||P ₀ -P ₁ || 106. If the listener is in the default position (nominal listening position) P ₀ 103 at time t ₀ , he will perceive the sound object (e.g., a mosquito) at the correct position A 101. If the user moves to position P ₁ 104 at time t ₁ , he will perceive the sound object at position B 102 if MPEG-H 3D Audio processing is applied as the current standard, which introduces the shown error δ _AB 105. That is, despite the movement of the listener's head, the sound object (e.g., a mosquito) will still be perceived as being located directly in front of the listener's head (i.e., as essentially moving with the listener's head). It should be noted that the introduced error δ _AB 105 occurs regardless of the orientation of the listener's head.

Fig. 2 shows an example of the system behavior with respect to the MPEG-H 3D Audio system 200 according to the present invention. In Fig. 2, the listener's head is at position P ₀ 203 at time t ₀ and moves to position P ₁ 204 at time t ₁ > t ₀ . Again, the dotted circles around positions P ₀ and P ₁ indicate the allowed 3DoF+ motion region (e.g. with a radius of 0.5 m). In point 201, it is shown that position A=B, which means that the transmitted position of the object (at time t ₀ and time t ₁ , i.e. it is assumed that the transmitted position of the object) is constant for some time. Position A=B 201 also indicates the position of the object rendered by MPEG-H 3D Audio at time t ₀ and time t ₁ . The vertical arrows extending upward from the positions P ₀ 203 and P ₁ 204 indicate the corresponding orientations (e.g., viewing directions) of the listener's head at the times t ₀ and t ₁ . If the listener is at the initial/default position (nominal listening position) P ₀ 203 at the time t ₀ , he will perceive the sound object (e.g., a mosquito) at the correct position A 201. If the user moves to the position P ₁ 203 at the time t ₁ , he will still perceive the sound object at the position B 201, which is similar to (e.g., substantially coincides with) the position A 201 in accordance with the present invention. Thus, the present invention provides for a change in the user's position over time (e.g., from position P ₀ 203 to position P ₁ 204), while at the same time providing for the perception of sound from the same (spatially fixed) location (e.g., position A=B 201, etc.). In other words, the sound object (e.g., a mosquito) moves relative to the listener's head in accordance with the movement of (e.g., negatively correlating with) the listener's head. This allows the user to move around the sound object (e.g., a mosquito) and perceive the sound object from different angles or even from different sides. The displacement of the user's head between position P ₀ and position P ₁ can be represented as r _offset = || P ₀ - P ₁ || 206.

Fig. 3 shows an example of an audio rendering system 300 according to the present invention. The audio rendering system 300 may correspond to or comprise a decoder, such as, for example, an MPEG-H 3D audio decoder. The audio rendering system 300 may comprise a scene shifting unit 310 with a corresponding audio scene shifting processing interface (for example, an interface for scene shifting data in accordance with the MPEG-H 3D Audio standard). The scene shifting unit 310 may output object positions 321 for rendering the corresponding audio objects. For example, the scene shifting unit may output object position metadata for rendering the corresponding audio objects.

The audio rendering system 300 may further comprise an audio object rendering module 320. For example, the rendering module may consist of hardware, software and/or any partial or complete processing performed by means of cloud computing, including various services such as software development platforms, servers, storages and software, via the Internet, often referred to as a "cloud", which are compatible with the description established by the MPEG-H 3D Audio standard. The audio object rendering module 320 may render audio objects for one or more (real or virtual) speakers in accordance with corresponding object positions (these object positions may be modified or further modified object positions described below). The audio object rendering module 320 may render audio objects for headphones and/or loudspeakers. That is, the audio object rendering module 320 may generate object waveforms according to a specified playback format. For this purpose, the audio object rendering module 320 can use compressed object metadata. Each object can be rendered for specific output channels according to its object position (e.g., a modified object position or a further modified object position). Thus, the object positions can also be referred to as channel positions of their audio objects. The audio object positions 321 can be included in the output information of the object position metadata or the scene offset metadata by the scene offset unit 310.

The processing according to the present invention may be compatible with the MPEG-H 3D Audio standard. Thus, it may be performed by means of an MPEG-H 3D Audio decoder, or more specifically by means of an MPEG-H scene shifter and/or an MPEG-H 3D Audio rendering module. Accordingly, the audio rendering system 300 of Fig. 3 may correspond to or comprise an MPEG-H 3D Audio decoder (i.e., a decoder compatible with the description established by the MPEG-H 3D Audio standard). In one example, the audio rendering system 300 may be an apparatus comprising a processor and a memory device coupled to the processor, wherein the processor is adapted to implement an MPEG-H 3D Audio decoder. In particular, the processor may be adapted to implement an MPEG-H scene shifter and/or an MPEG-H 3D Audio rendering module. Thus, the processor may be adapted to perform the processing steps described in the present invention (for example, steps S510-S560 of the method 500 described with reference to Fig. 5 ). In another example, the processing system or audio rendering system 300 may be executed in the cloud.

The audio rendering system 300 can obtain (for example, receive) listening location data 301. The audio rendering system 300 can obtain listening location data 301 via an input interface of an MPEG-H 3D Audio decoder.

The listening location data 301 may indicate the orientation and/or position (e.g., offset) of the listener's head. Thus, the listening location data 301 (which may also be referred to as position information) may comprise information about the orientation of the listener and/or information about the offset of the listener.

The listener offset information may indicate a displacement of the listener's head (e.g., from the nominal listening position). The listener offset information may correspond to or comprise an indication of the amplitude of the displacement of the listener's head from the nominal listening position, r _offset = || P ₀ - P ₁ || 206, as shown in Fig. 2. In the context of the present invention, the listener offset information indicates a small displacement of the listener's head position from the nominal listening position. For example, the absolute value of the displacement may be no more than 0.5 m. Typically, this represents a displacement of the listener's head from the nominal listening position that can be achieved by moving the listener's upper body and/or head. That is, the listener can perform the displacement without moving the lower body. For example, the listener's head can be displaced while the listener is sitting on a chair, as indicated above. The displacement can be expressed by means of various coordinate systems, such as, for example, Cartesian coordinates (e.g., in terms of x, y, z) or spherical coordinates (e.g., in terms of azimuth, elevation angle, radius). Alternative coordinate systems for expressing the displacement of the listener's head are also possible and are understood to be encompassed by the present invention.

The listener orientation information may indicate the orientation of the listener's head (e.g., the orientation of the listener's head relative to the nominal orientation/original orientation of the listener's head). For example, the listener orientation information may contain information about the yaw, pitch, and roll of the listener's head. In this document, the yaw, pitch, and roll may be specified relative to the nominal orientation.

The listening location data 301 may be continuously collected from a receiver, which may provide information regarding the user's translational movements. For example, the listening location data 301, which is used in a particular case in time, may have been recently collected from the receiver. The listening location data may be obtained/collected/generated based on information from sensors. For example, the listening location data 301 may be obtained/collected/generated by means of wearable and/or stationary equipment having suitable sensors. That is, the orientation of the listener's head may be detected by means of wearable and/or stationary equipment. Similarly, the displacement of the listener's head (e.g., from the nominal listening position) may be detected by means of wearable and/or stationary equipment. The wearable equipment may be, correspond to, and/or include, for example, a headset (e.g., an AR/VR headset). The stationary equipment may be, correspond to, and/or include, for example, camera sensors. The stationary equipment may be built into, for example, a television or a set-top box. In some embodiments, the listening location data 301 may be received from an audio encoder (for example, an MPEG-H 3D Audio-compatible encoder), which may have received (for example, received) information from the sensors.

In one example, wearable and/or stationary equipment for detecting listening location data 301 can be called tracking devices that support head position estimation/detection and/or head orientation estimation/detection. There are various solutions that allow precise tracking of user's head movements using computer or smartphone cameras (e.g., based on face recognition and tracking "FaceTrackNoIR", "opentrack"). Also, several virtual reality systems for head-mounted display (HMD) (e.g., HTC VIVE, Oculus Rift) have built-in user's head position tracking technology. Any of these solutions can be used in the context of the present invention.

It is also important to note that the distance of head displacement in real conditions does not have to correspond one-to-one to the displacement indicated by the listening location data 301. In order to achieve a hyper-realistic effect (e.g., an over-enhanced parallax effect of user motion), specific applications may use different sensor calibration settings or specify different mappings between motion in real and virtual spaces. Therefore, it can be expected that a small physical motion will result in a larger displacement in virtual reality in some use cases. In any case, it can be said that the displacement amplitudes in real conditions and in virtual reality (i.e., the displacement indicated by the listening location data 301) are positively correlated. Likewise, the displacement directions in real conditions and in virtual reality are positively correlated.

The audio rendering system 300 may further receive (object) position information 302 (e.g., object position data) and audio data 322. The audio data 322 may comprise one or more audio objects. The position information 302 may be part of the metadata for the audio data 322. The position information 302 may indicate corresponding object positions for the one or more audio objects. For example, the position information 302 may comprise an indication of the distance to the corresponding audio objects relative to the nominal listening position of the user/listener. The distance (radius) may be less than 0.5 m. For example, the distance may be less than 1 cm. If the position information 302 does not comprise an indication of the distance to a given audio object from the nominal listening position, the audio rendering system may set the distance from this audio object to the nominal listening position to a default value (e.g., 1 m). The position information 302 may further comprise indications of an elevation angle and/or azimuth of the corresponding audio objects.

Each position of the object can be used to render a corresponding audio object. Accordingly, the position information 302 and the audio data 322 can be contained in or form the audio content based on objects. The audio content (e.g., the audio objects/audio data 322 together with the position information 302) can be transmitted in an encoded audio bitstream. For example, the audio content can be in the format of a bitstream received via transmission over a network. In this case, as defined, the audio rendering system can receive the audio content (e.g., from the encoded audio bitstream).

In one example of the present invention, metadata parameters can be used to correct the processing of use cases with a backward-compatible enhancement for 3DoF and 3DoF+. The metadata can contain information about the offset of the listener in addition to information about the orientation of the listener. Such metadata parameters can be used by the systems shown in Figs. 2 and 3, as well as any other embodiments of the present invention.

A backward compatible enhancement may provide for an adjustment to the processing of use cases (e.g., implementations of the present invention) based on the normative MPEG-H 3D Audio scene displacement interface. This means that a legacy MPEG-H 3D Audio decoder/renderer will still provide output, albeit incorrectly. However, an enhanced MPEG-H 3D Audio decoder/renderer according to the present invention will correctly apply the extension data (e.g., extension metadata) and processing, and can therefore manage the scenario of objects located near the listener in a correct manner.

In one example, the present invention relates to providing data for small translational movements of the user's head in formats other than those noted below, and the formulas can be adapted accordingly. For example, the data can be provided in a format such as x, y, z coordinates (in a Cartesian coordinate system) instead of azimuth, elevation angle and radius (in a spherical coordinate system). An example of these coordinate systems relative to each other is shown in Fig. 4 .

In one example, the present invention relates to providing metadata (for example, information about the displacement of the listener is included in the listening location data 301 shown in Fig. 3 ) for inputting the translational movement of the listener's head. The metadata can be used, for example, for an interface for scene displacement data. The metadata (for example, information about the displacement of the listener) can be obtained by using a tracking device that supports 3DoF+ or 6DoF tracking.

In one example, metadata (e.g., information about the listener's displacement, in particular the listener's head displacement, or equivalently, the scene displacement) may be represented by the following three parameters: sd_azimuth , sd_elevation , and sd_radius , which refer to the azimuth, elevation angle, and radius (spherical coordinates) of the listener's head displacement (or scene displacement).

The syntax for these parameters is presented in the table below.

Table 264b - mpegh3daPositionalSceneDisplacementData() syntax

sd_azimuth This field specifies the azimuth position of the scene offset. This field can have values from -180 to 180.

αz _offset = (sd_azimuth - 128) ⋅ 1.5

αz _offset = min(max(αz _offset , -180), 180)

sd_elevation This field represents the elevation angle position of the scene offset. This field can have values from -90 to 90.

el _offset = (sd_elevation - 32) · 3.0

el _offset = min(max(el _offset , -90), 90)

sd_radius This field specifies the radius of the scene offset. This field can have values from 0.015626 to 0.25.

r _offset = (sd_radius + 1) / 16

In another example, metadata (for example, listener offset information) can be represented by the following three parameters sd_x , sd_y and sd_z in Cartesian coordinates, thereby reducing the processing of data from spherical coordinates to Cartesian coordinates. The metadata can be based on the following syntax:

As described above, the above syntax or equivalents of this syntax can signal information related to rotations around the x, y, z axis.

In one example of the present invention, the processing of scene offset angles for channels and objects can be improved by expanding the equations that take into account changes in the user's head position. That is, when processing object positions, information about the listener's offset can be taken into account (e.g., can be based, at least in part).

An example of a method 500 for processing position information indicating the position of an object for an audio object is shown in the block diagram of Fig. 5 . This method can be performed by a decoder, such as an MPEG-H 3D audio decoder. The audio rendering system 300 of Fig. 3 can serve as an example of such a decoder.

In a first step (not shown in Fig. 5 ), audio content is received, including an audio object and corresponding position information, for example from an encoded audio bitstream. The method may then further include decoding the encoded audio content to obtain the audio object and position information.

In step S510, information about the orientation of the listener is obtained (e.g. received). The information about the orientation of the listener may indicate the orientation of the head of the listener.

In step S520, information about the displacement of the listener is obtained (for example, received). The information about the displacement of the listener may indicate the displacement of the head of the listener.

In step S530 , the position of the object is determined from the position information. For example, the position of the object (e.g., within the azimuth, elevation angle, radius, or x, y, z or their equivalents) can be extracted from the position information. The determination of the position of the object can also be based, at least in part, on information regarding the geometry of the speaker arrangement of one or more (real or virtual) speakers in the listening environment. If the radius is not included in the position information for this audio object, the decoder can set a default radius value (e.g., 1 m). In some embodiments, the default value can depend on the geometry of the speaker arrangement.

It should be noted that steps S510, S520 and S520 may be performed in any order.

In step S540 , the position of the object determined in step S530 is modified based on the information about the displacement of the listener. This may be performed by applying a translation to the position of the object in accordance with the information about the displacement (for example, in accordance with the displacement of the head of the listener). Thus, as determined, the modification of the position of the object may be related to an adjustment of the position of the object for the displacement of the head of the listener (for example, a displacement from the nominal listening position). In particular, the modification of the position of the object based on the information about the displacement of the listener may be performed by a translation of the position of the object using a vector that is positively correlated with the amplitude and negatively correlated with the direction of the vector of the displacement of the head of the listener from the nominal listening position. An example of such a translation is schematically illustrated in Fig. 2 .

In step S550, the modified position of the object obtained in step S540 is further modified based on the orientation information of the listener. For example, this may be done by applying a rotational transformation to the modified position of the object in accordance with the orientation information of the listener. This rotation may be, for example, a rotation relative to the head of the listener or a nominal listening position. The rotational transformation may be performed by means of a scene shift algorithm.

As noted above, user offset compensation (i.e., modification of the object position based on the listener's offset information) is taken into account when applying a rotation transform. For example, applying a rotation transform may involve:

• calculation of the rotation transformation matrix (based on the user orientation, such as listener orientation information);

• converting the position of an object from spherical to Cartesian coordinates;

• applying a rotational transform to audio objects that compensates for the user's position deviation (i.e., to the modified position of the object); and

• converting the position of an object after a rotational transformation back from Cartesian to spherical coordinates.

In the next step S560 (not shown in Fig. 5 ), the method 500 may include rendering the audio object on one or more real or virtual speakers in accordance with the further modified position of the object. For this purpose, the further modified position of the object may be adjusted for the input format used by the MPEG-H 3D Audio rendering module (for example, the audio object rendering module 320 described above). The aforementioned one or more (real or virtual) speakers may be, for example, part of a headset or may be part of a speaker composition (for example, a 2.1 speaker composition, a 5.1 speaker composition, a 7.1 speaker composition, etc.). In some embodiments, the audio object may be rendered, for example, for the left and right speakers of the headset.

The purpose of the steps S540 and S550 described above is as follows. Namely, the modification of the position of the object and the further modification of the modified position of the object are performed in such a way that the sound object, after rendering on one or more (real or virtual) speakers in accordance with the further modified position of the object, is psychoacoustically perceived by the listener as arising from a fixed position relative to the nominal listening position. This fixed position of the sound object should be psychoacoustically perceived regardless of the displacement of the listener's head from the nominal listening position and regardless of the orientation of the listener's head relative to the nominal orientation. In other words, the sound object may be perceived as moving (translating) relative to the listener's head when the listener's head performs a displacement from the nominal listening position. Similarly, the sound object may be perceived as moving (rotating) relative to the listener's head when the listener's head performs an orientation change from the nominal orientation. In this way, the listener can perceive a close sound object at different angles and from different distances by means of his head movement.

The modification of the object position and the further modification of the modified object position in steps S540 and S550, respectively, may be performed in the context of a (rotational/translational) sound scene shift, for example by means of the sound scene shift unit 310 described above.

It should be noted that specific steps may be omitted depending on a given specific use case. For example, if the listening location data 301 contains only listener offset information (but does not contain listener orientation information, or only listener orientation information indicating that there is no deviation in the listener's head orientation from the nominal orientation), step S550 may be omitted. Then, rendering in step S560 will be performed in accordance with the modified object position determined in step S540. Similarly, if the listening location data 301 contains only listener orientation information (but does not contain listener offset information, or only listener offset information indicating that there is no deviation in the listener's head position from the nominal listening position), step S540 may be omitted. Then, step S550 will relate to modifying the object position determined in step S530 based on the listener orientation information. Rendering in step S560 will be performed in accordance with the modified position determined in step S550.

In general, the present invention proposes a position update for object positions obtained as part of object-based audio content (e.g. position information 302 together with audio data 322) based on listening location data 301 for a listener.

First, the object position (or channel position) p = (az,el,r) is determined. This may be performed in the context of (e.g., as part of) step 530 of method 500.

For channel-based signals, the radius r can be determined as follows:

- if the intended loudspeaker (channel for a channel-based input signal) exists in the playback loudspeaker layout and the distance to the playback layout is known, the radius r is set to the distance to the loudspeaker (e.g. in cm);

- if the intended loudspeaker does not exist in the playback loudspeaker arrangement, but the distance to the playback loudspeakers (e.g. from the nominal listening position) is known, the radius r is set to the maximum distance to the playback loudspeaker;

- if the intended loudspeaker does not exist in the playback loudspeaker layout and the distance to the playback loudspeaker is not known, the radius r is set to a default value (e.g. 1023cm).

For object-based signals, the radius r is determined as follows:

- if the distance to the object is known (e.g. from production facilities and from production formats and passed to prodMetadataConfig()), the radius r is set to the known distance to the object (e.g. passed via goa_bsObjectDistance[] (in cm) according to table AMD5.7 of the MPEG-H 3D Audio standard);

Table AMD5.7 - goa_Production_Metadata() syntax

- if the distance to the object is known from the position information (e.g. from the object metadata and passed to object_metadata()), the radius r is set to the distance to the object passed in the position information (e.g. to the radius[] (in cm) passed with the object metadata). The radius r can be passed according to the sections: "scaling object metadata" and "constraining object metadata" below.

Scaling object metadata

As an optional step in the context of object pose determination, the object pose p = (az,el,r) determined from the pose information can be scaled. This may involve applying a scaling factor to unscale the input encoder for each component. This can be done for each object. The actual scaling of the object pose can be implemented along with the pseudocode presented below:

descale_multidata()

{

for (o = 0; o < num_objects; o++)

azimuth[o] = azimuth[o] * 1.5;

for (o = 0; o < num_objects; o++)

elevation[o] = elevation[o] * 3.0;

for (o = 0; o < num_objects; o++)

radius[o] = pow(2.0, (radius[o] / 3.0)) / 2.0;

for (o = 0; o < num_objects; o++)

gain[o] = pow(10.0, (gain[o] - 32.0) / 40.0);

if (uniform_spread == 1)

{

for (o = 0; o < num_objects; o++)

spread[o] = spread[o] * 1.5;

}

else

{

for (o = 0; o < num_objects; o++)

spread_width[o] = spread_width[o] * 1.5;

for (o = 0; o < num_objects; o++)

spread_height[o] = spread_height[o] * 3.0;

for (o = 0; o < num_objects; o++)

spread_depth[o] = (pow(2.0, (spread_depth[o] / 3.0)) / 2.0) - 0.5;

}

for (o = 0; o < num_objects; o++)

dynamic_object_priority[o] = dynamic_object_priority[o];

}

Object Metadata Restriction

As an optional step in the context of determining the object pose (possibly scaled), the object pose p = (az,el,r) determined from the pose information may be constrained. This may involve applying a constraint to the decoded values for each component to keep the values within an acceptable range. This may be done for each object. The actual constraint on the object pose may be implemented according to the functionality of the pseudocode presented below:

limit_range()

{

minval = -180;

maxval = 180;

for (o = 0; o < num_objects; o++)

azimuth[o] = MIN(MAX(azimuth[o], minval), maxval);

minval = -90;

maxval = 90;

for (o = 0; o < num_objects; o++)

elevation[o] = MIN(MAX(elevation[o], minval), maxval);

minval = 0.5;

maxval = 16;

for (o = 0; o < num_objects; o++)

radius[o] = MIN(MAX(radius[o], minval), maxval);

minval = 0.004;

maxval = 5,957;

for (o = 0; o < num_objects; o++)

gain[o] = MIN(MAX(gain[o], minval), maxval);

if (uniform_spread == 1)

{

minval = 0;

maxval = 180;

for (o = 0; o < num_objects; o++)

spread[o] = MIN(MAX(spread[o], minval), maxval);

}

else

{

minval = 0;

maxval = 180;

for (o = 0; o < num_objects; o++)

spread_width[o] = MIN(MAX(spread_width[o], minval), maxval);

minval = 0;

maxval = 90;

for (o = 0; o < num_objects; o++)

spread_height[o] = MIN(MAX(spread_height[o], minval), maxval);

minval = 0;

maxval = 15.5;

for (o = 0; o < num_objects; o++)

spread_depth[o] = MIN(MAX(spread_depth[o], minval), maxval);

}

minval = 0;

maxval = 7;

for (o = 0; o < num_objects; o++)

dynamic_object_priority[o] = MIN(MAX(dynamic_object_priority[o], minval),

maxval);

}

After this, the defined (and optionally scaled and/or constrained) position of the object p = (az,el,r) can be converted to a given coordinate system, such as, for example, a coordinate system according to the "usual convention" in which azimuth 0° is at the right ear (positive values go counterclockwise) and elevation 0° is at the top of the head (positive values go downward). Thus, the position of the object p can be converted to a position p ' according to the "usual" convention. This gives a position of the object p ' with

p' = (az ' ,el ' ,r)

a' ' = az + 90 °

el' = 90 ° - el

with constant radius r .

At the same time, the listener's head offset, specified by the listener offset information ( az _offset , el _offset , r _offset ), can be converted to a given coordinate system. Using the "usual convention", this amounts to

az ' _offset = az _offset + 90°

el ' _offset = 90 ° - el _offset

with constant radius r _offset .

It should be noted that the conversion to a given coordinate system for both the object position and the listener's head offset may be performed in the context of step S530 or step S540.

Updating the actual position may be performed in the context of (e.g., as part of) step S540 of the method 500. Updating the position may include the following steps.

As a first step, the position of p , or if a translation into a given coordinate system has been performed, the position of p ', is translated into Cartesian coordinates (x, y, z). In the following, without implied constraint, the process will be described for the position of p ' in the given coordinate system. Also without implied constraint, the following orientation/direction of the coordinate axes can be assumed: the x-axis points to the right (shown from the listener's head in the nominal orientation), the y-axis points straight ahead, and the z-axis points straight up. At the same time, the offset of the listener's head, specified by the listener offset information ( az ' _offset , el ' _offset , r _offset ), is converted into Cartesian coordinates.

As a second step, the object's position in Cartesian coordinates is shifted (translated) in accordance with the listener's head displacement (scene displacement) in the manner described above. This can be accomplished by

x=r⋅ sin(el') cos(az') +r _offset ⋅ sin(el' _offset ) ⋅ cos(az' _offset )

y=r⋅ sin(el') sin(az') +r _offset ⋅ sin(el/ _offset ) ⋅ sin(az' _offset )

z=r⋅ cos(el') +r _offset ⋅ cos(el' _offset )

The above translational movement is an example of modifying the position of an object based on the listener displacement information in step S540 of the method 500.

The shifted position of an object in Cartesian coordinates is converted to spherical coordinates and may be called p ''. The shifted position of an object may be expressed in a given coordinate system according to the usual convention as p '' = (az '' ,el '' ,r ' ) .

When there are listener head displacements that provide a small change in the radius parameter (i.e. r ' ≈ r ), the modified object position p '' can be redefined as p '' = (az '' ,el '' ,r) .

In another example, when there are large listener head displacements that can provide a significant change in the radius parameter (i.e. r '>> r ), the modified object position p '' can also be defined as p '' = (az '' ,el '' ,r ' ) instead of p '' = (az '' ,el '' ,r) with a modified radius parameter r '.

The corresponding value of the modified radius parameter r ' can be obtained from the listener's head offset distance (i.e. r _offset =||P ₀ -P ₁ ||) and the initial radius parameter (i.e. r=||P ₀ -A||), (see, for example, Figs. 1 and 2). For example, the modified radius parameter r ' can be determined based on the following trigonometric relationship:

Mapping this modified radius parameter r ' to object/channel gains and applying them for subsequent audio rendering can significantly improve the effects of perceiving level changes due to user movements. By providing such a modification of the radius parameter r ', an "adaptive sweet spot" is provided. This would mean that the MPEG rendering system dynamically adjusts the position of the sweet spot according to the current location of the listener. In general, the rendering of an audio object according to a modified (or further modified) object position can be based on the modified radius parameter r '. In particular, the object/channel gains for rendering the audio object can be based on (e.g. modified based on) the modified radius parameter r '.

In another example, during setup and rendering of the playback speaker (e.g., in step S560 above) , scene shifting may be disabled. However, an optional enabling of scene shifting is available. This ensures that the 3DoF+ renderer creates a dynamically adjusted sweet spot according to the current location and orientation of the listener.

It should be noted that the stage of converting the object position and the listener's head displacement into Cartesian coordinates is optional, and the translational movement/shift (modification) in accordance with the listener's head displacement (scene displacement) can be performed in any suitable coordinate system. In other words, the choice of Cartesian coordinates in the above description should be understood as a non-limiting example.

In some embodiments, the processing of scene displacement (including modification of object position and/or further modification of the modified object position) may be enabled or disabled by a flag (field, element, setting bit) in the bitstream (e.g., useTrackingMode element). Subclauses "17.3 Interface for local loudspeaker setup and rendering" and "17.4 Interface for binaural room impulse responses (BRIRs)" in ISO/IEC 23008-3 contain descriptions of the useTrackingMode element activating the processing of scene displacement. In the context of the present invention, the useTrackingMode element shall determine (subclause 17.3) whether the processing of scene displacement values sent via the mpegh3daSceneDisplacementData() and mpegh3daPositionalSceneDisplacementData() interfaces shall or shall not occur. Alternatively or additionally (subclause 17.4), the useTrackingMode field shall specify whether a tracking device is connected and whether binaural rendering shall be processed in a special mode for tracking the user's head position, which means that processing of scene displacement values sent via the mpegh3daSceneDisplacementData() and mpegh3daPositionalSceneDisplacementData() interfaces shall be performed.

The methods and systems described in this document can be implemented as software, firmware and/or hardware. Some components can be implemented, for example, as software running on a digital signal processor or on a microprocessor. Other components can be implemented, for example, as hardware or as special-purpose integrated circuits. The signals that occur in the described methods and systems can be stored on media such as random access memory or optical storage media. They can be transmitted over networks such as radio networks, satellite networks, wireless networks or wired networks such as the Internet. Typical devices using the methods and systems described in this document are portable electronic devices or other consumer equipment that is used to store and/or generate audio signals.

Although reference is made in this document to MPEG and in particular MPEG-H 3D Audio, the present invention should not be construed as being limited to these standards. On the contrary, as will be apparent to those skilled in the art, the present invention may find advantageous application in other audio coding standards as well.

Furthermore, although reference is often made herein to small displacements of the listener's head position (e.g., from a nominal listening position), the present invention is not limited to small displacements of position and may generally be applied to arbitrary displacements of the listener's head position.

It should be noted that the description and drawings illustrate only the principles of the proposed methods, systems and apparatuses. Those skilled in the art will be able to implement various arrangements that, although not explicitly described or shown herein, implement the principles of the present invention and are included in its spirit and scope. Moreover, all examples and embodiments set forth herein are primarily expressly intended for explanatory purposes to assist the reader in understanding the principles of the proposed method. In addition, all statements in this document presenting principles, aspects and embodiments of the present invention, as well as specific examples thereof, are intended to cover their equivalents.

In addition to the above, various exemplary implementations and exemplary embodiments of the present invention will become apparent from the numbered embodiment examples (NEEs) listed below, which do not constitute claims.

The first PPVO relates to a method for decoding an encoded bitstream of an audio signal, wherein said method comprises: receiving, by an audio decoding apparatus 300, an encoded bitstream (302, 322) of an audio signal, wherein the encoded bitstream of the audio signal comprises encoded audio data (322) and metadata corresponding to at least one audio signal (302) of an object; decoding, by the audio decoding apparatus (300), the encoded bitstream (302, 322) of the audio signal to obtain a representation of several audio sources; receiving, by the audio decoding apparatus (300), listening location data (301); generating, by the audio decoding apparatus (300), audio object position data (321), wherein the audio object position data (321) describes several audio sources relative to the listening location based on the listening location data (301).

The second PPVO relates to the method according to the first PPVO, wherein the listening location data (301) are based on a first set of first translational position data and a second set of second translational position and orientation data.

The third PPVO relates to the method according to the second PPVO, wherein either the first translational position data or the second translational position data are based on at least one of a set of spherical coordinates or a set of Cartesian coordinates.

The fourth PPVO refers to the method according to the first PPVO, wherein the listening location data (301) is obtained via the input interface of the MPEG-H 3D Audio decoder.

The fifth PPVO relates to the method according to the first PPVO, wherein the encoded bitstream of the audio signal comprises MPEG-H 3D Audio bitstream syntax elements, and wherein the MPEG-H 3D Audio bitstream syntax elements comprise encoded audio data (322) and metadata corresponding to at least one audio signal (302) of the object.

The sixth PPVO relates to the method according to the first PPVO, which further includes rendering by the apparatus (300) of decoding sound on several loudspeakers of several sound sources, wherein the rendering process is compatible with at least the MPEG-H 3D Audio standard.

The seventh PPVO relates to the method according to the first PPVO, which further comprises converting by the sound decoding apparatus (300) based on the broadcast of the listening location data (301) of the position p corresponding to at least one sound signal (302) of the object, into a second position p '' corresponding to the positions (321) of the sound object.

The eighth PPVO refers to the method according to the seventh PPVO, where the position p’ for the positions of the sound object in a given coordinate system (e.g., according to the usual convention) is determined on the basis of:

p’=(az’,el’,r)

az ' = az + 90°

el ' = 90° - el

az ' _offset = az _offset + 90°

el ' _offset = 90° - el _offset

moreoverazcorresponds to the first parameter of azimuth,elcorresponds to the first parameter of the angle of ascent andrcorresponds to the first parameter of the radius, in this documentaz' corresponds to the second parameter of azimuth,el' corresponds to the second parameter of the angle of ascent andr' corresponds to the second parameter of the radius, andaz _offset corresponds to the third parameter of azimuth,el _offset corresponds to the third parameter of the angle of ascent, and at the same timeaz' _offset corresponds to the fourth parameter of azimuth,el' _offset corresponds to the fourth parameter of the angle of elevation.

The ninth PPVO refers to the method according to the eighth PPVO, wherein the shifted position p '' (321) of the sound object for the position (302) of the sound object is determined in Cartesian coordinates (x, y, z) based on:

x = r sin( el ') cos( az ') + x _offset

y = r sin( el ') sin( az ') + y _offset

z = r cos( el ') + z _offset

where the position in Cartesian coordinates (x, y, z) contains the parameters x, y and z, and where x _offset refers to the first parameter of the x-axis offset, y _offset refers to the first parameter of the y-axis offset and z _offset refers to the first parameter of the z-axis offset.

The tenth PPVO refers to the method according to the ninth PPVO, where the parameters x _offset , y _offset and z _offset are based on

x _offset =r _offset ∙sin(el' _offset ) cos(az' _offset )

y _{offset =} r _offset sin( el ' _offset ) sin( az ' _offset )

z _{offset =} r _offset ∙cos(el' _offset )

The eleventh PPVO refers to the method according to the seventh PPVO, while the az _offset parameter of the azimuth refers to the position of the azimuth offset of the scene and is based on:

az _offset = (sd_azimuth - 128) 1.5

az _offset = min(max( az _offset , -180), 180)

where sd_azimuth is a parameter indicating the MPEG-H 3DA scene azimuth offset, while the elevation angle parameter el _offset refers to the elevation angle position of the scene offset and is based on:

el _offset = (sd_elevation - 32) 3

el _offset = min(max( el _offset , -90), 90)

where sd_elevation is an elevation angle metadata parameter indicating the elevation angle offset of the MPEG-H 3DA scene, where the radius parameter r _offset refers to the scene offset radius and is based on:

r _offset = (sd_radius + 1) / 16

where sd_radius is a radius metadata parameter specifying the MPEG-H 3DA scene radius offset, and where X and Y parameters are scalar variables.

The twelfth PPVO refers to the method according to the tenth PPVO, wherein the parameter x _offset refers to the position sd_x of the scene offset deviation in the x-axis direction; the parameter y _offset refers to the position sd_y of the scene offset deviation in the y-axis direction; and the parameter z _offset refers to the position sd_z of the scene offset deviation in the z-axis direction.

The thirteenth PPVO relates to the method according to the first PPVO, which further includes interpolating, by the audio decoding apparatus, the first position data related to the listening location data (301) and the audio signal (102) of the object, at an update rate.

The fourteenth PPVO relates to the method according to the first PPVO, which further includes determining by the sound decoding apparatus 300 an effective entropy encoding of the data (301) of the listening location.

The fifteenth PPVO refers to the method according to the first PPVO, wherein the position data related to the listening location (301) are obtained based on information from sensors.

Claims (26)

1. A method for processing position information indicating the position of an object for an audio object, wherein the processing is performed using an MPEG-H 3D Audio decoder, wherein the position of the object is suitable for use in rendering the audio object, wherein the method comprises: obtaining listener orientation information indicating the orientation of the listener's head, wherein the listener orientation information comprises yaw, pitch, and roll information of the listener's head; obtaining listener offset information indicating the offset of the listener's head relative to the nominal listening position via the input interface of the MPEG-H 3D Audio decoder; determining the position of an object from position information; modifying the position of an object based on the listener's displacement information by applying translational movement to the position of the object; further modifying the modified object position based on the listener orientation information, wherein when the listener displacement information indicates a displacement of the listener's head from the nominal listening position due to a small position displacement, the small position displacement has an absolute value of 0.5 meters or less than 0.5 meters, the distance between the position of the sound object and the listening position after the listener's head displacement is equal to the distance between the modified position of the sound object and the nominal listening position; and rendering the sound object on one or more real or virtual speakers in accordance with the further modified position of the object. 2. The method according to paragraph 1, characterized in that: the modification of the position of the object and the additional modification of the modified position of the object are performed in such a way that the sound object, after rendering on one or more real or virtual speakers in accordance with the additional modified position of the object, is psychoacoustically perceived by the listener as arising from a fixed position relative to the nominal listening position, regardless of the displacement of the listener's head from the nominal listening position and the orientation of the listener's head relative to the nominal orientation. 3. The method according to paragraph 1, characterized in that: the modification of the object's position based on information about the listener's displacement is performed by means of a translational movement of the object's position equal to the displacement of the listener's head from the nominal listening position, but in the opposite direction. 4. The method according to paragraph 1, characterized in that: Listener displacement information indicates the displacement of the listener's head from the nominal listening position that can be achieved by movement of the listener's upper body and/or head. 5. The method according to paragraph 1, characterized in that it additionally includes: Detection of listener head orientation using wearable and/or fixed equipment. 6. The method according to paragraph 1, characterized in that it includes: detection of displacement of the listener's head from the nominal listening position using wearable and/or stationary equipment. 7. The method according to claim 1, characterized in that the distance between the modified position of the sound object and the listening position after the shift is converted into gain for modifying the level of the sound signal. 8. A non-volatile machine-readable medium containing commands that, when executed by a digital signal processing processor or microprocessor, cause the digital signal processing processor or microprocessor to execute the method according to paragraph 1. 9. An MPEG-H 3D Audio decoder for processing position information indicating the position of an object for an audio object, wherein the position of the object is usable for rendering the audio object, wherein the decoder comprises a processor and a memory device connected to the processor, wherein the processor is adapted to: obtaining listener orientation information indicating the orientation of the listener's head, wherein the listener orientation information comprises information about the yaw, pitch, and roll of the listener's head; obtaining listener offset information indicating the offset of the listener's head relative to the nominal listening position via the input interface of the MPEG-H 3D Audio decoder; determining the position of an object from position information; modifications of the object's position based on the listener's displacement information by applying translational movement to the object's position; further modifying the modified object position based on the listener orientation information, wherein when the listener displacement information indicates a displacement of the listener's head from the nominal listening position due to a small position displacement, the small position displacement has an absolute value of 0.5 meters or less than 0.5 meters, and the distance between the sound object position and the listening position after the listener's head displacement is equal to the distance between the modified sound object position and the nominal listening position; and rendering the sound object on one or more real or virtual speakers in accordance with the further modified position of the object.