CN115334366A - A modeling method for interactive immersive sound field roaming - Google Patents

Abstract

The invention provides a modeling method for interactive immersive sound field roaming, a sound field roaming method and a sound field roaming system. The sound field roaming method comprises the following steps: determining N first positions of N virtual musical instruments in a virtual sound field space and a second position of a virtual character in the virtual sound field space, wherein the virtual character is used for being operated by a user to stop or move in the virtual sound field space, and the N virtual musical instruments, the virtual sound field space and the virtual character are realized through virtual reality technology; determining relative position information between the N first positions and the second positions, wherein the N first positions are N virtual sound source positions, and the second positions are virtual listening positions; processing the N types of first audio signals by using a sound field space model according to the relative position information to obtain second audio signals; and responding to the playing operation of the user, and playing the second audio signal to the user.

Description

A modeling method for interactive immersive sound field roaming

technical field

The present invention relates to the field of performing arts science and technology, and more particularly relates to a modeling method of interactive immersive sound field roaming, a sound field roaming method and a sound field roaming system.

Background technique

Recreating the auditory effect of a real sound field space (such as a concert hall) in a virtual system is crucial to the experience of the audience and music listeners. For musicians in the performance industry, the development of rehearsal and performance simulation systems based on virtual platforms can help artists transfer the stage from offline to online, and help solve the current situation of difficulties in touring, brain drain, and difficulty in establishing cultural and artistic brands .

In the process of implementing the embodiments of the present invention, the inventor started from the field of music performance, combined with the needs of audiences, musicians, conductors and audio engineers, and found that there are still the following problems:

(1) For orchestra conductors and musicians, the position of the band parts in the prior art cannot be changed, and the real-time switching of the sound effect of the band parts position cannot be realized, which affects the efficiency of the performance evaluation of the orchestra.

(2) For the sound engineer, the existing technology cannot realize the real-time comparison and switching of the sound effects of different recording systems, nor can it realize the real-time control of the volume balance of different parts, and the music mixing efficiency is very low.

(3) For the auditory effect of the concert, the existing technology cannot simulate the real-time presentation of the acoustic effect at any location in the concert hall, nor can it simulate the sound field of different sound field spaces (such as different concert halls, different natural scenes and living environments). auditory effect.

Therefore, how to reproduce the acoustic effects of different sound field spaces is an urgent problem to be solved.

Contents of the invention

In view of the above problems, the present invention provides a modeling method for interactive immersive sound field roaming, a sound field roaming method and a sound field roaming system.

An aspect of the embodiments of the present invention provides an audible-based interactive immersive sound field roaming method, including: determining N first positions of N virtual musical instruments in the virtual sound field space, and determining the first positions of N virtual instruments in the virtual sound field A second position in space, wherein the avatar is used to be operated by the user to stop or move in the virtual sound field space; determining relative position information between the N first positions and the second position , wherein, the N first positions are N virtual sound source positions, the second position is a virtual listening position, and N is an integer greater than or equal to 1; according to the relative position information, the sound field space model is used to process N Types of first audio signals to obtain a second audio signal, wherein the sound field space model is used to simulate the propagation of the N types of first audio signals in physical space, the N types of first audio signals and the N types There is a one-to-one correspondence between the virtual musical instruments; in response to the user's playback operation, the second audio signal is played to the user.

According to an embodiment of the present invention, the sound field space model includes a direct sound processing model, an early reflection sound model, and a late reverberation sound model, and processing N types of first audio signals using the sound field space model to obtain a second audio signal includes: using The direct sound processing model performs attenuation processing on the N kinds of first audio signals to obtain a first output result; the first output result is input into the early reflection sound model for reflection processing to obtain a second output result; The first output result is input into the late reverberation sound model for reverberation processing to obtain a third output result; and the second audio signal is obtained according to the second output result and the third output result.

According to an embodiment of the present invention, the relative position information includes distance information, and the attenuation processing of the N types of first audio signals by using the direct sound processing model includes: using N distance attenuation curves according to the distance information to process the N types of first audio signals, wherein the N distance attenuation curves correspond to the N types of first audio signals one-to-one, and any two of the N distance attenuation curves are the same or different .

According to an embodiment of the present invention, the processing of the N types of first audio signals according to the distance information using N distance attenuation curves includes performing taper on at least one audio signal of the N types of first audio signals The attenuation processing specifically includes: for any one of the at least one audio signal, obtaining the propagation distance based on the internal space information of the virtual sound field space; using the virtual sound source position corresponding to the audio signal as the center of the sphere position, using the propagation distance as a radius to obtain the spherical propagation area of the audio signal; dividing the spherical propagation area into an inner corner region, an outer corner region, and a transition region between the inner corner region and the outer corner region; according to The actual area to which the second position belongs performs corresponding attenuation processing on the audio signal to obtain the first output result, wherein the actual area includes the inner corner area category, the outer corner area, and the transition any of the regions.

According to an embodiment of the present invention, M virtual sound sources are calculated according to the N virtual sound source positions and the geometry of the virtual sound field space; S number of virtual sound sources are calculated according to the second position and the geometry In the sound reflection path, M and S are respectively integers greater than or equal to 1; wherein, the input of the first output result into the early reflection sound model for reflection processing, and obtaining the second output result include: according to the M The virtual sound source and the S sound reflection paths perform reflection processing on the first output result to obtain the second output result.

According to an embodiment of the present invention, before obtaining the S sound reflection paths through the calculation, the method further includes: using the virtual person as a ray source, sending a virtual ray from the second position; detecting Auditory interaction information, wherein the auditory interaction information includes the distance between the virtual character and the wall in the virtual sound field space and the material information of the wall in the virtual sound field space.

According to an embodiment of the present invention, the late reverberation sound model includes K impulse response signals recorded from K physical environments, and the first output result is input into the late reverberation sound model for reverberation processing, Obtaining the third output result includes: invoking a first impulse response signal in response to the first virtual sound field space selected by the user from the K virtual sound field spaces, wherein the first virtual sound field space is based on the K virtual sound field spaces The first physical environment in the physical environment is constructed and obtained, K is an integer greater than or equal to 1; the first output result and the first impulse response signal are convoluted to obtain the third output result.

According to an embodiment of the present invention, the method further includes: moving the avatar to a third position in response to the user's first instruction to move the avatar; updating the virtual listening position to the first position Three positions: Re-perform the operations of determining the relative position information, obtaining the second audio signal, and playing the second audio signal to the user.

According to an embodiment of the present invention, the method further includes: moving the at least one virtual musical instrument to a fourth position in response to the user's second instruction to move the at least one virtual musical instrument; The corresponding position among the N virtual sound source positions is updated to the fourth position; and the operations of determining the relative position information, obtaining the second audio signal, and playing the second audio signal to the user are performed again.

Another aspect of the embodiments of the present invention provides an audible-based interactive immersive sound field roaming system, including: a position determination unit, configured to determine N first positions of N virtual instruments in the virtual sound field space, and The second position of the virtual character in the virtual sound field space, wherein the virtual character is used to be operated by the user to stop or move in the virtual sound field space; the relative position unit is used to determine the N first relative position information between the position and the second position, wherein the N first positions are N virtual sound source positions, the second position is a virtual listening position, and N is an integer greater than or equal to 1; A signal processing unit, configured to process N types of first audio signals using a sound field space model according to the relative position information to obtain a second audio signal, wherein the sound field space model is used to simulate the N types of first audio signals in Propagation in physical space, the N types of first audio signals correspond to the N types of virtual instruments one by one; an audio playback unit is configured to play the second audio to the user in response to the user's playback operation Signal.

Another aspect of the embodiments of the present invention provides a modeling method for interactive immersive sound field roaming, including: obtaining a direct sound processing model, and the direct sound processing model is used to attenuate N types of first audio signals to obtain The first output result, the N kinds of first audio signals are respectively propagated from N virtual sound source positions to the virtual listening position, N is an integer greater than or equal to 1; an early reflection model is obtained, and the early reflection model is used for Perform reflection processing on the first output result to obtain a second output result; obtain a late reverberation sound model, and the late reverberation sound model is used to perform reverberation processing on the first output result to obtain a third output result; set a main output bus, where the main output bus is used to obtain a second audio signal according to the second output result and the third output result, wherein the second audio signal is to simulate the N types of first audio signals in The resulting audio signal propagated in physical space.

Another aspect of the embodiments of the present invention provides an electronic device, including: one or more processors; a storage device for storing one or more programs, wherein, when the one or more programs are executed by the one When executed by one or more processors, one or more processors execute the method as described above.

Another aspect of the embodiments of the present invention also provides a computer-readable storage medium, on which executable instructions are stored, and when the instructions are executed by a processor, the processor executes the above-mentioned method.

One or more embodiments of the present invention can provide an audible virtual sound field space, and a virtual character that can be operated by the user. As the virtual character moves, it simulates the sound propagation phenomenon in the real environment to play audio for the user. The positions of the N types of virtual musical instruments and virtual characters are respectively determined and used as N virtual sound source positions and virtual listening positions. Then, relative position information between the N virtual sound source positions and the virtual listening position is determined. Next, the N kinds of first audio signals are processed by using the sound field space model to simulate the propagation effect of the first audio signals in N in the physical environment based on the relative position, to obtain the second audio signal. Finally the second audio signal is played to the user. In this way, the real-time interactive immersive sound field roaming function of performing arts performances is realized.

The above description is only an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the contents of the description, and in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable , the specific embodiments of the present invention are enumerated below.

Description of drawings

Through the following description of the embodiments of the present invention with reference to the accompanying drawings, the above content and other objects, features and advantages of the present invention will be more clear, in the accompanying drawings:

FIG. 1 schematically shows a flow chart of an audible-based interactive immersive sound field roaming method according to an embodiment of the present invention;

Fig. 2 schematically shows a flow chart of obtaining a second audio signal according to an embodiment of the present invention;

Fig. 3 schematically shows a flow chart of processing a first audio signal according to an embodiment of the present invention;

Fig. 4 schematically shows a flow chart of tapered attenuation processing according to an embodiment of the present invention;

Fig. 5 schematically shows a flow chart of reflection processing according to an embodiment of the present invention;

Fig. 6 schematically shows a flow chart of detecting auditory interaction information according to an embodiment of the present invention;

Fig. 7 schematically shows a flow chart of obtaining a third output result according to an embodiment of the present invention;

Fig. 8 schematically shows a flow chart of updating a virtual listening position according to an embodiment of the present invention;

Fig. 9 schematically shows a flow chart of updating the position of a virtual sound source according to an embodiment of the present invention;

Fig. 10 schematically shows a technical architecture diagram of a modeling method suitable for realizing interactive immersive sound field roaming according to an embodiment of the present invention;

Fig. 11 schematically shows a system development architecture diagram of a modeling method suitable for realizing interactive immersive sound field roaming according to an embodiment of the present invention;

Fig. 12 schematically shows a structural block diagram of an auralization-based interactive immersive sound field roaming system according to an embodiment of the present invention; and

Fig. 13 shows a schematic structural diagram of a computing device according to an embodiment of the present invention.

Detailed ways

Firstly, relevant terms involved in the embodiments of the present invention are described, so as to better understand the present invention.

Sonification: The technique of creating audible sound files from digital (analog, measured, or synthetic) data.

Interactive: the user can interact with the virtual object provided by the embodiment of the present invention through some operations, so as to provide the user with functions such as real-time sound field roaming, sound field switching, part position switching and audio processing.

Immersion: Simulate the real-time acoustic environment in real music performance, so that users can have an immersive listening experience in the sound field space, thus producing an immersive effect.

Sound field roaming: the process in which the virtual character moves in at least part of the virtual sound field space.

Reflected Sound: Sound waves in a room from ceilings and walls that contribute to higher sound pressure levels.

Direct sound: Refers to the sound that is transmitted directly from the sound source to the receiver in a straight line without any reflection.

Early reflections: also known as initial reflections. That portion of reflected sound that arrives immediately after the direct sound and is sonically beneficial.

Reverberation sound: the superposition of all primary and multiple reflections at the same time when the sound in the room reaches a steady state or the sound source continues to sound.

Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the invention may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

Embodiments of the present invention provide a modeling method for interactive immersive sound field roaming, a sound field roaming method, and a sound field roaming system. Taking the virtual concert hall as an example, it can start from the field of music performance and combine the needs of the audience, musicians, conductors and audio engineers. Based on the idea of providing a multi-functional, customizable, interactive and immersive concert hall, through The simulation algorithm of geometric acoustics and multi-engine cross-platform collaborative operation solve the technical problems of real-time simulation and simulation of the acoustic effect of the concert sound field. For orchestra conductors and musicians, it solves the technical problem that the sound effects of the band parts cannot be switched in real time. It can be used as a virtual rehearsal hall to simulate music performances of different arrangements and genres, improving the efficiency of orchestra performance evaluation. For audio engineers (such as sound engineers), it solves the technical problem of real-time comparison and switching of sound effects in different recording positions, realizes real-time control of the volume balance of different parts, and improves the work efficiency of sound engineers. For the auditory effect of the concert, it solves the technical problem of simulating the acoustic effect of any position in the concert hall in real time, and can simulate the auditory effect of different sound field spaces in real time.

Fig. 1 schematically shows a flow chart of an audible-based interactive immersive sound field roaming method according to an embodiment of the present invention.

As shown in Fig. 1 , the auralization-based interactive immersive sound field roaming method of this embodiment includes operation S110 to operation S140.

In operation S110, determine N first positions of N types of virtual instruments in the virtual sound field space, and second positions of the virtual character in the virtual sound field space, wherein the virtual character is used to be operated by the user to stop in the virtual sound field space or move.

Exemplarily, things in the real world are simulated in the virtual digital space to establish a realistic, virtual and interactive three-dimensional space environment, such as N kinds of virtual instrument models, virtual sound field space models and virtual character models . The avatar can roam in the virtual sound field space (move arbitrarily in at least part of the area). In other words, the virtual sound field space is a virtual three-dimensional space, which may include real three-dimensional space environment information.

In some embodiments, the system scene model is constructed with the Chinese National Orchestra (just an example) object and the concert hall scene, including 1 concert hall model, 1 scene roaming character model and 11 Chinese National Orchestra instrument models, musical instruments The models include Pipa, Daruan, Zhongruan, Sanxian and Yangqin in the plucked string instruments group, Bangdi, Nanxiao and Sheng in the wind instruments group, Erhu and Zhonghu in the stringed instruments group and chime bells in the percussion instrument group. The character roaming and musical instrument movement function scripts are written through the programming language C# supported by Unity. The material source is converted into an fbx file format and imported into the Unity system scene, and the material map is assigned to the physical model after importing.

For example, since the concert hall scene does not have conspicuous lamps and lanterns, but hides the light source in the building to achieve a warm and blended lighting effect, light map baking is used as the main method of scene lighting rendering. The system scene uses parallel light as the main light source, and at the same time uses dozens of simulated light types including point light source and area light to supplement the lighting of the stage area of the concert hall and the darkness of the auditorium area. In addition, the lighting network composed of multiple light probes is also applied to the system scene, and dynamically illuminates multiple moving objects in the concert hall, including characters and musical instrument models.

In operation S120, determine relative position information between N first positions and second positions, wherein, the N first positions are N virtual sound source positions, the second position is a virtual listening position, and N is greater than or equal to 1 an integer of .

Exemplarily, N first positions and second positions are mapped to position information in real (physical) space, so that N virtual sound source positions and virtual listening positions are mapped to real positions to simulate the propagation of audio signals, relative positions The information can reflect the relative position between the user and the instrument in real space.

In operation S130, according to the relative position information, use the sound field space model to process N types of first audio signals to obtain second audio signals, wherein the sound field space model is used to simulate the propagation of N types of first audio signals in physical space, N types The first audio signal is in one-to-one correspondence with the N types of virtual musical instruments.

Exemplarily, the sound field space model may have an immersive space sound field simulation framework based on the physical propagation principle of sound, and perform signal processing from three perspectives of sound emission, propagation path, and receiver. For example, the 11 kinds of first audio signals are audio files corresponding to 11 musical instrument models of the Chinese National Orchestra, and the audio files may be all audio signals of actual musical instrument performances in a piece of music recorded in advance.

In operation S140, the second audio signal is played to the user in response to the user's play operation.

Exemplarily, when the user performs the playback operation is not limited in the present invention, it may be before, during or after the execution of other operations other than operation S140, for example, the play button may be clicked in operation S110, or the play button may be clicked in operation S110 Before hitting the play button.

According to the embodiment of the present invention, an audible virtual sound field space can be provided, and users can interact to operate virtual characters and play audio, and simulate sound propagation phenomena in real environments (such as concert halls) to play audio for users. The positions of the N types of virtual musical instruments and virtual characters are respectively determined and used as N virtual sound source positions and virtual listening positions. Then, relative position information between the N virtual sound source positions and the virtual listening position is determined. Next, the N kinds of first audio signals are processed by using the sound field space model to simulate the propagation effect of the first audio signals in N in the physical environment based on the relative position, to obtain the second audio signal. Finally the second audio signal is played to the user. In this way, the real-time interactive immersive sound field roaming function of performing arts performances is realized.

Fig. 2 schematically shows a flow chart of obtaining a second audio signal according to an embodiment of the present invention. Fig. 3 schematically shows a flow chart of processing a first audio signal according to an embodiment of the present invention.

As shown in FIG. 2 , in operation S130 , using the sound field space model to process N types of first audio signals, and obtaining second audio signals includes operations S210 to S240 . Among them, the sound field space model includes a direct sound processing model, an early reflection sound model and a late reverberation sound model.

In operation S210, the N types of first audio signals are attenuated by using the direct sound processing model to obtain a first output result.

Exemplarily, in the virtual sound field space, sound (the first audio signal) is set as a point sound source, and its position coordinates are given to N types of virtual instrument models in the scene. Direct sound refers to the part of the energy that the sound source directly transmits to the receiver (virtual character) without any reflection under free field conditions, and the sound energy is attenuated by the surrounding environment during the propagation process. In the theory of physical propagation, the attenuation of point sound source energy follows the inverse square law, and the amplitude is proportional to the reciprocal of the propagation distance, that is, every time the distance doubles, the amplitude will decrease by 6dB. This makes the energy radiate outward, as the distance increases, the spread range will become larger and smaller, and the spread energy will become smaller and smaller.

Referring to Figure 3, playing the first audio signal of a virtual instrument can be called a playback event. In the structural hierarchy of the direct sound processing model, each first audio signal corresponds to an audio track. After direct sound processing and early reflection sound processing It is called a dry track, and it is called a wet track after direct sound processing and post-reverb sound processing.

The two playback events shown in Figure 3 are the same. In some embodiments, the output of the direct sound processing model can be used as the input of the early reflection sound model and the late reverberation sound model respectively. It is also possible to set two direct sound processing models corresponding to the early reflection sound model and the late reverberation sound model respectively.

In operation S220, input the first output result into the early reflection acoustic model for reflection processing, and obtain the second output result.

For example, the sound wave (audio signal) will collide with the surrounding medium during the continuous propagation from the sound source. During this process, part of the energy is absorbed by the medium material, part of the energy continues to propagate forward, and the other part of the energy will be reflected . The first few collision reflections of sound waves are defined as early reflections. There is a certain time delay between the early reflections and the direct sound, and their propagation directions are various. The time difference, directionality, and sound energy information reflected in the early reflections are recognized by the listener (that is, the virtual character), forming a preliminary judgment of its own sense of direction and positioning in the space it is in, and further revealing the space of the room. Size, shape, and produce different acoustic effects with the change of wall material.

In operation S230, the first output result is input into the late reverberation sound model for reverberation processing, and a third output result is obtained.

As an example, sound waves continue to propagate in a real concert hall. Every time they encounter an obstacle, they undergo reflection and absorption, and the remaining sound wave energy continues to propagate. After a large number of multiple reflections and absorptions, the remaining sound wave energy in the room The sum is called reverberation. After the sound source stops producing sound, the reverberation sound continues to be emitted, and slowly disappears with reflection and absorption. By listening to the late reverberation of different spaces, the volume of the space and its unique architectural acoustic information will be intuitively felt. The content of low-frequency sound in reverberation sound is always higher than that of high-frequency sound, and the decay time is longer than that of high-frequency sound. This is because low-frequency sound has a longer wavelength, and it is easier to bypass obstacles without being reflected, and it is also less likely to be blocked by obstacles. absorbed. And the sound will be attenuated when it propagates in the air, but under the same propagation conditions, the attenuation of low-frequency sound is lower than that of high-frequency sound.

In operation S240, a second audio signal is obtained according to the second output result and the third output result.

Referring to FIG. 3 , the second output result and the third output result are combined into the main output bus, and the second audio signal is output from the main output bus. A bus is a route for audio signals, either to another bus or directly to an output.

In some embodiments, the final output of the main output bus includes 11 dry track instrument directs, early reflections, and wet reverbs. That is, the first output results are respectively entered into the main output bus, the early reflection sound model and the late reverberation sound model.

According to an embodiment of the present invention, audio dry and wet sounds are processed separately. The audio dry sound can contain rich distance and orientation information after early reflection processing, and is sent directly to the main output bus. Reverb sounds do not contain dry sounds. According to the actual law of sound propagation, the energy of pure reverberant sound increases with the increase of propagation distance, and the distance difference between the listener and the sound source will assist in the formation of spatial position information. As the sound source propagates in the virtual space, dry sound and wet sound The sound will be smoothly rendered as a reverberation sound effect carrying comprehensive information in the process of spreading to the listener.

According to an embodiment of the present invention, the above-mentioned relative position information includes distance information, and in operation S210, the direct acoustic processing model is used to perform attenuation processing on N types of first audio signals, and obtaining the first output result includes: using N distance attenuation curves according to the distance information N types of first audio signals are processed to obtain a first output result, wherein the N distance attenuation curves are in one-to-one correspondence with the N types of first audio signals, and any two of the N distance attenuation curves are the same or different.

The distance modeling work for simulating the natural sound attenuation is completed in this step, the first audio signal in N can be classified, and the corresponding distance attenuation curve is created according to the classification results to construct the distance attenuation model. The maximum attenuation point is determined by the maximum attenuation distance value, and a spherical attenuation range is formed around each sound source with the maximum distance value as the radius. The attenuation curve can be customized, and control points can be added for fine adjustment. Distance decay curves include linear curve, constant curve, logarithmic curve, power curve and S-curve. For example, the logarithmic curve is selected here, and a curve with a more realistic sense of hearing is simulated by interpolation.

In some embodiments, in order to simulate the effect of air absorption, a recursive filter is selected for some high-frequency and low-frequency frequencies.

Fig. 4 schematically shows a flow chart of tapered attenuation processing according to an embodiment of the present invention.

As shown in FIG. 4 , this embodiment includes performing tapered attenuation processing on at least one audio signal among the N types of first audio signals, specifically including: performing operations S410 to any one of the at least one audio signal. Operation S440.

In operation S410, a propagation distance is obtained based on internal space information of the virtual sound field space.

Exemplarily, the internal space information may be three-dimensional spatial parameters of the virtual sound field space, such as the size of the internal space, the layout of a building, or the layout of all parties in a concert (such as the stage, auditorium, and band, etc.). The propagation distance may be the distance from the sound source to a certain wall inside the virtual sound field space.

In operation S420, the position of the virtual sound source corresponding to the audio signal of this type is used as the position of the center of the sphere, and the propagation distance is used as the radius to obtain a spherical propagation area of the audio signal of the type.

In operation S430, the spherical propagation area is divided into an inner corner area, an outer corner area, and a transition area between the inner corner area and the outer corner area.

In operation S440, perform corresponding attenuation processing on the audio signal according to the actual area to which the second position belongs, and obtain a first output result, wherein the actual area includes any area among inner corner area category, outer corner area and transition area.

According to the embodiment of the present invention, the direct sound of the point sound source reveals the initial virtual sound source position information in the virtual sound field space, and some musical instruments that emphasize directivity are set in the sound cone attenuation mode to carry more changes in the direction of the listener. Generate interactive stage acoustic information. In conical falloff, a sphere centered at the geometric center of the instrument model and with a propagation distance of radius is divided into inner corners, outer corners, and transition regions. In the inner corner area, the volume of the output bus does not attenuate, and in the outer corner area, the output volume is attenuated, and the filtering effect reaches the highest level set by the system. In the transition area between the inner and outer corners, linear interpolation is used to make the bus output volume drop. The directionality of instrument propagation in spatial audio is accomplished through tapered attenuation. Finally, during system operation, as the listener's orientation changes (such as forward, sideways, and back), different degrees of volume attenuation and filtering effects are presented. , the listener can directly feel the change process of the sound with the direction.

In some embodiments, plug-in algorithms, geometric modeling, filter design and other methods can be used comprehensively to calculate and simulate the spatialized early reflections of the concert hall model, and initially restore the spatial sound field information of the concert hall model.

Fig. 5 schematically shows a flow chart of reflection processing according to an embodiment of the present invention.

As shown in FIG. 5 , the reflection processing in this embodiment includes operation S510 to operation S530 . Wherein, operation S530 is one embodiment of operation S220.

In the virtual sound field space, the position information of the roaming listener (virtual listening position) is a factor considered in processing the audio signal. The listener is allowed to roam around in the concert hall, and at the same time receive the early reflection sound wave information from all directions, and the reception of the early reflection sound information is related to the various reflection walls (that is, the obstacles in the sound wave propagation process). Distance information, direction information, material information, etc. are directly related, and together constitute the auditory perception of the listener, which affects their judgment on the sense of space and immersion. This part can be detected in real time and calculated for feedback.

In operation S510, M virtual sound sources are calculated according to the N virtual sound source positions and the geometry of the virtual sound field space.

Exemplarily, acoustic reflection geometry modeling can be performed. For example, the virtual source technology based on the multi-tap time-varying delay line calculates the spatialized early reflection sound. Its premise is that the reflective surface is regarded as infinite and ideally rigid. At this time, the reflection model achieves physical accuracy. In the simulation of the virtual source method, an image sound source is formed at an equidistant distance from the sound source after each reflecting surface, and the line connecting it with the sounding body is perpendicular to the reflecting surface. The order of early reflections increases with the complexity of the reflective surfaces in the room geometry. The original sound source is reflected to each surface in the room geometry, resulting in first-order reflections first, and then recursively obtaining all higher-order reflections from the previous order. The highest order of the early reflections simulated here is the fourth order, and each order of reflection can be called each virtual sound source.

Exemplarily, the geometry includes three-dimensional geometric space information of the virtual sound field space, such as size, shape and other information.

In operation S520, S sound reflection paths are calculated according to the second position and the geometry, and M and S are integers greater than or equal to 1, respectively.

The virtual sound source and the sound source are related to the geometric shape of the virtual sound field space, which means that if the two remain stationary, the information of all virtual sound sources remains unchanged, so all virtual sound sources can be pre-calculated, but they are not related to the moving listener. The real-time sound reflection path calculation related to the listener is performed iteratively and individually as the listener position changes.

Exemplarily, the space surface material in the virtual sound field space may be preset. The wall material determines how much energy is filtered out by sound waves as they pass through obstacles. In this concert hall system, the simulation of the sound absorption phenomenon of the wall material is completed through the design of the frequency segment filter. The spatial surface material model is completed based on four-band filter attenuation. The absorption frequency bands are divided into low, medium-low, medium-high, and high. The default mapping intervals are shown in Table 1.

Table 1 Absorption frequency band mapping interval

type name frequency range low frequency <250Hz Medium and low frequency >250Hz and <1,000Hz Medium and high frequency >1,000Hz and <4,000Hz high frequency >4,000Hz

Each reflective surface in the model obtained by acoustic reflection geometric modeling is given a space surface material, and the analog signal is continuously filtered. In the actual operation of the virtual sound field space, the sound absorption effect of all materials will increase with the successive arrival of sound waves. overlay. Different filtering parameters are set for the walls, reflectors, and seats inside the virtual sound field space. These values are consistent with the actual material absorption coefficients that can be found, so as to ensure that the acoustic properties of real physical materials are accurately copied into the virtual scene. .

In operation S530, reflection processing is performed on the first output result according to the M virtual sound sources and the S sound reflection paths to obtain a second output result.

According to the embodiment of the present invention, the second output result reveals the size and shape of the room, which is directly related to the distance, direction and material information of various reflection obstacles in the propagation path, and together constitutes the listener's judgment on the spatial position, This part should be detected and calculated in real time.

Fig. 6 schematically shows a flow chart of detecting auditory interaction information according to an embodiment of the present invention.

Before operation S520, as shown in FIG. 6 , in this embodiment, detecting auditory interaction information includes operation S610 to operation S620.

In operation S610, a virtual character is used as a ray source, and a virtual ray is emitted from a second position.

Exemplarily, the virtual ray can simulate the irradiation of light to detect the surrounding environment according to the propagation and feedback of the virtual ray.

In operation S620, the auditory interaction information is detected through the virtual ray, wherein the auditory interaction information includes the distance between the virtual character and the wall in the virtual sound field space and material information of the wall in the virtual sound field space.

Exemplarily, the method of ray detection is used to systematically detect the orientation information between the sound source and the receiver and the environment information around the propagation path. On the basis of the acoustic reflection geometric model, when the virtual sound field is running, the virtual character will send rays around when roaming, and the information related to the auditory interaction, such as the distance from the character to the wall, and the sound reflection material of the walls around the character, will be It is detected in real time and enters the algorithm process. Rays are re-sent every time there is a change in the system, such as when a sound source is activated and deactivated, when a roaming character changes position, or when the geometry of buildings around the source and receiver changes.

According to the embodiment of the present invention, the sound reflection path can be accurately calculated according to the distance from the person to the wall, the sound reflection material of the wall around the person, and the like.

In some embodiments, considering that the actual geometric surface of the room is not completely an infinite rigid body in an ideal state, but has a boundary, so in addition to the early reflection of the acoustic surface, more physical propagation phenomena occur at the boundary, For example, diffraction, transmission, etc. also need to be taken into account. The definition of diffraction is the physical phenomenon that the sound wave deviates from the original straight line when it encounters an obstacle. It is specifically manifested as the phenomenon that the sound wave bends when it bypasses the edge of the obstacle. The size of the diffraction is related to the wavelength of the sound and the size of the obstacle. When the size of the obstacle is too large relative to the wavelength, the degree of diffraction of the sound wave is relatively large. The diffraction model based on the ray method combines the uniform diffraction theory to define the visible area, reflection area and shadow area. The shaded area, where sound waves are audible but the sound source is not visible. Transmission specifically describes the obstruction of the propagation of a sound source by obstacles between the source and the listener. The degree to which the filter is applied to simulate transmission, i.e. sets the array associated acoustic material and transmission loss values.

Fig. 7 schematically shows a flow chart of obtaining a third output result according to an embodiment of the present invention.

As shown in FIG. 7 , in operation S630 , the first output result is input into the late reverberation sound model for reverberation processing, and obtaining the third output result includes operation S710 to operation S720 .

In operation S710, invoking a first impulse response signal in response to the first virtual sound field space selected by the user from K virtual sound field spaces, wherein the first virtual sound field space is constructed and obtained according to the first physical environment among the K physical environments, K is an integer greater than or equal to 1.

Exemplarily, the user may be provided with K virtual sound field space models constructed in one-to-one correspondence with the K physical environments. The physical environment includes three-dimensional spatial information.

The room at this time can be compared to the concept of "system" in the field of signal processing, and more specifically, it can be regarded as a linear time-invariant system. Here, the audio dry sound signal can be regarded as the input of the system, and the sound with reverberation produced after passing through the room effect is regarded as the output of the system. When multiple different audio dry sound signals are input, the output obtained is a separate Input the sum of the output results of these audio dry sound signals, and the input time does not affect the output result. Furthermore, if the input signal covers all frequencies, then the obtained output signal will naturally include the response of the system to all frequencies. In a digital signal, the input signal is called an impulse, and the output signal of the system is called an impulse response. When an impulse is input in a specific room, the obtained impulse response contains all the spatial information of the room.

Exemplarily, when constructing K virtual sound field space models, three-dimensional spatial information and three-dimensional scenes of famous concert halls, natural scenes and living environments in the world can be copied. And the K virtual sound field space models can be selected by the user to switch in real time.

The virtual sound field space can simulate spaces such as Amsterdam Concert Hall, Berlin Concert Hall, Boston Concert Hall, Chicago Concert Hall, glaciers, bridge holes, caves and indoor courts. In the signal preprocessing stage, for example, the impulse response signals selected include the impulse response signals recorded and collected in the Amsterdam Concert Hall, Berlin Concert Hall, Boston Concert Hall, and Chicago Concert Hall. Each concert hall includes a left voice in monophonic format. channel signal and a right channel signal together constitute stereo. Another example is that the impulse response signal includes some natural and living environment scenes, aiming to provide users with the opportunity to study the applicable environment of experimental music works, such as glaciers, bridge holes, caves and indoor courts, etc. The audio system is stereo.

In operation S720, perform convolution calculation on the first output result and the first impulse response signal to obtain a third output result.

For many reasons, the proportion of low-frequency sound in reverberation sound is higher, and because the radiation directivity of low-frequency sound waves is not as obvious as that of high-frequency sound waves, it can be said that reverberation sound has less obvious directional characteristics than early reflection sound. Therefore, when simulating the late reverberation sound of the concert hall system, obtaining the sampling information of the concert hall space, and using the acoustic parameters to control the reverberation effector with the convolution algorithm can reduce the consumption cost of the central processor of the computer.

Exemplarily, by setting different impulse response signals with spatial information and performing convolution operation with the input audio dry sound signal, the sense of space of different sound field environments can be truly reproduced.

Before using the convolution algorithm, a necessary step is to upmix the two mono audio to a stereo format pulse signal. All the processed stereo impulse response signals are convolved with the same stereo music dry sound signal, and the output is a music signal with a sense of space.

Referring to Figure 3, similar to the work of the early reflection sound simulation, multiple auxiliary buses with convolution reverb effects were added to the project, and each impulse response signal was applied as convolution data to form a concert hall reverberation effect. Impulse response signal transcoding is completed offline. When the concert hall system is running, the preprocessed impulse response signal is directly convolved with the input audio dry sound, and more digital signal processing work involving real-time feedback is also completed at the same time. In this system, they are specifically the calling and switching of the "state" of the audio interaction event. The spatial reverberation effect of each actual concert hall is predefined as a global state. When the system is running, the global state will be triggered following the operator's instructions, and then assigned to the corresponding audio object. The preset audio parameters change in the state will be applied to the audio object.

When the user selects a concert hall scene, the system will respond to its corresponding audio state, and the audio auxiliary bus mounted on the convolution reverb of this concert hall will be activated and sent to the main output bus of the audio link . Inside each auxiliary bus loaded with convolution reverb effects, the input level, channel configuration, balance control of the impulse response signal, and the dry sound level, reverb level, and frequency equalization of the convolution reverb operation , filtering, delay time and other technical parameters have been properly adjusted to ensure that the reverb sound is not distorted and balanced, and can be switched smoothly and naturally.

Referring to Fig. 3, the overall hierarchical structure includes 11 instrument dry sound tracks and 11 instrument pure reverberation sound tracks. Among them, the dry sound track can be directly sent to the main output bus as the direct sound, and at the same time sent to the early reflection auxiliary bus equipped with the early reflection simulation plug-in to form the early reflection sound of the concert hall system. 11 instrument pure reverb tracks are directly sent to the convolution reverb auxiliary bus mounted with convolution reverb effect plug-ins. After algorithm processing and parameter adjustment, the pure wet reverb of many famous concert halls is formed.

Convolution reverb has continuity problems in restoring architectural sound fields, specifically, given that a single application of convolution reverb can only convolve one impulse response signal with dry sound, and the impulse response signal is actually based on the It is formed by recording or simulation calculation at a certain point. Therefore, strictly speaking, what is heard at this time is the spatial audio information felt by standing at a certain point in the concert hall. When the auditory environment changes from a flat 2D scene to a 3D scene, which increases the functional experience of scene characters roaming in the environment, as the listening position changes in real time, the convolution reverberation sound experienced by the listener at this time will not change. It can accurately reflect the real spatial sense of hearing, that is to say, the convolution reverberation effect rendered offline is static, does not have real-time changing spatial position information, and does not support the presentation of dynamic changes in sound involving head rotation. More sound interaction information related to the movement state of scene roaming characters should be added to the reverberation sound field simulation.

Referring to FIG. 3 and operation S210 to operation S240, the audio dry sound and the wet sound obtained by the convolution reverberation processing are processed separately. The audio dry corresponds to the original instrument track with a distance decay model and a cone decay model, and their physical propagation model contains rich distance and orientation information, which is directly fed into the main output bus. The convolution reverb aux bus outputs only the reverb sound without any dry sound. According to the actual law of sound propagation, the energy of pure reverberant sound increases with the increase of propagation distance, and the distance difference between the listener and the sound source will assist in the formation of spatial position information. As the sound source propagates in the virtual space, dry sound and wet sound The sound will be smoothly rendered as a reverberation sound effect carrying comprehensive information in the process of spreading to the listener.

Fig. 8 schematically shows a flow chart of updating a virtual listening position according to an embodiment of the present invention.

As shown in FIG. 8 , updating the virtual listening position in this embodiment includes operation S810 to operation S830 .

In operation S810, the avatar is moved to a third position in response to the user's first instruction to move the avatar.

Exemplarily, the user can operate the virtual character to roam in the virtual sound field space. Multiple users can operate their corresponding virtual characters to roam, and each virtual character is independent of each other.

Exemplarily, the camera is specially set to a first-person perspective. The avatar can be manipulated to roam, stand and listen in different locations of the concert hall. In order to better restore the experience of identifying the listening position in the real world, the system adds a head rotation function. The audience can control the head rotation of the virtual roaming character through the keyboard arrow keys, so as to perceive the change of the direction of the sound source in the process. Bring an immersive experience to the concert hall audience through the combination of audio-visual senses.

In operation S820, the virtual listening position is updated to the third position.

In operation S830, the operations of determining the relative position information, obtaining the second audio signal, and playing the second audio signal to the user are re-performed. That is, re-execute operation S120 to operation S140.

Considering that comparing seats in different parts is the most urgent need of the audience, the concert hall system realizes the interaction between the audience and the sound field through real-time auditory simulation. During the roaming process, as the position of the character changes, the interaction between the character and nearby buildings and sound sources is calculated by the computer in real time, and used as a ray feedback carrying sound information. What can happen is that the audience roams through the different locations to get a feel for the concert, and they can also walk up to the stage and experience the orchestra conductor's perspective at work, standing up close to each instrument and learning about its acoustic properties.

It should be noted that the positional information, geometry or internal space in the virtual space (virtual sound field space) has a mapping relationship with the real space, and the processing, propagation and playback of audio signals in the described virtual space have the same characteristics as in the real space through simulation. Effect.

Fig. 9 schematically shows a flow chart of updating the position of a virtual sound source according to an embodiment of the present invention.

As shown in FIG. 9 , updating the virtual sound source position in this embodiment includes operation S910 to operation S930 .

In operation S910, at least one virtual musical instrument is moved to a fourth position in response to a user's second instruction to move the at least one virtual musical instrument.

In operation S920, the corresponding position of the at least one virtual instrument in the N virtual sound source positions is updated to a fourth position.

Exemplarily, each virtual musical instrument may include multiple virtual musical instruments of this category. One or more virtual instruments can be moved in the virtual sound field space to realize voice part placement adjustment. The virtual sound source position coordinates are assigned to N types of virtual instrument models in the scene, and change with the movement of the instrument models during runtime.

In operation S930, the operations of determining the relative position information, obtaining the second audio signal, and playing the second audio signal to the user are re-performed. That is, re-execute operation S120 to operation S140.

Exemplarily, bands, conductors and musicians can rehearse online (the first audio signal can be pre-recorded, and can also be acquired and processed during real-time performance), without the need for personal experience in the real world. Various classic orchestra position modes can be preset for the operator to switch with one button, and drag and drop to adjust the position of the instrument during the performance. This function can be used to study the stage acoustics of classical instruments, improved instruments and innovative instruments in different positions.

According to the embodiment of the present invention, for the orchestra and the conductor, the online simulated rehearsal can be realized by replaying the audio after moving at least one virtual instrument. In the simulated rehearsal, functions such as musical instrument acoustic simulation, part position adjustment, and concert hall sound field real-time switching are realized.

Fig. 10 schematically shows a technical architecture diagram of a modeling method suitable for realizing interactive immersive sound field roaming according to an embodiment of the present invention. Fig. 11 schematically shows a system development architecture diagram of a modeling method suitable for realizing interactive immersive sound field roaming according to an embodiment of the present invention.

Referring to Figure 10 and Figure 11, this embodiment is based on digital twins, virtual reality, sound field simulation, interactive immersion and other technical means to roam the virtual sound field space for customized interactive immersion, such as N kinds of virtual instruments, virtual sound field space and virtual characters Realized through digital twin and virtual reality technology.

In order to reproduce the architectural acoustics of a real concert hall, virtual reality technology and binaural room impulse response are used to simulate an acoustic environment (ie, an interactive immersive sound field) based on sound propagation principles, geometric feature models and acoustic materials.

Referring to Figure 10 and Figure 11 , combined with one or more embodiments described in Figures 1 to 9, this embodiment can provide a modeling method for interactive immersive sound field roaming based on audibility, in order to realize the user's sound field roaming Purpose, through the modeling method, the user's sound field roaming can be realized. The method includes: obtaining a direct acoustic processing model, the direct acoustic processing model is used to attenuate N types of first audio signals to obtain a first output result, and the N types of first audio signals propagate from N virtual sound source positions to virtual Listening position, N is an integer greater than or equal to 1; obtain the early reflection sound model, and the early reflection sound model is used to perform reflection processing on the first output result to obtain the second output result; obtain the late reverberation sound model, and use the late reverberation sound model Reverberate the first output result to obtain the third output result; set the main output bus, the main output bus is used to obtain the second audio signal according to the second output result and the third output result, wherein the second audio signal is An audio signal obtained by simulating propagation of N types of first audio signals in a physical space.

It should be noted that, referring to FIG. 10 and FIG. 11 , and in combination with one or more embodiments described in FIG. 1 to FIG. 9 , one or more steps of the sound field roaming method of the present disclosure are based on one of the corresponding modeling methods or a plurality of steps to achieve, and will not go into details here.

As a scene rendering platform, the 3D development engine Unity is integrated with professional modeling software and interactive audio engine Wwise. At the same time, the developed UI system is also carried on Unity. Finally, the above content is integrated into an application system to achieve interactive and immersive sound field roaming Method's concert hall system. As shown in Figure 11, the communication between Wwise and Unity is based on the logic of audio event packaging, and all audio materials, events and state properties are packaged into sound libraries. Through the API, it can be sent to Unity, where a series of event commands can be invoked with a C# script.

Exemplarily, the currently defined synchronizer logic may include two state groups, which are respectively used to control the group playback of instrument tracks and the dynamic recall and bypass of the convolution reverberation auxiliary bus. The rules for defining the playback status of instrument tracks are: for the mute status of the wind instrument group, the dry/wet sound of Bangdi, Nanxiao and Sheng is set to negative infinity, and the dry/wet sound of other instruments is set to 0; For the mute state of the instrument group, the dry/wet sound of Pipa, Zhongruan, Daruan, Sanxian and Yangqin is set to negative infinity, the dry/wet sound of other instruments is set to 0, and so on. The definition rule for the dynamic control state of convolution reverberation is: the current convolution reverberation auxiliary bus of 5 concert hall reverberation, 4 natural scenes and living environment is set to 0, and the other convolution reverberation auxiliary buses are set to negative infinity, The early reflections aux bus is set to 0, and the bypassed reverb for each aux bus is set to minus infinity.

For professional audio engineers, a series of audio features have been designed for professionals to practice their skills in a virtual workplace, but also for music lovers to experience and explore. Given that a DAW is the most familiar working environment for audio professionals, an interactive control system in the form of a DAW is a core requirement. This concert hall system has created several comprehensive control panels that support custom adjustments, and all parameter variable changes will have an effect on the virtual orchestra performance without causing pauses or stuttering.

For mixing engineers and music-acoustic researchers, it allows adjusting the volume of each instrument track in an orchestra, and the system supports selective playback and mute when mixing specific instrument groups or adjusting the overall sound level of a production. The system also supports switching and bypassing of reverb effects.

The main work of recording engineers and stage technicians is to deal with various microphones, and the selection and matching between multiple microphones requires special analysis and design. The task of recording engineers is to deliver first-hand music, but it is difficult to grow quickly without learning in person on site. One solution is to simulate recording online and make full preparations. Thus, the system supports switching and critical listening to audio files recorded with microphones of different pickup types and frequency responses. This function can also contribute to the establishment of digital microphone libraries.

In some embodiments, a monitoring system construction method can be proposed for the application experience side, that is, a head tracker is used in the system, which can track the rotation direction and angle of the head when the person is listening to the sound in real time, so as to simulate the rotation of the head. The effect of changing the direction of the sound source produced. When the user uses the headset to play back, place the head tracker in the middle of the beam directly above the headset, connect and pair it to the computer via Bluetooth.

Based on the binaural effect, the head tracking technology obtains accurate head position information through the head tracker, processes information such as filtering and delay, sound reflection, and sound field displacement, and completes the actual spatial sound field without adding sound coloration. binaural realization. A practical function that head tracking technology also includes is personalized custom head modeling. Its core technical principle is to model the head-related transfer function. The HRTF function describes the propagation of sound waves from the spatial sound source to the two ears. Physical processes, including the effects of physiological structures (head, torso, auricle, etc.) on sound waves such as diffraction, scattering, and diffraction. It can also be said that HRTF reflects the change in amplitude and phase of sound waves during the transmission from the sound source to the ears. Data such as the listener's head circumference and the distance between the ears are obtained through the synchronization tracker, and the interaural delay and the filtering and gain required for each ear are calculated and simulated to compensate for the body filtering effect in the real sound field.

According to the embodiment of the present invention, on the one hand, the concert hall scene is built by means of virtual reality, and interactive functions are realized; on the other hand, the auralization process from the sound source through the space to the receiver is realized through sound transmission and sound propagation simulation , finally presented in the form of binaural audio. The functional design of the concert hall system is based on the needs of different groups for music performances. For concert audiences, a series of functions such as sound field exploration, sound image positioning, and virtual space simulation are designed to achieve immersion, realism, and binaural positioning in a simulated real space. For orchestras and conductors, in order to solve the dilemma of orchestras on global tours, help the transformation of offline-online performance forms, and realize online simulated rehearsals, the acoustic simulation of musical instruments, the adjustment of voice parts, and the real-time switching of the sound field of the concert hall are designed. Function. For audio engineers, this system is based on the requirements of simulating digital audio workstations, making it easier for mixers and sound engineers to work online and practice skills, and designs a user interactive control system that can tune in real time.

Based on the above audible-based interactive immersive sound field roaming method, the present invention also provides an audible-based interactive immersive sound field roaming system. The device will be described in detail below with reference to FIG. 12 .

Fig. 12 schematically shows a structural block diagram of an auralization-based interactive immersive sound field roaming system 1200 according to an embodiment of the present invention.

As shown in FIG. 12 , an auralization-based interactive immersive sound field roaming system 1200 may include a position determination unit 1210 , a relative position unit 1220 , a signal processing unit 1230 and an audio playback unit 1240 .

The position determining unit 1210 may perform operation S110 for determining N first positions of N types of virtual instruments in the virtual sound field space, and second positions of the virtual character in the virtual sound field space, wherein the virtual character is used to be operated by the user to stop or move in the virtual sound field space.

The location determination unit 1210 may perform operation S810 to operation S820, and operation S910 to operation S920 will not be repeated here.

The relative position unit 1220 may perform operation S120 for determining relative position information between N first positions and second positions, wherein the N first positions are N virtual sound source positions, and the second position is a virtual listening position , N is an integer greater than or equal to 1.

The signal processing unit 1230 may perform operation S130, for processing N types of first audio signals using a sound field space model according to the relative position information to obtain a second audio signal, wherein the sound field space model is used to simulate N types of first audio signals in physical For propagation in space, N types of first audio signals correspond to N types of virtual musical instruments one-to-one.

The signal processing unit 1230 may also perform operation S210 to operation 240, operation S410 to operation 440, operation S510 to operation 530, operation S610 to operation 620, operation S710 to operation 720, which will not be repeated here.

The audio playing unit 1240 may perform operation S140 for playing the second audio signal to the user in response to the user's playing operation.

FIG. 13 shows a schematic structural diagram of a computing device according to an embodiment of the present invention, and the specific embodiment of the present invention does not limit the specific implementation of the computing device.

As shown in FIG. 13 , the computing device may include: a processor (processor) 1302 , a communication interface (Communications Interface) 1304 , a memory (memory) 1306 , and a communication bus 1308 .

in:

The processor 1302 , the communication interface 1304 , and the memory 1306 communicate with each other through the communication bus 1308 .

The communication interface 1304 is used to communicate with network elements of other devices such as clients or other servers.

The processor 1302 is configured to execute the program 1310, and may specifically execute the relevant steps in the above embodiment of the object grasping method.

Specifically, the program 1310 may include program codes including computer operation instructions.

The processor 1302 may be a central processing unit CPU, or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention. The computing device includes one or more processors, which may be of the same type, such as one or more CPUs. It can also be a different type of processor, such as one or more CPUs and one or more ASICs.

The memory 1306 is used to store the program 1310 . The memory 1306 may include a high-speed RAM memory, and may also include a nonvolatile memory (nonvolatile memory), such as at least one disk memory.

The program 1310 may be specifically configured to enable the processor 1302 to execute the object grasping method in any of the above method embodiments. For the specific implementation of each step in the program 1310, refer to the corresponding steps and the corresponding descriptions in the units in the above-mentioned object grasping embodiment, and details are not repeated here. Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the foregoing method embodiments, and will not be repeated here.

The present invention also provides a computer-readable storage medium, which may be included in the device/apparatus/system described in the above embodiments. It can also exist independently without being assembled into the equipment/device/system. The above-mentioned computer-readable storage medium carries one or more programs, and when the above-mentioned one or more programs are executed, the method according to the embodiment of the present invention is realized.

The algorithms or displays presented herein are not inherently related to any particular computer, virtual system, or other device.

Various generic systems can also be used with the teachings based on this. The structure required to construct such a system is apparent from the above description. Furthermore, embodiments of the present invention are not directed to any particular programming language. It should be understood that various programming languages can be used to implement the content of the present invention described herein, and the above description of specific languages is for disclosing the best mode of the present invention.

In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure the understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, in order to streamline the present disclosure and to facilitate an understanding of one or more of the various inventive aspects, various features of the embodiments of the invention are sometimes grouped together into a single implementation examples, figures, or descriptions thereof. This method of invention, however, is not to be interpreted as reflecting an intention that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment.

Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art can understand that the modules in the device in the embodiment can be adaptively changed and arranged in one or more devices different from the embodiment. Modules or units or components in the embodiments may be combined into one module or unit or component, and furthermore may be divided into a plurality of sub-modules or sub-units or sub-assemblies. All features disclosed in this specification (including accompanying claims, abstract and drawings) and any method or method so disclosed may be used in any combination, except that at least some of such features and/or processes or units are mutually exclusive. All processes or units of equipment are combined. Each feature disclosed in this specification (including accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will understand that although some embodiments herein include some features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the invention. And form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all functions of some or all components according to the embodiments of the present invention. The present invention can also be implemented as an apparatus or an apparatus program (for example, a computer program and a computer program product) for performing a part or all of the methods described herein. Such a program for realizing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet site, or provided on a carrier signal, or provided in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means can be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the execution sequence.

Claims (11)

1. An audibility-based interactive immersive sound field roaming method, comprising:

determining N first positions of N virtual musical instruments in a virtual sound field space and a second position of a virtual character in the virtual sound field space, wherein the virtual character is used for being operated by a user to stop or move in the virtual sound field space;

determining relative position information between the N first positions and the second positions, wherein the N first positions are N virtual sound source positions, the second positions are virtual listening positions, and N is an integer greater than or equal to 1;

processing N first audio signals by using a sound field space model according to the relative position information to obtain second audio signals, wherein the sound field space model is used for simulating the propagation of the N first audio signals in a physical space, and the N first audio signals are in one-to-one correspondence with the N virtual musical instruments;

and responding to the playing operation of the user, and playing the second audio signal to the user.

2. The method according to claim 1, wherein the sound field spatial model comprises a direct sound processing model, an early reflected sound model and a late reverberant sound model, and the processing the N types of first audio signals with the sound field spatial model to obtain the second audio signals comprises:

performing attenuation processing on the N first audio signals by using the direct sound processing model to obtain a first output result;

inputting the first output result into the early reflected sound model for reflection processing to obtain a second output result;

inputting the first output result into the later reverberation model for reverberation processing to obtain a third output result;

and obtaining the second audio signal according to the second output result and the third output result.

3. The method as recited in claim 2, wherein the relative position information comprises distance information, and wherein the attenuation processing the N first audio signals using the direct sound processing model comprises:

and processing the N types of first audio signals according to the distance information by utilizing N distance attenuation curves, wherein the N distance attenuation curves correspond to the N types of first audio signals one to one, and any two curves in the N distance attenuation curves are the same or different.

4. The method according to claim 3, wherein said processing the N first audio signals according to the distance information using N distance attenuation curves comprises cone attenuation processing at least one of the N first audio signals, in particular comprising: for any of the at least one audio signal,

obtaining a propagation distance based on the internal space information of the virtual sound field space;

taking the position of a virtual sound source corresponding to the audio signal as the position of a sphere center, and taking the propagation distance as a radius to obtain a spherical propagation area of the audio signal;

dividing the spherical propagation region into an inner angle region, an outer angle region, and a transition region between the inner angle region and the outer angle region;

and performing corresponding attenuation processing on the audio signal according to an actual region to which the second position belongs to obtain the first output result, wherein the actual region comprises any one of the inner corner region type, the outer corner region and the transition region.

5. The method of claim 2, wherein:

calculating to obtain M virtual sound sources according to the N virtual sound source positions and the geometric forms of the virtual sound field spaces;

calculating S sound reflection paths according to the second position and the geometric form, wherein M and S are integers which are larger than or equal to 1 respectively;

wherein, the inputting the first output result into the early stage reflected sound model for reflection processing to obtain a second output result includes:

and performing reflection processing on the first output result according to the M virtual sound sources and the S sound reflection paths to obtain a second output result.

6. The method of claim 5, wherein prior to said calculating S sound reflection paths, the method further comprises:

taking the virtual character as a ray source, and emitting virtual rays from the second position;

and detecting auditory interaction information through the virtual ray, wherein the auditory interaction information comprises the distance between the virtual character and the wall in the virtual sound field space and the material information of the wall in the virtual sound field space.

7. The method of claim 2, wherein the late reverberation sound model includes K impulse response signals obtained from K recordings of physical environments, the inputting the first output result into the late reverberation sound model for reverberation processing, and obtaining a third output result includes:

responding to a first virtual sound field space selected by the user from K virtual sound field spaces, and calling a first impulse response signal, wherein the first virtual sound field space is obtained according to a first physical environment construction in the K physical environments, and K is an integer greater than or equal to 1;

and performing convolution calculation on the first output result and the first impulse response signal to obtain a third output result.

8. The method of claim 1, wherein the method further comprises:

in response to a first instruction from the user to move the virtual character, causing the virtual character to move to a third location;

updating the virtual listening position to the third position;

and re-executing the operations of determining the relative position information, obtaining the second audio signal and playing the second audio signal to the user.

9. The method of claim 1, wherein the method further comprises:

in response to a second instruction from the user to move at least one virtual instrument, causing the at least one virtual instrument to move to a fourth position;

updating the corresponding position of the at least one virtual musical instrument in the N virtual sound source positions to the fourth position;

and re-executing the operations of determining the relative position information, obtaining the second audio signal and playing the second audio signal to the user.

10. An audibility-based interactive immersive sound field roaming system, comprising:

a position determination unit for determining N first positions of N kinds of virtual musical instruments in a virtual sound field space, and a second position of a virtual character in the virtual sound field space, wherein the virtual character is used for being operated by a user to stop or move in the virtual sound field space;

a relative position unit, configured to determine relative position information between the N first positions and the second position, where the N first positions are N virtual sound source positions, the second position is a virtual listening position, and N is an integer greater than or equal to 1;

a signal processing unit, configured to process N types of first audio signals by using a sound field space model according to the relative position information, to obtain a second audio signal, where the sound field space model is used to simulate propagation of the N types of first audio signals in a physical space, and the N types of first audio signals are in one-to-one correspondence with the N types of virtual musical instruments;

and the audio playing unit is used for responding to the playing operation of the user and playing the second audio signal to the user.

11. A method of modeling interactive immersive sound field roaming, comprising:

obtaining a direct sound processing model, wherein the direct sound processing model is used for carrying out attenuation processing on N types of first audio signals to obtain a first output result, the N types of first audio signals are respectively transmitted to virtual listening positions from N virtual sound source positions, and N is an integer greater than or equal to 1;

obtaining an early phase reflected sound model, wherein the early phase reflected sound model is used for performing reflection processing on the first output result to obtain a second output result;

obtaining a late reverberation sound model for performing reverberation processing on the first output result to obtain a third output result;

and setting a main output bus, wherein the main output bus is used for obtaining a second audio signal according to the second output result and the third output result, and the second audio signal is obtained by simulating the propagation of the N types of first audio signals in a physical space.

Images