JPWO2022023417A5 - - Google Patents

Description

The present invention relates to headphone equalization and room adaptation for binaural reproduction in augmented reality (AR).

Selective hearing (SH) refers to the ability of a listener to direct attention to a particular sound source or sources in an auditory scene. In turn, this means that the listener's focus on sources of no interest is reduced.

Human listeners can therefore communicate in loud environments, usually by exploiting different aspects: when listening with two ears, there are time and level differences that depend on the direction, as well as different spectral colorings of sound that depend on the direction. The latter allows hearing to determine the direction of sound sources and separate different sound sources, even when listening with one ear.

Time and level differences alone are not enough to determine the exact location of a sound source, and locations with the same time and level differences are located on a hyperbolic surface. The resulting localization ambiguity is called the cone of confusion. In a room, each sound source is reflected by boundary surfaces. Each of these so-called mirror sources is located on a further hyperbolic surface. Human hearing combines information about the direct sound and the associated reflections to the auditory event, thus resolving the confusion ambiguity. At the same time, the reflections that belong to the sound source increase the perceived loudness of the sound source.

Also, in the case of natural sound sources, especially speech, signal parts of different frequencies are combined in time. In binaural hearing, all of these aspects are used together. Moreover, large disturbance sources that can be well localized can be actively ignored, so to speak.

In the literature, the concept of selective hearing is related to other terms such as assisted hearing [1], virtual hearing environments and amplified hearing environments [2]. Assisted hearing is a broader term that includes virtual, amplified and SH applications.

According to the prior art, classical hearing devices operate mainly in a monophonic manner, i.e. the signal processing for the left and right ears is completely independent in terms of frequency response and dynamic compression. As a result, time, level and frequency differences between the ear signals are lost.

Modern, so-called binaural hearing devices combine the correction factors of two hearing devices. Often they have several microphones, but usually only the microphone with the "most speech-like" signal is selected and no explicit beamforming is calculated. In complex hearing situations, the desired and undesired sound signals are amplified in the same way, and therefore focusing on the desired sound component is not supported.

In the field of hands-free devices, for example for telephone, several microphones are already used today, so-called beams are calculated from the individual microphone signals, sounds coming from the direction of the beam are amplified and sounds from other directions are reduced. Today's methods learn constant sounds in the background (e.g. engine and wind noise in a car) and learn large disturbances that are well localizable via further beams and subtract these from the used signal (e.g. generalized sidelobe canceller). Sometimes telephone systems use detectors that detect static characteristics of speech and suppress everything that is not structured like speech. In hands-free devices, only mono signals are finally transmitted, losing interesting spatial information in the transmission path to capture the situation and to provide the illusion as if "one was there", especially in the case of several talkers having mutual calls. By suppressing non-speech signals, important information about the acoustic environment of the other party is lost, which can hinder communication.

Naturally, humans can consciously focus on "selective listening" to individual sound sources in their environment. An automated system for selective hearing using artificial intelligence (AI) must first learn the underlying concepts. Automatic decomposition of an acoustic scene (scene decomposition) first requires the detection and classification of all active sound sources, and then separating them so that they can be further processed, amplified, or attenuated as separate audio objects.

The research field of auditory scene analysis seeks to detect and classify, based on recorded audio signals, temporally located sound events such as steps, clapping, or voices, as well as more global acoustic scenes such as concerts, restaurants, or supermarkets. In this case, current methods exclusively use methods from the fields of artificial intelligence (AI) and deep learning. This involves data-driven learning of deep neural networks that learn on a large training volume to detect characteristic patterns within audio signals [70]. Inspired by progress in the research fields of image processing (computer vision) and speech processing (natural language processing), among others, a mixture of convolutional neural networks for two-dimensional pattern detection in spectrogram representations and recurrent layers (recurrent neural networks) for temporal modeling of sounds is used in principle.

Speech analysis has a set of specific challenges to address. Due to their complexity, deep learning models are very data intensive. In contrast to the research fields of image processing and speech processing, the datasets available for speech processing are relatively small. The largest dataset is the AudioSet dataset from Google [83], with about 2 million sound examples and 632 different sound event classes, and most datasets used in research are significantly smaller. This small amount of training data can be addressed, for example, using transfer learning, where a model pre-trained on a large dataset is then fine-tuned on a smaller dataset with new classes determined for the use case (fine-tuning) [77]. Furthermore, methods from semi-supervised learning are exploited to include in the training also unannotated speech data, which is generally available in large quantities.

A further important difference compared to image processing is that in the case of simultaneously audible sound events, there is no masking of sound objects (as in the case of images) but rather a complex phase-dependent overlap. Current algorithms in deep learning use so-called "attention" mechanisms, which allow, for example, the model to focus the classification on a specific time segment or frequency range [23]. The detection of sound events is further complicated by the high variance regarding their duration. Algorithms should be able to robustly detect very short events, such as gunshots, and long events, such as a passing train.

Due to the strong dependence of the models on the acoustic conditions in the recording of the training data, the models often show unexpected behavior in new acoustic environments, which are different, for example, in terms of spatial reverberation or microphone positioning. Various solutions have been developed to alleviate this problem. For example, data augmentation methods try to achieve higher robustness and invariance of the models through the simulation of different acoustic conditions [68] and the artificial overlap of different sound sources. Furthermore, the parameters of complex neural networks can be adjusted in different ways so that overtraining and specialization on the training data is avoided and at the same time a better generalization to unseen data is achieved. In recent years, different algorithms have been proposed for "domain adaptation" [67] to adapt previously trained models to new application conditions. For the in-headphone usage scenario planned in this project, the real-time capability of the sound source detection algorithm is of fundamental importance. Here, a trade-off must necessarily be made between the complexity of the neural network and the maximum possible number of computational operations on the underlying computing platform. Even if the duration of a sound event is long, it needs to be detected as quickly as possible in order to start the corresponding sound source separation.

At Fraunhofer IDMT, much work has been done in the field of automatic sound source detection in recent years. In the research project "Stadtram", a distributed sensor network was developed that can measure noise levels and classify between 14 different acoustic scenes and event classes based on audio signals recorded at different locations in a city [69]. In this case, the processing at the sensors is performed in real time on an embedded platform Raspberry Pi 3. In previous work, a novel approach for data compression of spectrograms based on autoencoder networks was considered [71]. Recently, through the use of methods from deep learning in the field of music signal processing (music information retrieval), great progress has been made in applications such as music transcription [76], [77], chord detection [78], and instrument detection [79]. In the field of industrial sound processing, new datasets have been established and deep learning methods are being used, for example, to monitor the acoustic condition of electric motors [75].

The scenario addressed in this embodiment assumes several sound sources whose number and type are initially unknown and may change continuously. For source separation, several sound sources with similar characteristics, such as several loudspeakers, are a particularly big challenge [80].

To achieve high spatial resolution, several microphones must be used in the form of an array [72]. In contrast to conventional sound recordings, either mono (one channel) or stereo (two channels), such a recording scenario allows for precise localization of sound sources around the listener.

Source separation algorithms typically leave behind artifacts such as distortion and crosstalk between the sources [5], which can generally be perceived as disruptive by the listener. By remixing the tracks, such artifacts can be partially masked and therefore reduced [10].

To enhance the "blind" source separation, additional information such as the detected number and type of sources or their estimated spatial location is often used (informed source separation [74]). For a conference with several active speakers, current analysis systems are able to simultaneously estimate the number of speakers, determine their respective temporal activity, and subsequently separate them by source separation [66].

At the Fraunhofer IDMT, much work has been done in recent years on perception-based evaluation of sound source separation algorithms [73].

In the field of music signal processing, real-time capable algorithms have been developed for separating isolated and associated instruments, using base frequency estimates of isolated instruments as additional information [81]. An alternative approach for separating singing from complex musical compositions based on deep learning methods is proposed in [82]. Specialized source separation algorithms have also been developed for applications in the context of industrial audio analysis [7].

Headphones greatly affect the acoustic perception of the surroundings. Depending on the headphone construction, the sound incident towards the ear is attenuated to different degrees. In-ear headphones completely block the ear channel [85]. Closed headphones that surround the pinna also strongly acoustically isolate the listener from the external environment. Open and semi-open headphones allow sound to pass through completely or partially [84]. In many applications in daily life, it is desirable for headphones to isolate unwanted surrounding sounds to a greater extent than is possible for their construction type.

Furthermore, the effects of external interference can be attenuated by active noise control (ANC). This is achieved by recording the incoming sound signals by the microphones of the headphones and then reproducing them by the loudspeakers so that these sound parts and the sound parts penetrating the headphones cancel each other out by interference. Overall, this can achieve a strong acoustic isolation from the surroundings. However, in many everyday situations this is associated with dangers, so it is desirable to be able to intelligently turn on this function on demand.

The first product allows the microphone signal to be passed through headphones to reduce passive isolation. So, besides the prototype [86], there are already commercial products advertising the feature of "transparent hearing". For example, Sennheiser offers the feature with its AMBEO headset [88], and Bragi with its product "Dash Pro". However, this possibility is only the beginning. In the future, this functionality should be significantly expanded to be able to make individual signal parts (e.g. only the voice signal or the alarm signal) exclusively hearable on demand, in addition to turning the ambient sound completely on and off. The French company Orosound allows the person wearing the headset "chilled earphone" to adapt the strength of the ANC using a slider [89]. In addition, the voice of the conversation partner can also be induced during the activated ANC. However, this only works if the conversation partner is located face to face in the 60° cone. Direction-independent adaptation is not possible.

US2015195641 (see [91]) discloses a method implemented to generate an auditory environment for a user. In this case, the method includes receiving a signal representative of the user's ambient auditory environment and processing the signal using a microprocessor to identify at least one sound type of a plurality of sound types in the ambient auditory environment. The method further includes receiving a user preference for each of the plurality of sound types, modifying the signal for each sound type in the ambient auditory environment, and outputting the modified signal to at least one loudspeaker to generate the auditory environment for the user.

Headphone equalization and room adaptation (or spatial/spatial adaptation or spatial/spatial compensation) for binaural reproduction in augmented reality (AR) are important issues.

In a typical scenario, a human listener wears acoustically (partially) transparent headphones and hears ambient sounds through the headphones. Furthermore, an additional sound source is played through the headphones, said sound source being embedded in the real surroundings in such a way that the listener cannot distinguish between the real acoustic scene and the additional sounds.

Typically, the direction the head rotates and the position of the listener in the room (or space) are determined by tracking (6 degrees of freedom (DoF)). Research shows that good results (i.e. externalization and accurate localization) are achieved if the room acoustics of the recording and playback rooms match or if the recording is adapted to the playback room.

In this case, an exemplary solution can be implemented as follows:

In the first step, the BRIR is measured without headphones, either in an individualized manner or using an artificial head with a probe microphone.

In the second step, an analysis of the room characteristics of the recording room is performed based on the measured BRIR.

In a third step, measurements of the headphone transfer function are made using an artificial head, either in an individualized manner or by means of a co-located probe microphone, from which the equalization function is determined .

Optionally, in a fourth step, measurements of the room characteristics of the reproduction room, analysis of the acoustic characteristics of the reproduction room, and adaptation of the BRIR to the reproduction room may be performed.

Then, in a further step, a convolution (or folding) of the source, augmented with a correctly positioned and optionally adapted BRIR, is performed to obtain two raw channels. Convolution of the raw channels with an equalization function to obtain a headphone signal.

Finally, in a further step, playback of the headphone signal is performed via headphones.

However, the problem is that when headphones are worn, the effect of the pinna on the BRIR is eliminated; i.e. the BRIR is different than without the headphones. This results in a different natural sound source being heard than without the headphones, but a virtual augmented sound source being reproduced as if the headphones were not there.

US Patent Publication No. 2015195641

It would be desirable to provide a concept that allows for a simple, fast and efficient determination of the room characteristics of a regeneration room.

Embodiments of the present invention are provided below.

Accordingly, claim 1 provides a system according to an embodiment of the invention, claim 19 provides a method, and claim 20 provides a computer program.

A system according to an embodiment of the invention comprises an analyzer for determining a plurality of binaural room impulse responses and a loudspeaker signal generator for generating at least two loudspeaker signals in response to the plurality of binaural room impulse responses and in response to a source signal of at least one sound source, the analyzer being configured to determine the plurality of binaural room impulse responses such that each of the plurality of binaural room impulse responses takes into account an effect resulting from headphones being worn by a user.

Further provided is a method according to an embodiment of the present invention, the method comprising:
determining a plurality of binaural room impulse responses;
- generating at least two loudspeaker signals in response to a plurality of binaural room impulse responses and in response to a source signal of at least one sound source;
Includes.

The plurality of binaural room impulse responses are determined such that each of the plurality of binaural room impulse responses takes into account the effect resulting from a user wearing headphones.

Furthermore, a computer program according to an embodiment of the present invention is provided having program code for executing the above-mentioned method.

Next, a preferred embodiment of the present invention will be described with reference to the drawings.

FIG. 1 illustrates a system according to one embodiment. FIG. 1 illustrates a further system for assisting selective hearing according to a further embodiment. FIG. 1 illustrates a further system for assisting selective hearing, further including a user interface. FIG. 1 illustrates a system for assisting selective hearing that includes a hearing device having two corresponding loudspeakers. FIG. 1 illustrates a system for assisting selective hearing including a housing structure and two loudspeakers. FIG. 1 illustrates a system for assisting selective hearing that includes headphones with two loudspeakers. FIG. 1 illustrates a system according to one embodiment including a remote device 190 including a detector and locator as well as a sound type classifier and a signal portion modifier and signal generator. FIG. 1 illustrates a system according to one embodiment, including five subsystems. FIG. 2 illustrates a corresponding scenario according to one embodiment. FIG. 1 illustrates a scenario according to an embodiment with four external sound sources. FIG. 2 is a diagram illustrating a processing workflow for SH applications according to an embodiment.

Figure 1 illustrates a system according to one embodiment.

The system includes an analyzer 152 for determining a number of binaural room impulse responses.

Furthermore, the system includes a loudspeaker signal generator 154 for generating at least two loudspeaker signals in response to the multiple binaural room impulse responses and in response to the source signal of at least one sound source.

The analyzer 152 is configured to determine a plurality of binaural room impulse responses such that each of the plurality of binaural room impulse responses takes into account effects resulting from a user wearing headphones.

In one embodiment, for example, the system may include headphones, which may be configured to output at least two loudspeaker signals.

According to one embodiment, for example, a headphone may include at least two headphone capsules and at least one microphone for measuring sound in each of the two headphone capsules, for example, the at least one microphone for measuring sound may be arranged in each of the two headphone capsules, where for example the analyzer 152 may be configured to perform a determination of a plurality of binaural room impulse responses using the measurements of the at least one microphone in each of the two headphone capsules. A headphone intended for binaural reproduction always comprises at least two headphone capsules (for example for different frequency ranges), and three or more capsules may also be provided.

In one embodiment, for example, at least one microphone in each of the two headphone capsules may be configured to generate one or more recordings of a sound situation in a reproduction room (or space) prior to playback of the at least two loudspeaker signals by the headphones, determine an estimate of a raw audio signal of at least one sound source from the one or more recordings, and determine a binaural room impulse response of a plurality of binaural room impulse responses for the sound source in the reproduction room.

According to one embodiment, for example, at least one microphone of each of the two headphone capsules may be configured to generate, during playback of the at least two loudspeaker signals by the headphones, one or more further recordings of the sound situation in the reproduction room, subtract the augmented signal from these one or more further recordings, determine an estimate of the raw sound signal from the one or more sound sources, and determine a binaural room impulse response of the multiple binaural room impulse responses for the sound sources in the reproduction room.

In one embodiment, for example, the analyzer 152 may be configured to determine acoustic room characteristics of a reproduction room and to adapt the multiple binaural room impulse responses according to the acoustic room characteristics.

According to one embodiment, for example, at least one microphone may be placed in each of the two headphone capsules to measure sounds near the entrance of the ear canal.

In one embodiment, for example, the system may include one or more additional microphones outside the two headphone capsules to measure the sound conditions in the playback room.

According to one embodiment, for example, the headphones may include a headphone band , for example, at least one of the one or more further microphones being located on the headphone band .

In one embodiment, for example, the loudspeaker signal generator 154 may be configured to generate at least two loudspeaker signals by convolving each of a plurality of binaural room impulse responses with a source signal of one or more of the source signals.

According to one embodiment, for example, the analyzer 152 may be configured to determine at least one of a plurality of binaural room impulse responses (or some or all of the binaural room impulse responses) in response to headphone movement.

In an embodiment, the system may include a sensor for determining the movement of the headphones. For example, the sensor may be a sensor such as an accelerometer with at least 3DoF (three degrees of freedom) to capture head rotation. For example, a 6DoF sensor (six degrees of freedom sensor) may be used.

Certain embodiments of the present invention address the technical challenge that in an auditory environment, which is often very loud, certain sounds in the auditory environment are disturbing and selective hearing is desired. While the human brain itself can perform selective hearing to some extent, intelligent technical assistants can significantly improve selective hearing. Just as glasses help many people in modern life to better perceive their environment, there are hearing aids for hearing, but even people with normal hearing can benefit from assistance from intelligent systems in many situations. To realize an "intelligent hearable" (auditory device or hearing aid), a technical system needs to analyze the (acoustic) environment and identify individual sound sources so that they can be processed separately. Research has already been conducted on this issue, but the prior art has not achieved the analysis and processing of the entire acoustic environment in real time (transparent to the ear) and with high sound quality (content that sounds indistinguishable from the normal acoustic environment).

Improved concepts for machine listening are provided below.

In the first step, measurements of the BRIR using headphones are performed either individually using a probe microphone or using headphones.

In the second step, an analysis of the room characteristics of the recording room is performed based on the measured BRIR.

Optionally, for example in a third step, before playback, at least one built-in microphone in each shell records the actual sound situation in the playback room. From these recordings, estimates of the raw sound signals of one or more sound sources are determined , and the BRIR of each of the sound sources/sources in the playback room is determined . From this estimate, the acoustic room characteristics of the playback room are determined , to which the BRIR in the recording room is adapted.

Optionally, for example in a further step, during playback, at least one built-in microphone in each shell records the actual sound situation in the playback room. From these recordings, the augmented signal is first subtracted, then an estimate of the raw sound signal of one or more sound sources is determined , and the BRIR of each of the sound sources/sources in the playback room is determined . From this estimate, the acoustic room characteristics of the playback room are determined , and the BRIR of the playback room is adapted thereto.

In a further step, a convolution of the source, augmented with a correctly positioned and optionally adapted BRIR, is performed to obtain the headphone signal.

Finally, in a further step, playback of the headphone signal is performed via headphones.

In one embodiment, for example, at least one microphone is placed within each headphone capsule to measure sounds near the entrance of the ear canal.

According to one embodiment, additional microphones are optionally placed on the outside of the headphones, possibly even above the headphone band , to measure and analyze the sound situation in the playback room.

In the embodiment, the same natural and augmented sound sources are realized.

The embodiment recognizes that measuring headphone characteristics is not required.

Thus, the embodiment provides a concept for measuring the room characteristics of a playback room.

Some embodiments provide starting values and (post)optimization of room adaptation. The concepts provided also work when the room acoustics of the reproduction room change, e.g. when the listener moves to a different room (or space).

Among other things, the embodiment is based on installing different technologies for assisting hearing in a technical system and then combining them in such a way that an improvement in sound and quality of life (e.g. desired sounds are louder, undesired sounds are softer, speech intelligibility is better) is achieved for people with normal hearing and for people with hearing loss.

Figure 2 illustrates a system for supporting selective hearing in one embodiment.

The system includes a detector 110 for detecting source signal portions of one or more sound sources by using at least two received microphone signals of the auditory environment (or listening environment).

In addition, the system includes a position determiner 120 for assigning position information to each of the one or more sound sources.

The system further includes an audio type classifier 130 for assigning an audio signal type to the audio source signal portion of each of the one or more audio sources.

Furthermore, the system includes a signal portion modifier 140 for modifying the source signal portion of at least one of the one or more sound sources depending on the sound signal type of the source signal portion of the at least one sound source to obtain a modified sound signal portion of the at least one sound source.

The analyzer 152 and the loudspeaker signal generator 154 of FIG. 1 together form the signal generator 150.

The analyzer 152 of the signal generator 150 is configured to generate a plurality of binaural room impulse responses for each of the one or more sound sources, the plurality of binaural room impulse responses being dependent on the position information of the sound source and the orientation of the user's head.

The loudspeaker signal generator 154 of the signal generator 150 is configured to generate at least two loudspeaker signals in response to a plurality of binaural room impulse responses and in response to the modified audio signal portion of at least one sound source.

According to one embodiment, for example, detector 110 may be configured to detect source signal portions of one or more sound sources by using a deep learning model.

In one embodiment, for example, position determiner 120 may be configured to determine position information for each of one or more sound sources as a function of the captured image or recorded video.

According to one embodiment, for example, the position determiner 120 may be configured to determine position information for each of the one or more sound sources in response to the video by detecting lip movements of a person in the video and assigning position information to a source signal portion of one of the one or more sound sources in response to the lip movements.

In one embodiment, for example, detector 110 may be configured to determine one or more acoustic characteristics of the auditory environment in response to at least two received microphone signals.

According to one embodiment, for example, the signal generator 150 may be configured to determine multiple binaural room impulse responses as a function of one or more acoustic characteristics of the auditory environment.

In one embodiment, for example, the signal portion modifier 140 may be configured to select at least one sound source whose sound source signal portion is to be modified in response to a previously learned user scenario and to modify it in response to the previously learned user scenario.

According to one embodiment, for example, the system may include a user interface 160 for selecting a previously learned user scenario from a group of two or more previously learned user scenarios. FIG. 3 illustrates such a system according to one embodiment, further including such a user interface 160.

In an embodiment, for example, the detector 110 and/or the locator 120 and/or the sound type classifier 130 and/or the signal modifier 140 and/or the signal generator 150 may be configured to perform parallel signal processing using a Hough transform, or using multiple VLSI chips, or by using multiple memristors.

According to one embodiment, for example, the system may include a hearing device 170 that functions as a hearing aid for a user with limited hearing capabilities and/or impaired hearing, the hearing device including at least two loudspeakers 171, 172 for outputting at least two loudspeaker signals. Figure 4 shows such a system according to one embodiment including such a hearing device 170 with two corresponding loudspeakers 171, 172.

In one embodiment, for example, the system may include at least two loudspeakers 181, 182 for outputting at least two loudspeaker signals and a housing structure 183 for housing the at least two loudspeakers, at least one housing structure 183 being suitable for being fixed to a user's head 185 or any other body part of the user. FIG. 5a shows such a housing structure 183 and a corresponding system including two loudspeakers 181, 182.

According to one embodiment, for example, the system may include a headphone 180 including at least two loudspeakers 181, 182 for outputting at least two loudspeaker signals. Figure 5b shows a corresponding headphone 180 having two loudspeakers 181, 182 according to one embodiment.

In one embodiment, for example, the detector 110 and the locator 120 as well as the sound type classifier 130 and the signal portion modifier 140 and the signal generator 150 may be integrated into the headphones 180 .

6, for example, the system may include a remote device 190 including the detector 110 and the locator 120 as well as the sound type classifier 130 as well as the signal portion modifier 140 and the signal generator 150. In this case, for example, the remote device 190 may be spatially separated from the headphones 180.

In one embodiment, for example, the remote device 190 may be a smartphone.

Embodiments do not necessarily use microprocessors, but rather use parallel signal processing steps such as Hough transforms, VLSI chips, or memristors for an energy-efficient implementation, especially for artificial neural networks.

In an embodiment, the auditory environment is captured and reproduced spatially, using two or more signals for the representation of the input signal on the one hand, and also spatial reproduction on the other hand.

In an embodiment, the signal separation is performed by a deep learning (DL) model (e.g. CNN, RCNN, LSTM, Siamese network), simultaneously processing information from at least two microphone channels, at least one microphone in each hearable. According to the invention, several output signals (corresponding to the individual sound sources) are determined by mutual analysis together with their respective spatial positions. If the recording means (microphones) are connected to the head, the position of the object changes with the movement of the head. This allows a natural focus on important/unimportant sounds, for example by orienting oneself towards the sound object.

In some embodiments, the algorithm for signal analysis is based, for example, on deep learning architectures. Alternatively, it uses variation by analysis units or variation by separate networks for aspect localization, detection and sound separation. An alternative use of generalized cross-correlation (correlation vs. time offset) accommodates frequency-dependent shadowing/separation by the head, improving localization, detection and sound source separation.

According to one embodiment, different source categories (e.g., speech, vehicles, male/female/voice of children, warning tones, etc.) are learned by the detector in a training phase. Here, the source separation network is also trained on the target stimuli for high signal quality as well as the localization network with high accuracy of localization.

For example, the training steps described above use multi-channel speech data, and a first training round is typically performed in a laboratory using simulated or recorded speech data. This is followed by training runs in different natural environments (e.g., a room, a classroom, a train station, an (industrial) production environment, etc.), i.e. transfer learning and domain adaptation are performed.

Alternatively or additionally, the position detector can be coupled to one or more cameras to also determine the visual position of the sound source/source. In case of speech, lip movements are correlated with the audio signal coming from the sound source separator to achieve more accurate localization.

After training, we have a DL model with the network architecture and associated parameters.

In some embodiments, the auralization is performed by binaural synthesis. Binaural synthesis offers the additional advantage that, although it cannot completely remove undesirable components, it can reduce them to a level that is perceptible but not disturbing. This has the additional advantage of perceiving unexpected additional sources (warning signals, speech bubbles, ...) that would be missed if they were completely turned off.

According to some embodiments, the analysis of the auditory environment is used not only to separate objects, but also to analyze acoustic characteristics (e.g. reverberation time, initial time gap). These characteristics are then used in binaural synthesis to adapt pre-stored (possibly individualized) binaural room impulse responses (BRIRs) to the real room (or space). By reducing the room divergence, the listener has a significantly reduced listening effort when understanding the optimized signal. Minimizing the room divergence affects the externalization of auditory events and therefore the veracity of spatial audio reproduction in the monitoring room. For speech understanding or for general understanding of the optimized signal, there are no known solutions in the prior art.

In an embodiment, a user interface is used to determine which sound source is selected. According to the invention, this is done by pre-learning different user scenarios such as "amplify voice directly in front" (conversation with one person), "amplify voice in ±60 degree range" (group conversation), "suppress music and amplify music"(don't want to hear concert goers), "silence everything" (want to be left alone), "suppress all voices and warning tones", etc.

Some embodiments are agnostic to the hardware used, i.e., open and closed headphones can be used. Signal processing may be built into the headphones, an external device, or the smartphone. Optionally, in addition to playing acoustically recorded and processed signals, signals can be played directly from the smartphone (e.g., music, phone calls).

In another embodiment, an ecosystem for "AI-assisted selective listening" is provided. The embodiment refers to "personalized auditory reality" (PARty). In such a personalized environment, the listener can amplify, reduce or modify defined acoustic objects. A series of analysis and synthesis processes should be performed to create a wholesome experience adapted to individual requirements. Research into targeted transformation stages forms an essential component for this.

Some embodiments provide analysis of the real sound environment and detection of individual acoustic objects, isolation, tracking and editability of available objects, as well as reconstruction and playback of modified acoustic scenes.

In an embodiment, detection of sound events, separation of sound events, and suppression of some sound events are achieved.

In an embodiment, AI methods (particularly deep learning-based methods) are used.

Embodiments of the present invention contribute to technological developments for recording, signal processing, and playback of spatial audio.

For example, embodiments create spatiality and three-dimensionality in multimedia systems with interacting users.

In this case, the embodiment is based on research knowledge of perceptual and cognitive processing of spatial hearing/listening.

Some embodiments use two or more of the following concepts:

Scene decomposition: This involves spatial acoustic detection of the real environment, as well as parameter estimation and/or position-dependent sound field analysis.

Scene representation: This involves the representation and identification, and/or efficient representation and storage of objects and/or the environment.

Scene combination and reproduction: This includes the adaptation and transformation of objects and environments, and/or the rendering and auralization.

Quality assessment: This includes technical and/or auditory quality measurements.

Microphone positioning: This involves the application of a microphone array and appropriate audio signal processing.

Signal conditioning: This includes feature extraction as well as dataset generation for machine learning (ML).

Estimation of room and ambient acoustics: This involves in-situ measurement and estimation of room acoustic parameters and/or providing room acoustic signatures for source separation and ML.

Auralization: This includes spatial audio reproduction with auditory adaptation to the environment and/or validation and evaluation and/or functional proof and quality estimation.

Figure 8 illustrates a corresponding scenario according to one embodiment.

Embodiments combine concepts for sound source detection, classification, separation, localization, and enhancement, highlighting recent advances in each field and illustrating connections between them.

In the following, we provide a consistent concept that can combine/detect/classify/localize and separate/enhance sound sources to provide the flexibility and robustness required for real-world SH.

Furthermore, the embodiments provide a low-latency concept suitable for real-time performance when dealing with the dynamics of real-world auditory scenes.

Some embodiments use concepts from deep learning, machine hearing, and smart headphones (smart hearables) to allow the listener to selectively modify the auditory scene.

Embodiments provide the listener with the possibility to selectively enhance, attenuate, suppress or modify sound sources within an auditory scene using hearing devices such as headphones, earphones etc.

Figure 9 shows a scenario in one embodiment with four external sound sources.

In Fig. 9, the user is the center of the auditory scene. In this case, four external sound sources (S1-S4) are active around the user. The user interface allows the listener to influence the auditory scene. Sources S1-S4 can be attenuated, improved or suppressed by their corresponding sliders. As can be seen in Fig. 2, the listener can define sound sources or sound events that should be kept in the auditory scene or suppressed from it. In Fig. 2, the urban background noise should be suppressed, but alarms or phone calls should be kept. At all times, the user has the possibility to play (or play) additional audio streams such as music or radio through the auditory device.

The user is usually the center of the system and controls the auditory scene by means of a control unit. The user can modify the auditory scene by means of a user interface as shown in FIG. 9 or by means of any type of interaction such as voice control, gestures, gaze direction, etc. Once the user has provided feedback to the system, the next step consists of a detection/classification/location stage. In some cases, only detection is necessary, for example if the user wants to keep any speech occurring in the auditory scene. In other cases, classification may be necessary, for example if the user wants to keep fire alarms in the auditory scene, but not telephone ringing or office noise. In some cases, only the location of the source is relevant to the system. This is for example the case of the four sound sources in FIG. 9, where the user can decide to remove or attenuate sound sources coming from a certain direction, regardless of the type or characteristics of the source.

Figure 10 shows the processing workflow for SH applications according to an embodiment.

First, the separation enhancement stage in Fig. 10 modifies the auditory scene. This is done by suppressing, attenuating, or enhancing a particular sound source (e.g., or a particular sound source). As shown in Fig. 10, an additional processing option in SH is noise control, which has the objective of removing or minimizing background noise in the auditory scene. Perhaps the most common and widespread technique for noise control is active noise control (ANC) [11].

Selective hearing is distinguished from virtual and augmented auditory environments by restricting selective hearing to applications where only real sound sources are modified in the auditory scene, without any attempt to add virtual sound sources to the scene.

From a machine hearing perspective, selective hearing applications require techniques to automatically detect, localize, classify, separate, and enhance sound sources. To further clarify the terminology related to selective hearing, we define the following terms and highlight their differences and relationships:

In embodiments, for example, source localization is used, which refers to the ability to detect the position of a sound source within an auditory scene. In the context of audio processing, source location usually refers to the direction of arrival (DOA) of a given sound source, which can be given either as a 2D coordinate (azimuth) or a 3D coordinate if it includes the elevation angle. Some systems also estimate the distance from the sound source to the microphone as position information [3]. In the context of music processing, location often refers to the panning of the sound source in the final mix, usually given as an angle in degrees [4].

According to an embodiment, sound source detection is used with reference to the ability to determine , for example, whether any instances of a given sound source type are present in an auditory scene. An example of a detection task is determining whether a speaker is present in a scene. In this context, determining the number of speakers in a scene or the identity of the speakers is outside the scope of sound source detection. Detection can be understood as a binary classification task where the classes correspond to "source present" and "source absent".

In an embodiment, for example, sound source classification is used, which assigns a class label from a set of predefined classes to a given sound source or a given sound event. An example of a classification task is to determine whether a given sound source corresponds to speech, music, or environmental noise. Sound source classification and detection are closely related concepts. In some cases, the classification system includes a detection stage by considering "no class" as one of the possible labels. In these cases, the system learns to detect the presence or absence of a sound source implicitly and is not forced to assign a class label in the absence of sufficient evidence that any of the sound sources is active.

According to an embodiment, source separation is used with reference to, for example, the extraction of a given sound source from a sound mixture or an auditory scene. An example of source separation is the extraction of a singing voice from a sound mixture where, apart from the singer, other instruments are playing simultaneously [5]. Source separation becomes relevant in selective listening scenarios, since it allows to suppress sound sources that are not of interest to the listener. Some sound separation systems implicitly perform a detection task before extracting the sound source from the mixture. However, this is not necessarily the rule, and therefore highlights the distinction between these tasks. Moreover, separation often serves as a pre-processing stage for other types of analysis, such as source emphasis [6] or classification [7].

In an embodiment, for example, audio source identification is used, which goes one step further and aims to identify specific instances of audio sources within an audio signal. Speaker identification is perhaps the most common use of audio source identification today. In this task, the goal is to identify whether a particular speaker is present in the scene. In the example of Figure 1, the user has selected "Speaker X" as one of the audio sources held in the auditory scene. This requires techniques that go beyond voice detection and classification, and requires speaker-specific models that allow this accurate identification.

According to an embodiment, for example source enhancement is used to refer to a process that increases the salience of a given source in an auditory scene [8]. For a speech signal, the goal is often to increase its perceptual quality and intelligibility. A common scenario for speech enhancement is the denoising of speech corrupted by noise [9]. In the context of music processing, source enhancement is related to the concept of remixing and is often performed to make one instrument (source) more prominent in the mix. Remixing applications often use a sound separation front-end to access individual sources and modify the characteristics of the mixture [10]. Although a source separation stage can be performed before source enhancement, this is not always the case, and therefore we also highlight the distinction between these terms.

In the field of sound source detection, classification and identification, for example, some embodiments use one of the following concepts, such as acoustic scene and event detection and classification [18]. In this context, a method has been proposed for audio event detection (AED) in a domestic environment, where the goal is to detect the time boundary of a given sound event within 10 seconds of recording [19], [20]. In this particular case, 10 sound event classes were considered, including cat, dog, talking, alarm, and tap water. Methods for polyphonic sound event (several simultaneous events) detection have also been proposed in the literature [21], [22]. In [21], a method for polyphone sound event detection is proposed in which a total of 61 sound events from real-world contexts are detected using a binary activity detector based on a bidirectional long short-term memory (BLSTM) recurrent neural network (RNN).

For example, to handle weakly labeled data, some embodiments incorporate a temporal attention mechanism to focus on specific regions of the signal for classification [23]. The problem of noisy labels in classification is particularly relevant for selective hearing applications where class labels are highly diverse and high-quality annotation is very costly [24]. Noisy labels in sound event classification tasks are addressed in [25], where a noise-robust loss function based on categorical cross-entropy is presented, as well as a method to evaluate both noisy and manually labeled data. Similarly, [26] presents a system for sound event classification based on a convolutional neural network (CNN) that incorporates a validation step of noisy labels based on the prediction consensus of the CNN on multiple segments of training examples.

For example, some embodiments provide simultaneous detection and localization of sound events. Thus, some embodiments perform detection as a multi-label classification task, such as in [27], where the location is given as the 3D coordinates of the direction of arrival (DOA) of each sound event.

Some embodiments use concepts of voice activity detection and speaker recognition/identification for SH. Voice activity detection has been addressed in noisy environments using denoising autoencoders [28], recurrent neural networks [29], or as end-to-end systems using raw waveforms [30]. For speaker recognition applications, numerous systems have been proposed in the literature [31], most of which focus on increasing robustness to different conditions, for example with data augmentation or improved embeddings that facilitate recognition [32]-[34]. Some of the embodiments therefore use these concepts.

Further embodiments use concepts for instrument classification for sound event detection. Instrument classification in both monophonic and polyphonic settings has been addressed in [35], [36]. In [35], dominant instruments in 3-s audio segments are classified among 11 instrument classes and several aggregation techniques are proposed. Similarly, [37] proposes a method for instrument activity detection that can detect instruments with a finer time resolution of 1 s. A significant amount of research has been carried out in the field of singing voice analysis. In particular, methods such as [38] have been proposed for the task of detecting segments in recordings where the singing voice is active. Some embodiments use these concepts.

Some of the embodiments use one of the concepts described below for source localization. Source localization is closely related to the problem of source counting, since the number of sources in an auditory scene is usually unknown in real-world applications. Some systems work under the assumption that the number of sources in a scene is known. This is the case, for example, of the model presented in [39], which uses a histogram of active intensity vectors to localize the sources. From a supervised perspective, [40] proposes a CNN-based algorithm to estimate the DOA of multiple speakers in an auditory scene using a phase map as an input representation. In contrast, some works in the literature jointly estimate the number of sources in a scene and their location information. This is the case in [41], where a system for multi-speaker localization in noisy and reverberant environments is proposed. The system uses a complex-valued Gaussian Mixture Model (GMM) to estimate both the number of sources and their localization. The concepts described there are used by some of the embodiments.

Sound source localization algorithms can be computationally demanding, as they often involve scanning a large space around the auditory scene [42]. To reduce the computational requirements in localization algorithms, some embodiments use concepts that reduce the search space, such as by using clustering algorithms [43] or by performing a multi-resolution search with established methods, such as those based on the steered response power phase transform (SRP-PHAT) [42]. Other methods impose sparsity constraints and assume that only one sound source dominates in a given time-frequency region [44]. Recently, an end-to-end system for azimuth detection directly from raw waveforms has been proposed in [45]. Some embodiments use these concepts.

Some embodiments use concepts described below for sound source separation (SSS), particularly from the field of speech and music separation.

In particular, some embodiments use the concept of speaker-independent separation: separation is performed in situ without any prior information about the speakers in the scene [46]. Some embodiments also estimate the spatial location of the speakers to perform the separation [47].

Given the importance of computational performance in selective hearing applications, research conducted with the specific goal of achieving low latency is particularly important. Several studies have been proposed to perform low-latency speech separation (<10 ms) with little available training data [48]. To avoid the delays caused by framing analysis in the frequency domain, some systems approach the separation problem by carefully designing filters applied in the time domain [49]. Other systems achieve low-latency separation by directly modeling the time-domain signal using an encoder-decoder framework [50]. In contrast, some systems attempt to reduce the framing delay in frequency-domain separation approaches [51]. These concepts are adopted by some embodiments.

Some embodiments use concepts for music sound separation (MSS) to extract music sources from audio mixtures [5], such as those for reed instrument accompaniment separation [52]. These algorithms attempt to take the most prominent sound source in a mixture and separate it from the remaining accompaniments, regardless of its class label. Some embodiments use concepts for singing voice separation [53]. In most cases, either a specific source model [54] or a data-driven model [55] is used to capture the characteristics of singing voices. Although systems such as those proposed in [55] do not explicitly incorporate a classification or detection stage to achieve separation, the data-driven nature of these approaches allows these systems to implicitly learn to detect singing voices with a certain accuracy before separation. Another class of algorithms in the music domain attempts to perform separation using only the location of the sound sources, without attempting to classify or detect the sound sources before separation [4].

Some embodiments use concepts of active noise control (ANC), such as active noise cancellation (ANC). ANC systems primarily aim to remove background noise for headphone users by introducing and canceling a noise-removing signal [11]. ANC can be considered as a special case of SH and faces similarly stringent performance requirements [14]. Some works have focused on active noise control in specific environments, such as automobile cabins [56] or industrial scenarios [57]. The work in [56] analyzes the removal of different types of noise, such as road noise and engine noise, and calls for a unified noise control system that can address different types of noise. Some works have focused on developing ANC systems to remove noise over specific spatial regions. In [58], ANC over spatial regions is addressed using spherical harmonics as basis functions to represent the noise field. Some embodiments use the concepts described herein.

Some of the embodiments use concepts for audio source extension.

In the context of speech enhancement, one of the most common applications is the enhancement of speech corrupted by noise. Many studies have focused on phase processing for single-channel speech enhancement [8]. From the perspective of deep neural networks, the problem of speech denoising has been addressed by denoising autoencoders in [59], by nonlinear regression problems between clean and noisy speech using deep neural networks (DNNs) in [60], and by end-to-end systems using Generative Adversarial Networks (GANs) in [61]. Often, speech enhancement is applied as a front-end of automatic speech recognition (ASR) systems, as in the case of [62], which approaches speech enhancement with an LSTM RNN. Speech enhancement is also often performed in conjunction with source separation techniques, which may involve first extracting the speech and then applying enhancement techniques to the separated speech signal [6]. The concepts described herein are used by some embodiments.

In most cases, source enhancement in the context of music refers to applications for creating music remixes. In contrast to speech enhancement, where it is often assumed that the speech is only impaired by noise sources, music applications almost always assume that other sound sources (musical instruments) are being played simultaneously with the source to be enhanced. For this reason, music remix applications are always provided to be preceded by a source separation stage. For example, in [10], early jazz recordings were remixed by applying lead accompaniment and harmonic percussion separation techniques to achieve a better tonal balance in the mixture. Similarly, [63] studied the use of different singing voice separation algorithms to modify the relative volume of the singing voice and the backing track, showing that an increase of 6 dB was possible by introducing a slight but audible distortion in the final mixture. In [64], the authors study how to improve the music perception of cochlear implant users by applying source separation techniques to achieve a new mix. The concepts described there are used by some embodiments.

One of the biggest challenges in selective hearing applications relates to the stringent requirements on processing time. To maintain naturalness and perceived quality for the user, the complete processing workflow needs to be performed with minimal delay. The maximum acceptable latency of the system depends heavily on the application and the complexity of the auditory scene. For example, McPherson et al. propose 10 ms as an acceptable latency criterion for interactive music interfaces [12]. For music performance over a network, the authors of [13] report that delays become perceptible in the range of 20-25 ms to 50-60 ms. However, active noise control/cancellation (ANC) techniques require ultra-low latency processing for better performance. In these systems, the amount of acceptable latency depends on both frequency and attenuation, but can be as low as 1 ms for an attenuation of about 5 dB for frequencies below 200 Hz [14]. A final consideration in SH applications refers to the perceived quality of the modified auditory scene. A significant amount of work has been devoted to methodologies for reliable assessment of speech quality in various applications [15], [16], [17]. However, a challenge in SH is managing the clear trade-off between processing complexity and perceptual quality. Some embodiments use the concepts described therein.

Some embodiments use concepts for counting/counting and localization as described in [41], localization and detection as described in [27], separation and classification as described in [65], and separation and counting as described in [66].

Some embodiments use concepts to increase the robustness of current machine listening methods, as described in [25], [26], [32], [34], and new emerging directions include domain adaptation [67] and training on datasets recorded on multiple devices [68].

Some embodiments can handle raw waveforms and use concepts to improve the computational efficiency of machine listening methods such as those described in [48] or concepts described in [30], [45], [50], [61].

Some embodiments provide an integrated optimization scheme that detects/classifies/localizes and separates/enhances in a combined manner so that sound sources in a scene can be selectively modified, while the independent detection, separation, localization, classification, and enhancement methods are reliable and provide the robustness and flexibility required for SH.

Some embodiments are suitable for real-time processing with a good trade-off between algorithmic complexity and performance.

Some embodiments combine ANC with machine hearing. For example, an auditory scene is first classified and then ANC is selectively applied.

Further embodiments are provided below.

To augment a real auditory environment with virtual sound objects, the transfer functions from each location of the sound object to each location of the listener in the room must be well known.

The transfer function maps the characteristics of the sound source and the direct sound between the object and the user, as well as all reflections that occur in the room. To ensure correct spatial audio reproduction for the actual room acoustics in which the listener is currently located, the transfer function must additionally map with sufficient accuracy the room acoustics characteristics in the listener's room.

In an audio system suitable for the representation of individual audio objects at different locations in a room, the challenge is proper detection and separation of the individual audio objects when multiple audio objects are present, and the audio signals of each object overlap at the recording or listening position in the room. The room acoustics and audio signal overlap change when the objects and/or listening positions in the room change.

Due to the relative motion, the estimation of the room acoustics parameters must be performed quickly enough. Here, low estimation latency is more important than high accuracy. High accuracy is required when the positions of the sound sources and receivers do not change (static case). In the proposed system, the room acoustic parameters, as well as the room geometry and listener positions, are estimated or extracted from a stream of audio signals. The audio signals are recorded in a real environment, where the sound source(s) and receiver(s) can move in any direction and the sound source(s) and/or receiver(s) can arbitrarily change their orientation.

The audio signal stream may be the result of any microphone setup, including one or more microphones. The stream is fed to a signal processing stage for pre-processing and/or further analysis. The output is then fed to a feature extraction stage, which estimates room acoustic parameters, e.g. T60 (reverberation time), DRR (direct-to-reverberant ratio), etc.

The second data stream is generated by a 6DoF sensor ("6 degrees of freedom": three dimensions of room position and line of sight) that captures the orientation and position of the microphone setup. The position data stream is fed into a 6DoF signal processing stage for pre-processing or further analysis.

The output of the 6DoF signal processing, the audio feature extraction stage, and the preprocessed microphone streams are fed into a machine learning block, where the auditory space, i.e. the listening room (size, geometry, reflective surfaces) and the position of the microphone field within the room, are estimated. Furthermore, to enable more robust estimation, a user behavior model is applied. This model takes into account the limitations of human motion (e.g. continuous movement, speed, etc.) as well as the probability distribution of different types of motion.

Some embodiments achieve blind estimation of room acoustic parameters using arbitrary microphone arrangements and by adding user position and pose information, as well as by analyzing the data using machine learning methods.

For example, systems according to embodiments can be used for acoustically augmented reality (AAR), where a virtual room impulse response must be synthesized from estimated parameters.

Some embodiments include the removal of reverberation from the recorded signal. An example of such an embodiment is hearing aids for people with normal hearing and for people with hearing impairments. In this case, with the help of estimated parameters, reverberation can be removed from the input signal of the microphone setup.

A further application is the spatial synthesis of sound scenes generated in rooms other than the current auditory space. For this purpose, the room acoustics parameters that are part of the sound scene are adapted to the room acoustics parameters of the auditory space.

In the case of binaural synthesis, for this purpose the available BRIRs are adapted to the different acoustic parameters of the auditory space.

In one embodiment, an apparatus for determining one or more room acoustics parameters is provided.

The device is configured to acquire microphone data including one or more microphone signals.

Further, the device is configured to obtain tracking data regarding the user's position and/or orientation.

Additionally, the apparatus is configured to determine one or more room acoustics parameters as a function of the microphone data and the tracking data.

According to one embodiment, for example, the device may be configured to use machine learning to determine one or more room acoustics parameters as a function of the microphone data and the tracking data.

In an embodiment, for example, the device may be configured to use machine learning, in that the device may be configured to use a neural network.

According to one embodiment, for example, the device may be configured to use cloud-based processing for machine learning.

In one embodiment, for example, one or more room acoustics parameters may include reverberation time.

According to one embodiment, for example, one or more room acoustics parameters may include a directivity to reverberation ratio.

In one embodiment, for example, the tracking data may include x, y, and z coordinates to label the user's location.

According to an embodiment, for example, tracking data may include pitch, yaw, and roll coordinates to label the user's orientation.

In an embodiment, for example, the device may be configured to transform one or more microphone signals from the time domain to the frequency domain, e.g., the device may be configured to extract one or more features of the one or more microphone signals in the frequency domain, and the device may be configured to determine one or more room acoustic parameters as a function of the one or more features.

According to one embodiment, for example, the device may be configured to use cloud-based processing to extract one or more features.

In one embodiment, for example, the device may include a microphone arrangement of several microphones for recording several microphone signals.

According to one embodiment, for example, the microphone arrangement may be configured to be worn on the user's body.

In an embodiment, for example, the above-mentioned system may further include the above-mentioned apparatus for determining one or more room acoustics parameters.

According to one embodiment, for example, the signal portion modifier 140 and/or the signal generator 150 may be configured to perform the generation of at least one of the plurality of binaural room impulse responses for each of the one or more sound sources depending on the at least one of the one or more room acoustics parameters.

Figure 7 shows a system according to one embodiment, which includes five subsystems (subsystems 1-5).

Subsystem 1 includes a microphone setup of one, two or more individual microphones, which can be combined into a microphone field if one or more microphones are available. The positioning and relative placement of the microphones/microphones with respect to each other can be arbitrary. The microphone configuration can be part of a device worn by the user or can be a separate device placed in the room of interest.

Subsystem 1 also includes a tracking device that measures the user's translational position in the room and the user's head pose. It can measure up to 6DoF (x, y, z, pitch, yaw, roll).

The tracking device may be placed on the user's head or may be split into several sub-devices to measure the required DoF, and may or may not be placed on the user.

Thus, subsystem 1 represents an input interface including a microphone signal input interface 101 and a position information input interface 102.

Subsystem 2 includes signal processing of the recorded microphone signals. This includes frequency conversion and/or time domain based processing. Furthermore, it includes methods for combining different microphone signals to realize field processing. Feedback from system 4 is possible to adapt the parameters of the signal processing in subsystem 2. The signal processing block of the microphone signals may be part of the device in which the microphone is embedded or may be part of a separate device. It may also be part of cloud-based processing.

Furthermore, the subsystem 2 includes signal processing for the recorded tracking data. This includes frequency transformation and/or time domain based processing. Furthermore, it includes methods to improve the technical quality of the signal by using noise suppression, smoothing, interpolation and extrapolation. Furthermore, it includes methods to derive higher level information. This includes speed, acceleration, path direction, idle time, movement range and movement path. It also includes prediction of near future movement path and near future speed. The signal processing block of the tracking signal may be part of the tracking device or may be part of a separate device. It may also be part of cloud based processing.

Subsystem 3 includes extraction of processed microphone features.

The feature extraction block may be part of the user's wearable device, or it may be part of a separate device, or it may be part of cloud-based processing.

Subsystems 2 and 3, together with their modules 111 and 121, for example, implement detector 110, voice type classifier 130, and signal portion modifier 140. For example, subsystem 3, module 121 may output the result of voice classification to subsystem 2, module 111 (feedback). For example, subsystem 2, module 112 implements position determiner 120. Furthermore, in one embodiment, subsystems 2 and 3 may also implement signal generator 150, for example by subsystem 2, module 111 generating binaural room impulse response and loudspeaker signals.

Subsystem 4 contains methods and algorithms to estimate room acoustic parameters using the processed microphone signals, extracted microphone signal features, and processed tracking data. The output of this block is the room acoustic characteristic parameters as idle data, as well as control and variation of parameters of microphone signal processing in subsystem 2. Machine learning block 131 may be part of the user's device or a separate device. It may also be part of cloud-based processing.

Furthermore, Subsystem 4 includes post-processing of the room acoustic idle data parameters (e.g., in block 132). This includes outlier detection, combining individual parameters into new parameters, smoothing, extrapolation, interpolation, and validation. This block also gets information from Subsystem 2, including the near future position of the user in the room, to estimate the near future acoustic parameters. This block may be part of the user's device or a separate device. It may also be part of cloud-based processing.

Subsystem 5 includes storage and allocation of room acoustics parameters for downstream systems (e.g., in memory 141). Parameter allocation may be realized just-in-time and/or time-responsively stored. Storage may be performed in a device located on or near the user, or in a cloud-based system.

A use case for an embodiment of the present invention is described below.

One embodiment use case is home entertainment, involving a user in a home environment. For example, a user wants to focus on a particular playback device, such as a TV, radio, PC, tablet, etc., and wants to suppress other sources of disturbance (other users' or children's devices, construction noise, street noise). In this case, the user positions himself close to the preferred playback device and selects the device or its location. Regardless of the user's location, the selected device, or sound source location, is acoustically emphasized until the user cancels his selection.

For example, the user moves closer to a target sound source. The user selects the target sound source via an appropriate interface, and the hearable accordingly adapts the sound playback based on the user position, the user's gaze direction, and the target sound source, ensuring that the target sound source is well understood even in the presence of noise interference.

Alternatively, the user moves closer to a particularly disruptive sound source. The user selects this disruptive sound source via an appropriate interface, and the hearable adjusts the audio playback accordingly based on the user's position, the user's gaze direction, and the disruptive sound source to explicitly accommodate the disruptive sound source.

A further use case for a further embodiment is a cocktail party where the user is positioned between several speakers.

In the presence of many talkers, for example, the user may wish to focus on one (or more) of them and tune out or attenuate other sources of disturbance. In this use case, control of the hearable should require little interaction from the user. Control of the strength of selectivity based on biosignals or detectable indicators of speech difficulty (frequent questions, foreign language, strong diamond words) would be optional.

For example, speakers are randomly distributed and move relative to the listener. In addition, there are periodic pauses in speech, new speakers are added or others leave the scene. Perhaps external disturbing sounds, such as music, are relatively loud. The selected speaker is acoustically highlighted and is recognized again after a pause in speech, a change in his or her position or posture.

For example, the hearable recognizes speakers in the user's vicinity. With appropriate controllability (e.g., gaze direction, attention control) the user can select a preferred speaker. The hearable adapts the audio playback according to the user's gaze direction and the selected target sound source, allowing the target sound source to be fully understood even in the presence of noise interference.

Alternatively, if a user is directly addressed by a (previously) non-preferred speaker, the user must at least be audible to ensure natural communication.

Another use case of another embodiment is in an automobile where the user is located in his/her (or her) car. While driving, the user wants to actively direct his/her acoustic attention to specific playback devices such as a navigation device, a radio, or a conversation partner so that he/she can better understand them next to interfering noises (wind, motor, passengers).

For example, the user and the target sound source are placed at fixed locations in a car. The user is stationary relative to the reference system, but the vehicle itself is moving. This requires an adaptive tracking solution. The selected sound source location is acoustically highlighted until the user cancels the selection or a warning signal stops the device functioning.

For example, a user enters a car and the surroundings are detected by the device. With appropriate controllability (e.g. speed awareness), the user can switch between target sound sources and the hearable adapts the sound playback according to the user's gaze direction and the selected target sound source so that the target sound source can be fully understood even in the presence of noise interference.

Alternatively, for example, a traffic-related warning signal interrupts the normal flow and cancels the user's selection. Resumption of normal flow is then performed.

Another use case for further embodiments is live music and concerns guests at a live music event. For example, guests at a concert or live music performance may wish to increase their focus on the performance with the help of those who can hear, and wish to ignore other guests who behave in a disruptive manner. Furthermore, the audio signal itself may be optimized, for example to balance unfavorable listening positions or room acoustic effects.

For example, the user is located among many sources of disturbance. In most cases, however, the performance is relatively large: the target sound sources are located at fixed positions or at least in a predefined area, while the user is highly mobile (e.g., the user may be dancing). The selected sound source position is acoustically emphasized until the user cancels the selection or until a warning signal stops the device functioning.

For example, the user selects a stage area or musicians as the target sound source. With appropriate controllability, the user can define the stage/musician position and the hearable adapts the sound playback according to the user's gaze direction and the selected target sound source so that the target sound source can be fully understood even in the case of noise interference.

Alternatively, for example, warning information (e.g., evacuation in case of an outdoor event, oncoming thunderstorm) and warning signals can interrupt the normal flow and cancel the user's selection, after which there is a resumption of the normal flow.

A further use case of another embodiment is for major events and guests at major events. Thus, at major events (e.g., football stadiums, ice hockey stadiums, large concert halls, etc.), hearables can be used to highlight the voices of family and friends that would otherwise be drowned out by crowd noise.

For example, a big event with many attendees takes place in a stadium or a large concert hall. A group (family, friends, school class) attends the event and is located outside or inside the event location where a large crowd is roaming. One or more children lose eye contact with the group and seek out the group despite high noise levels caused by noise. The user then turns off the voice recognition and the hearable no longer amplifies the sound.

For example, a person in the group selects the lost child's voice on the hearable. The hearable finds the voice. The hearable then amplifies the voice and the user can recover the lost child (faster) based on the amplified voice.

Alternatively, for example, a missing child may also wear a hearable and select a parent's voice. The hearable amplifies the parent's voice. The amplification allows the child to locate the parents. This allows the child to walk back to the parent. Alternatively, for example, a missing child may also wear a hearable and select a parent's voice. The hearable finds the parent's voice(s) and announces the distance to the voice(s). This helps the child find the parent. Optionally, playback of an artificial voice from the hearable may be provided for the distance announcement.

For example, coupling of hearables for selective amplification of sound is provided and sound profiles are stored.

A further use case of a further embodiment is recreational sports and concerns recreational athletes. Listening to music during sports is popular; however, it also involves dangers. Warning signals or other road users may not be heard. In addition to playing music, the hearables can react to warning signals or voices and temporarily interrupt the music playback. In this context, a further use case is sports in small groups. The hearables of a sports group can be connected to ensure good communication during sports while suppressing other disturbing noises.

For example, users can be mobile and possible warning signals are overlapped by many sources of disturbance. The problem is that not all warning signals are potentially relevant to the user (remote sirens in the city, horns in the street). This causes the hearable to automatically stop music playback and acoustically emphasize the communication partner's warning signal until the user cancels the selection. After which the music will play normally.

For example, a user is engaged in sports and is listening to music via a hearable. A warning signal or voice regarding the user is automatically detected and the hearable pauses the music playback. The hearable adapts the audio playback to fully understand the target sound source/acoustic environment. The hearable then continues playing the music automatically (e.g., after the warning signal ends) or as requested by the user.

Or, for example, athletes in a group can connect their hearables, optimizing speech intelligibility between group members and suppressing other distracting noises.

Another use case of another embodiment is the suppression of snoring, which concerns all those who want sleep that is disturbed by snoring. People whose partners are snoring have trouble sleeping as their night rest is disturbed. The hearable provides a sense of security since it suppresses the snoring sounds, guarantees a night's rest and provides security in the home. At the same time, the hearable lets other sounds through (babies crying, alarm sounds, etc.) so that the user is not completely acoustically isolated from the outside world. For example, snore detection is provided.

For example, a user has trouble sleeping due to snoring sounds. By using the hearable, the user can sleep better again, which has a stress reducing effect.

For example, a user wears the hearable while sleeping. He/she switches the hearable to a sleep mode that suppresses all snoring sounds. After falling asleep, he/she turns the hearable off again.

Alternatively, it can suppress other sounds such as construction noise, lawnmower noise, etc. while you sleep.

A further use case of a further embodiment is a diagnostic device for the user in daily life. The hearable records preferences (e.g. which sound source is selected, which attenuation/amplification is selected) and creates a profile with trends over the period of use. This data may make it possible to draw conclusions regarding changes in hearing ability. The aim is to detect hearing loss as soon as possible.

For example, a user carries the device with them in their daily life or the mentioned use case for months or years. The hearable creates analytics based on the selected settings and outputs warnings and recommendations to the user.

For example, a user wears a hearable for an extended period of time (months to years). The device creates an analysis based on the hearing preferences, and the device outputs recommendations and warnings in case of the onset of hearing loss.

A further use case of another embodiment is as a therapeutic device, relating to users with hearing impairment in their daily life. In its role as a transitional device on the way to a hearing device, potential patients are assisted as early as possible, thus treating dementia preventatively. Other possibilities are use as a concentration trainer (e.g. in the case of ADHS), treatment of tinnitus, and stress reduction.

For example, a listener has a hearing problem or attention deficit and temporarily/temporarily uses a hearable as a hearing device. Depending on the hearing problem, it is alleviated by the hearing device, for example, by amplification of all signals (hardness of hearing), high selectivity of preferred sound sources (attention deficit), playback of therapeutic sounds (treatment of tinnitus).

The user, independently or based on a physician's advice, selects the form of treatment, makes the desired adjustments, and the hearable carries out the selected treatment.

Alternatively, the hearable can detect hearing problems from the UC-PRO1 and automatically adapt playback based on the detected problem and notify the user.

A further use case of a further embodiment is public sector work and concerns public sector employees. Public sector employees who are subject to high levels of noise during their work (hospitals, pediatricians, airport counters, educators, restaurant industry, service counters, etc.) wear hearables for better communication of one or a few people's speech with emphasis, for example through stress reduction, and for better safety at work.

For example, employees experience high levels of noise in their work environment and need to talk to clients, patients, or coworkers despite the background noise without being able to switch to a quieter environment. Hospital employees experience high levels of noise from medical devices ringing and beeping (or any other work-related noise) and must still be able to communicate with patients or coworkers. Pediatricians and educators must work amidst noise or screaming children and be able to talk to parents. At an airport counter, employees have difficulty understanding airline passengers when the noise level is high in the airport concourse. A waiter has difficulty hearing customers' orders amidst the noise of a frequented restaurant. Then, for example, the user turns off the voice selection and the hearable no longer amplifies the voice.

For example, a person turns on an equipped hearable. The user sets the hearable to audio selection for nearby voices, and the hearable amplifies the nearby voice, or a few nearby voices, while simultaneously suppressing background noise. The user then better understands the relevant voice.

Alternatively, a person sets the hearable to continuous noise suppression. The user turns on the function to detect and amplify available sound. Thus, the user can continue working with a lower level of noise. When directly addressed from within x meters, the hearable then amplifies the sound. Thus, the user can converse with others with a low level of noise. After the conversation, the hearable goes back to noise suppression mode, and after work, the user turns the hearable off again.

Another use case of another embodiment is passenger transportation, and concerns users of automobiles for the transportation of passengers. For example, users and drivers of passenger transport vehicles want their passengers to be as distracted as possible while driving. Even though passengers are a major source of disturbance, communication with them is necessary from time to time.

For example, the user or driver and the disturbance sources are located at fixed positions in a car. The user is stationary relative to the reference system, but the vehicle itself is moving. This requires an adaptive tracking solution. Thus, passenger sounds and conversations are acoustically suppressed by default unless communication is taking place.

For example, the hearable suppresses noise for the occupants by default. The user can manually override the suppression through appropriate control possibilities (voice recognition, buttons in the vehicle). The hearable then adapts the audio playback according to the selection.

Alternatively, the hearable could detect when the passenger is actively talking to the driver and temporarily halt noise suppression.

Another use case of further embodiments is school and education, involving teachers and students in a class. In one example, the hearable has two roles and the functions of the devices are partially combined. The teacher/speaker's device suppresses interfering noise and amplifies speech/questions from the students. The hearable of the listener may also be controlled via the teacher's device. Thus, particularly important content can be highlighted without the need to speak louder. Students can set their hearables in such a way that they can understand the teacher better and filter out disruptive classmates.

For example, the teacher and students are located in defined areas in a closed space (this is the rule). If all devices are coupled to each other, the relative positions are interchangeable, which simplifies source separation. The selected sound source is acoustically emphasized until the user (teacher/student) cancels the selection or until a warning signal interrupts the functioning of the device.

For example, a teacher or speaker presents content and the device suppresses distracting noise. The teacher wants to hear a student's question and changes the focus of the hearable (automatically or via suitable control possibilities) to the person with the question. After the communication, all sounds are suppressed again. Furthermore, it may be provided that students who feel, for example, that they are being disturbed by their classmates, tune them out acoustically. For example, a student sitting far away from the teacher may have the teacher's voice amplified.

Alternatively, for example, teacher and student devices may be combined. Selectivity of the student device may be temporarily controlled via the teacher device. For particularly important content, the teacher changes the selectivity of the student device to amplify his or her voice.

A further use case of another embodiment is military and concerns soldiers. On the one hand, verbal communication between soldiers in the field is done via radio, on the other hand, via voice and direct contact. Radio is mostly used when communication is between different units and subgroups. A certain radio etiquette is often used. Ejection and direct contact are mostly done to communicate within a unit or group. During a soldier's mission, difficult acoustic conditions (e.g. people screaming, weapon noise, bad weather) may exist that can impair both communication paths. Radio devices with earphones are often part of the soldier's equipment. In addition to the purpose of voice reproduction, they also provide a protection function against higher levels of sound pressure. These devices are often equipped with a microphone to bring environmental signals to the carrier's ear. Active noise suppression is also part of such systems. Enhancement/extension of the functional range allows the soldier's voice-ejection and direct contact in noisy environments by intelligent attenuation of interference noise and selective emphasis of voice with directional reproduction. For this purpose, the relative position of the soldier in the room/field must be known. Furthermore, the voice signal and the interfering noise must be separated from each other spatially and by content. The system must be able to handle high SNR levels as well, from low whispers to screams and explosions. The advantages of such a system are: oral communication between soldiers in noisy environments, preservation of hearing protection, possibility of waiving radio etiquette, and (as it is not a radio solution) interception security.

For example, voice and direct contact between soldiers on a mission can be complicated by interference noise. This problem is now being addressed by short-range and longer-range wireless solutions. New systems enable voice and direct contact in the near field with intelligent spatial emphasis of each speaker and attenuation of ambient noise.

For example, a soldier is on a mission. Voices and sounds are automatically detected and the system amplifies them with simultaneous attenuation of background noise. The system adapts the spatial audio playback to ensure that the target sound source is fully understandable.

Or, for example, the system could know which soldiers are in a group and only let through the voice signals of those group members.

A further use case of a further embodiment relates to security guards and security officers. Thus, for example, hearables can be used to disrupt major events (celebrations, demonstrations) for preemptive detection of crimes. The selectivity of the hearable is controlled by keywords, for example calls for help or calls for violence. This presupposes a content analysis of the audio signal (for example speech recognition).

For example, a security guard is surrounded by many loud sound sources, and the guard and all sound sources may be moving; a person calling for help may not be able to hear them under normal hearing conditions or can only hear them to a limited extent (poor SNR). A manually or automatically selected sound source is acoustically enhanced until the user cancels the selection. Optionally, a virtual sound object is placed at the location/direction of the sound source of interest to make its location (e.g. in case of a one-off call for help) easy to find.

For example, the hearable detects a sound source that has a potential danger. The guard selects (e.g., by selecting on a tablet) which sound source or event he wants to follow. The hearable then adapts the audio playback so that the sound source can be well understood and localized even in the presence of noise interference.

Alternatively, for example, if the target sound source is silent, a localization signal may be placed towards/within the distance of the sound source.

Another use case of another embodiment is communication on stage, and concerns musicians. On stage, in a rehearsal or concert (e.g. band, orchestra, chorus, music), a single instrument (group) may not be heard due to difficult acoustic conditions, even if it is still audible in the other environment. This impairs the dialogue, since important (accompanying) sounds are no longer perceptible. Hearables can highlight these sounds and make them hearable again, thus improving or ensuring the dialogue of the individual musicians. This use can reduce the noise exposure of the individual musicians, preventing hearing loss, for example by attenuating the drums, and also allows the musicians to hear all important things at the same time.

For example, a musician without hearables can no longer hear at least one other voice on stage. In this case, hearables may be used. After the rehearsal or concert is over, the user turns off and then removes the hearables.

In one example, a user turns on the hearable. The user selects one or more desired instruments to be amplified. When creating music together, the selected musical instruments are amplified and thus made audible again by the hearable. After creating the music, the user turns off the hearable again.

In another example, a user turns on a hearable. The user selects a desired instrument that they would like to have the volume reduced. When creating music together, the volume of the selected instrument is reduced by the hearable so that the user can only hear it at a moderate volume.

For example, instrument profiles can be stored in a hearable.

Another use case for further embodiments is source separation as a software module for hearing devices in the ecosystem sense, which concerns the manufacturer of the hearing device, or the user of the hearing device. The manufacturer can use source separation as an additional tool for the hearing device and offer it to the customer. Thus, the hearing device can also benefit from the development. Licensing models for other markets/devices (headphones, mobile phones, etc.) are also conceivable.

For example, a user of a hearing device has difficulty separating different sound sources in a complex hearing situation, for example to focus on a particular speaker. In order to be able to listen selectively without an external additional system (e.g. transfer of signals from a mobile radio set via Bluetooth, selective transfer of signals in a classroom via FM equipment or inductive hearing equipment), the user uses a hearing device with an additional function for selective listening. Thus, without any external effort, the user can focus on individual sound sources through sound source separation. Finally, the user turns off the additional function and continues to listen normally with the hearing device.

For example, a hearing device user acquires a new hearing device with an integrated additional feature for selective hearing. The user sets up the hearing device with the feature for selective hearing. The user then selects a profile (e.g. amplify the loudest/closest source, amplify speech recognition of specific sounds in the personal surroundings (e.g. UC-CE5-Major Events, etc.). The hearing device amplifies each sound source according to the set profile, simultaneously suppressing background noise on request, and the hearing device user hears individual sound sources from a complex auditory scene and not just the "noise"/clutter of sound sources.

Alternatively, the user of the hearing device acquires an add-on feature for selective hearing, e.g. as software for his hearing device. The user installs the add-on feature in his hearing device. Then the user configures the feature for selective hearing in the hearing device. The user selects a profile (amplify the loudest/closest sound source and amplify speech recognition of specific sounds from the personal surroundings (e.g. UC-CE5-Major Events)) and the hearing device amplifies each sound source according to the configured profile, while simultaneously suppressing background noise on demand. In this case the user of the hearing device hears individual sound sources from a complex hearing scene and not just a clutter of "noise"/sound sources.

For example, hearables can provide memorizable voice profiles.

A further use case of a further embodiment concerns professional sports and athletes in competitions. In sports such as biathlon, triathlon, cycling, marathon, etc., professional athletes rely on coaches for information or to communicate with teammates. However, there are situations where they want to protect themselves from loud sounds (shooting in biathlon, loud applause, party horns, etc.) to be able to concentrate. The hearable can be adapted to the respective sport/athlete to allow a fully automatic selection of the relevant sound source (detection of specific voices, volume limiting for typical disturbing noises).

For example, users are very mobile and the type of disturbing noise depends on the sport. Due to intense physical tension, control of the device by the athlete is not possible or only to a limited extent. However, in most sports there are prescribed procedures (biathlon: running, shooting) and important communication partners (trainer, teammates) can be predefined. Noise is suppressed in general or at certain stages of the activity. Communication between the athlete and teammates and coaches is always emphasized.

For example, athletes use hearables that are specially tuned for their type of sport. The hearables fully automatically (pre-tuned) suppress disturbing noises, especially in situations that require high levels of attention in each type of sport. In addition, the hearables fully automatically (pre-tuned) highlight trainers and team members when they are within hearing range.

A further use case of further embodiments is hearing training, for music students, professional musicians, and hobbyist musicians. In music rehearsals (e.g., in orchestras, bands, ensembles, music classes), hearables can be used selectively to track individual voices in a filtered way. Especially at the beginning of a rehearsal, it is useful to hear the final recording of a piece and track your own voice. In some compositions, hearing foreground voices alone does not allow you to hear background voices well. Hearables can selectively highlight voices based on instruments, etc., for more targeted practice.

Music students (who wish to do so) could use hearables to train their hearing abilities and selectively prepare for exams by gradually minimizing individual emphasis until they are finally able to extract individual sounds from complex pieces without aid.

A further possible use case is karaoke, for example when Singstar or similar is not available nearby. The vocals can be suppressed from the music as needed to hear only the instrumental version for signing karaoke.

For example, a musician may begin learning the sounds from a song by listening to a recording of the music via a playback medium such as a CD player. When the user is done practicing, they turn off the hearables again.

In one example, a user turns on the hearable. They select the desired instruments they want to amplify. The hearable will amplify the sound of the musical instruments and reduce the volume of the remaining musical instruments when listening to music, thus allowing the user to better track their own voice.

In another example, a user turns on the hearable. Selects the desired instrument that they wish to suppress. When listening to a song, the audio of the selected song is suppressed so that only the remaining audio is heard. Thus, the user can practice audio on their instrument with other audio without being distracted by audio from the recording.

In an embodiment, the hearable can provide a stored instrument profile.

Another use case for another embodiment is safety at work, which concerns workers in noisy environments. Workers in noisy environments, such as machine halls or construction sites, must protect themselves from the noise, but must also be able to perceive warning signals and communicate with their colleagues.

For example, the user is located in a very loud environment and the desired sound source (warning signal, colleague) may be significantly softer than the disturbing noise. The user may be mobile. However, the disturbing noise is often stationary. Similar to hearing protection, the noise is permanently reduced and the hearable highlights the warning signal completely automatically. Communication with the colleague is ensured by the amplification of the loudspeaker sound source.

For example, a user is at work and uses a hearable as hearing protection. A warning signal (e.g. a fire alarm) is acoustically accentuated, causing the user to stop working if necessary.

Or, for example, the user is at work and uses the hearable as hearing protection. When there is a need to communicate with a colleague, a communication partner is selected and acoustically enhanced with the help of a suitable interface (here, for example, eye control).

Another use case of a further embodiment is source separation as a software module for a live translator, with respect to the user of the live translator. The live translator translates spoken foreign languages in real time and can benefit from an upstream software module for source separation. In particular, in the case of multiple speakers, the software module can extract the target speaker and potentially improve the translation.

For example, the software module is part of a live translator (a dedicated device or an app on a smartphone). The user can, for example, select a target speaker via the device's display. Advantageously, the user and the target sound source do not move or move only slightly during the translation. The selected sound source position is acoustically highlighted, thus potentially improving the translation.

For example, a user may wish to converse in a foreign language or listen to a speaker of a foreign language. The user selects the target speaker via an appropriate interface (e.g., a GUI on the display) and the software module optimizes the recording for further use in the translator.

A further use case of another embodiment is safety in rescue operations, for firefighters, civil protection, police, emergency services. For the mitigating forces, good communication is essential to successfully handle the mission. Despite the high ambient noise, hearing protection is often not possible, since the relief makes communication impossible. For example, firefighters must be able to accurately transmit orders and understand them, despite the loud motor sounds that occur in part via radios. The relief forces are therefore exposed to loud noises that do not allow compliance with hearing protection regulations. On the one hand, the hearable provides hearing protection for the relief forces, and on the other hand still allows communication between the relief forces. Furthermore, with the help of the hearable, the relief forces are not acoustically isolated from the environment when carrying the helmet/protective equipment and can therefore provide better support. They can communicate better and also estimate their own danger better (for example, hear the type of fire that is occurring).

For example, a user is subjected to strong ambient noise and therefore cannot wear hearing protection and must still be able to communicate with others. Hearables are used. After the mission is completed or the dangerous situation has ended, the user removes the hearables again.

For example, a user wears a hearable during a mission. The hearable is turned on. The hearable suppresses background noise and amplifies the speech of nearby colleagues and other speakers (e.g., fire victims).

Or, a user could wear a hearable while on a mission. They would turn the hearable on and it would suppress background noise and amplify the voice of their colleagues over the radio.

Where applicable, the hearable is specially designed to meet structural compliance for operation in accordance with operational specifications. In some cases, the hearable includes an interface to a wireless device.

Although some aspects are described in the context of a device, it will be understood that said aspects also represent a description of a corresponding method, such that blocks or structural components of the device should also be understood as corresponding method steps or features of method steps. Similarly, aspects described in the context of or as method steps also represent a description of corresponding blocks or details or features of the corresponding device. Some or all of the method steps may be performed using hardware devices such as microprocessors, programmable computers, or electronic circuits. In some embodiments, some or some of the most important method steps may be performed by such devices.

Depending on the specific implementation requirements, embodiments of the invention can be implemented in hardware or in software. Implementation may be done using digital storage media, such as floppy disks, DVDs, Blu-ray disks, CDs, ROMs, PROMs, EPROMs, EEPROMs or flash memories, hard disks or any other magnetic or optical memory on which electronically readable control signals are stored that cooperate or can cooperate with a programmable computer system so that the respective method is executed. This is why the digital storage medium may be computer readable.

Thus, some embodiments according to the invention include a data carrier that includes electronically readable control signals that can cooperate with a programmable computer system to perform any of the methods described herein.

In general, embodiments of the invention may be implemented as a computer program product having program code that is effective to perform any of the methods when the computer program product is run on a computer.

The program code may, for example, be stored on a machine-readable carrier.

Other embodiments include a computer program for performing any of the methods described herein, said computer program being stored on a machine readable carrier. In other words, an embodiment of the inventive method is therefore a computer program having a program code for performing any of the methods described herein, when the computer program runs on a computer.

Thus, a further embodiment of the method of the present invention is a data carrier (or digital storage medium or computer readable medium) on which is recorded a computer program for performing any of the methods described herein. The data carrier, digital storage medium or recording medium is typically tangible or non-volatile.

A further embodiment of the inventive method is therefore a data stream or a sequence of signals representing a computer program for performing any of the methods described herein. The data stream or signal sequence can for example be configured to be transmitted over a data communication link, for example via the Internet.

Further embodiments include a processing unit, e.g., a computer or programmable logic device, configured or adapted to perform any of the methods described herein.

A further embodiment includes a computer having installed thereon a computer program for performing any of the methods described herein.

Further embodiments according to the invention include a device or system configured to transmit a computer program for performing at least one of the methods described herein to a receiver. The transmission may be, for example, electronic or optical. The receiver may be, for example, a computer, a mobile device, a memory device, or a similar device. The device or system may, for example, include a file server for transmitting the computer program to the receiver.

In some embodiments, a programmable logic device (e.g., a field programmable gate array, FPGA) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field programmable gate array may cooperate with a microprocessor to perform any of the methods described herein. In general, the methods are performed in some embodiments by any hardware device. The hardware device may be any universally applicable hardware, such as a computer processor (CPU), or may be method-specific hardware, such as an ASIC.

The above-described embodiments merely represent illustrations of the principles of the present invention. It is understood that others skilled in the art will appreciate modifications and variations of the configurations and details described herein. As such, it is intended that the present invention be limited only by the scope of the following claims and not by the specific details presented herein by way of the description and discussion of the embodiments.

References [1] V. Valimaki, A. Franck, J. Ramo, H. Gamper, and L. Savioja, “Assisted listening using a headset: Enhancing audio perception in real, augmented, and virtual environment,” IEEE Signal Processing Magazine, volume 32, no. 2, pp. 92-99, March 2015

[2] K. Brandenburg, E. Cano, F. Klein, T. Kollmer, H. Lukashevich, A. Neidhardt, U. Sloma, and S. Werner, “Plausible augmentation of auditory scenes using dynamic binaural synthesis for personalized auditory “realities,” in Proc. of AES International Conference on Audio for Virtual and Augmented Reality, August 2018

[3] S. Argentieri, P. Dans, and P. Soures, “A survey on sound source localization in robotics: From binural to array processing methods,” Computer Speech Language, volume 34, no. 1, pp. 87-112, 2015

[4]D. FitzGerald, A. Liutkus, and R. Badeau, “Projection-based demixing of spatial audio,” IEEE/ACM Trans. on Audio, Speech, and Language Processing, volume 24, no. 9, pp. 1560-1572, 2016

[5] E. Cano, D. FitzGerald, A. Liutkus, M. D. Plumbley, and F. Stoter, “Musical source separation: An introduction,” IEEE Signal Processing Magazine, volume 36, no. 1, pp. 31-40, January 2019

[6] S. Gannot, E. Vincent, S. Markovich-Golan, and A. Ozerov, “A consolidated perspective on multimicrophone speech enhancement and source separation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, volume 25, no. 4, pp. 692-730, April 2017

[7] E. Cano, J. Nowak, and S. Grollmisch, “Exploring sound source separation for acoustic condition monitoring in industrial scenarios,” in Proc. of 25th European Signal Processing Conference (EUSIPCO), August 2017, pp. 2264-2268

[8] T. Gerkmann, M. Krawczyk-Becker, and J. Le Roux, “Phase processing for single-channel speech enhancement: History and recent advances,” IEEE Signal Processing Magazine, volume 32, no. 2, pp. 55-66, March 2015

[9] E. Vincent, T. Virtanen, and S. Gannot, Audio Source Separation and Speech Enhancement. Wiley, 2018

[10]D. Matz, E. Cano, and J. Abesser, “New sonorities for early jazz recordings using sound source separation and automatic mixing tools,” in Proc. of the 16th International Society for Music Information Retrieval Conference. Malaga, Spain: ISMIR, October 2015, pp. 749-755

[11] S. M. Kuo and D. R. Morgan, “Active noise control: a tutorial review,” Proceedings of the IEEE, volume 87, no. 6, pp. 943-973, June 1999

[12] A. McPherson,R. Jack, and G. Moro, “Action-sound latency: Are our tools fast enough?” in Proceedings of the International Conference on New Interfaces for Musical Expression, July 2016

[13]C. Rottondi, C. Chafe, C. Allocchio, and A. Sarti, “An overview on networked music performance technologies,” IEEE Access, volume 4, pp. 8823-8843, 2016

[14] S. Liebich, J. Fabry, P. Jax, and P. Vary, “Signal processing challenges for active noise cancellation headsphones,” in Speech Communication; 13th ITG-Symposium, October 2018, pp. 1-5

[15] E. Cano, J. Liebetrau, D. Fitzgerald, and K. Brandenburg, “The dimensions of perceptual quality of sound source separation,” in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), April 2018, pp. 601-605

[16] P. M. Delgado and J. Herre, “Objective assessment of spatial audio quality using directional loudness maps,” in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2019, pp. 621-625

[17] C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “An algorithm for intelligence prediction of time-frequency weighted noise speech,” IEEE Transactions on Audio, Speech, and Language Processing, volume 19, no. 7, pp. 2125-2136, September 2011

[18] M. D. Plumbley, C. Kroos, J. P. Bello, G. Richard, D. P. Ellis, and A. Mesaros, Proceedings of the Detection and Classification of Acoustic Scenes and Events 2018 Workshop (DCASE2018). Tampere University of Technology. Laboratory of Signal Processing, 2018

[19] R. Serizel, N. Turpault, H. Eghbal-Zadeh, and A. Parag Shah, “Large-Scale Weakly Labeled Semi-Supervised Sound Event Detection in Domestic Environments,” July 2018, submitted to DCASE2018 Workshop

[20] L. JiaKai, “Mean teacher convolution system for dcase 2018 task 4,” DCASE2018 Challenge, Tech. Rep. ,September 2018

[21] G. Parascandolo, H. Huttunen, and T. Virtanen, “Recurrent neural networks for polyphonic sound event detection in real life recordings,” in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), March 2016, pp. 6440-6444

[22] E. C, Cakir and T. Virtanen, “End-to-end polyphonic sound event detection using convolutional recurring neural networks with learned time-frequency representation input,” in Proc. of International Joint Conference on Neural Networks (IJCNN), July 2018, pp. 1-7

[23] Y. Xu, Q. Kong, W. Wang, and M. D. Plumbley, “Large-Scale Weakly Supervised Audio Classification Using Gated Convolutional Neural Network,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada, 2018, pp. 121-125

[24] B. Frenay and M. Verleysen, “Classification in the presence of label noise: A survey,” IEEE Transactions on Neural Networks and Learning Systems, volume 25, no. 5, pp. 845-869, May 2014

[25] E. Fonseca, M. Plakal, D. P. W. Ellis, F. Font, X. Favory, and X. Serra, “Learning sound event classifiers from web audio with noise labels,” in Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK, 2019

[26] M. Dorfer and G. Widmer, “Training general-purpose audio tagging networks with noisy labels and iterative self-verification,” in Proceedings of the Detection and Classification of Acoustic Scenes and Events 2018 Workshop (DCASE2018), Surrey, UK, 2018

[27] S. Adavanne, A. Politis, J. Nikunen, and T. Virtanen, “Sound event localization and detection of overlapping sources using convolutional recurring neural networks,” IEEE Journal of Selected Topics in Signal Processing, pp. 1-1, 2018

[28] Y. Jung, Y. Kim, Y. Choi, and H. Kim, “Joint learning using denoising variational autoencoders for voice activity detection,” in Proc. of Interspeech, September 2018, pp. 1210-1214

[29] F. Eyben, F. Weninger, S. Squartini, and B. Schuller, “Real-life voice activity detection with LSTM recurring neural networks and an application to hollywood movies,” in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing, May 2013, pp. 483-487

[30] R. Zazo-Candil, T. N. Sainath, G. Simko, and C. Parada, “Feature learning with raw-waveform CLDNNs for voice activity detection,” in Proc. of INTERSPEECH, 2016

[31] M. McLaren, Y. Lei, and L. Ferrer, “Advances in deep neural network approaches to speaker recognition,” in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), April 2015, pp. 4814-4818

[32]D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-vectors: Robust DNN embeddings for speaker recognition,” in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), April 2018, pp. 5329-5333

[33] M. McLaren, D. Castan, M. K. Nandwana, L. Ferrer, and E. Yilmaz, “How to train your speaker embeddings extractor,” in Odyssey, 2018

[34] S. O. Sadjadi, J. W. Pelecanos, and S. Ganapathy, “The IBM speaker recognition system: Recent advances and error analysis,” in Proc. of Interspeech, 2016, pp. 3633-3637

[35] Y. Han, J. Kim, and K. Lee, “Deep convolutional neural networks for predominant instrument recognition in polyphonic music,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, volume 25, no. 1, pp. 208-221, January 2017

[36] V. Lonstanlen and C. -E. Cella, “Deep convolutional networks on the pitch spiral for musical instrument recognition,” in Proceedings of the 17th International Society for Music Information Retrieval Conference. New York, USA: ISMIR, 2016, pp. 612-618

[37] S. Gururani, C. Summers, and A. Lerch, “Instrument activity detection in polyphonic music using deep neural networks,” in Proceedings of the 19th International Society for Music Information Retrieval Conference. Paris, France: ISMIR, September 2018, pp. 569-576

[38] J. Schlutter and B. Lehner, “Zero mean convolutions for level-invariant singing voice detection,” in Proceedings of the 19th International Society for Music Information Retrieval Conference. Paris, France: ISMIR, September 2018, pp. 321-326

[39] S. Delikaris-Manias, D. Pavlidi, A. Mouchtaris, and V. Pulkki, “DOA estimation with histogram analysis of spatially constrained active intensity vectors,” in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), March 2017, pp. 526-530

[40] S. Chakrabarty and E. A. P. Habets, “Multi-speaker DOA estimation using deep convolutional networks trained with noise signals,” IEEE Journal of Selected Topics in Signal Processing, volume 13, no. 1, pp. 8-21, March 2019

[41] X. Li, L. Girin, R. Horaud, and S. Gannot, “Multiple-speaker localization based on direct-path features and likelihood maximization with spatial sparsity regulation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, volume 25, no. 10, pp. 1997-2012, October 2017

[42] F. Grondin and F. Michael, “Lightweight and optimized sound source localization and tracking methods for open and closed microphone array configurations,” Robotics and Autonomous Systems, volume 113, pp. 63-80, 2019

[43]D. Yook, T. Lee, and Y. Cho, “Fast sound source localization using two-level search space clustering,” IEEE Transactions on Cybernetics, volume 46, no. 1, pp. 20-26, January 2016

[44]D. Pavlidi, A. Griffin, M. Puigt, and A. Mouchtaris, “Real-time multiple sound source localization and counting using a circular microphone array,” IEEE Transactions on Audio, Speech, and Language Processing, volume 21, no. 10, pp. 2193-2206, October 2013

[45] P. Vecchiotti, N. Ma, S. Squartini, and G. J. Brown, “End-to-end binural sound localization from the raw waveform,” in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2019, pp. 451-455

[46] Y. Luo, Z. Chen, and N. Mesgarani, “Speaker-independent speech separation with deep attractor network,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, volume 26, no. 4, pp. 787-796, April 2018

[47] Z. Wang, J. Le Roux, and J. R. Hershey, “Multi-channel deep clustering: Discriminative spectral and spatial embeddings for speaker-independent speech.” separation,” in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), April 2018, pp. 1-5

[48] G. Naithani, T. Barker, G. Parascandolo, L. BramslLw, N. H. Pontoppidan, and T. Virtanen, “Low latency sound source separation using convolutional recurring neural networks,” in Proc. of IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), October 2017, pp. 71-75

[49] M. Sunohara, C. Haruta, and N. Ono,”Low-latency real-time blind source separation for hearing aids based on time-domain implementation of online independent vector analysis with truncation of non-causal components,” in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), March 2017, pp. 216-220

[50] Y. Luo and N. Mesgarani, “TaSNet: Time-domain audio separation network for real-time, single-channel speech separation,” in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), April 2018, pp. 696-700

[51] J. Chua, G. Wang, and W. B. Kleijn, “Convolutive blind source separation with low latency,” in Proc. of IEEE International Workshop on Acoustic Signal Enhancement (IWAENC), September 2016, pp. 1-5

[52] Z. Rafii, A. Liutkus, F. Stoter, S. I. Mimilakis, D. FitzGerald, and B. Pardo, “An overview of lead and accommodation separation in music,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, volume 26, no. 8, pp. 1307-1335, August 2018

[53] F. -R. Stoter, A. Liutkus, and N. Ito, “The 2018 signal separation evaluation campaign,” in Latent Variable Analysis and Signal Separation, Y. Deville, S. Gannot, R. Mason, M. D. Plumbley, and D. Ward, Eds. Cham: Springer International Publishing, 2018, pp. 293-305

[54] J. -L. Durrieu, B. David, and G. Richard, “A musically stimulated midlevel representation for pitch estimation and musical audio source separation,” Selected Topics in Signal Processing, IEEE Journal of, volume 5, no. 6, pp. 1180-1191, October 2011

[55] S. Uhlich, M. Porcu, F. Giron, M. Enenkl, T. Kemp, N. Takahashi, and Y. Mitsufuji, “Improving music source separation based on deep neural networks through data augmentation and network blending,” in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017

[56] P. N. Samarasinghe, W. Zhang, and T. D. Abhayapala, “Recent advances in active noise control inside automobile cabins: Toward quieter cars,” IEEE Signal Processing Magazine, volume 33, no. 6, pp. 61-73, November 2016

[57] S. Papini, R. L. Pinto, E. B. Medeiros, and F. B. Coelho, “Hybrid approach to noise control of industrial exhaust systems,” Applied Acoustics, volume 125, pp. 102-112, 2017

[58] J. Zhang, T. D. Abhayapala, W. Zhang, P. N. Samarasinghe, and S. Jiang, “Active noise control over space: A wave domain approach,” IEEE/ACM Transactions on Audio, Speech, and Language. Processing, volume 26, no. 4, pp. 774-786, April 2018

[59] X. Lu, Y. Tsao, S. Matsuda, and C. Hori, “Speech enhancement based on deep denoising autoencoder,” in Proc. of Interspeech, 2013

[60] Y. Xu, J. Du, L. Dai, and C. Lee, “A regression approach to speech enhancement based on deep neural networks,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, volume 23, no. 1, pp. 7-19, January 2015

[61] S. Pascual, A. Bonafonte, and J. Serra, “SEGAN: speech enhancement generative adversarial network,” in Proc. of Interspeech, August 2017, pp. 3642-3646

[62] F. Weninger, H. Erdogan, S. Watanabe, E. Vincent, J. Le Roux, J. R. Hershey, and B. Schuller, “Speech enhancement with LSTM recurrent neural networks and its application to noise-robust ASR,” in Latent Variable Analysis and Signal Separation, E. Vincent, A. Yeredor, Z. Koldovsky, and P. Tichavsky, Eds. Cham: Springer International Publishing, 2015, pp. 91-99

[63] H. Wierstorf, D. Ward, R. Mason, E. M. Grais, C. Hummersone, and M. D. Plumbley, “Perceptual evaluation of source separation for remixing music,” in Proc. of Audio Engineering Society Convention 143, October 2017

[64] J. Pons, J. Janer, T. Rode, and W. Nogueira, “Remixing music using source separation algorithms to improve the musical experience of cochlear implant users,” The Journal of the Acoustical Society of America, volume 140, no. 6, pp. 4338-4349, 2016

[65] Q. Kong, Y. Xu, W. Wang, and M. D. Plumbley, “A joint separation-classification model for sound event detection of weakly labeled data,” in Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), March 2018

[66] T. v. Neumann, K. Kinoshita, M. Delcroix, S. Araki, T. Nakatani, and R. Haeb-Umbach, “All-neural online source separation, counting, and diarization for meeting analysis,” in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2019, pp. 91-95

[67] S. Gharib, K. Drossos, E. Cakir, D. Serdyuk, and T. Virtanen, “Unsupervised adversarial domain adaptation for acoustic scene classification,” in Proceedings of the Detection and Classification of Acoustic Scenes and Events Workshop (DCASE), November 2018, pp. 138-142

[68] A. Mesaros, T. Heittola, and T. Virtanen, “A multi-device dataset for urban acoustic scene classification,” in Proceedings of the Detection and Classification of Acoustic Scenes and Events Workshop, Surrey, UK, 2018

[69] J. Abesser, M. Gotze, S. Kuhnlenz, R. Grafe, C. Kuhn, T. Clauss, H. Lukashevich, “A Distributed Sensor Network for Monitoring Noise Level and Noise Sources in Urban Environments,” in Proceedings of the 6th IEEE International Conference on Future Internet of Things and Cloud (FiCloud), Barcelona, Spain, pp. 318-324. ,2018

[70] T. Virtanen, M. D. Plumbley, D. Ellis (Eds.), “Computational Analysis of Sound Scenes and Events,” Springer, 2018

[71] J. Abesser, S. Ioannis Mimilakis, R. Grafe, H. Lukashevich, “Acoustic scene classification by combining autoencoder-based dimensionality reduction and convolutional neural net-works,” in Procedures of the 2nd DCASE Workshop on Detection and Classification of Acoustic Scenes and Events, Munich, Germany, 2017

[72] A. Avni, J. Ahrens, M. Geierc, S. Spors, H. Wierstorf, B. Rafaely, “Spatial perception of sound fields recorded by spherical microphone arrays with varying spatial resolution,” Journal of the Acoustic Society of America, 133(5), pp. 2711-2721, 2013

[73] E. Cano, D. FitzGerald, K. Brandenburg, “Evaluation of quality of sound source separation algorithms: Human perception vs quantitative metrics,” in Proceedings of the 24th European Signal Processing Conference (EUSIPCO), pp. 1758-1762, 2016

[74] S. Marchand, “Audio scene transformation using informed source separation,” The Journal of the Acoustical Society of America, 140(4), p. 3091, 2016

[75] S. Grollmisch, J. Abesser, J. Liebetrau, H. Lukashevich, “Sounding industry: Challenges and datasets for industrial sound analysis (ISA),” in Proceedings of the 27th European Signal Processing Conference (EUSIPCO) (submitted), A Coruna, Spain, 2019

[76] J. Abesser, M. Muller, “Fundamental frequency contour classification: A comparison between hand-crafted and CNN-based features,” in Proceedings of the 44th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2019

[77] J. Abesser, S. Balke, M. Muller, “Improving bass salience estimation using label propagation and transfer learning,” in Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR), Paris, France, pp. 306-312, 2018

[78] C. -R. Nagar, J. Abesser, S. Grollmisch, “Towards CNN-based acoustic modeling of seventh chords for recognition chord recognition,” in Proceedings. of the 16th Sound & Music Computing Conference (SMC) (submitted), Malaga, Spain, 2019

[79] J. S. Gomez, J. Abesser, E. Cano, “Jazz solo instrument classification with convolutional neural networks, source separation, and transfer learning”, in Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR), Paris, France, pp. 577-584, 2018

[80] J. R. Hershey, Z. Chen, J. Le Roux, S. Watanabe, “Deep clustering: Discriminative embeddings for segmentation and separation,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 31-35, 2016

[81] E. Cano, G. Schuller, C. Dittmar, “Pitch-informed solo and accommodation separation towers its use in music education applications”, EURASIP Journal on Advances in Signal Processing, 2014:23, pp. 1-19

[82] S. I. Mimilakis, K. Drossos, J. F. Santos, G. Schuller, T. Virtanen, Y. Bengio, “Monaural Singing Voice Separation with Skip-Filtering Connections and Recurrent Inference of Time-Frequency Mask,” in Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Calgary, Canada, S. 721-725, 2018

[83] J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, M. Ritter, “Audio Set: An ontology and human-labeled dataset for audio events,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, USA, 2017

[84] Kleiner, M. "Acoustics and Audio Technology,". 3rd ed. USA: J. Ross Publishing, 2012

[85] M. Dickreiter, V. Dittel, W. Hoeg, M. Wohr, M. ,,Handbuch der Tonstudiotechnik,” A.medienakademie (Eds). 7th edition, Vol.1., Munich: K.G. Saur Verlag, 2008

[86] F. Muller, M. Karau. ,,Transparent hearing,” in: CHI ,02 Extended Abstracts on Human Factors in Computing Systems (CHI EA '02), Minneapolis, USA, pp. 730-731, April 2002

[87] L. Vieira. “Super hearing: a study on virtual prototyping for healables and hearing aids,” Master Thesis, Aalborg University, 2018. Available: https://projecter. aau. dk/projecter/files/287515943/MasterThesis_Luis. pdf

[88] Sennheiser, “AMBEO Smart Headset,” [Online]. Available:
https://de-de. sennheiser. com/finalstop [Accessed: March 1, 2019]

[89] Orosound “Tilde Earphones” [Online]. Available:
https://www. orosound. com/tilde-earphones/ [Accessed; March 1, 2019]

[90] Brandenburg, K. , Cano Ceron, E. , Klein, F. , Kollmer, T. , Lukashevich, H. , Neidhardt, A. , Nowak, J. , Sloma, U. , and Werner, S. ,,,Personalized auditorium reality,” in 44. Jahrestagung fur Akustik (DAGA), Garching bei Munchen, Deutsche Gesellschaft fur Akustik (DEGA), 2018

[91] US 2015 195641 A1, Application date: January 6, 2014; published on July 9, 2015.

Claims (20)

an analyzer (152) for determining a plurality of binaural room impulse responses;
a loudspeaker signal generator (154) for generating at least two loudspeaker signals in response to the plurality of binaural room impulse responses and in response to a source signal of at least one sound source;
Equipped with
the analyzer (152) is configured to determine the plurality of binaural room impulse responses such that each of the plurality of binaural room impulse responses takes into account an effect resulting from a user wearing headphones.
system. The system comprises the headphones,
the headphones are configured to output the at least two loudspeaker signals;
The system of claim 1 . The headphones include two headphone capsules and at least one microphone for measuring sound in each of the two headphone capsules;
the at least one microphone is configured to measure the sound and is disposed in each of the two headphone capsules;
the analyzer (152) is configured to perform the determination of the plurality of binaural room impulse responses using the measurements of the at least one microphone in each of the two headphone capsules.
3. A system according to claim 1 or 2. the at least one microphone in each of the two headphone capsules is configured to generate one or more recordings of a sound situation in a reproduction room prior to reproduction of the at least two loudspeaker signals by the headphones, determine an estimate of a first sound signal of at least one sound source from the one or more recordings, and determine a binaural room impulse response of the plurality of binaural room impulse responses for the sound source in the reproduction room.
The system of claim 3. the at least one microphone in each of the two headphone capsules is configured to generate, during playback of the at least two loudspeaker signals by the headphones, one or more further recordings of the sound situation in the reproduction room, subtract an augmented signal from these one or more further recordings, determine the estimate of the first sound signal from one or more sound sources, and determine the binaural room impulse response of the plurality of binaural room impulse responses for the sound sources in the reproduction room.
The system of claim 4. the analyzer (152) is configured to determine an acoustic room characteristic of the reproduction room and to adapt the plurality of binaural room impulse responses in response to the acoustic room characteristic.
6. A system according to claim 4 or 5. the at least one microphone is disposed in each of the two headphone capsules to measure sounds near the entrance of the ear canal;
A system according to any one of claims 4 to 6. the system includes one or more further microphones outside the two headphone capsules for measuring the sound situation in the reproduction room;
A system according to any one of claims 4 to 7. the headphones include a headphone band , and at least one of the one or more further microphones is disposed on the headphone band .
The system of claim 8. the loudspeaker signal generator (154) is configured to generate the at least two loudspeaker signals by convolving each of the plurality of binaural room impulse responses with a source signal of a plurality of one or more source signals.
A system according to any one of claims 1 to 9. the analyzer (152) is configured to determine at least one of the plurality of binaural room impulse responses in response to a movement of the headphones.
A system according to any one of claims 1 to 10. The system includes a sensor for determining movement of the headphones.
The system of claim 11. 1. A system for assisting selective hearing, comprising:
a detector (110) for detecting source signal portions of one or more sound sources using at least two received microphone signals of the auditory environment;
a position determiner (120) for assigning position information to each of said one or more sound sources;
a sound type classifier (130) for assigning a sound signal type to the sound source signal portions of each of the one or more sound sources;
a signal portion modifier (140) for modifying the source signal portion of the at least one sound source of the one or more sound sources depending on the sound signal type of the source signal portion of the at least one sound source to obtain a modified sound signal portion of the at least one sound source;
Further equipped with
said analyzer (152) and said loudspeaker signal generator (154) together form a signal generator (150);
the analyzer (152) of the signal generator (150) is configured to generate the plurality of binaural room impulse responses, the plurality of binaural room impulse responses being a plurality of binaural room impulse responses for each of the one or more sound sources, dependent on the position information of the sound sources and on a user's head orientation;
the loudspeaker signal generator (154) of the signal generator (150) is configured to generate the at least two loudspeaker signals in response to the plurality of binaural room impulse responses and in response to the modified audio signal portion of the at least one sound source.
The system of claim 5 . the detector (110) is configured to detect the source signal portions of the one or more sound sources by using a deep learning model.
The system of claim 13. the location determiner (120) is configured to determine the location information for each of the one or more sound sources in response to captured images or recorded video.
15. A system according to claim 13 or 14. the signal portion modifier (140) is configured to select the at least one sound source whose sound source signal portion is to be modified in response to a previously learned user scenario, and to modify the sound source in response to the previously learned user scenario.
A system according to any one of claims 13 to 15. The system includes a remote device (190) including the detector (110), the location determiner (120), the sound type classifier (130), the signal portion modifier (140), and the signal generator (150);
the remote device is spatially separated from the headphones;
17. A system according to any one of claims 13 to 16. The system of claim 17, wherein the remote device (190) is a smartphone. 1. A method comprising:
determining, by an analyzer (152) of the system , a plurality of binaural room impulse responses;
generating at least two loudspeaker signals by a loudspeaker signal generator (154) of the system in response to the plurality of binaural room impulse responses and in response to a source signal of at least one sound source;
Including,
the plurality of binaural room impulse responses are determined by the analyzer (152) such that each of the plurality of binaural room impulse responses takes into account an effect due to a user wearing headphones.
method. 20. A computer program having a program code for causing the execution of the method according to claim 19 , when the computer program is executed by a computer or signal processor .