[go: up one dir, main page]

EP3225037A1 - Method and apparatus for generating a directional sound signal from first and second sound signals - Google Patents

Method and apparatus for generating a directional sound signal from first and second sound signals

Info

Publication number
EP3225037A1
EP3225037A1 EP14771598.1A EP14771598A EP3225037A1 EP 3225037 A1 EP3225037 A1 EP 3225037A1 EP 14771598 A EP14771598 A EP 14771598A EP 3225037 A1 EP3225037 A1 EP 3225037A1
Authority
EP
European Patent Office
Prior art keywords
frequency
directional
signal
generating
sound
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP14771598.1A
Other languages
German (de)
French (fr)
Other versions
EP3225037B1 (en
Inventor
Hauke Krüger
Bernd Geiser
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Binauric Se
Original Assignee
Binauric Se
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Binauric Se filed Critical Binauric Se
Publication of EP3225037A1 publication Critical patent/EP3225037A1/en
Application granted granted Critical
Publication of EP3225037B1 publication Critical patent/EP3225037B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04RLOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R3/00Circuits for transducers, loudspeakers or microphones
    • H04R3/005Circuits for transducers, loudspeakers or microphones for combining the signals of two or more microphones
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04RLOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R2430/00Signal processing covered by H04R, not provided for in its groups
    • H04R2430/20Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic
    • H04R2430/25Array processing for suppression of unwanted side-lobes in directivity characteristics, e.g. a blocking matrix

Definitions

  • the present invention generally relates to the field of sound acquisition. More particularly, the present invention relates to a method and an apparatus for generating a directional sound signal from first and second sound signals, which are generated by a first and a second microphone, which are separated by a distance.
  • microphone arrays proved to be useful. They are designed to attenuate possible noise and interference components while retaining the desired source signal by exploiting different spatial (or directional) characteristics of the different signal sources (see, e.g., J. Benesty, J. Chen, and Y. Huang, “Microphone Array Signal Processing,” Heidelberg: Springer, 2008 for an overview).
  • a simple, yet efficient approach is the first-order differential microphone array described in G. Elko and A.-T. N. Pong, "A simple adaptive first-order differential microphone,” in IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 169 to 172, October 1995.
  • This microphone array which is schematically and exemplarily shown in Fig.
  • a possible target device is a wireless loudspeaker with two integrated miniature digital micro-electromechanical system (MEMS) microphone cap- sules which facilitate handsfree audio communication.
  • MEMS micro-electromechanical system
  • Fig. 1 shows schematically and exemplarily a differential microphone array according to G. Elko and A.-T. N. Pong.
  • Two closely spaced omnidirectional microphones M 1 and M2 are used to capture the acoustic environment.
  • the corresponding digital signals x t (k) and x 2 (fe) are sampled with a rate of f s .
  • the signals x f (fc) and x (fc) can be interpreted as "forward and backward facing cardioid" signals as the respective directional responses of Eqs. (1 ) and (2) form cardioid shapes (see Fig . 3 in G. Elko and A.-T. N. Pong).
  • a method for generating a directional sound signal from first and second sound signals, which are generated by a first and a second microphone, which are separated by a distance comprises:
  • the generating of the second differential sound signal comprises generating a difference signal of the first and the second sound signals and a frequency-selective processing that depends on a steering angle, which indicates a desired direction of maximum attenuation of the frequency-dependent directional response pattern, wherein the frequency-selective processing adjusts the actual direction of maximum attenuation of the frequency-dependent directional response pattern to correspond to the steering angle substantially independent of frequency over the frequency range of the directional sound signal.
  • the present invention is based on the idea that by employing these steps, a (substantially) frequency invariant notch characteristic can be obtained even for larger microphone distances. A larger distance also helps to confine the noise gain of the array. Therefore, the array becomes practically usable even for higher sampling rates (e.g., 16kHz).
  • the term "difference signal" as used herein also includes the case where one or both of the first and the second sound signals is/are further temporally delayed, for example, by means of a fractional delay filter h T (k), as described in section 2 above.
  • the frequency-selective processing comprises weighting the difference signal with an approximated steering factor that is independent of frequency to generate a weighted difference signal and correcting for the approximation by adding a correction signal that is generated from the difference signal in dependence of frequency and the steering angle.
  • the generation of the correction signal comprises applying two separate operations, one being dependent on frequency and independent of the steering angle and one being dependent on the steering angle but independent of frequency.
  • the generation of the correction signal comprises filtering the difference signal with a filter that is dependent on frequency and independent of the steering angle to generate a filtered difference signal. It is further preferred that the generation of the correction signal further comprises weighting the filtered difference signal with a factor that is dependent on the steering angle and independent of frequency.
  • the factor is determined by using a polynomial approximation that is evaluated with the steering angle.
  • the method further comprises filtering the directional sound signal with a low-pass filter to generate a filtered directional sound signal.
  • the approximated steering factor for a time instance is adapted for the following time instance by adding an adaptation value that is scaled by a stepsize param- eter, wherein the stepsize parameter is adapted in dependence of estimated energies of coherent and incoherent sound components.
  • the energy of the incoherent sound components is approximated by the estimated short-term energy of the directional sound signal and the energy of the coherent sound components is approximated by a fraction of the estimated short-term energy of the difference signal.
  • the method further comprises estimating a relative gain of the first and the second microphone and equalizing power levels of the first and the second microphone based on the relative gain.
  • the relative gain is determined based on recursively estimated variances of the first and second sound signals.
  • the first and the second microphone are omnidirectional microphones.
  • an apparatus for generating a directional sound signal from first and second sound signals, which are generated by a first and a second microphone, which are separated by a distance comprising:
  • first generating means for generating first and second differential sound signals based on the first and second sound signals
  • generating of the second differential sound signal comprises generating a difference signal of the first and the second sound signals and a frequency-selective processing that depends on a steering angle, which indicates a desired direction of max- imum attenuation of the frequency-dependent directional response pattern, wherein the frequency-selective processing adjusts the actual direction of maximum attenuation of the frequency-dependent directional response pattern to correspond to the steering angle substantially independent of frequency over the frequency range of the directional sound signal.
  • a first and a second microphone which are separated by a distance and generate first and second sound signals
  • Fig . 1 shows schematically and exemplarily a differential microphone array according to G. Elko and A.-T. N. Pong,
  • Fig . 2 illustrates the optimal steering factor vs. a linear approximation
  • H(a), (l>) 2j - e- J ⁇ (c ⁇ +1 sin I—— (1 + cos0) 1— a ⁇ sin I—— (1— cos0)
  • the steering angle a should, ideally, be adapted to match the interference or noise incidence angle ⁇ . This is discussed in section 6.
  • the factors ⁇ ( ⁇ ) and ⁇ ( ⁇ ) can be computed by marginalization of the 2-dimensional function ⁇ ( ⁇ , ⁇ ) and appropriate normalization.
  • the factor ⁇ ( ⁇ ) can now be regarded as the frequency response of a fixed filter. It can be transformed to the time domain via periodic extension, inverse DFT, cyclic shifting (to enforce causality) and an appropriate shortening to a desired length.
  • the resulting FIR filter coefficients 3 ⁇ 4 DEQ (fe), e.g., of order 16 are independent of the steering angle .
  • the angular dependency is then reintroduced with a polynomial approximation (e.g., order 4) of the second factor ⁇ ( ⁇ ) after a variable transformation from to lin (a), i.e., ⁇ ⁇ ( ⁇ ⁇ ( ⁇ )) ⁇ ⁇ ( ⁇ ( ⁇ ⁇ ⁇ ))-
  • the distorted notch curve of the standard differential array (Fig. 3 (a)) not only limits the ability to suppress interfering sound sources, but it can even compromise the accurate NLMS adaptation of the steering angle (see section 6).
  • a smaller microphone distance D could be used so that the product ⁇ in Eq. (6) remains sufficiently small.
  • the downside of this approach is a stronger highpass effect of the array which, in turn, requires heavier output equalization with a more pronounced lowpass filter h (k). In a real system, this leads to a higher amplification of the microphone noise, particularly at low frequencies.
  • D 1.8 cm
  • less than half of the original distance is required to obtain a comparably straight directional response. This, however, comes at the cost of a significantly increased noise gain (+10dB) over a wide frequency range.
  • the goal of a notch adaptation algorithm is to automatically align the notch angle of the differential array with the incidence angle ⁇ of the (main) interferer.
  • the standard approach to adapt the factor a (or lin if directional equalization is used) and therefore the notch angle is the (normalized) least mean square (NLMS) algorithm.
  • the goal here is to minimize the power of the output signal y(fe), i.e. where, usually, 0 ⁇ a ⁇ 1, i.e., 180° > a ⁇ 90° is enforced.
  • the stepwise update this method is (e.g., H. Puder) ⁇
  • This equation represents the error signal of a single-tap adaptive filter with a noisy input.
  • the noise signal n(k) is due to the incoherent (ambient) noise that cannot be suppressed.
  • the coherent contribution to y(fe) should ideally be zero.
  • an error signal e(k) appears at the output.
  • the optimal (adaptive) stepsize parameter is
  • the best approximation of £ ⁇ n 2 (fc) ⁇ is the level ⁇ 2 of the microphone array's output y(fe) while for £ ⁇ e 2 (fc) ⁇ , the assumption of a fixed attenuation factor for the backward cardioid signal is made, i.e. £ ⁇ e 2 (fc) ⁇ ⁇ ⁇ ⁇ i 2 b .
  • 0.01 (assumed attenuation of 20 dB) and the adaptive stepsize parameter is hence with the recursively estimated short term powers ⁇ , 2 and 8 , which leads to the new NLMS update rule
  • a(k) a(k - 1) + ⁇ — — - x b (fe) - y(fe)
  • the adaptation can be deliberately slowed down by the factor 0 ⁇ ⁇ 1 to avoid artifacts that stem from the single-tap prediction which does not apply any smoothing.
  • the combination of the proposed NLMS notch adaptation with the directional equalizer of section 5 is straight forward.
  • the equalizer can indirectly influence and enhance the notch adaptation via the array output signal y(fe).
  • the performance of the proposed fast notch adaptation algorithm is contrasted with the conventional NLMS using a fixed stepsize in Fig. 5.
  • the graph illustrates the adaptation process for a synthetic sound field with a single sound source that arrives from changing angles ⁇ .
  • the adaptation should not drift towards the 90° boundary but rather maintain the previously identified steering factor a.
  • the underlying assumption is that an interferer does not move while being inactive.
  • the fast version of the constant stepsize NLMS (Eq. (12)) (gray curve) for example drifts towards 90° easily in case of activity of the desired sound source, but even the slower version (blue curve) is not able to maintain a once identified steering factor in all situations.
  • the described differential microphone array (including the proposed enhancements) has been implemented on a signal processor of a wireless loudspeaker (Binauric Boom Boom) which is, at the same time, a handsfree communication device.
  • the microphones offer SNRs of more than 60 dB which open up the possibility of a differential microphone array with a sufficiently low noise level.
  • An example application scenario is a handsfree call in an office where another colleague is working on the opposite side of the desk. The colleague's noise (typing, voice, etc.) can then be canceled out when placing a call with Boom Boom.
  • the signal processing software has been developed with the help of the RTProc rapid real-time prototyping framework (see H. Kriiger and P. Vary) - the developer interface for algorithm parameterization is shown in Fig. 6.
  • a Matlab prototype based on framewise processing
  • several other versions have been subsequently developed: A parameterizable C version, a C version with generated parameter tables, a C version based on fixed point arithmetic with an emulated instruction set and generated parameter tables, and finally optimized assembler code for the signal processor with generated parameter tables. All versions can be verified against each other and there is the possibility to step back to Matlab and add or modify features.
  • a frequency invariant notch characteristic can be obtained even for larger microphone distances.
  • a larger distance also helps to confine the noise gain of the array. Therefore, the array becomes practically usable even for higher sampling rates (e.g., 16kHz).
  • the noise (or interferer) suppression works more reliable in a broader range of acoustic scenarios. Also, the desired source does not compromise the notch adaptation anymore. Moreover, the combination with the directional equalizer leads to a more stable direction of arrival tracking.
  • a single unit or device may fulfill the functions of several items recited in the claims.
  • the mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Landscapes

  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Otolaryngology (AREA)
  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Acoustics & Sound (AREA)
  • Signal Processing (AREA)
  • Circuit For Audible Band Transducer (AREA)

Abstract

The present invention relates to a method for generating a directional sound signal (y(k)) from first and second sound signals (x 1 (k), x 2 (k)), which are generated by a first and a second microphone (M1, M2), which are separated by a distance (D). The method comprises generating first and second differential sound signals (x f (k), X b,DEQ (k)) based on the first and second sound signals (x 1 (k), x 2 (k)), and generating the directional sound signal (y(k)) according to a frequency-dependent directional response pattern based on the first and second differential sound signals (x f (k), X b,DEQ (k)). The generating of the second differential sound signal (X b,DEQ (k)) comprises generating a difference signal (xb(k)) of the first and the second sound signals (x 1 (k), x 2 (k)) and a frequency-selective processing that depends on a steering angle (a), which indicates a desired direction of maximum attenuation of the frequency-dependent directional response pattern, wherein the frequency-selective processing adjusts the actual direction of maximum attenuation of the frequency-dependent directional response pattern to correspond to the steering angle (α) substantially independent of frequency (ω) over the frequency range of the directional sound signal (y(k)).

Description

Method and apparatus for generating a directional sound signal from first and second sound signals
FIELD OF THE INVENTION
The present invention generally relates to the field of sound acquisition. More particularly, the present invention relates to a method and an apparatus for generating a directional sound signal from first and second sound signals, which are generated by a first and a second microphone, which are separated by a distance.
BACKGROUND OF THE INVENTION
1. Introduction
For sound acquisition in realistic acoustic environments, microphone arrays proved to be useful. They are designed to attenuate possible noise and interference components while retaining the desired source signal by exploiting different spatial (or directional) characteristics of the different signal sources (see, e.g., J. Benesty, J. Chen, and Y. Huang, "Microphone Array Signal Processing," Heidelberg: Springer, 2008 for an overview). A simple, yet efficient approach is the first-order differential microphone array described in G. Elko and A.-T. N. Pong, "A simple adaptive first-order differential microphone," in IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 169 to 172, October 1995. This microphone array, which is schematically and exemplarily shown in Fig. 1 allows to place two symmetrical notches (directions of maximum attenuation) at angles of Θ and 360° - Θ. Herein, three independent enhancements to the original method are proposed and a practical implementation for handsfree communication is described. A possible target device is a wireless loudspeaker with two integrated miniature digital micro-electromechanical system (MEMS) microphone cap- sules which facilitate handsfree audio communication.
2. Differential Microphone Array
First, the originally proposed method shall be reviewed with reference to Fig. 1 , which shows schematically and exemplarily a differential microphone array according to G. Elko and A.-T. N. Pong. Two closely spaced omnidirectional microphones M 1 and M2 are used to capture the acoustic environment. The corresponding digital signals xt(k) and x2(fe) are sampled with a rate of fs . Due to the small distance D between M 1 and M2, a coherent mutual subtraction - for convenience, acausal filters are assumed herein; in practice, appropriate signal alignment is required as marked by Ό' in Figs. 1 and 4 - of the aligned signals, i.e. , xf(k) = Xj fc)— x2(fe) * hT(k) (1 )
*b(fc) = x2(X) - * hT(k) (2) can be achieved. Thereby, hT(k) is a fractional delay filter with a delay time of T = D/c (c: speed of sound). This value corresponds to the sound propagation time from one microphone to the other. The signals xf(fc) and x (fc) can be interpreted as "forward and backward facing cardioid" signals as the respective directional responses of Eqs. (1 ) and (2) form cardioid shapes (see Fig . 3 in G. Elko and A.-T. N. Pong). For example xf(fe) does not contain any sound components from the rear direction (Θ = 180°) while x (fc) does not contain components from the front direction (Θ = 0°). Both signals xf(fe) and x (fc) are finally combined according to yEQ(fe) = h (k) * (Xf(fe) - a xb(fe)) (3) whereby the low-pass equalizer ¾EQ(fe) compensates for the highpass effect of the coherent subtraction operations and the scalar factor a can be used to control the notch an- gle(s): a = 0 corresponds to a (double) notch angle of 180° while a = 1 corresponds to two symmetrical notch angles of 90° and 270°. Due to the symmetry, all angle specifica- tions are restricted to 0° ... 180° in the following, whereby any statement for a specific angle Θ also applies to its symmetrical counterpart 360° - Θ. In the common operation scenario of the microphone array, the desired sound source lies in the front half plane (Θ = 0° ... 90°) while the undesired noise or interference source(s) lie(s) in the rear half plane (0 = 90° ... 180°). A common problem with differential microphone arrays are the tolerances of the employed microphones, leading to a "microphone mismatch" and therefore noise amplification (see M . Buck and M . Fte er, "First Order Differential Microphone Arrays for Automotive Applications," in Proceedings of International Workshop on Acoustic Echo and Noise Control (IWAENC), September 2001 ). The digital MEMS microphones in the possible tar- get device usually exhibit relatively constant frequency responses; therefore, individual microphone equalization is preferably not necessary for the envisaged application. However, their power levels may still vary to a certain extent due to mounting and assembly tolerances, which is disadvantageous since it is preferred to have fully matched input levels in order to utilize the full potential of the method. SUMMARY OF THE INVENTION
3. Objects and Solutions
It is an object of the present invention to provide a method and an apparatus for generating a directional sound signal from first and second sound signals, which are generated by a first and a second microphone, which are separated by a distance, which inter alia allow for a wider operating frequency and help to reduce a noise gain.
In a first aspect of the present invention, a method for generating a directional sound signal from first and second sound signals, which are generated by a first and a second microphone, which are separated by a distance, is presented, wherein the method comprises:
- generating first and second differential sound signals based on the first and second sound signals, and generating the directional sound signal according to a frequency-dependent directional response pattern based on the first and second differential sound signals,
wherein the generating of the second differential sound signal comprises generating a difference signal of the first and the second sound signals and a frequency-selective processing that depends on a steering angle, which indicates a desired direction of maximum attenuation of the frequency-dependent directional response pattern, wherein the frequency-selective processing adjusts the actual direction of maximum attenuation of the frequency-dependent directional response pattern to correspond to the steering angle substantially independent of frequency over the frequency range of the directional sound signal.
The present invention is based on the idea that by employing these steps, a (substantially) frequency invariant notch characteristic can be obtained even for larger microphone distances. A larger distance also helps to confine the noise gain of the array. Therefore, the array becomes practically usable even for higher sampling rates (e.g., 16kHz). The term "difference signal" as used herein, also includes the case where one or both of the first and the second sound signals is/are further temporally delayed, for example, by means of a fractional delay filter hT(k), as described in section 2 above.
It is preferred that the frequency-selective processing comprises weighting the difference signal with an approximated steering factor that is independent of frequency to generate a weighted difference signal and correcting for the approximation by adding a correction signal that is generated from the difference signal in dependence of frequency and the steering angle.
It is further preferred that the generation of the correction signal comprises applying two separate operations, one being dependent on frequency and independent of the steering angle and one being dependent on the steering angle but independent of frequency.
It is preferred that the generation of the correction signal comprises filtering the difference signal with a filter that is dependent on frequency and independent of the steering angle to generate a filtered difference signal. It is further preferred that the generation of the correction signal further comprises weighting the filtered difference signal with a factor that is dependent on the steering angle and independent of frequency.
It is preferred that the factor is determined by using a polynomial approximation that is evaluated with the steering angle.
It is further preferred that the method further comprises filtering the directional sound signal with a low-pass filter to generate a filtered directional sound signal.
It is preferred that the approximated steering factor for a time instance is adapted for the following time instance by adding an adaptation value that is scaled by a stepsize param- eter, wherein the stepsize parameter is adapted in dependence of estimated energies of coherent and incoherent sound components.
Employing these steps provides a fast notch adaptation algorithm, with which the noise (or interferer) suppression works more reliable in a broader range of acoustic scenarios. Also, the desired source does not compromise the notch adaptation anymore. Moreover, the combination with the directional equalizer leads to a more stable direction of arrival tracking.
It is further preferred that the energy of the incoherent sound components is approximated by the estimated short-term energy of the directional sound signal and the energy of the coherent sound components is approximated by a fraction of the estimated short-term energy of the difference signal.
It is preferred that the method further comprises estimating a relative gain of the first and the second microphone and equalizing power levels of the first and the second microphone based on the relative gain.
Employing these steps enables the array to work reliably despite certain assembly and sensor tolerances. It is further preferred that the relative gain is determined based on recursively estimated variances of the first and second sound signals.
It is preferred that the first and the second microphone are omnidirectional microphones.
In a second aspect of the present invention, an apparatus for generating a directional sound signal from first and second sound signals, which are generated by a first and a second microphone, which are separated by a distance, is presented, wherein the apparatus comprises:
first generating means for generating first and second differential sound signals based on the first and second sound signals, and
- second generating means for generating the directional sound signal according to a directional response pattern based on the first and second differential sound signals, wherein the generating of the second differential sound signal comprises generating a difference signal of the first and the second sound signals and a frequency-selective processing that depends on a steering angle, which indicates a desired direction of max- imum attenuation of the frequency-dependent directional response pattern, wherein the frequency-selective processing adjusts the actual direction of maximum attenuation of the frequency-dependent directional response pattern to correspond to the steering angle substantially independent of frequency over the frequency range of the directional sound signal. In a third aspect of the present invention, a system is presented, wherein the system comprises:
a first and a second microphone, which are separated by a distance and generate first and second sound signals, and
an apparatus as defined in claim 13. In a fourth aspect of the present invention, computer program comprising program code means, which, when run on a computer controlling the apparatus according to claim 13, perform the steps of the method according to any of claims 1 to 12 is presented. It shall be understood that the method of claim 1 , the apparatus of claim 13, the system of claims 14, and the computer program of claim 15 have similar and/or identical preferred embodiments, in particular, as defined in the dependent claims.
It shall be understood that a preferred embodiment of the invention can also be any combination of the dependent claims with the respective independent claim.
BRIEF DESCRIPTION OF THE DRAWINGS
These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. In the following drawings:
Fig . 1 shows schematically and exemplarily a differential microphone array according to G. Elko and A.-T. N. Pong,
Fig . 2 illustrates the optimal steering factor vs. a linear approximation,
Fig . 3 shows directional frequency responses (rear half plane) of a wideband (/s = 16 kHz) array with D = 1.8 cm steered towards a = 135° using the linear approximation of Eq. (7) and the directional equalizer of Fig. 4,
Fig . 4 shows schematically and exemplarily a differential microphone array with directional equalizer according to an embodiment of the present invention, illustrates schematically and exemplarily the performance of the proposed fast notch adaptation algorithm (D = 1.8 cm, fs = 16 kHz). The optimal values of lin for the incidence angles used herein are: lin(90°) = 1 , alin(135°) ~ 0.17, alin(180°) = 0, alin(0°) = undef, and shows schematically and exemplarily a developer interface for the exemplarily implemented array based on the RTProc rapid real time prototyping framework (see H. Kriiger and P. Vary, "RTPROC: A System for Rapid Real-Time Prototyping in Audio Signal Processing," in Proceedings of IEEE/ACM International Symposium on Distributed Simulation and Real Time Applications, October 2008).
DETAILED DESCRIPTION OF EMBODIMENTS 4. Input Level Alignment
In order to fully match the input levels of the microphones M 1 and M2, an input level alignment procedure is devised herein.
First, the relative gain gc = J¾/c¾2 is computed on the basis of the recursively estimated input channel variances <x? and <x? . Then, correction gains are obtained
else else which are finally recursively smoothed before they are applied to the respective input channels. With these gains, the "louder" channel's level is reduced to the "quieter" channel's level.
5. Directional Equalization
In the following, it will become clear that the directional response of the original method described in G. Elko and A.-T. N. Pong is distorted, i.e., the actual notch angle clearly deviates from the desired angle, in particular for high frequencies. To correct this behavior, we herein propose an efficient "directional equalization" approach.
It can be shown (proof omitted) that the angle-dependent transfer function of the differential array (without the output equalizer) is
. ω£> \ (ωΌ , (5)
H(a), (l>) = 2j - e-J^(c→+1 sin I—— (1 + cos0) 1— a · sin I—— (1— cos0)
2c 2c with the (angular) frequency ω = Ω · fs = 2nf and the respective angle of observation φ. Now this transfer function should become zero for a specific angle, i.e., the so-called steering angle a. Hence, by requiring Η(ω, φ = a) = 0, the optimal steering factor a can be deduced from Eq. (5):
Sln \ ~2c~ ' cosa) I
si .n (ωΌ
I -~— (1— cosa) I (6)
which obviously depends on a and on the frequency ω. The steering angle a should, ideally, be adapted to match the interference or noise incidence angle Θ . This is discussed in section 6.
We note that if we approach the spatial alias frequency, i.e., ω→ ωΑ =— , we have α(ω, α)→ 1 regardless of the angle a, i.e., the steering angle is not controllable at ωΑ, which renders it a natural cutoff frequency for the entire array. The frequency dependency of a is, however, a contradiction to the scalar multiplication operation in Eq. (3). Therefore, usually, small values of ωΌ are assumed and a linear approximation of Eq. (6) is used (see H. Puder, "Acoustic noise control: An overview of several methods based on applications in hearing aids," in IEEE Pacific Rim Conference on Comm. Computers and Signal Processing, pages. 871 to 876, August 2009): cosa
cosa (7)
This approximation is frequency independent which is sufficient for many applications. However, alin(a) deviates more and more from the optimal α(ω, α) as ω approaches ωΑ, which is illustrated in Fig. 2. The effect already becomes relevant for frequencies well below ωΑ and also varies with the angle a. For a clear demonstration, a = 135° will be used in the following examples, because the deviation from the expected behavior is very prominent in this case. The directional response of a differential microphone array using the linear approximation according to Eq. (7) is shown in Fig. 3 (a).
To correct the obvious deformation of the notch characteristic, we propose to integrate the optimal (frequency and angle dependent) steering factor α(ω, a) into the time domain realization of the differential array. In principle, this can be achieved by replacing the scalar multiplication operation (factor a in Eq. (3) and in Fig. 1 ) with a filtering operation, whereby the angle dependent filter transfer functions are given by α(ω, ). For a more efficient implementation with low memory and computational requirements, we consider the approximation error of the steering factor:
Αα(ω, ) = α(ω, )— aVm( ) (8)
Fortunately, Δ (ω, ) is separable with good accuracy:
Δ (ω, ) ~ Δ (ω) · Δ (α) (9)
The factors Δ (ω) and Δ (α) can be computed by marginalization of the 2-dimensional function Δ (ω, α) and appropriate normalization. The factor Δ (ω) can now be regarded as the frequency response of a fixed filter. It can be transformed to the time domain via periodic extension, inverse DFT, cyclic shifting (to enforce causality) and an appropriate shortening to a desired length. The resulting FIR filter coefficients ¾DEQ(fe), e.g., of order 16, are independent of the steering angle . The angular dependency is then reintroduced with a polynomial approximation (e.g., order 4) of the second factor Δ (α) after a variable transformation from to lin(a), i.e., ΡΔα(αηη(α)) ~ Δα(α(απη))-
The resulting differential microphone array with directional equalization is shown schematically and exemplarily in Fig. 4, whereby the "directionally equalized" backward cardi- oid signal
~ ~1{aVm( ) + Δ (ω) · Δ (α)} * Xb(fc)
= alin(a) xh(k) + Aa(a)-1{Δα(ω)} * xb(fe) ( 0) ¾ alin(a) xh(k) + PAa(alin(a)) ¾DEQ(fe) * xb(fe) where ί'"1(·) denotes the inverse Fourier transform. The effect of directional equalizing can be observed in Fig. 3 (b), which displays an almost frequency-invariant behavior over the entire wideband frequency range.
The practical operation of the modified version of the microphone array (Fig. 4) does not significantly differ from the conventional version (Fig. 1 ): The desired notch angle is still easily controlled by adapting the scalar factor lin. As an additional step, the polynomial ΡΔα( 1ίη) must be evaluated. 5.1 Discussion
The distorted notch curve of the standard differential array (Fig. 3 (a)) not only limits the ability to suppress interfering sound sources, but it can even compromise the accurate NLMS adaptation of the steering angle (see section 6). It should be noted that other approaches exist to cope with distorted notch characteristics. For example, a smaller microphone distance D could be used so that the product ωΌ in Eq. (6) remains sufficiently small. The downside of this approach is a stronger highpass effect of the array which, in turn, requires heavier output equalization with a more pronounced lowpass filter h (k). In a real system, this leads to a higher amplification of the microphone noise, particularly at low frequencies. For the present example with D = 1.8 cm, less than half of the original distance is required to obtain a comparably straight directional response. This, however, comes at the cost of a significantly increased noise gain (+10dB) over a wide frequency range.
If a subband (or frequency domain) realization of the differential array is used (see, e.g., H. Puder), the subband steering factors can be individually adapted which, naturally, helps to compensate the notch curve distortion of Fig. 3. The problem has also been identified by M. Buck and M. Fte er, where it is proposed to combine the advantages of the differential array with that of a superdirective endfire array. A more general proposal for frequency-invariant beamforming techniques was made in L. C. Parra, "Steerable frequency-invariant beamforming for arbitrary arrays," Journal of the Acoustical Society of America, Vol. 1 19, No. 6, pages 3839 to 3847, 2006.
6. Fast Notch Adaptation
The goal of a notch adaptation algorithm is to automatically align the notch angle of the differential array with the incidence angle Θ of the (main) interferer. In the common appli- cation scenario, the undesired noise or interference sources are assumed to lie in the rear half plane, i.e., Θ = 90° ... 180°. The standard approach to adapt the factor a (or lin if directional equalization is used) and therefore the notch angle is the (normalized) least mean square (NLMS) algorithm. The goal here is to minimize the power of the output signal y(fe), i.e. where, usually, 0≤ a≤ 1, i.e., 180° > a≥ 90° is enforced. The stepwise update this method is (e.g., H. Puder) μ
a(k) = a(k - 1) +— · xb(fc) · y(k)
(12) with the short term power estimate σ of the backward cardioid signal and the constant stepsize μ.
For our proposed notch adaptation algorithm, which is inspired by the "optimum stepsize NLMS" of M. Pawig, G. Enzner, and P. Vary, "Adaptive sampling rate correction for acoustic echo control in voice-over-IP," IEEE Transactions on Signal Processing, Vol. 58, pages 189 to 199, January 2010, we separate the coherent and the incoherent (ambient) components of the acoustic environment. The array output (before the lowpass equalizer) therefore becomes y(fe) = Xf,c(fe) - - Xb,c(fc) + Xf,a(fe) - a Xb,a(fc)
(13) e(fe) n(fe)
This equation represents the error signal of a single-tap adaptive filter with a noisy input. In the context of the differential array, the noise signal n(k) is due to the incoherent (ambient) noise that cannot be suppressed. The coherent contribution to y(fe) should ideally be zero. However, due to the instantaneous misadaptation of the factor a, an error signal e(k) appears at the output. According to adaptive filter theory, the optimal (adaptive) stepsize parameter (see P. Vary and R. Martin, "Digital Speech Transmission - Enhancement, Coding and Error Concealment", Chichester: Wiley, 2006, (13.56)) is
E{e2 (k)}
μ°ρ1 E{n2 (k)} + E{e2 (k)} (14)
From the signals available, the best approximation of £{n2(fc)} is the level σ2 of the microphone array's output y(fe) while for £{e2(fc)}, the assumption of a fixed attenuation factor for the backward cardioid signal is made, i.e. £{e2(fc)} ~ κ · i2 b . We set κ = 0.01 (assumed attenuation of 20 dB) and the adaptive stepsize parameter is hence with the recursively estimated short term powers σ, 2 and 8 , which leads to the new NLMS update rule
K
a(k) = a(k - 1) + ξ— — - xb(fe) - y(fe)
(16)
The adaptation can be deliberately slowed down by the factor 0≤ ξ≤ 1 to avoid artifacts that stem from the single-tap prediction which does not apply any smoothing. The combination of the proposed NLMS notch adaptation with the directional equalizer of section 5 is straight forward. The equalizer can indirectly influence and enhance the notch adaptation via the array output signal y(fe).
Although the frequency dependency of a(k) is disregarded in the NLMS update, this slight error is immediately compensated for at the next sample instant when the DEQ is in turn adapted to the new steering factor a{k + 1) (or lin(fe + 1) in this case).
6.1 Evaluation & Discussion
The performance of the proposed fast notch adaptation algorithm is contrasted with the conventional NLMS using a fixed stepsize in Fig. 5. The graph illustrates the adaptation process for a synthetic sound field with a single sound source that arrives from changing angles Θ . In the example, only the angle of Θ = 0° is associated with a desired source. In this case, the adaptation should not drift towards the 90° boundary but rather maintain the previously identified steering factor a. The underlying assumption is that an interferer does not move while being inactive. The fast version of the constant stepsize NLMS (Eq. (12)) (gray curve) for example drifts towards 90° easily in case of activity of the desired sound source, but even the slower version (blue curve) is not able to maintain a once identified steering factor in all situations. The proposed adaptive NLMS (Eq. (16)) (green curve) mostly solves this problem while converging almost instantaneously towards new interfering sound sources. However, all three approaches exhibit an unstable behavior for Θ = 135°. These surprisingly strong fluctuations, which are clearly audible and also disturbing, are explained by intermittent high frequency sounds in the interferer source (such as fricatives in speech). If a high frequency sound event occurs from Θ = 135°, the adaptation algorithm (which minimizes the output power) shifts the distorted notch characteristic of Fig. 3 (a) to the left, i.e., toward = 90° or alin = 1. Low frequency sounds yield the expected result ( = 135° or alin ~ 0.17) instead. If the DEQ algorithm from section 5 is activated (red curve in Fig. 5), the unstable adaptation behavior vanishes almost completely.
7. Implemented System Using fixed point arithmetic, the described differential microphone array (including the proposed enhancements) has been implemented on a signal processor of a wireless loudspeaker (Binauric Boom Boom) which is, at the same time, a handsfree communication device. Two miniature digital MEMS microphone capsules are placed on the top of the device with D = 1.8 cm. The microphones offer SNRs of more than 60 dB which open up the possibility of a differential microphone array with a sufficiently low noise level. An example application scenario is a handsfree call in an office where another colleague is working on the opposite side of the desk. The colleague's noise (typing, voice, etc.) can then be canceled out when placing a call with Boom Boom.
The signal processing software has been developed with the help of the RTProc rapid real-time prototyping framework (see H. Kriiger and P. Vary) - the developer interface for algorithm parameterization is shown in Fig. 6. Beginning with a Matlab prototype (based on framewise processing), several other versions have been subsequently developed: A parameterizable C version, a C version with generated parameter tables, a C version based on fixed point arithmetic with an emulated instruction set and generated parameter tables, and finally optimized assembler code for the signal processor with generated parameter tables. All versions can be verified against each other and there is the possibility to step back to Matlab and add or modify features. The measured complexity of the assembler code is approximately 7 MIPS for the wideband sampling rate ( s = 16kHz).
8. Conclusions A number of algorithmic enhancements for the standard differential microphone according to G. Elko and A.-T. N. Pong have been proposed herein: The described input level alignment procedure enables the array to work reliably despite certain assembly and sensor tolerances.
With the proposed directional equalizer, a frequency invariant notch characteristic can be obtained even for larger microphone distances. A larger distance also helps to confine the noise gain of the array. Therefore, the array becomes practically usable even for higher sampling rates (e.g., 16kHz).
With the fast notch adaptation algorithm, the noise (or interferer) suppression works more reliable in a broader range of acoustic scenarios. Also, the desired source does not compromise the notch adaptation anymore. Moreover, the combination with the directional equalizer leads to a more stable direction of arrival tracking.
The implementation of the proposed techniques in a new commercial handsfree communication device was described. Finally, the subjective listening impression confirms that, with the new algorithms, interfering sound sources are suppressed much more reliably than with the conventional microphone array. The now frequency-invariant notch charac- teristic is not the only reason for this. Rather, only the combined application of both DEQ and fast notch adaptation facilitates the clearly improved direction of arrival tracking and interference suppression.
Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.
In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality.
A single unit or device may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.
Any reference signs in the claims should not be construed as limiting the scope.

Claims

CLAIMS:
1 . A method for generating a directional sound signal (y(fe)) from first and second sound signals (xj fc), x2 (fe)), which are generated by a first and a second microphone (M 1 , M2), which are separated by a distance (D), wherein the method comprises:
- generating first and second differential sound signals (xf(fc) , based on the first and second sound signals (x^k), x2 {k)), and
generating the directional sound signal (y(fe)) according to a frequency-dependent directional response pattern based on the first and second differential sound signals
wherein the generating of the second differential sound signal comprises generating a difference signal (xb(fc)) of the first and the second sound signals (xj fc), x2 (fe)) and a frequency-selective processing that depends on a steering angle (a), which indicates a desired direction of maximum attenuation of the frequency-dependent directional response pattern, wherein the frequency-selective processing adjusts the actual direction of maximum attenuation of the frequency-dependent directional response pattern to correspond to the steering angle (a) substantially independent of frequency (ω) over the frequency range of the directional sound signal (y(fe)).
2. The method according to claim 1 , wherein the frequency-selective processing comprises weighting the difference signal (xb(fc)) with an approximated steering factor ( lin(a)) that is independent of frequency (ω) to generate a weighted difference signal ( lin(a) · xb(fe)) and correcting for the approximation by adding a correction signal (ΡΔα( 1ίη(α)) · ¾DEQ(fe) * Xb(fe)) that is generated from the difference signal (xb(fc)) in dependence of frequency (ω) and the steering angle ( ).
3. The method according to claim 2, wherein the generation of the correction signal (Ράα(αΐίη(α ) · * xb(& ) comprises applying two separate operations, one being dependent on frequency (ω) and independent of the steering angle ( ) and one being dependent on the steering angle ( ) but independent of frequency (ω).
4. The method according to claim 2 or 3, wherein the generation of the correction signal (ΡΔα (απη(α)) * xb(k)) comprises filtering the difference signal (xb(fe)) with a filter (¾DEQ(fe)) that is dependent on frequency (ω) and independent of the steering angle ( ) to generate a filtered difference signal (¾DEQ(fe) * a¾0 )-
5. The method according to claim 4, wherein the generation of the correction signal (ΡΔα( 1ίη(α)) · ¾DEQ(fe) * Xb(fc)) further comprises weighting the filtered difference signal (hO (k) * x (fe)) with a factor (ίΔα (αηη (α) ) that is dependent on the steering angle ( ) and independent of frequency (ω).
6. The method according to claim 5, wherein the factor (ίΔα (αηη (α) ) 's determined by using a polynomial approximation that is evaluated with the steering angle ( ) or the approximated steering factor ( lin(a) ).
7. The method according to any of claims 1 to 6, wherein the method further comprises filtering the directional sound signal (y(fc)) with a low-pass filter (¾EQ(fe)) to generate a filtered directional sound signal (yEQ(fc)).
8. The method according to any of claims 2 to 7, wherein the approximated steering factor ( lin(a)) for a time instance (k - 1) is adapted for the following time instance (k) by adding an adaptation value that is scaled by a stepsize parameter (μορ1), wherein the stepsize parameter (μορ1) is adapted in dependence of estimated energies of coherent and incoherent sound components.
9. The method according to claim 8, wherein the energy of the incoherent sound components is approximated by the estimated short-term energy (<xj;) of the directional sound signal (y(fc)) and the energy of the coherent sound components is approximated by a fraction (κσ^) of the estimated short-term energy (<¾b) of the difference signal
(xb(fe)).
10. The method according to any of claims 1 to 9, wherein the method further comprises estimating a relative gain (gc) of the first and the second microphone (M 1 , M2) and equalizing power levels of the first and the second microphone (M 1 , M2) based on the relative gain (gc).
1 1 . The method according to claim 10, wherein the relative gain (gc) is determined based on recursively estimated variances (¾ , <¾2 ) of the first and second sound signals
(x^fc), x2(fc)).
12. The method according to any of claims 1 to 1 1 , wherein the first and the second microphone (M 1 , M2) are omnidirectional microphones.
13. An apparatus (2) for generating a directional sound signal (y(fe)) from first and second sound signals (xj fc), x2(fc)), which are generated by a first and a second microphone (M 1 , M2), which are separated by a distance (D), wherein the apparatus (2) comprises:
first generating means for generating first and second differential sound signals (xf(fe), Xb,DEQ(fc)) based on the first and second sound signals (xj fc), x2(fc)), and
second generating means for generating the directional sound signal (y(fe)) according to a directional response pattern based on the first and second differential sound signals (xf(fc), xb,DEQ(fe)),
wherein the generating of the second differential sound signal comprises generating a difference signal (xb(fe)) of the first and the second sound signals (xj fc), x2(fe)) and a frequency-selective processing that depends on a steering angle (a), which indicates a desired direction of maximum attenuation of the frequency-dependent directional response pattern, wherein the frequency-selective processing adjusts the actual direction of maximum attenuation of the frequency-dependent directional response pattern to correspond to the steering angle ( ) substantially independent of frequency (ω) over the frequency range of the directional sound signal (y(fe)).
14. A system (1 ), wherein the system (1 ) comprises:
a first and a second microphone (M 1 , M2), which are separated by a distance (D) and generate first and second sound signals (xj fc), x2(fc)), and
an apparatus (2) as defined in claim 13.
15. A computer program comprising program code means, which, when run on a computer controlling the apparatus (2) according to claim 13, perform the steps of the method according to any of claims 1 to 12.
EP14771598.1A 2014-09-23 2014-09-23 Method and apparatus for generating a directional sound signal from first and second sound signals Active EP3225037B1 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/EP2014/070243 WO2016045706A1 (en) 2014-09-23 2014-09-23 Method and apparatus for generating a directional sound signal from first and second sound signals

Publications (2)

Publication Number Publication Date
EP3225037A1 true EP3225037A1 (en) 2017-10-04
EP3225037B1 EP3225037B1 (en) 2019-05-08

Family

ID=51585120

Family Applications (1)

Application Number Title Priority Date Filing Date
EP14771598.1A Active EP3225037B1 (en) 2014-09-23 2014-09-23 Method and apparatus for generating a directional sound signal from first and second sound signals

Country Status (2)

Country Link
EP (1) EP3225037B1 (en)
WO (1) WO2016045706A1 (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7139628B2 (en) * 2018-03-09 2022-09-21 ヤマハ株式会社 SOUND PROCESSING METHOD AND SOUND PROCESSING DEVICE
GB2575491A (en) * 2018-07-12 2020-01-15 Centricam Tech Limited A microphone system
US11728905B2 (en) * 2020-10-05 2023-08-15 CUE Audio, LLC Method and system for digital communication over an acoustic channel

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9202475B2 (en) * 2008-09-02 2015-12-01 Mh Acoustics Llc Noise-reducing directional microphone ARRAYOCO

Also Published As

Publication number Publication date
EP3225037B1 (en) 2019-05-08
WO2016045706A1 (en) 2016-03-31

Similar Documents

Publication Publication Date Title
US11109163B2 (en) Hearing aid comprising a beam former filtering unit comprising a smoothing unit
CN111128210B (en) Method and system for audio signal processing with acoustic echo cancellation
US8194880B2 (en) System and method for utilizing omni-directional microphones for speech enhancement
US10657981B1 (en) Acoustic echo cancellation with loudspeaker canceling beamformer
EP2207168B1 (en) Robust two microphone noise suppression system
US9723422B2 (en) Multi-microphone method for estimation of target and noise spectral variances for speech degraded by reverberation and optionally additive noise
AU2011334840B2 (en) Apparatus and method for spatially selective sound acquisition by acoustic triangulation
JP5805365B2 (en) Noise estimation apparatus and method, and noise reduction apparatus using the same
JP5331201B2 (en) Audio processing
CN103098132A (en) Sound source separator device, sound source separator method, and program
EP3692529B1 (en) An apparatus and a method for signal enhancement
TWI465121B (en) System and method for utilizing omni-directional microphones for speech enhancement
US11373668B2 (en) Enhancement of audio from remote audio sources
EP3225037B1 (en) Method and apparatus for generating a directional sound signal from first and second sound signals
Xiao et al. Effect of target signals and delays on spatially selective active noise control for open-fitting hearables
US11019433B2 (en) Beam former, beam forming method and hearing aid system
Geiser et al. A differential microphone array with input level alignment, directional equalization and fast notch adaptation for handsfree communication
US10692514B2 (en) Single channel noise reduction
Masuyama et al. Causal distortionless response beamforming by alternating direction method of multipliers
Yong et al. Effective binaural multi-channel processing algorithm for improved environmental presence
Marquardt et al. Incorporating relative transfer function preservation into the binaural multi-channel wiener filter for hearing aids
Lombard et al. Combination of adaptive feedback cancellation and binaural adaptive filtering in hearing aids
HK1190260A (en) Apparatus and method for spatially selective sound acquisition by acoustic triangulation
HK1190260B (en) Apparatus and method for spatially selective sound acquisition by acoustic triangulation

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20170810

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAX Request for extension of the european patent (deleted)
GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20181121

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

Ref country code: AT

Ref legal event code: REF

Ref document number: 1132097

Country of ref document: AT

Kind code of ref document: T

Effective date: 20190515

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602014046410

Country of ref document: DE

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20190508

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190908

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190808

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190809

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190808

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1132097

Country of ref document: AT

Kind code of ref document: T

Effective date: 20190508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602014046410

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

26N No opposition filed

Effective date: 20200211

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190923

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190930

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190930

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190923

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20190930

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190930

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190908

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20140923

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20190508

REG Reference to a national code

Ref country code: DE

Ref legal event code: R082

Ref document number: 602014046410

Country of ref document: DE

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20240226

Year of fee payment: 10

Ref country code: GB

Payment date: 20240227

Year of fee payment: 10

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20240227

Year of fee payment: 10

REG Reference to a national code

Ref country code: DE

Ref legal event code: R119

Ref document number: 602014046410

Country of ref document: DE

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20240923

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20250401

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20240923