[go: up one dir, main page]

EP1257146B1 - Verfahren und System zur Schallbearbeitung - Google Patents

Verfahren und System zur Schallbearbeitung Download PDF

Info

Publication number
EP1257146B1
EP1257146B1 EP02009933A EP02009933A EP1257146B1 EP 1257146 B1 EP1257146 B1 EP 1257146B1 EP 02009933 A EP02009933 A EP 02009933A EP 02009933 A EP02009933 A EP 02009933A EP 1257146 B1 EP1257146 B1 EP 1257146B1
Authority
EP
European Patent Office
Prior art keywords
sound
microphone array
source
location
enhancing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Lifetime
Application number
EP02009933A
Other languages
English (en)
French (fr)
Other versions
EP1257146A3 (de
EP1257146A2 (de
Inventor
David John Benjamin Pearce
James Alexander Rex
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Motorola Solutions Inc
Original Assignee
Motorola Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Motorola Inc filed Critical Motorola Inc
Publication of EP1257146A2 publication Critical patent/EP1257146A2/de
Publication of EP1257146A3 publication Critical patent/EP1257146A3/de
Application granted granted Critical
Publication of EP1257146B1 publication Critical patent/EP1257146B1/de
Anticipated expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R3/00—Circuits for transducers, loudspeakers or microphones
    • H04R3/005—Circuits for transducers, loudspeakers or microphones for combining the signals of two or more microphones

Definitions

  • the present invention relates to sound signal processing.
  • the invention relates to enhancing sound signal quality for use in a sound processing device.
  • pick-up of noise and reverberation may be reduced by employing an array of remote microphones.
  • the remote microphones' outputs are processed and then combined into a single signal that has an improved signal quality.
  • the best processing parameters to use in the microphone array depend on the location of the desired sound source relative to the microphones. Many arrays reject noise by having much higher sensitivity to sound coming from the direction (or region) of the desired source than to sounds from elsewhere, where noise sources are located. However, the source's relative location is often unknown initially, and may change over time. For example, people often move their heads whilst speaking, and if an array is in a hand-held device, it is quite likely to move somewhat during use.
  • the location of a sound source relative to a microphone array can be estimated by measuring certain characteristics of the signals received from that source by the array of microphones.
  • An example of one such characteristic is cross-power spectral phase.
  • the other method of location of a sound source relative to a microphone array is estimation by visually imaging the source, with a camera mounted on the same base as the array.
  • the camera image is processed to locate the source within it, using known visual characteristics of the source. For example, a speaker's mouth may be located by looking for his/her face in the image.
  • Granted US patent US-A-4984087 shows a prior art microphone apparatus for a video camera.
  • the methods and system of the invention are for use in portable sound processing devices
  • the methods and system according to the present invention allow processing of sound signals that result in increased quality.
  • the output signal has decreased content of acoustic noise and reverberation caused by surroundings.
  • the output signal is also less sensitive to changes of location of a sound source relative to the microphones. These changes of location are caused both by movement of the sound source, and movement of the device equiped with these microphones.
  • the sound source would be a person speaking and the device might be held in his/her hand.
  • a method for enhancing sound signal quality as defined in claim 1.
  • the method comprises steps of receiving sound from sound source by a microphone array and detecting a movement of this microphone array.
  • the movement is detected by a platform motion sensor.
  • Current location of the sound source relative to the microphone array is estimated on the basis of the sound signal from the microphone array and signals from the platform motion sensor. Results of this estimation are used for enhancing output sound signal quality.
  • An advantage of the present invention is that the motion sensor can continously track the motion of the microphone array, and use this to adjust the current estimate of the relative location of the desired sound source.
  • the changes of relative location of the sound source and microphone array can be tracked even when there is no sound signal.
  • the sound-based source location estimation can only track changes in source location when sound is present.
  • a system for enhancing sound signal quality as defined in claim 5.
  • the system comprises a microphone array and a platform motion sensor that are connected to a source location estimator. Additionally the microphone array is connected to a sound enhancing processor. The sound enhancing processor is connected to a source location estimator.
  • a “movement signal” herein below refers to a signal generated and provided by a platform motion sensor.
  • a “microphone array” mentioned herein below refers to a set of microphones that are a part of a portable device. Such a portable device is typically in a user's hand, during use, and may be subject to various rotational and translational movements.
  • a sound from a sound source 412 which may be moving, is received by a microphone array 402.
  • a platform motion sensor 400 detects a movement of the microphone array 402.
  • a sound signal is provided to a sound enhancing processor 408 and to a source location estimator 406.
  • the movement signal is provided to the source location estimator 406.
  • step 112 the sound enhancing processor 408 enhances the quality of the sound signal. This enhancement may be performed according to methods known in the art, and is not described further here.
  • step 104 on the basis of this sound signal, an estimation of a location of the sound source 412 relative to the microphone array 402 is performed. For the purpose of this estimation, long time intervals of the sound signal are analysed.
  • step 106 on the basis of the movement signal, an estimation of a change of position of the microphone array 402 is performed. For the purpose of this estimation, short time intervals of the movement signal are analysed.
  • step 108 on the basis of results of estimations performed in steps 104 and 106, a current change of source's relative location is estimated.
  • step 110 the estimated current location of the sound source 412 relative to the microphone array 402 is obtained, by combining the current change of source's relative location from step 108 with the location of the sound source 412 relative to the microphone array 402.
  • Results of the estimation obtained in step 110 and the enhanced signal received in step 112 can be used for further processing in other devices, e.g. speech recognition devices.
  • step 200 a sound from a sound source 412, which may be moving, is received by a microphone array 402.
  • platform motion sensor 400 detects a movement of the microphone array 402. Afterwards, a sound signal is provided to a sound enhancing processor 408 and to a source location estimator 406. At the same time the movement signal is provided to the source location estimator 406.
  • step 204 on the basis of this sound signal, an estimation of a location of the sound source 412 relative to the microphone array 402 is performed. For the purpose of this estimation, long time intervals of the sound signal are analysed. Results of this estimation are provided to the sound enhancing processor 408.
  • step 212 the sound enhancing processor 408 enhances the quality of the sound signal using the results of the estimation performed in step 204.
  • step 206 on the basis of the movement signal, an estimation of a change of position of the microphone array 402 is performed. For the purpose of this estimation, short time intervals of the movement signal are analysed.
  • step 208 on the basis of results of estimations performed in steps 204 and 206, a current change of the source's relative location is estimated.
  • step 210 an estimation of current location of the sound source 412 relative to the microphone array 402 is performed, by combining the current change of the source's relative location from step 208 with the location of the sound source 412 relative to the microphone array 402.
  • Results of the estimation obtained in step 210 and the enhanced signal received in step 212 can be used for further processing in other devices, e.g. speech recognition devices.
  • step 300 a sound from a sound source 412, which may be moving, is received by a microphone array 402.
  • platform motion sensor 400 detects a movement of the microphone array 402.
  • a sound signal is provided to a sound enhancing processor 408 and to a source location estimator 406.
  • the movement signal is provided to the source location estimator 406.
  • step 304 on the basis of this sound signal, an estimation of a location of the sound source 412 relative to the microphone array 402 is performed. For the purpose of this estimation, long time intervals of the sound signal are analysed. Simultaneously, in step 306, on the basis of the movement signal an estimation of a change of position of the microphone array 402 is performed. For the purpose of this estimation, short time intervals of the movement signal are analysed.
  • step 308 on the basis of results of estimations performed in steps 304 and 306, a current change of the source's relative location is estimated.
  • step 310 by combining the current change of source's relative location from step 308 with the location of the sound source 412 relative to the microphone array 402, the estimated current location of the sound source 412 relative to the microphone array 402 is obtained.
  • step 312 the sound enhancing processor 408 enhances the quality of the sound signal using the results of the estimation performed in step 310.
  • the sound signal of enhanced quality obtained in step 312 can be used for further processing in other devices, e.g. speech recognition devices.
  • the long time interval referenced in steps 104, 204 and 304 is typically in the range from 10 ms up to 1 s.
  • the short time interval referenced in steps 106, 206 and 306 is typically in the range from 0.1 ⁇ s up to 10 ⁇ s.
  • FIG. 4 depicts a system for enhancing quality of an output sound signal, operative in accordance with the embodiment of the present invention as illustrated in Fig. 3.
  • a system for enhancing sound signal quality comprises a microphone array 402, that is connected to a sound enhancing processor 408, and to a source location estimator 406.
  • the system also comprises a platform motion sensor 400 that is connected to the source location estimator 406.
  • the source location estimator 406 is connected to the sound enhancing processor 408.
  • the microphone array 402 consists of two or more microphones 404.
  • An array of at least two microphones can provide output signals that can be used to estimate both the direction and distance of a source of sound.
  • the platform motion sensor 400 comprises at least one gyroscope and/or accelerometer.
  • the platform motion sensor 400 is fixed to the microphone array 402, or to any other part of the portable sound processing device 410 that forms one unit with the microphone array 402.
  • the source location estimator 406 and sound enhancing processor 408 have respectively a first signal output connection and a second signal output connection.
  • the invention uses the signal from platform motion sensor 400 to provide an indication of movements of the microphone array 402. Therefore it is necessary that sensor 400 and microphone array 402 be linked so that the signal from sensor 400 relates to the motion of microphone array 402.
  • a link may simply include sensor 400 being attached to array 402, or both being attached to the same housing. Therefore, although figure 4 shows elements 400, 402, 406 and 408 all in one unit, elements 406 and 408 may in fact be located elsewhere.
  • a system in accordance with the invention, or the methods of the invention may be used in various portable devices.
  • the invention is usable in portable radio communication devices. Therefore the system may be used in a mobile telephone or a portable or mobile PMR radio.
  • the invention also may be used in a personal digital assistant (PDA) or laptop computer, linked for example by a radio or infra-red communication link to a cellular network.
  • PDA personal digital assistant
  • Such a network may be in a building, or a cellular telephone, or UMTS/3G network.
  • the invention may form part of a Distributed Speech Recognition (DSR) system.
  • DSR Distributed Speech Recognition
  • some speech processing would be performed remotely, i.e. the processing would be carried out at at least two different locations.

Landscapes

  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Otolaryngology (AREA)
  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Acoustics & Sound (AREA)
  • Signal Processing (AREA)
  • Circuit For Audible Band Transducer (AREA)
  • Measurement Of Velocity Or Position Using Acoustic Or Ultrasonic Waves (AREA)
  • Electrophonic Musical Instruments (AREA)
  • Devices For Supply Of Signal Current (AREA)

Claims (8)

  1. Verfahren zum Verbessern einer Tonsignalqualität zur Verwendung in einem portablen Tonverarbeitungsgerät, wobei das Verfahren die folgenden Schritte aufweist:
    - Empfangen (300) eines Tons von einer Tonquelle (412) durch ein Mikrophon-Array (402);
    - Detektieren (302) einer Bewegung des Mikrophon-Arrays (402) durch einen Plattformbewegungssensor (400) und Erzeugen eines Bewegungssignals, das eine solche Bewegung anzeigt;
    - Vorsehen eines Tonsignals von dem Mikrophon-Array (402) für einen Tonverbesserungsprozessor (408) und für eine Quellenstandortschätzeinrichtung (406);
    - Vorsehen des Bewegungssignals von dem Plattformbewegungssensor (400) an die Quellenstandortschätzeinrichtung (406);
    - wobei die Quellenstandortschätzeinrichtung (406) einen momentanen Standort der Tonquelle (412) relativ zu dem Mikrophon-Array (402) unter Verwendung des Tonsignals von dem Mikrophon-Array und des Signals, das durch den Plattformbewegungssensor bereitgestellt wird, schätzt;
    - Vorsehen eines Ergebnisses der Schätzung des momentanen Standorts der Tonquelle (412) relativ zu dem Mikrophon-Array (402) für den Tonverbesserungsprozessor (408);
    - wobei der Tonverbesserungsprozessor (408) die Tonsignalqualität verbessert (312);
    - Vorsehen des verbesserten Tonsignals für ein Tonverarbeitungsgerät.
  2. Verfahren gemäß Anspruch 1, wobei die Schätzung des momentanen Standorts der Tonquelle relativ zu dem Mikrophon-Array die folgenden Schritte aufweist:
    - Schätzen (206) einer Positionsänderung des Mikrophon-Arrays auf der Basis des Bewegungssignals von dem Plattformbewegungssensor;
    - Schätzen (208) einer momentanen Änderung eines Quellenrelativstandorts auf der Basis eines Ergebnisses der Schätzung des Standorts der Tonquelle relativ zu dem Mikrophon-Array und der Schätzung der Positionsänderung des Mikrophon-Arrays;
    - Kombinieren (210) der momentanen Änderung des Quellenrelativstandorts mit dem Standort der Tonquelle relativ zu dem Mikrophon-Array.
  3. Verfahren gemäß Anspruch 1 oder Anspruch 2, wobei zum Zwecke der Schätzung eines Standorts der Tonquelle relativ zu dem Mikrophon-Array Langzeitintervalle des Tonsignals analysiert werden, wobei die Langzeitintervalle in dem Bereich von 10ms bis 1s liegen.
  4. Verfahren gemäß einem vorhergehenden Anspruch, wobei zum Zwecke der Schätzung der Positionsänderung des Mikrophon-Arrays Kurzzeitintervalle des Bewegungssignals analysiert werden, wobei die Kurzzeitintervalle in dem Bereich von 0,1µs bis 10µs liegen.
  5. Vorrichtung zum Verbessern einer Tonsignalqualität zur Verwendung in einem portablen Tonverarbeitungsgerät (410), wobei die Vorrichtung aufweist:
    einen Tonverbesserungsprozessor (408);
    eine Quellenstandortschätzeinrichtung (406), die mit dem Tonverbesserungsprozessor verbunden ist;
    ein Mikrophon-Array (402), das mit dem Tonverbesserungsprozessor (408) und der Quellenstandortschätzeinrichtung (406) verbunden ist; und
    einen Plattformbewegungssensor (400), der mit der Quellenstandortschätzeinrichtung (406) verbunden ist;
    wobei das Mikrophon-Array (402) derart betreibbar ist, dass es dem Tonverbesserungsprozessor (408) und der Quellenstandortschätzeinrichtung (406) ein Tonsignal bereitstellt;
    wobei der Plattformbewegungssensor (400) derart betreibbar ist, dass er der Quellenstandortschätzeinrichtung (406) ein Bewegungssignal bereitstellt;
    wobei die Quellenstandortschätzeinrichtung (406) derart betreibbar ist, dass sie einen momentanen Standort der Tonquelle (412) relativ zu dem Mikrophon-Array (402) unter Verwendung des Tonsignals von dem Mikrophon-Array und des Bewegungssignals, das durch den Plattformbewegungssensor bereitgestellt wird, schätzt;
    wobei die Quellenstandortschätzeinrichtung derart betreibbar ist, dass sie ein Ergebnis der Schätzung des momentanen Standorts der Tonquelle (412) relativ zu dem Mikrophon-Array (402) dem Tonverbesserungsprozessor (408) bereitstellt;
    wobei der Tonverbesserungsprozessor (408) derart betreibbar ist, dass er die Tonsignalqualität verbessert (302) und ein verbessertes Tonsignal einem Tonverarbeitungsgerät bereitstellt.
  6. Vorrichtung gemäß Anspruch 5, wobei der Plattformbewegungssensor (400) an dem Mikrophon-Array fixiert ist.
  7. Vorrichtung gemäß Anspruch 5 oder Anspruch 6, wobei das Tonverarbeitungsgerät ein Spracherkennungsmodul ist.
  8. Mobiltelefon, portables oder mobiles PMR-Funkgerät, Handcomputer (PDA = Personal Digital Assistant) oder Laptop-Computer, das/der die Vorrichtung zum Verbessern der Tonsignalqualität gemäß einem der Ansprüche 5 bis 8 umfasst.
EP02009933A 2001-05-03 2002-05-03 Verfahren und System zur Schallbearbeitung Expired - Lifetime EP1257146B1 (de)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
GB0110861A GB2375276B (en) 2001-05-03 2001-05-03 Method and system of sound processing
GB0110861 2001-05-03

Publications (3)

Publication Number Publication Date
EP1257146A2 EP1257146A2 (de) 2002-11-13
EP1257146A3 EP1257146A3 (de) 2004-01-07
EP1257146B1 true EP1257146B1 (de) 2006-07-19

Family

ID=9913962

Family Applications (1)

Application Number Title Priority Date Filing Date
EP02009933A Expired - Lifetime EP1257146B1 (de) 2001-05-03 2002-05-03 Verfahren und System zur Schallbearbeitung

Country Status (5)

Country Link
EP (1) EP1257146B1 (de)
AT (1) ATE333774T1 (de)
DE (1) DE60213155T2 (de)
GB (1) GB2375276B (de)
PT (1) PT1257146E (de)

Families Citing this family (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6675027B1 (en) 1999-11-22 2004-01-06 Microsoft Corp Personal mobile computing device having antenna microphone for improved speech recognition
US7383181B2 (en) 2003-07-29 2008-06-03 Microsoft Corporation Multi-sensory speech detection system
US7447630B2 (en) 2003-11-26 2008-11-04 Microsoft Corporation Method and apparatus for multi-sensory speech enhancement
US7499686B2 (en) 2004-02-24 2009-03-03 Microsoft Corporation Method and apparatus for multi-sensory speech enhancement on a mobile device
US7574008B2 (en) 2004-09-17 2009-08-11 Microsoft Corporation Method and apparatus for multi-sensory speech enhancement
US7283850B2 (en) 2004-10-12 2007-10-16 Microsoft Corporation Method and apparatus for multi-sensory speech enhancement on a mobile device
US7346504B2 (en) 2005-06-20 2008-03-18 Microsoft Corporation Multi-sensory speech enhancement using a clean speech prior
US7680656B2 (en) 2005-06-28 2010-03-16 Microsoft Corporation Multi-sensory speech enhancement using a speech-state model
US7406303B2 (en) 2005-07-05 2008-07-29 Microsoft Corporation Multi-sensory speech enhancement using synthesized sensor signal
US7930178B2 (en) 2005-12-23 2011-04-19 Microsoft Corporation Speech modeling and enhancement based on magnitude-normalized spectra
US9185488B2 (en) * 2009-11-30 2015-11-10 Nokia Technologies Oy Control parameter dependent audio signal processing
JP5407848B2 (ja) * 2009-12-25 2014-02-05 富士通株式会社 マイクロホンの指向性制御装置
EP2638694A4 (de) * 2010-11-12 2017-05-03 Nokia Technologies Oy Tonverarbeitungsvorrichtung
US20120183156A1 (en) * 2011-01-13 2012-07-19 Sennheiser Electronic Gmbh & Co. Kg Microphone system with a hand-held microphone
CN104244142B (zh) * 2013-06-21 2018-06-01 联想(北京)有限公司 一种麦克风阵列、实现方法及电子设备
GB2521175A (en) * 2013-12-11 2015-06-17 Nokia Technologies Oy Spatial audio processing apparatus
WO2017218399A1 (en) 2016-06-15 2017-12-21 Mh Acoustics, Llc Spatial encoding directional microphone array
US10477304B2 (en) 2016-06-15 2019-11-12 Mh Acoustics, Llc Spatial encoding directional microphone array

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3025359A (en) * 1960-02-25 1962-03-13 Gulton Ind Inc Vibration-compensated pressure sensitive microphone
US4984087A (en) * 1988-05-27 1991-01-08 Matsushita Electric Industrial Co., Ltd. Microphone apparatus for a video camera
WO1995013690A1 (fr) * 1993-11-08 1995-05-18 Sony Corporation Detecteur d'angle et appareil de lecture audio utilisant ledit detecteur
DE19943872A1 (de) * 1999-09-14 2001-03-15 Thomson Brandt Gmbh Vorrichtung zur Anpassung der Richtcharakteristik von Mikrofonen für die Sprachsteuerung

Also Published As

Publication number Publication date
GB0110861D0 (en) 2001-06-27
ATE333774T1 (de) 2006-08-15
GB2375276A (en) 2002-11-06
EP1257146A3 (de) 2004-01-07
EP1257146A2 (de) 2002-11-13
PT1257146E (pt) 2006-11-30
DE60213155T2 (de) 2007-07-26
DE60213155D1 (de) 2006-08-31
GB2375276B (en) 2003-05-28

Similar Documents

Publication Publication Date Title
EP1257146B1 (de) Verfahren und System zur Schallbearbeitung
JP4694700B2 (ja) 話者の方向をトラッキングする方法およびシステム
CN112019976B (zh) 用于处理音频信号的设备和方法
CN100370830C (zh) 用于音频/图像的说话者检测和定位的方法和装置
JP3598932B2 (ja) 話者方向検出回路及びそれに用いる話者方向検出方法
US8635066B2 (en) Camera-assisted noise cancellation and speech recognition
US7813923B2 (en) Calibration based beamforming, non-linear adaptive filtering, and multi-sensor headset
CN106782589B (zh) 移动终端及其语音输入方法和装置
CN111641794B (zh) 声音信号采集方法和电子设备
US8868413B2 (en) Accelerometer vector controlled noise cancelling method
US20180132042A1 (en) Laser microphone utilizing mirrors having different properties
US20060182291A1 (en) Acoustic processing system, acoustic processing device, acoustic processing method, acoustic processing program, and storage medium
WO2014161309A1 (zh) 一种移动终端实现声源定位的方法及装置
TW202147862A (zh) 強烈雜訊干擾存在下穩健的揚聲器定位系統與方法
US20030171932A1 (en) Speech recognition
JP2007221300A (ja) ロボット及びロボットの制御方法
WO2021244056A1 (zh) 一种数据处理方法、装置和可读介质
EP4220637B1 (de) Verfahren und vorrichtung zur mehrkanal-audiosignalerfassung und system
CN110572600A (zh) 一种录像处理方法及电子设备
CN111007462A (zh) 定位方法、定位装置、定位设备及电子设备
US20030031327A1 (en) Method and apparatus for providing multiple output channels in a microphone
EP1266538B1 (de) Räumliches schallsteuerungssystem
US8831681B1 (en) Image guided audio processing
JP2000311077A (ja) 音声情報入力装置
JP2002108391A (ja) 音声入力装置

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LI LU MC NL PT SE TR

AX Request for extension of the european patent

Free format text: AL;LT;LV;MK;RO;SI

PUAL Search report despatched

Free format text: ORIGINAL CODE: 0009013

AK Designated contracting states

Kind code of ref document: A3

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LI LU MC NL PT SE TR

AX Request for extension of the european patent

Extension state: AL LT LV MK RO SI

RIC1 Information provided on ipc code assigned before grant

Ipc: 7H 04R 1/40 B

Ipc: 7H 04R 3/00 A

Ipc: 7G 10K 11/34 B

17P Request for examination filed

Effective date: 20040707

17Q First examination report despatched

Effective date: 20040805

AKX Designation fees paid

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LI LU MC NL PT SE TR

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE CH CY DE DK ES FI FR GR IE IT LI LU MC NL PT SE TR

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT;WARNING: LAPSES OF ITALIAN PATENTS WITH EFFECTIVE DATE BEFORE 2007 MAY HAVE OCCURRED AT ANY TIME BEFORE 2007. THE CORRECT EFFECTIVE DATE MAY BE DIFFERENT FROM THE ONE RECORDED.

Effective date: 20060719

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20060719

Ref country code: BE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20060719

Ref country code: CH

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20060719

Ref country code: LI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20060719

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REF Corresponds to:

Ref document number: 60213155

Country of ref document: DE

Date of ref document: 20060831

Kind code of ref document: P

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20061019

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20061030

REG Reference to a national code

Ref country code: SE

Ref legal event code: TRGR

REG Reference to a national code

Ref country code: PT

Ref legal event code: SC4A

Free format text: AVAILABILITY OF NATIONAL TRANSLATION

Effective date: 20061006

ET Fr: translation filed
PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: NL

Payment date: 20070412

Year of fee payment: 6

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20070420

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: IT

Payment date: 20070518

Year of fee payment: 6

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20070531

REG Reference to a national code

Ref country code: PT

Ref legal event code: MM4A

Free format text: LAPSE DUE TO NON-PAYMENT OF FEES

Effective date: 20080204

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20061020

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20080204

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20070503

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20080530

Year of fee payment: 7

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20070503

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20081201

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20080503

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20060719

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20070503

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20060719

REG Reference to a national code

Ref country code: FR

Ref legal event code: ST

Effective date: 20100129

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20090602

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20080424

Year of fee payment: 7

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20091201

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: SE

Payment date: 20160506

Year of fee payment: 15

REG Reference to a national code

Ref country code: SE

Ref legal event code: EUG

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20170504