EP2254104B1 - Method for automatic recognition of a change in a situation - Google Patents
Method for automatic recognition of a change in a situation Download PDFInfo
- Publication number
- EP2254104B1 EP2254104B1 EP10004126.8A EP10004126A EP2254104B1 EP 2254104 B1 EP2254104 B1 EP 2254104B1 EP 10004126 A EP10004126 A EP 10004126A EP 2254104 B1 EP2254104 B1 EP 2254104B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- values
- behaviour
- situation
- learned
- objects
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Not-in-force
Links
- 238000000034 method Methods 0.000 title claims description 39
- 230000006399 behavior Effects 0.000 claims description 83
- 238000010972 statistical evaluation Methods 0.000 claims description 17
- 230000006870 function Effects 0.000 claims description 10
- 238000012935 Averaging Methods 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 claims description 3
- 241001465754 Metazoa Species 0.000 claims description 2
- 238000003384 imaging method Methods 0.000 claims description 2
- 239000002245 particle Substances 0.000 claims description 2
- 230000000875 corresponding effect Effects 0.000 description 24
- 238000012544 monitoring process Methods 0.000 description 9
- 238000001514 detection method Methods 0.000 description 6
- 230000035945 sensitivity Effects 0.000 description 4
- 230000015572 biosynthetic process Effects 0.000 description 3
- 238000001454 recorded image Methods 0.000 description 3
- 238000013480 data collection Methods 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 230000003542 behavioural effect Effects 0.000 description 1
- 230000001447 compensatory effect Effects 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 239000000446 fuel Substances 0.000 description 1
- 230000012447 hatching Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000002269 spontaneous effect Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/01—Detecting movement of traffic to be counted or controlled
- G08G1/04—Detecting movement of traffic to be counted or controlled using optical or ultrasonic detectors
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/01—Detecting movement of traffic to be counted or controlled
- G08G1/0104—Measuring and analyzing of parameters relative to traffic conditions
Definitions
- the invention relates to a method for automatic detection of a situation change within a spatial area to be monitored by means of sensors.
- the invention also relates to a device for this purpose.
- the most commonly used sensors for monitoring are cameras that record the area to be monitored.
- the captured image of a camera is then forwarded in most cases to a central, where it is then displayed on a screen for an operator.
- a central where it is then displayed on a screen for an operator.
- For the operator there is the problem that with an increase in the number of cameras and the number of images that need to be monitored, are increased. Ultimately, this leads to increased complexity, which in the worst case is at the expense of safety.
- US 2003/0107650 A1 a monitoring and security system which automatically issues a warning when corresponding events that were previously programmed occur in the monitoring of an area.
- this system can alert you to possible shoplifting if a person opens a plastic bag inside the area to be monitored (here a shop).
- the system is set up in such a way that it picks up the noise within the store and examines it for the characteristic acoustic pattern of an opening plastic bag.
- So z. B. in the US 2004/0130620 Al discloses a method for video analysis, in which a plurality of image sensors record an overlapping area, the objects within the surveillance area moving objects can be tracked by means of a corresponding tracking algorithm, even across the recording limits of an image sensor.
- a behavior of the objects is then determined for at least one of the fields at least one of its object properties is learned by evaluating the values of the object property stored in the field by means of statistical methods. This results in a corresponding behavior of the objects with respect to the monitored property of the objects for each field of the entire spatial area.
- a behavior in terms of speed can be learned what z. B. expressed in terms of average speed.
- the learned behavior of at least one field is compared with currently determined values of the corresponding field, whereby based on the comparison on a situation change is closed.
- the behavior in step b) is learned by the last ascertained values of the corresponding field, which lie within a certain period of time. So it is z. B. conceivable that only those values are taken into account when learning the behavior that within the last z. For example, five minutes were recorded. This makes it possible for the system to re-learn the changed situation after a certain time and to view it as typical behavior.
- a behavior is learned for a plurality of discrete periods, which is the basis for the comparison for recognizing a situation change in step c). It is conceivable that a behavior of a period that lies farther back in the comparison is given a lower weighting than those periods that were not so long ago, so that young periods receive a higher weighting than older ones.
- a position indication is determined which represents the corresponding spatial position of the object within the spatial area.
- This so-called determination position which indicates the position at which the value was determined within the spatial area at the corresponding object, is then used to determine the field with which the determined value is linked.
- the above-described determination position is converted into a uniform position coordinate system or uniform world coordinates so that the position information of the determined values can be compared with one another. In this way, the values can be entered in the corresponding fields regardless of the sensor position.
- step c) it is particularly advantageous if the detection of a situation change in step c) takes place as a function of the comparison between a learned behavior from step b) and a behavior based on the statistical evaluation of currently determined values.
- a current behavior for one or more fields is continuously determined and compared with a previously learned behavior for the corresponding fields. This makes it possible to detect changes in the situation that affect the behavior of the objects within one or more fields, with respect to one or more object properties.
- a situation change is detected as a function of the comparison between the learned behavior from step b) and current values of a trajectory of a specific object.
- values relating to one or more object properties of a specific object are detected, while the object moves on a specific path (trajectory) through the area to be monitored. These values recorded in this way are then assigned to a specific trajectory of the object and compared with the learned behavior of the other objects. If the behavior with respect to one or more object properties of the specific object deviates significantly compared with the learned behavior of the other objects, it is possible to conclude that the situation has changed.
- individual objects can be identified, which behave atypically compared to the typical behavior of the entire scene, which also represents a situation change.
- mean values, sums, products and / or standard deviations are formed for the statistical evaluation, which can then be compared with one another. In principle, all arithmetic operations are possible which the expert would use for statistical purposes.
- Particularly suitable sensors for monitoring the spatial area are imaging sensors which are advantageously equipped with image recognition methods, radar sensors or RFID transponders.
- objects to be monitored come in particular people, animals, ie living beings in general, road, rail, air or water vehicles into consideration as well as particles and other objects, It is particularly advantageous if the sensors mentioned for determining properties such as activity of Objects that are set to speed the objects or their direction.
- the threshold value can be designed in such a way that sudden changes in the behavior of the objects are recognized as a situation change.
- threshold values can also be stored for a field if several typical behaviors arise with regard to an object property.
- the invention relates to a computer program product for carrying out the above method and a device for this purpose.
- Fig. 1a and 1b sketchy show the inclusion of a road junction, which is to be used as an embodiment of the further embodiments.
- the intersection is monitored by two video cameras, where Fig. 1a the image of the first video camera can be seen from a first angle while Fig. 1b See the image of a second video camera from a different angle.
- values of specific object properties are identified from the recorded image information.
- the objects are vehicles passing through the intersection.
- the speed and direction are recognized by the image recognition software for each vehicle and stored accordingly.
- per vehicle which is in the field of view of the cameras, continuously determined both the speed and the direction.
- the recorded image area is subdivided into a plurality of fields 1.
- this is only sketchily indicated and usually extends over the entire scene.
- the position at which the corresponding values were determined is also determined. So that these values can also be utilized independently of the location of the corresponding camera, this position information is converted into a world coordinate system and stored accordingly.
- a vehicle which by the field of view of the two cameras in Fig. 1a and 1b a plurality of speed and direction values at different positions or a plurality of speed and direction values at the same positions, since the values can be detected by a plurality of different sensors.
- Each speed and direction value is now assigned the corresponding field 1 within which the determined position lies, based on its position.
- exactly one such field can be assigned to each measured speed and direction value.
- Fig. 2 shows.
- the hatched fields 2 are the fields in which object values of different objects have been stored. The white fields, however, have no stored values.
- a derived card-like structure is in Fig. 3 to recognize.
- Fig. 3 This is a data structure in which the mean value of the values stored in this field was formed in each field. In this embodiment, this was done with the object property "Speed". But it can also derive other card-like structures, such. For example, those in which the standard deviation is determined as a statistical evaluation for each field.
- the values are determined for each field whose property is to be derived statistically. In this embodiment, these would be the speed values. Then the statistical evaluation is carried out for each field. Since the statistical evaluation is carried out separately for each field, this results in different fields for each field Fig. 3 represented by a different hatching in the individual fields.
- the behavior of the objects can be determined with respect to certain object properties within the scene, whereby the learned behavior is understood as a typical behavior of the objects. Now changes the scene by z. If, for example, the speed becomes slower or faster, this ultimately also results in the statistical evaluation so that a situation change can then be concluded on the basis of a comparison between the learned behavior and the current values. Thus, such a situation change is ultimately always a deviation of the behavior of the objects from the typical behavior of the objects within this scene with respect to at least one object property.
- Fig. 4 shows by way of example the comparison between the learned behavior and a current behavior with respect to the two object properties speed (S) and direction (D) on the basis of the statistical evaluation "averaging" (S ⁇ , D ⁇ ).
- map-type data structures have been learned as described above. For this purpose, data structures with respect to the speed and direction of the objects within the observed scene were learned. From the determined values for the speed S and the direction D, the average value was then derived for each field, which is referred to as card-like data structure 41 and 42 in FIG Fig. 4 can be seen.
- a common behavior was derived, which is also stored in the form of a card-like data structure 43 and 44 respectively.
- a common learned behavior can be done with the help of different weights z. B. younger learned behavior to be considered more than older.
- the system is self-learning and it adapts to the new situation.
- the system automatically adapts to the given situation without having to readjust it. If the traffic jam clears up again after a while, the average speed increases abruptly, which is again recognized as a change in the situation, as is the behavior of the objects from the typical behavior learned differs.
- the system can recognize both the formation of a congestion and its resolution afterwards without any problems and manual intervention and adapts by learning the standard deviation also to expected levels of change.
- the average value (mean) in a large scene often varies only slightly, although e.g. has jammed on a roadway, e.g. because the traffic jam initially forms only on one lane and then "spreads" to the other lanes or the jam forms only in one direction, while the opposite lane is free. This can lead to compensatory changes that cancel each other out, so that no change in the situation is detected in the sum.
- the standard deviation is stored for each field, which is then used for a threshold value comparison, so as to be able to determine selectively for individual fields situation changes.
- a sensitivity can be set so that in some fields situation changes result, since here the standard deviations are relatively low, while in other fields this is recognized as normal behavior due to relatively large standard deviations.
- the method of the invention adjusts the standard deviation in the fields where a large change occurs over time (eg, when a congestion occurs), while the other fields remain sensitive to changes (eg, the oncoming lane where no congestion has occurred ).
- Fig. 5 schematically shows another embodiment in which a particular or specific object was tracked by means of a corresponding tracking algorithm.
- a trajectory 51 is recorded by the object to be observed, which has a plurality of data points. At these data points were then z. B. corresponding values of object properties to be measured and stored. These data points are then compared to the learned behavior of the overall scene to determine if the object behaves in a typical or atypical manner relative to the overall behavior of all objects. This is done at the same time for all objects in the scene individually.
- each data point of the trajectory is compared with the underlying card-like data structure, in particular with the fields in which the corresponding data point falls.
- this is shown in a scene with the data point 52.
- the corresponding object property here the speed
- Fig. 5 can be with this embodiment in Fig. 5 recognize if z. B, Cyclists move very fast through a scene or pedestrians who run aimlessly or suddenly start to run or walk over otherwise barely used areas like meadows. Also can be seen with this system a ghost driver. But it can also monitor other objects such as aircraft so that when monitoring an airport z. For example, situations may be identified where service vehicles are moving in unusual areas or at atypical speeds. It is also possible to detect launches of rarely used runways as well as changes in the starting direction and the use of a test vehicle with this method.
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Chemical & Material Sciences (AREA)
- Analytical Chemistry (AREA)
- Traffic Control Systems (AREA)
- Image Analysis (AREA)
Description
Die Erfindung betrifft ein Verfahren zum automatischen Erkennen einer Situationsänderung innerhalb eines mittels Sensoren zu überwachenden räumlichen Gebietes. Die Erfindung betrifft ebenfalls eine Vorrichtung hierzu.The invention relates to a method for automatic detection of a situation change within a spatial area to be monitored by means of sensors. The invention also relates to a device for this purpose.
Häufig wird unter dem Aspekt der Sicherheit die mitunter sehr komplexe Anforderung gestellt, Orte oder Gebiete, in denen sich viele Akteure oder Objekte bewegen und miteinander interagieren, mittels entsprechender Sensoren zu überwachen. Beispiele für solche Orte oder Gebiete sind Straßen, Parks, Bahnhöfe, öffentliche Plätze, Flughäfen, Autobahnen oder Einkaufszentren. Innerhalb dieser Orte oder Gebiete interagieren die sich daran befindlichen Akteure oder Objekte miteinander, die beispielsweise Fußgänger, Fahrradfahrer, PKWs, Schiffe, Flugzeuge, Sportler oder Kunden sein können. Dabei wird meist bei der Überwachung insbesondere darauf abgestellt, dass die in dem Gebiet befindlichen Objekte oder Akteure innerhalb der festgelegten Rahmenbedingungen bewegen, so dass atypische oder gar bedrohliche Ereignisse durch die jeweiligen Überwacher (z. B. Fluglotsen, Sicherheitsdienste) schnell erkannt werden können,Frequently, from the point of view of security, the sometimes very complex requirement is made to monitor places or areas in which many actors or objects move and interact with each other by means of corresponding sensors. Examples of such places or areas are streets, parks, railway stations, public squares, airports, highways or shopping malls. Within these locations or areas, the actors or objects attached to each other interact with each other, which may be, for example, pedestrians, cyclists, cars, ships, aircraft, athletes or customers. In this case, surveillance is particularly geared in particular to ensuring that the objects or actors located in the area move within the specified boundary conditions, so that atypical or even threatening events can be quickly detected by the respective supervisors (eg air traffic controllers, security services),
Die zur Überwachung am häufigsten verwendeten Sensoren sind dabei Kameras, die das zu überwachende räumliche Gebiet aufnehmen. Das so aufgenommene Bild einer Kamera wird dann in den meisten Fällen an eine Zentrale weitergeleitet, wo es dann auf einem Bildschirm für einen Operator dargestellt wird. Gerade bei Gebieten mit einer sehr großen räumlichen Ausdehnung kommt es sehr schnell zu einer großen Anzahl von benötigten Kameras, um alle Bereiche des zu überwachenden Gebietes erfassen und abdecken zu können. Für den Operator stellt sich dabei das Problem, dass mit einer Erhöhung der Anzahl der Kameras auch die Anzahl der Bilder, die überwacht werden müssen, erhöht werden. Dies führt letztlich zu einer gesteigerten Komplexität, die im ungünstigsten Fall zu Lasten der Sicherheit geht.The most commonly used sensors for monitoring are cameras that record the area to be monitored. The captured image of a camera is then forwarded in most cases to a central, where it is then displayed on a screen for an operator. Especially in areas with a very large spatial extent it comes very quickly to a large number of required cameras to capture and cover all areas of the area to be monitored. For the operator, there is the problem that with an increase in the number of cameras and the number of images that need to be monitored, are increased. Ultimately, this leads to increased complexity, which in the worst case is at the expense of safety.
Um diese Überwachungskomplexität zu verringern, gibt es zum Einen die Möglichkeit, die Anzahl der zu überwachenden Bilder bzw. allgemein gesprochen, die Anzahl der zur Überwachung benötigten Informationen, zu reduzieren, was letztlich zum Verlust von Informationen führen kann. Zum Anderen gibt es die Möglichkeit, mit Hilfe von Bilderkennungsverfahren die Bildsignale zu analysieren und auf entsprechend programmierte Signaturen hin zu filtern.On the one hand, in order to reduce this monitoring complexity, there is the possibility of reducing the number of images to be monitored, or generally speaking, the number of information required for monitoring, which can ultimately lead to the loss of information. On the other hand, there is the possibility of analyzing the image signals with the aid of image recognition methods and filtering them on to suitably programmed signatures.
So beschreibt beispielsweise die
Nachteilig dabei ist insbesondere die Tatsache, dass dieses System nur solche Ereignisse erkennt, auf die es speziell programmiert wurde. So ist dieses System nicht ohne Weiteres auf eine andere Szenerie oder Umgebung übertragbar.The disadvantage here is in particular the fact that this system recognizes only those events to which it has been specially programmed. So this system is not easily transferable to another scenery or environment.
Auch bei der Verkehrskontrolle bzw. bei der Verkehrslagenerfassung spielt Überwachung eine wichtige Rolle. So wird auf deutschen Autobahnen mittels entsprechender Sensoren ständig die Verkehrsdichte erfasst, um so z. B. Verkehrsleiteinrichtungen, wie sie auf der Autobahn A2 installiert sind, in Abhängigkeit der Verkehrsdichte ansteuern zu können.Surveillance also plays an important role in traffic control and traffic situation detection. Thus, the traffic density is constantly detected on German motorways by means of appropriate sensors, so as. B. traffic guidance devices, as they are installed on the A2 motorway to be able to control depending on the traffic density.
Zu dieser Thematik ist aus der
Nachteilig dabei ist bei diesem System wie auch bei den anderen aus dem Stand der Technik bekannten Systemen, die Tatsache, dass Signaturen der zu detektierenden Ereignisse vom Operator festgelegt werden müssen. Außerdem werden in den Wochen nach der Installation meist häufig weitere Wartungsdurchläufe nötig, um durch Änderungen im beobachteten Bereich, durch Wetteränderungen, Jahreszeiten usw. das System neu anzupassen. Dabei sind diese Systeme meist nur in der Lage, genau festgelegte Vorgänge innerhalb ihres Bereiches zu erkennen.A disadvantage of this system as well as the other known from the prior art systems, the fact that signatures of the events to be detected must be set by the operator. In addition, in the weeks following installation, often additional maintenance runs are required to re-adjust the system through changes in the observed area, weather changes, seasons, and so on. These systems are usually only able to detect exactly defined processes within their range.
So wird z. B. in der
Auch die Dokumente
Im Hinblick auf die aus dem Stand der Technik bekannten Nachteile ist es Aufgabe der vorliegenden Erfindung, ein verbessertes Verfahren zum Detektieren von Ereignissen innerhalb eines zu überwachenden räumlichen Gebietes anzugeben.In view of the disadvantages known from the prior art, it is an object of the present invention to provide an improved method for detecting events within a spatial area to be monitored.
Die Aufgabe wird mit dem Verfahren der eingangs genannten Art gelöst, wobei das zu überwachende räumliche Gebiet in eine Mehrzahl von Feldern unterteilt wird, mit den Schritten:
- a) Erfassen von Werten mittels der Sensoren hinsichtlich mindestens einer Eigenschaft von Objekten, die sich innerhalb des zu überwachenden räumlichen Gebietes befinden, und deren Felder, innerhalb dessen der jeweilige Wert der Eigenschaft erfasst wurde,
- b) Lernen zumindest eines Verhaltens der Objekte bezüglich mindestens einer der Objekteigenschaften anhand einer statistischen Auswertung der ermittelten Werte für mindestens ein Feld, und
- c) Erkennen einer Situationsänderung in Abhängigkeit eines Vergleichs zwischen dem erlernten Verhalten mindestens eines Feldes und zumindest einem aktuell ermitteltem Wert des mindestens einen Feldes bezüglich mindestens einer der Objekteigenschaften.
- a) detecting values by means of the sensors with regard to at least one property of objects which are located within the spatial area to be monitored and their fields within which the respective value of the property was detected,
- b) learning at least one behavior of the objects with regard to at least one of the object properties on the basis of a statistical evaluation of the determined values for at least one field, and
- c) detecting a situation change in dependence on a comparison between the learned behavior of at least one field and at least one currently determined value of the at least one field with respect to at least one of the object properties.
Damit wird es möglich, ein Gebiet mit entsprechenden Sensoren zu überwachen, ohne das es speziell auf bestimmte Ereignisse, die es erkennen soll, programmiert werden muss. Dazu werden mit Hilfe der Sensoren, die das Gebiet überwachen, Werte von den Objekten erfasst, die sich innerhalb des zu überwachenden räumlichen Gebietes befinden. Die erfassten Werte sind dabei Werte entsprechender Eigenschaften der Objekte, die mit Hilfe der Sensoren erfasst bzw. ermittelt oder gemessen werden können. Eine solche Objekteigenschaft könnte dabei z. B. die Geschwindigkeit, Richtung oder aber auch ganz allgemein gesprochen die Aktivität eines Objektes innerhalb eines entsprechenden Feldes sein. Es ist aber auch denkbar, dass weitere Informationen über die Objekte als Eigenschaften ermittelt werden können, wie z. B. der Treibstoffvorrat eines Flugzeuges, je nachdem was für eine Art von Sensoren verwendet wird.This makes it possible to monitor an area with corresponding sensors without having to program it specifically for certain events it is to recognize. For this purpose, using the sensors monitoring the area, values are acquired from the objects which are located within the spatial area to be monitored. The recorded values are values of corresponding properties of the objects which can be detected or determined or measured with the aid of the sensors. Such an object property could be z. As the speed, direction or in general terms, the activity of an object within a corresponding field. But it is also conceivable that further information about the objects can be determined as properties, such. As the fuel supply of an aircraft, depending on what kind of sensors is used.
Darüber hinaus wird beim Erfassen der Werte ermittelt, innerhalb welchen Feldes der entsprechende Wert erfasst wurde, so dass sich ein Datenpaar aus einem Wert einer Objekteigenschaft und dazugehörigen Feld des räumlichen Gebietes ergibt. Im nächsten Schritt wird dann für mindestens eines der Felder ein Verhalten der Objekte bezüglich mindestens einer ihrer Objekteigenschaften gelernt, indem die in dem Feld hinterlegten Werte der Objekteigenschaft mittels statistischer Methoden ausgewertet werden. Somit ergibt sich für jedes Feld des gesamten räumlichen Gebietes ein entsprechendes Verhalten der Objekte bezüglich der überwachten Eigenschaft der Objekte.In addition, when acquiring the values, it is determined within which field the corresponding value was acquired, so that a data pair results from a value of an object property and associated field of the spatial area. In the next step, a behavior of the objects is then determined for at least one of the fields at least one of its object properties is learned by evaluating the values of the object property stored in the field by means of statistical methods. This results in a corresponding behavior of the objects with respect to the monitored property of the objects for each field of the entire spatial area.
So kann beispielsweise bei der Überwachung einer Straße für jedes Feld ein Verhalten hinsichtlich der Geschwindigkeit gelernt werden, was sich z. B. in Form der Durchschnittsgeschwindigkeit ausdrücken lässt. Um nun eine Veränderung der beobachten Situation erkennen zu können, die sich letztlich aus einer spontanen Verhaltensänderung einer oder mehrerer Objekte innerhalb der Gesamtszene ergibt, wird das erlernte Verhalten mindestens eines Feldes mit aktuell ermittelten Werten des entsprechenden Feldes verglichen, wobei anhand des Vergleiches auf eine Situationsänderung geschlossen wird.Thus, for example, when monitoring a road for each field, a behavior in terms of speed can be learned what z. B. expressed in terms of average speed. In order to be able to detect a change in the observed situation, which ultimately results from a spontaneous behavioral change of one or more objects within the overall scene, the learned behavior of at least one field is compared with currently determined values of the corresponding field, whereby based on the comparison on a situation change is closed.
Dabei kann selbstverständlich nicht nur ein Verhalten bezüglich einer Eigenschaft und bezüglich eines Feldes gelernt werden, sondern auch mehrere Verhalten pro Feld und Eigenschaft. Solche Konstellationen treten z.B. in Kreuzungsbereichen auf, in denen es z.B. zwei Häufungspunkte für die Geschwindigkeit gibt, und zwar einmal für Fahrzeuge und einmal für Fußgänger. Die aktuell ermittelten Werte werden dann sowohl mit dem einen als auch mit dem anderen Verhalten verglichen (Schritt c)), wobei eine Situationsänderung dann angenommen wird, wenn sich die aktuellen Werte unter keines der Verhalten subsumieren lassen.Of course, not only a behavior regarding a property and a field can be learned, but also several behaviors per field and property. Such constellations occur e.g. in crossing areas where e.g. There are two accumulation points for speed, once for vehicles and once for pedestrians. The currently determined values are then compared with both the one and the other behavior (step c)), wherein a change of situation is assumed if the current values can not be subsumed under any of the behaviors.
So erkennt das oben genannte Verfahren z. B. eine sprunghafte Änderung der Durchschnittsgeschwindigkeit der Fahrzeuge auf der Straße, so dass somit die Bildung eines Staus bzw. das Auflösen eines Staus erkannt wird. Der Vorteil dieses Verfahrens besteht dabei darin, dass durch die Aufteilung des räumlichen Gebietes in eine Mehrzahl von Feldern lokale Situationsänderungen innerhalb eines großen räumlichen Gebietes ohne Weiteres detektierbar sind sowie die Integration mehrerer Sensoren, die das räumliche Gebiet aus unterschiedlichen Sichtweisen aufnehmen, problemlos möglich ist. Darüber hinaus ist bei diesem Verfahren keine spezielle Programmierung auf entsprechend zu detektierende Ereignissen notwendig.So recognizes the above method z. B. a sudden change in the average speed of the vehicles on the road, so that thus the formation of a jam or the resolution of a jam is detected. The advantage of this method is that by the division of the spatial area in a plurality of fields local situation changes within a large spatial area are readily detectable and the integration of multiple sensors that record the spatial area from different perspectives, is easily possible. In addition, in this method, no special programming on corresponding events to be detected is necessary.
Vorteilhafter Weise wird das Verhalten in Schritt b) durch die zuletzt ermittelten Werte des entsprechenden Feldes gelernt, die innerhalb eines bestimmten Zeitraumes liegen. So ist es z. B. denkbar, dass nur solche Werte beim Lernen des Verhaltens berücksichtigt werden, die innerhalb der letzten z. B. fünf Minuten erfasst wurden. Dadurch wird es möglich, dass das System nach einer gewissen Zeit die sich geänderte Situation neu lernt und als typisches Verhalten ansieht.Advantageously, the behavior in step b) is learned by the last ascertained values of the corresponding field, which lie within a certain period of time. So it is z. B. conceivable that only those values are taken into account when learning the behavior that within the last z. For example, five minutes were recorded. This makes it possible for the system to re-learn the changed situation after a certain time and to view it as typical behavior.
Es ist aber auch denkbar, dass für mehrere diskrete Zeiträume jeweils ein Verhalten gelernt wird, das dem Vergleich zum Erkennen einer Situationsänderung in Schritt c) zugrunde gelegt wird, Dabei ist es denkbar, dass ein Verhalten eines Zeitraumes, der weiter zurückliegt, bei dem Vergleich eine geringere Gewichtung erhält, als jene Zeiträume, die noch nicht so lange her sind, so dass junge Zeiträume eine höhere Gewichtung erhalten als ältere.However, it is also conceivable that a behavior is learned for a plurality of discrete periods, which is the basis for the comparison for recognizing a situation change in step c). It is conceivable that a behavior of a period that lies farther back in the comparison is given a lower weighting than those periods that were not so long ago, so that young periods receive a higher weighting than older ones.
Besonders vorteilhaft ist es aber auch, dass jeweilige Verhalten der Mehrzahl der Zeiträume zu einem gemeinsamen Verhalten zusammenzuführen, wobei auch hier ältere Zeiträume eine andere Gewichtung bekommen können als jüngere, Durch das Zusammenführen verschiedener erlernter Verhalten unterschiedlichster Zeiträume kann somit ein typisches Verhalten der Objekte innerhalb der zu beobachtenden Szene abgebildet werden. Dabei können die unterschiedlich gelernten Verhalten z. B. mittels Mittelwertbildung zusammengeführt werden.However, it is also particularly advantageous for the respective behavior of the majority of the time periods to be combined to form a common behavior, in which case older periods can also be given a different weighting than younger ones. By combining different learned behaviors of the most varied time periods, a typical behavior of the objects within the to be observed scene to be observed. The differently learned behavior z. B. be merged by means of averaging.
An dieser Stelle sei erwähnt, dass pro Feld und Zeitraum mehrere Verhalten entsprechend der Objekteigenschaften erlernt werden können, So ist es denkbar, dass für eine Objekteigenschaft 1 ein Verhalten gelernt wird und gleichzeitig für eine Objekteigenschaft 2, die z. B. mittels anderer Sensoren aufgenommen wird, ebenfalls ein Verhalten gelernt wird, so dass sich für jede Objekteigenschaft ein entsprechend gelerntes Verhalten ergibt. Diese pro Objekteigenschaft erlernten Verhalten können dann sowohl einzeln betrachtet als auch zusammengeführt werden.At this point it should be mentioned that per field and time period several behaviors can be learned in accordance with the object properties. Thus it is conceivable that a behavior is learned for an
Vorteilhaft ist es, wenn vor dem Vergleich in Schritt c) die Felder maskiert werden, d. h. ausschließlich die Felder zum Vergleich herangezogen werden, für die überhaupt entsprechende Werte hinterlegt sind, Dabei bleiben alle die Felder unberücksichtigt, für die keine oder nur eine sehr geringe Anzahl von Werten hinterlegt sind.It is advantageous if the fields are masked before the comparison in step c), d. H. Only those fields are used for comparison, for which corresponding values are stored at all, whereby all those fields for which no or only a very small number of values are stored are ignored.
Ganz besonders vorteilhaft ist es, wenn zusätzlich zu der Erfassung der Werte eine Positionsangabe ermittelt wird, welche die entsprechende Ortsposition des Objektes innerhalb des räumlichen Gebietes repräsentiert. Diese sog. Ermittlungsposition, welche die Position angibt, an der der Wert innerhalb des räumlichen Gebietes bei dem entsprechenden Objekt ermittelt wurde, wird dann zur Ermittlung des Feldes, mit der der ermittelte Wert verknüpft wird, herangezogen.It is particularly advantageous if, in addition to the detection of the values, a position indication is determined which represents the corresponding spatial position of the object within the spatial area. This so-called determination position, which indicates the position at which the value was determined within the spatial area at the corresponding object, is then used to determine the field with which the determined value is linked.
Damit mehrere Sensoren auch überlappende Bereiche des zu überwachenden Gebietes abdecken können, ist es ganz besonders vorteilhaft, wenn diese oben beschriebene Ermittlungsposition in ein einheitliches Positionskoordinatensystem bzw. in einheitliche Weltkoordinaten umgerechnet wird, damit die Positionsangaben der ermittelten Werte untereinander vergleichbar sind. So können dann die Werte unabhängig von der Sensorposition in die entsprechenden Felder eingetragen werden.In order for several sensors to be able to cover overlapping areas of the area to be monitored, it is particularly advantageous if the above-described determination position is converted into a uniform position coordinate system or uniform world coordinates so that the position information of the determined values can be compared with one another. In this way, the values can be entered in the corresponding fields regardless of the sensor position.
In einer konkreten Ausführungsform ist es ganz besonders vorteilhaft, wenn das Erkennen einer Situationsänderung in Schritt c) in Abhängigkeit des Vergleiches zwischen einem erlernten Verhalten aus Schritt b) und einem anhand statistischer Auswertung von aktuell ermittelten Werten aktuellem Verhalten erfolgt. Dabei wird kontinuierlich ein aktuelles Verhalten für eines oder mehrere Felder ermittelt und mit einem zuvor erlernten Verhalten für die entsprechenden Felder verglichen. Dadurch wird es möglich, Situationsänderungen daran zu erkennen, dass sich das Verhalten der Objekte innerhalb einer oder mehrerer Felder entsprechend ändert, und zwar bezüglich einer oder mehrerer Objekteigenschaften.In a specific embodiment, it is particularly advantageous if the detection of a situation change in step c) takes place as a function of the comparison between a learned behavior from step b) and a behavior based on the statistical evaluation of currently determined values. Here, a current behavior for one or more fields is continuously determined and compared with a previously learned behavior for the corresponding fields. This makes it possible to detect changes in the situation that affect the behavior of the objects within one or more fields, with respect to one or more object properties.
In einer anderen Ausführungsform ist es ganz besonders vorteilhaft, wenn eine Situationsänderung in Abhängigkeit des Vergleiches zwischen dem erlernten Verhalten aus Schritt b) und aktuellen Werten einer Trajektorie eines konkreten Objektes erkannt wird. Dabei werden Werte bezüglich einer oder mehrerer Objekteigenschaften eines konkreten Objektes erfasst, während sich das Objekt auf einer bestimmten Bahn (Trajektorie) durch das zu überwachende Gebiet bewegt. Diese so aufgenommenen Werte werden dann einer bestimmten Trajektorie des Objektes zugeordnet und mit dem erlernten Verhalten der anderen Objekte verglichen. Weicht das Verhalten bezüglich einer oder mehrerer Objekteigenschaften des konkreten Objektes gegenüber dem erlernten Verhalten der anderen Objekte signifikant ab, so kann auf eine Situationsänderung geschlossen werden. Somit können auch einzelne Objekte identifiziert werden, die sich gegenüber dem typischen Verhalten der gesamten Szene atypisch verhalten, was ebenfalls eine Situationsänderung darstellt.In another embodiment, it is particularly advantageous if a situation change is detected as a function of the comparison between the learned behavior from step b) and current values of a trajectory of a specific object. In this case, values relating to one or more object properties of a specific object are detected, while the object moves on a specific path (trajectory) through the area to be monitored. These values recorded in this way are then assigned to a specific trajectory of the object and compared with the learned behavior of the other objects. If the behavior with respect to one or more object properties of the specific object deviates significantly compared with the learned behavior of the other objects, it is possible to conclude that the situation has changed. Thus, individual objects can be identified, which behave atypically compared to the typical behavior of the entire scene, which also represents a situation change.
Vorteilhafterweise werden für die statistische Auswertung Mittelwerte, Summen, Produkte und/oder Standardabweichungen gebildet, die dann miteinander verglichen werden können. Dabei sind grundsätzlich alle arithmetischen Operationen möglich, die der Fachmann für statistische Zwecke heranziehen würde. Als Sensoren zur Überwachung des räumlichen Gebietes kommen insbesondere bildgebende Sensoren, die vorteilhafterweise mit Bilderkennungsverfahren ausgestattet sind, Radarsensoren oder RFID Transponder in Betracht. Als zu überwachende Objekte kommen insbesondere Menschen, Tiere, also Lebewesen im Allgemeinen, Straßen-, Schienen-, Luft- oder Wasserfahrzeuge in Betracht sowie Partikel und sonstige Objekte, Besonders vorteilhaft ist es dabei, wenn die genannten Sensoren zum Ermitteln von Eigenschaften wie Aktivität der Objekte, die Geschwindigkeit der Objekte oder deren Richtung eingerichtet sind.Advantageously, mean values, sums, products and / or standard deviations are formed for the statistical evaluation, which can then be compared with one another. In principle, all arithmetic operations are possible which the expert would use for statistical purposes. Particularly suitable sensors for monitoring the spatial area are imaging sensors which are advantageously equipped with image recognition methods, radar sensors or RFID transponders. As objects to be monitored come in particular people, animals, ie living beings in general, road, rail, air or water vehicles into consideration as well as particles and other objects, It is particularly advantageous if the sensors mentioned for determining properties such as activity of Objects that are set to speed the objects or their direction.
Darüber hinaus ist es ganz besonders vorteilhaft, wenn der Vergleich in Schritt c) mittels eines Schwellenwertvergleiches erfolgt, so dass nicht bereits kleinste Änderungen zum Erkennen einer Situationsänderung führen. So kann der Schwellenwert derart ausgelegt sein, dass sprunghafte Veränderungen im Verhalten der Objekte als Situationsänderung erkannt werden. So kann für jedes Feld bzw. jede Zelle ein Schwellenwert ermittelt werden, der dann für jedes Feld hinterlegt wird, wobei sich der Schwellenwert z.B. aus dem Produkt der Standardabweichung des jeweiligen Feldes mit einem Empfindlichkeitsfaktor wie folgt ergeben kann:
mit Si, j als Schwellenwert in dem Feld (i, j), mit Vobj (i, j) als das erlernte Verhalten bezüglich einer Eigenschaft der Objekte im Feld (i, j), mit kempf als eine Empfindlichkeitsfaktor, der für die gesamte Szene bzw. Karte gilt und mit σi, j als Standardabweichung in dem Feld (i, j) bezüglich der erlernten Objekteigenschaft. Durch k kann in der gesamten Karte eine Empfindlichkeit eingestellt werden, die dann aber in jedem Feld zu einem anderen Ergebnis in Abhängigkeit von σ führt. So kann in örtlich nahe beieinander liegenden Feldern in einem Feld eine Situationsänderung erkannt werden, in benachbarten Feldern jedoch nicht, weil hier beispielsweise die Standardabweichung σ sehr hoch ist (z.B. in Kreuzungsgebieten). Somit kann der Schwellwert für jede Zelle einzeln und dynamisch geregelt werden, was das Verfahren noch genauer macht.Moreover, it is particularly advantageous if the comparison in step c) by means of a threshold comparison takes place so that not even the smallest changes lead to the recognition of a situation change. Thus, the threshold value can be designed in such a way that sudden changes in the behavior of the objects are recognized as a situation change. Thus, for each field or each cell, a threshold value can be determined, which is then stored for each field, whereby the threshold value can for example result from the product of the standard deviation of the respective field with a sensitivity factor as follows:
with S i, j as a threshold in the field (i, j), with V obj (i, j) as the learned behavior with respect to a property of the objects in the field (i, j), with k as a sensitivity factor for the entire scene or map holds and with σ i, j as the standard deviation in the field (i, j) with respect to the learned object property. Through k, a sensitivity can be set in the entire map, which then leads to a different result in each field as a function of σ. Thus, a situation change can be detected in fields close to one another in a field, but not in adjacent fields, because here, for example, the standard deviation σ is very high (eg in intersections). Thus, the threshold for each cell can be controlled individually and dynamically, making the process even more accurate.
Dabei können für ein Feld auch mehrere Schwellwerte hinterlegt sein, wenn sich bezüglich einer Objekteigenschaft mehrere typische Verhalten ergeben.In this case, several threshold values can also be stored for a field if several typical behaviors arise with regard to an object property.
Des Weiteren betrifft die Erfindung ein Computerprogrammprodukt zum Ausführen des vorstehenden Verfahren sowie eine Vorrichtung hierzu.Furthermore, the invention relates to a computer program product for carrying out the above method and a device for this purpose.
Die Erfindung wird anhand der beigefügten Zeichnungen beispielhaft erläutert. Es zeigen:
- Fig. 1a, 1b
- skizzenhafte Darstellung einer aufgenommenen Szene;
- Fig. 2
- Grundschema der Datenstruktur;
- Fig. 3
- Grundschema von abgeleiteten Informationen;
- Fig. 4
- schematische Darstellung der Situationserkennung über mehrere Zeitbereiche;
- Fig. 5
- skizzenhafte Darstellung eines Trajektorienvergleiches.
- Fig. 1a, 1b
- sketchy representation of a recorded scene;
- Fig. 2
- Basic scheme of the data structure;
- Fig. 3
- Basic scheme of derived information;
- Fig. 4
- schematic representation of the situation detection over several time ranges;
- Fig. 5
- sketchy representation of a trajectory comparison.
Die
Mittels einer entsprechenden Software, die nicht Gegenstand des vorliegenden erfindungsgemäßen Verfahrens ist, werden aus den aufgenommenen Bildinformationen Werte bestimmter Objekteigenschaften identifiziert. In diesen Ausführungsbeispielen sind die Objekte Fahrzeuge, welche die Kreuzung passieren, Dabei wird von der Bilderkennungssoftware für jedes Fahrzeug die Geschwindigkeit sowie die Richtung erkannt und entsprechend abgespeichert. Somit wird pro Fahrzeug, welches sich im Sichtfeld der Kameras befindet, kontinuierlich sowohl die Geschwindigkeit als auch die Richtung ermittelt.By means of appropriate software, which is not the subject of the present inventive method, values of specific object properties are identified from the recorded image information. In these embodiments, the objects are vehicles passing through the intersection. The speed and direction are recognized by the image recognition software for each vehicle and stored accordingly. Thus, per vehicle, which is in the field of view of the cameras, continuously determined both the speed and the direction.
Wie in
Jedem Geschwindigkeits- und Richtungswert wird nun anhand seiner Position das entsprechende Feld 1 zugeordnet, innerhalb dessen die ermittelte Position liegt. Somit kann jedem gemessenen Geschwindigkeits- und Richtungswert genau ein solches Feld zugeordnet werden. Dies ist schematisch in
Aus dieser Anordnung ergibt sich eine kartenartige Datenstruktur, aus der sich dann mit Hilfe von statistischen Auswertungen weitere kartenartige Datenstrukturen ermitteln lassen, aus denen dann das Verhalten der Objekte innerhalb der Szene abgeleitet werden kann.This arrangement results in a card-like data structure from which further card-like data structures can then be determined with the aid of statistical evaluations, from which the behavior of the objects within the scene can then be derived.
Eine solche abgeleitete kartenartige Struktur ist in
Dabei werden zunächst für jedes Feld die Werte ermittelt, deren Eigenschaft statistisch abgeleitet werden soll. In diesem Ausführungsbeispiel wären das die Geschwindigkeitswerte. Dann wird für jedes Feld die statistische Auswertung durchgeführt. Da die statistische Auswertung für jedes Feld separat durchgeführt wird, ergeben sich somit für jedes Feld auch unterschiedliche Werte, die in der
Durch eine solche statistische Auswertung, wie sie in die
Es sei beispielhaft einmal angenommen, die mittlere Geschwindigkeit liegt dabei je nach Feld zwischen 50 und 60 km/h. Kommt es nun zu einem Stau, so verringert sich naturgemäß die Geschwindigkeit der Fahrzeuge drastisch, was von den Sensoren erkannt wird. Verglichen mit dem erlernten Verhalten, welches eine mittlere Geschwindigkeit zwischen 50 und 60 km/h als typisches Verhalten anzeigt, ist eine Geschwindigkeit von 10 oder 20 km/h eines Objektes innerhalb dieser Szene atypisch, so dass auf eine Situationsänderung geschlossen werden kann. Darüber hinaus würde sich in diesem Fall auch die statistische Auswertung bezüglich der Standardabweichung sprunghaft verändern. Somit kann sehr schnell auf ein atypisches Verhalten der Objekte geschlossen werden, ohne dass zuvor das System wissen musste, worauf es zu achten hat. Dabei kann der Stau allein aus der Veränderung der Standardabweichung erkannt werden, da Stop&Go eine viel größere Standardabweichung erzeugt, als fließender Verkehr, wobei dies auch schon bei relativ kleinen bzw. schwachen Veränderungen festzustellen ist.For example, suppose the average speed is between 50 and 60 km / h, depending on the field. If there is a traffic jam, naturally the speed of the vehicles decreases drastically, which is detected by the sensors. Compared with the learned behavior, which indicates a mean speed between 50 and 60 km / h as typical behavior, is a speed of 10 or 20 km / h of an object within this scene atypical, so that a situation change can be inferred. In addition, in this case, the statistical evaluation with respect to the standard deviation would change abruptly. Thus, an atypical behavior of the objects can be concluded very quickly without the system first having to know what to look for. The congestion can be detected solely from the change in the standard deviation, since Stop & Go generates a much larger standard deviation than flowing traffic, and this is already noticeable in relatively small or weak changes.
Über drei in der Vergangenheit liegende diskrete Zeiträume t-3, t-2 und t-1 wurden kartenartige Datenstrukturen gelernt, wie sie zuvor beschrieben wurden. Dazu wurden Datenstrukturen bezüglich der Geschwindigkeit und der Richtung der Objekte innerhalb der beobachteten Szene gelernt. Aus den ermittelten Werten für die Geschwindigkeit S und die Richtung D wurde dann für jedes Feld der Mittelwert abgeleitet, was als kartenartige Datenstruktur 41 und 42 in
Aus diesen drei abgeleiteten kartenartigen Datenstrukturen in den Zeiträumen t-3 bis t-1 wurde dann ein gemeinsames Verhalten abgeleitet, welches ebenfalls in Form einer kartenartigen Datenstruktur 43 bzw. 44 abgespeichert wird. Bei der Bildung eines gemeinsamen erlernten Verhaltens kann dabei mit Hilfe unterschiedlicher Gewichtungen z. B. jüngeres erlerntes Verhalten stärker berücksichtigt werden als älteres.From these three derived card-like data structures in the periods t -3 to t -1 then a common behavior was derived, which is also stored in the form of a card-
Darüber hinaus wurde aus momentan aktuell ermittelten Werten bezüglich der Geschwindigkeit und der Richtung ein aktuelles Verhalten gelernt, welches sich in den abgeleiteten kartenartigen Datenstrukturen 45 und 46 in
Als Ergebnis dieses Vergleiches kommt in diesem Ausführungsbeispiel ein konkreter Skalar heraus, welcher die Unterschiede zwischen dem aktuellen und dem erlernten Verhalten repräsentiert. Es ist dabei leicht zu erkennen, dass je größer der Unterschied zwischen erlerntem und aktuellem Verhalten ist, je größer auch der Skalar ist. Des Weiteren lässt sich aus dieser Anordnung erkennen, dass kleine Änderungen sich nicht wesentlich auf das Gesamtergebnis auswirken, so dass nicht gleich auf eine Situationsänderung geschlossen wird, sobald die Werte lokal auch nur minimal abweichen.As a result of this comparison, a concrete scalar emerges in this embodiment representing the differences between the current and learned behaviors. It is easy to see that the larger the difference between the learned and current behavior, the larger the scalar is. Furthermore, it can be seen from this arrangement that small changes do not significantly affect the overall result, so that it is not immediately on a situation change is concluded, as soon as the values vary locally even minimally.
Des Weiteren ist in dieser Anordnung zu erkennen, dass bei anhaltender Situationsänderung das System selbstlernend ist und es sich der neuen Situation anpasst. Am obigen Beispiel des Staus orientiert, bedeutet dies, dass zwar der Stau als Situationsänderung zunächst erkannt wird (z.B. anhand einer großen Standardabweichung), nach einer gewissen Zeit aber die langsame Geschwindigkeit des Staus als typisches Verhalten gelernt wird (die Standardabweichung wird wieder kleiner). Das System passt sich somit automatisch ohne dass es neu justiert werden muss an die gegebene Situation an. Löst sich der Stau nach einer gewissen Weile wieder auf, so erhöht sich sprunghaft die Durchschnittsgeschwindigkeit, was ebenfalls wieder als Situationsänderung erkannt wird, da nun wiederum das Verhalten der Objekte von dem erlernten typischen Verhalten abweicht. Somit kann das System sowohl die Bildung eines Staus als auch dessen Auflösung hinterher ohne Probleme und manuelles Eingreifen erkennen und passt sich durch Mitlernen der Standardabweichung auch an zu erwartende Änderungsstärken an.Furthermore, it can be seen in this arrangement that, if the situation changes, the system is self-learning and it adapts to the new situation. Based on the above example of congestion, this means that although the congestion is initially recognized as a change in situation (eg by means of a large standard deviation), after a certain time the slow speed of the congestion is learned as a typical behavior (the standard deviation becomes smaller again). The system automatically adapts to the given situation without having to readjust it. If the traffic jam clears up again after a while, the average speed increases abruptly, which is again recognized as a change in the situation, as is the behavior of the objects from the typical behavior learned differs. Thus, the system can recognize both the formation of a congestion and its resolution afterwards without any problems and manual intervention and adapts by learning the standard deviation also to expected levels of change.
Auch ist es mit diesem Verfahren möglich, auf bestimmte äußere Einflüsse wie Witterungsbedingungen oder Ausfall von bestimmten Sensoren entsprechend zu reagieren, ohne dass es dafür eines manuellen Eingriffs bedarf. Fällt z. B. ein Sensor aus oder wird der Sensor von einem Fremdkörper teilweise überdeckt, so wird dies zwar von dem Verfahren zunächst als Situationsänderung erkannt, im weiteren Verlauf jedoch als typisch wahrgenommen. Für den späteren Verlauf hat somit der Ausfall bzw. der teilweise Ausfall eines Sensors keine Bedeutung mehr.It is also possible with this method to respond to certain external influences such as weather conditions or failure of certain sensors accordingly, without the need for manual intervention. Falls z. B. a sensor or the sensor is partially covered by a foreign body, this is indeed recognized by the method first as a change in situation, but perceived in the further course as typical. For the later course of the failure or the partial failure of a sensor has no meaning.
An dieser Stelle sei angemerkt, dass sowohl die statistische Auswertung "Mittelwertbildung" als auch die genannten Objekteigenschaften nicht auf diese begrenzt sind und nicht einschränkend zu verstehen sind.It should be noted at this point that both the statistical evaluation "averaging" and the mentioned object properties are not limited to these and should not be understood as limiting.
Dabei kann auch hier mit Schwellenwerten gearbeitet werden, die für jedes Verhalten und der bezüglichen Standardabweichung ermittelt werden. So schwankt der Durchschnittswert (Mittelwert) in einer großen Szene oft nur gering, obwohl sich z.B. auf einer Fahrbahn ein Stau gebildet hat, z.B. weil sich der Stau zunächst nur auf einer Fahrbahn bildet und sich dann auf die anderen Fahrbahnen "ausbreitet" bzw. sich der Stau nur in eine Richtung bildet, während die Gegenfahrbahn frei ist. So kann es zu kompensatorischen Änderungen kommen, die sich gegenseitig aufheben, so dass in der Summe keine Situationsänderung detektiert wird.It is also possible here to work with threshold values which are determined for each behavior and the relative standard deviation. Thus, the average value (mean) in a large scene often varies only slightly, although e.g. has jammed on a roadway, e.g. because the traffic jam initially forms only on one lane and then "spreads" to the other lanes or the jam forms only in one direction, while the opposite lane is free. This can lead to compensatory changes that cancel each other out, so that no change in the situation is detected in the sum.
So ist es besonders vorteilhaft, wenn für jedes Feld die Standardabweichung gespeichert wird, die dann für einen Schwellwertvergleich herangezogen wird, um so selektiv für einzelne Felder Situationsänderungen feststellen zu können. Durch eine Faktorisierung der Standardabweichung kann ein Empfindlichkeit eingestellt werden, so dass sich in manchen Feldern Situationsänderungen ergeben, da hier die Standardabweichungen relativ gering sind, während in anderen Feldern dies als normales Verhalten erkannt wird, was auf relativ große Standardabweichungen zurückzuführen ist.So it is particularly advantageous if the standard deviation is stored for each field, which is then used for a threshold value comparison, so as to be able to determine selectively for individual fields situation changes. By factoring the standard deviation, a sensitivity can be set so that in some fields situation changes result, since here the standard deviations are relatively low, while in other fields this is recognized as normal behavior due to relatively large standard deviations.
So passt das erfindungsgemäße Verfahren die Standardabweichung in den Feldern, in denen eine starke Änderung auftritt entsprechend über die Zeit an (z.B. bei Entstehung eines Staus), während die anderen Feldern für Veränderungen empfindlich bleiben (z.B. die Gegenfahrbahn, bei der noch kein Stau aufgetreten ist).Thus, the method of the invention adjusts the standard deviation in the fields where a large change occurs over time (eg, when a congestion occurs), while the other fields remain sensitive to changes (eg, the oncoming lane where no congestion has occurred ).
Dazu wird jeder Datenpunkt der Trajektorie mit der darunter liegenden kartenartigen Datenstruktur, insbesondere mit den Feldern verglichen, in die der entsprechende Datenpunkt fällt. Am Beispiel von
Beispielsweise lässt sich mit diesem Ausführungsbeispiel in
Claims (18)
- Method for automatic recognition of a change in situation within a spatial area to be monitored by means of sensors, the monitored spatial area being subdivided into a plurality of fields, comprising the steps of:a) detecting values by means of the sensors with respect to at least one property of objects which are located within the spatial area to be monitored, and their fields, within which the respective value of the property has been detected,b) learning at least one behaviour of the objects with reference to at least one of the object properties with the aid of a statistical evaluation of the determined values for at least one field, andc) detecting a change in situation as a function of a comparison between the learned behaviour of at least one field, and at least one currently determined value of the at least one field with reference to at least one of the object properties.
- Method according to Claim 1, characterized by learning the behaviour in step b) with the aid of the statistical evaluation of the values last determined within a period.
- Method according to Claim 1 or 2, characterized by learning the behaviour for more than one period and recognizing a change in situation by comparison between the learned behaviour of the respective periods and the current values.
- Method according to Claim 3, characterized by determining a common learned behaviour as a function of the behaviour learned in the respective periods, and recognizing a change in situation by comparison between the common learned behaviour and the current values.
- Method according to Claim 4, characterized by determining the common learned behaviour by averaging.
- Method according to one of the preceding claims, characterized by comparing in step c) exclusively those fields for which at least one value is stored with reference to one of the object properties.
- Method according to one of the preceding claims, characterized by detecting a determination position which indicates the position of the object within the spatial area during the determination of a value of a property, and determining the corresponding field as a function of the determination position.
- Method according to Claim 7, characterized by converting the determination position into a unified position coordinate system and determining the corresponding field as a function of the determination position within the unified position coordinate system.
- Method according to one of the preceding claims, characterized by detecting a change in situation in step c) as a function of the comparison between the learned behaviour from step b) and a behaviour which is current with the aid of statistical evaluation of currently determined values, as current values.
- Method according to one of the preceding claims, characterized by recognizing a change in situation in step c) as a function of the comparison between the learned behaviour from step b) and current values of at least one of the object properties of a trajectory of a particular object while taking account of the fields which are being traversed by the trajectory of the object.
- Method according to one of the preceding claims, characterized by images of mean values, sums, products and/or standard deviations as statistical evaluation.
- Method according to one of the preceding claims, characterized by imaging sensors, in particular image recognition methods, radar sensors and/or RFID transponders as sensors.
- Method according to one of the preceding claims, characterized by humans, animals, road, rail, air and/or watercraft and particles as objects.
- Method according to one of the preceding claims, characterized by activity, speed and/or direction as a determinable property of the objects.
- Method according to one of the preceding claims, characterized by comparison of step c) by means of a threshold comparison.
- Method according to Claim 15, characterized by determining a threshold for each field as a function of at least one behaviour with reference to at least one object property and the standard deviation with reference to the object property and the respective field.
- Computer program product with program code means for carrying out the method according to one of the preceding claims when the computer program product is run on a computing machine.
- Device for recognition of a change in situation having at least one sensor which is set up to determine the values of at least one property of objects which are situated within a spatial area to be monitored, and having a data processing system which is set up to carry out the method according to one of Claims 1 to 16.
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
DE200910021765 DE102009021765A1 (en) | 2009-05-18 | 2009-05-18 | Method for automatic detection of a situation change |
Publications (3)
Publication Number | Publication Date |
---|---|
EP2254104A2 EP2254104A2 (en) | 2010-11-24 |
EP2254104A3 EP2254104A3 (en) | 2012-11-07 |
EP2254104B1 true EP2254104B1 (en) | 2014-06-11 |
Family
ID=42262035
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP10004126.8A Not-in-force EP2254104B1 (en) | 2009-05-18 | 2010-04-19 | Method for automatic recognition of a change in a situation |
Country Status (2)
Country | Link |
---|---|
EP (1) | EP2254104B1 (en) |
DE (1) | DE102009021765A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9552725B2 (en) | 2000-08-28 | 2017-01-24 | Inrix Global Services Limited | Method and system for modeling and processing vehicular traffic data and information and applying thereof |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7620402B2 (en) | 2004-07-09 | 2009-11-17 | Itis Uk Limited | System and method for geographically locating a mobile device |
DE102010053150B4 (en) * | 2010-11-30 | 2018-02-01 | Deutsches Zentrum für Luft- und Raumfahrt e.V. | Method and device for evaluating a traffic situation |
DE102021207997A1 (en) | 2021-07-26 | 2023-01-26 | Robert Bosch Gesellschaft mit beschränkter Haftung | Method for checking completeness of a model of traffic dynamics at a traffic junction |
Family Cites Families (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5091780A (en) * | 1990-05-09 | 1992-02-25 | Carnegie-Mellon University | A trainable security system emthod for the same |
JPH06153202A (en) * | 1992-10-29 | 1994-05-31 | F M T:Kk | Abnormality monitoring device |
US5448484A (en) * | 1992-11-03 | 1995-09-05 | Bullock; Darcy M. | Neural network-based vehicle detection system and method |
DE4300651A1 (en) * | 1993-01-08 | 1994-07-14 | Refit Ev | Determination of road traffic data |
DE59702873D1 (en) | 1996-03-25 | 2001-02-15 | Mannesmann Ag | Method and system for traffic situation detection by stationary data acquisition device |
US20060074546A1 (en) * | 1999-04-19 | 2006-04-06 | Dekock Bruce W | System for providing traffic information |
JP3243234B2 (en) * | 1999-07-23 | 2002-01-07 | 松下電器産業株式会社 | Congestion degree measuring method, measuring device, and system using the same |
US6587781B2 (en) * | 2000-08-28 | 2003-07-01 | Estimotion, Inc. | Method and system for modeling and processing vehicular traffic data and information and applying thereof |
US20030107650A1 (en) | 2001-12-11 | 2003-06-12 | Koninklijke Philips Electronics N.V. | Surveillance system with suspicious behavior detection |
EP1563686B1 (en) | 2002-11-12 | 2010-01-06 | Intellivid Corporation | Method and system for tracking and behavioral monitoring of multiple objects moving through multiple fields-of-view |
JP2005173668A (en) * | 2003-12-08 | 2005-06-30 | Hitachi Ltd | Abnormality decision system for life activity pattern and apparatus therefor |
JP4471362B2 (en) * | 2004-08-25 | 2010-06-02 | パナソニック株式会社 | Surveillance camera device |
JP4368767B2 (en) * | 2004-09-08 | 2009-11-18 | 独立行政法人産業技術総合研究所 | Abnormal operation detection device and abnormal operation detection method |
US8009193B2 (en) * | 2006-06-05 | 2011-08-30 | Fuji Xerox Co., Ltd. | Unusual event detection via collaborative video mining |
-
2009
- 2009-05-18 DE DE200910021765 patent/DE102009021765A1/en not_active Withdrawn
-
2010
- 2010-04-19 EP EP10004126.8A patent/EP2254104B1/en not_active Not-in-force
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9552725B2 (en) | 2000-08-28 | 2017-01-24 | Inrix Global Services Limited | Method and system for modeling and processing vehicular traffic data and information and applying thereof |
Also Published As
Publication number | Publication date |
---|---|
EP2254104A3 (en) | 2012-11-07 |
DE102009021765A1 (en) | 2010-11-25 |
EP2254104A2 (en) | 2010-11-24 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
DE102017203838B4 (en) | Process and system for detecting the surroundings | |
DE102014008578B4 (en) | Method for determining position data for use in the operation of a vehicle system of a motor vehicle and position data acquisition and distribution system | |
EP2422332B1 (en) | Method for classifying an object as an obstacle | |
DE102018218220A1 (en) | Control device for a vehicle | |
DE102009033431B4 (en) | Method and device for the dynamic control of a signaling system | |
DE102014009059B4 (en) | Side-rear alarm system for vehicles and alarm control method for this | |
DE102014201159A1 (en) | Method and device for classifying a behavior of a pedestrian when crossing a roadway of a vehicle and personal protection system of a vehicle | |
DE102011010864A1 (en) | Method for predicting collision between lorry and e.g. pedestrian in dead angular area during driving on highway, involves computing collision probability between vehicle and object by intersecting vehicle and object accessibility amounts | |
EP1327969B1 (en) | Vehicle with means for recognising traffic road signs | |
DE102018221054B4 (en) | Method for providing map data in a motor vehicle, motor vehicle and central data processing device | |
DE102020003073B3 (en) | Method and device for automated driving of a vehicle and vehicle | |
DE102017212644A1 (en) | Sensor system and method for determining at least one traffic situation | |
EP2110798A1 (en) | Method for monitoring a vehicle path for a pre-defined type of vehicle | |
WO2019162794A1 (en) | Method and system for detecting parking spaces which are suitable for a vehicle | |
DE102019211098B4 (en) | Method, device and computer program for determining a traffic light phase of a traffic light of a traffic light system | |
DE102014212216A1 (en) | Method and driver assistance system for detecting a Fahrzeugumfel des | |
WO2023036545A1 (en) | Method and control circuit for checking whether a presently active driving mode is being operated within its odd, and system and back-end server | |
EP2254104B1 (en) | Method for automatic recognition of a change in a situation | |
DE102016202829A1 (en) | Device for transverse guidance support for a road-bound vehicle | |
DE102017212513A1 (en) | Method and system for detecting a free area within a parking lot | |
DE102018251778A1 (en) | Method for assisting a motor vehicle | |
EP3770879A1 (en) | Method, device and computer program for processing data about a traffic system | |
DE102009037461A1 (en) | Assistance system for increasing traffic safety | |
DE102010003375A1 (en) | Surrounding evaluation system, particularly for use in speed monitoring systems in vehicle, has electronic controller and sensor mediums for detecting objects in surrounding of vehicle | |
DE102022206348A1 (en) | Method and system for determining sensor and/or localization errors |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
AK | Designated contracting states |
Kind code of ref document: A2 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: AL BA ME RS |
|
PUAL | Search report despatched |
Free format text: ORIGINAL CODE: 0009013 |
|
AK | Designated contracting states |
Kind code of ref document: A3 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: AL BA ME RS |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G08G 1/01 20060101ALI20121002BHEP Ipc: G08G 1/04 20060101AFI20121002BHEP |
|
17P | Request for examination filed |
Effective date: 20130507 |
|
RBV | Designated contracting states (corrected) |
Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
INTG | Intention to grant announced |
Effective date: 20140128 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D Free format text: NOT ENGLISH |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D Free format text: LANGUAGE OF EP DOCUMENT: GERMAN |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 672570 Country of ref document: AT Kind code of ref document: T Effective date: 20140715 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 502010007155 Country of ref document: DE Effective date: 20140724 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140611 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140611 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140912 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140911 |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: VDEP Effective date: 20140611 |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140611 Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140611 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140611 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140611 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140611 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140611 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20141013 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140611 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140611 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140611 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20141011 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140611 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 502010007155 Country of ref document: DE |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140611 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140611 |
|
26N | No opposition filed |
Effective date: 20150312 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 502010007155 Country of ref document: DE Effective date: 20150312 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140611 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R082 Ref document number: 502010007155 Country of ref document: DE Representative=s name: GRAMM, LINS & PARTNER PATENT- UND RECHTSANWAEL, DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20150419 Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140611 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
GBPC | Gb: european patent ceased through non-payment of renewal fee |
Effective date: 20150419 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: MM4A |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GB Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20150419 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20150430 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20150430 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: ST Effective date: 20151231 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: FR Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20150430 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20150419 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MM01 Ref document number: 672570 Country of ref document: AT Kind code of ref document: T Effective date: 20150419 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: AT Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20150419 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140611 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140611 Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20100419 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140611 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140611 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20150430 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140611 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140611 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20200317 Year of fee payment: 11 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R119 Ref document number: 502010007155 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: DE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20211103 |