JPH10113343A - Method and apparatus and system for recognizing actions and actions - Google Patents

Abstract

(57) [Problem] To provide a method and an apparatus for automatically recognizing the operation and behavior of a mobile object such as a human, an animal, or a machine. A measuring device (2, 3) for measuring a state change accompanying an action or a behavior of a subject (1) is attached. In the signal processing device 7 for recognizing the operation or the action, the correlation between the reference data included in the feature amount database 6 of the operation or the action stored in advance and the extracted feature amount is calculated. Is output as a recognition result.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to the measurement of the movement and behavior of an object (hereinafter referred to as a subject) whose state changes, such as a human, an animal, or a machine. The present invention relates to a method, an apparatus using the method, and a system including the apparatus.

[0002] In the present specification, the action or action of a subject is abbreviated as "action / action".

[0003]

2. Description of the Related Art The basic prior art relating to measurement of human movement and work is described in "Illustrated Ergonomics" by Kageyu Noro, Japan Standards Association, February 14, 1990, pp. 538 to 5
It is described on page 44. This prior art discloses a method of measuring human motion or work by VTR or direct visual observation.

[0004] A conventional technique for measuring human motion or work using an acceleration sensor is disclosed in Japanese Patent Publication No. 7-96014. In this prior art, a vibration waveform obtained from an acceleration sensor attached to a human body is subjected to A / D conversion for each operation, and an obtained discrete A / D conversion value vibration pattern table is prepared. Is compared with the vibration pattern table at a predetermined timing in synchronization with a timing signal output from the clock unit. According to this conventional technique, the recognition processing can be performed at the same operation as the operation for producing the vibration pattern table or at the same operation speed.

[0005]

The former prior art for observing a subject's movement, action, or work by VTR or direct visual observation has the following problems.

(1) The analysis or direct observation of the VTR requires the observer to record the position of the body of the subject and the work being performed from time to time, which requires a great deal of time for the observer.

(2) The blind spot is not observed by the VTR or directly visually.

(3) When the subject moves, the observer must track the subject.

(4) The subject becomes conscious of the line of sight of the VTR and the observer.

(5) In the measurement of the position of the joint of the subject, merely measuring the position of the joint and reproducing the same cannot automatically recognize the action or work, and the reproduced operation is performed by a human. You have to be aware.

The latter conventional technique has the following problems.

(6) Due to the difference in pitch that is common in walking and running motions, if the speed is different even in the same “walking” motion such as “sloppy walking” and “walking fast,” the vibration pattern table And the input vibration waveform have different lengths in the time axis direction, so no correlation can be detected.
They mistakenly recognize it as a different operation. Therefore, in order to correctly recognize an operation having a different pitch, it is necessary to newly add a vibration pattern table corresponding to a different pitch. Furthermore, since the pitch usually changes continuously, many vibration patterns are required for more accurate recognition. Therefore, there is a problem that a very large vibration pattern table is required for more accurate recognition.

(7) Even when recognizing an operation in which a plurality of actions are combined, such as "gauging with a fan while walking" or "walking in a running train", "gauging with a fan while walking" And "walk in the running train" need to be registered as one vibration pattern. Furthermore, in the case of "floating with a fan while walking", the vibration waveform differs depending on whether the fan is lifted downward or upward when the foot touches the ground. It is necessary to register a new pattern. Also in this case, there is a problem that a very large vibration pattern table is required for actual recognition.

(8) Only the acceleration applied to the human body is measured, and this is recognized as data. For this reason, it is difficult to recognize a motion that has a characteristic in angular acceleration such as twisting the body.

(9) Recognition targets are limited to intermittent operations. Generally, a human action is achieved by performing a plurality of actions. For example, in the case of “sit on a chair”, the operation changes as “walk” → “stop” → “sit”. There is no disclosure of a technique for correctly recognizing one action including a plurality of actions.

(10) In the conventional recognition method, no consideration is given to a method of expressing the recognition result in an easily understandable manner.

The present invention has been made in consideration of the above-described problems, and has a recognition method capable of more accurately recognizing the motion and behavior of a moving object (subject) such as a human, an animal, and a machine. , An apparatus using the method, and a system including the apparatus.

More specifically, an object of the present invention is to provide a recognition method, apparatus, and system capable of realizing the following technical problems.

(1) Automatically recognize an action, action or work so as not to burden the observer. (2) There is no influence of blind spots due to the shadow of the object.

(3) Observation is not hindered even if the subject moves.

(4) No burden is imposed on the subject.

(5) It is possible to recognize or estimate the motion or action of the subject and the work content from the measurement result, in addition to the simple motion measurement.

(6) It is not affected by the operation speed.

(7) Recognition is possible even when a plurality of operations are superimposed.

(8) Recognize more types of operations by referring to phenomena other than acceleration.

(9) Recognizing, associating or estimating the behavior of the subject (workpiece) and the work content based on the history of the operation, instead of the intermittent operation measurement.

(10) Express the recognition result of the action / action in an easy-to-understand manner using computer animation. The computer animation is controlled based on the recognition result of the action / action.

[0028]

In order to achieve the above object, a recognition device for recognizing a motion / behavior of a subject according to the present invention is attached to the subject, and the motion / recognition device of the subject is provided.
Measuring means for observing a state change accompanying an action, feature quantity extracting means for extracting a feature quantity in the observation result, storage means for storing a feature quantity of an action / action to be recognized by the recognition device; It has a recognizing means for recognizing the motion / action of the subject from the feature quantity extracted from the result and the stored feature quantity, and an output means for outputting the recognition result.

More specifically, the recognition method, apparatus and system according to the present invention include, for example, the following means.

(1) A measuring instrument for automatically measuring a state change accompanying the movement or action of the subject is attached to the subject, and the measurement data is transmitted to the observer.

(2) A change in the state of the subject is measured by an acceleration sensor, a speed sensor, a position sensor, or the like, which does not rely on video or vision.

(3) The observation result is transmitted by radio or the like.

(4) The state change is measured at a small number of observation points where the state change accompanying the action or action is noticeable.

(5) A feature amount of an action or action is extracted in advance, compared with the feature quantity extracted from the measurement data, and an action or action represented by a feature quantity having a high correlation is output as a recognition result. .

(6) A waveform previously measured as representing one operation is normalized by the time (operation time) from the start to the end of the operation. In the case of normalization, the time from the start to the end of the operation is set to "1". For example, in the case of walking, two steps are one cycle. After that, characteristic components of each operation are extracted by functionalization or Fourier transform, wavelet transform, etc., and the coefficients are converted when the functions are converted, and the extracted components are converted when the Fourier transform is performed.
It is registered as a feature value of an operation to be referred to. Next, when recognizing the operation, the start and end of the operation are recognized from the input waveform, and the input waveform is normalized by the duration of the operation,
After that, coefficients or feature components are extracted by performing the above-described functionalization, Fourier transform, wavelet transform, or the like, and recognition is performed by comparing the extracted coefficients or components with a previously registered feature amount. (7) Recognition of the observation result is performed by using a result obtained by superimposing feature amounts of different operations. For example, the feature amount of the “walking” motion and the feature amount of the “fan fan” are superimposed.

(8) When observing a motion or action, not only acceleration data but also other physical quantities such as speed, position, angular acceleration, angular velocity, rotation angle, and biological information associated with the motion or action of the subject. Data is also acquired and used for recognition processing.

(9) The action associated with the action history is registered as the associative action, and the action history of the subject is compared by comparing the observed and recognized action history with the associative action.

(10) The selection of the feature amount component and the control of the strength of the computer animation generation method are performed using the recognition result of the motion / action of the subject.

According to each of the above means, for example, the following effects can be achieved.

(1) Since the measurement data is automatically sent, no burden is imposed on the observer.

(2) A sensor that does not generate blind spots.

(3) Due to the radio, no matter where the subject moves, the observer need not track.

(4) No burden is imposed on the subject because the number of observation points is small. Further, since the visual observation is not performed, the subject is not affected by the line of sight of the observer.

(5) Because of the automatic recognition, there is no need for the observer to directly handle the measured raw data.

(6) Since the time axis is normalized, recognition can be performed without being affected by the operation speed. (7) Since a combination of feature amounts is also taken into consideration, a more complicated operation can be recognized.

(8) Recognition is performed using a physical quantity other than acceleration, so that it is possible to recognize even an operation in which a feature does not appear much in acceleration.

(9) An action performed by a combination of a plurality of actions (hereinafter referred to as a task) can also be recognized.

(10) The recognition result can be presented to the observer in an easy-to-understand manner.

[0049]

DESCRIPTION OF THE PREFERRED EMBODIMENTS One embodiment of the present invention will be described below with reference to FIG.
This will be described below. In the present embodiment, the subject 1 will be described as a target of recognition of an action or action (hereinafter abbreviated as action / action).

The apparatus for recognizing actions and actions according to the present embodiment includes measuring instruments 2 and 3 attached to the subject 1 and an A / D converter 4 for digitizing the measurement results from the measuring instruments 2 and 3.
A feature value extraction unit 5 for extracting feature values from digitized measurement results, a feature value database 6 storing feature values of various operations and actions measured and extracted in advance, and a feature value extraction unit 5 A signal processing unit 7 for recognizing an action / action indicated by the obtained feature amount using data stored in the feature amount database 6, and an output unit 8 for outputting a recognition result.

The feature quantity extraction unit 5 and the signal processing unit 7 perform, for example, feature quantity extraction processing and operation /
The present invention is realized by an information processing apparatus including a memory for storing a program for executing an action recognition process, and a DSP or an MPU for executing the program. The output unit 8 is realized by, for example, a display device including a liquid crystal panel or a CRT.

In the present embodiment, the measuring instruments 2 and 3 for measuring the state change accompanying the operation / behavior are attached to the subject 1. In the example of FIG. 1, a state change measuring instrument 2 attached to the waist position and a state change measuring instrument 3 attached to the arm are shown. These state change measuring instruments are attached to places where the state change is prominent. For example, when a state change to be observed appears on the foot, it is preferable to attach a state change measuring device to the foot. The number of measuring instruments may be one or more.

The results measured by the state change measuring devices 2 and 3 are converted into digital signals every moment by the A / D converter 4. The converted digital signal is processed by the signal characteristic amount extraction unit 5, and a characteristic characteristic of the signal is extracted. This feature extraction process will be described with reference to FIG.

Here, as an example of a measuring device for measuring a state change, an acceleration sensor attached to the waist as shown in FIG. 2 will be described. This acceleration sensor measures the acceleration of the human body along the height direction. The output results 20 from the acceleration sensor are human “walking”, “running”, “squatting”,
The time-series data examples 21 to 24 unique to each obtained by the operation of “landing” are shown. In the case of this example, the data 21 and 22 indicate a periodic acceleration change due to walking or running, the data 23 indicates a single-shot change, and the data 24 indicates a horizontal change. Is observed.

After the above data is digitized by the A / D converter 4, a frequency spectrum 25 is obtained as a result of time-frequency analysis (Fourier transform as an example) which is one method of signal analysis. The analysis results corresponding to the above data 21 to 24 are frequency spectra 26, 27, 28, 29
become that way. The bar graph of each analysis result represents the spectrum intensity of the frequency component obtained by performing the Fourier transform. The frequency characteristics of each operation are different from each other, and this difference is a feature amount of each operation.

In the present embodiment, a feature quantity serving as reference data in the signal processing unit 7 for recognizing an action / action is extracted from data relating to the action or action whose state is known in advance and registered. Keep it. When registering this reference data, the reference data is stored in the database 6 of the feature amount of the action / action via the path 9 in FIG. 1 (process 30 in FIG. 2).

Signal processing unit 7 for recognizing operation / action
Then, the feature amount data 10 which is sent from the feature amount extraction unit 5 every moment and originates from the current motion / action of the subject 1
And the reference data 11 of the feature amounts of various actions / actions already registered in the database 6
Compare with In other words, the currently transmitted feature amount is correlated with the various feature amounts stored in the database 6, and the operation / action corresponding to the feature amount having the highest correlation is the operation currently performed by the subject 1. / Action, and the output is output by the output unit 8.

In the present invention, the method for obtaining the correlation is not particularly limited, but for example, there is a method as shown in FIG. In this method, first, feature amount data 10 which is a spectrum of a measured waveform corresponding to the movement / action of the subject 1
, A frequency component F (m) normalized so as to satisfy the following expression 1 is obtained. Similarly, each frequency component G (m) corresponding to the reference data 11 to be correlated and normalized so as to satisfy the following Expression 2 is obtained.

[0059]

(Equation 1)

[0060]

(Equation 2)

Next, a function H (m) is defined by the following equation 3 as a function indicating the correlation between the two data 10 and 11.

[0062]

H (m) = G (m): (if F (m)> G (m)) or = F (m): (if G (m) ≦ F (m)) (number) 3) The function H (m) indicates the overlapping portion 290 of both data, and if H (m) satisfies the following equation 4, it is determined that there is a correlation.

[0063]

(Equation 4)

Here, the maximum value of H (m) is 1.0.
Therefore, when it is required that they are exactly the same, α is set to 1.0, but when something similar is required to some extent, α is set to a value slightly smaller than 1.0. And When comparing with a plurality of reference data, the largest integrated value of H (m) is determined as the action / action performed by the subject 1, or according to the value of the integrated value. Determined as a possible candidate.

Further, in Equation (4), all frequency components of the spectrum are set to have the same weight, but different weights may be assigned to each component. For example,
If a spectrum specific to the operation is generated, the weight of the portion may be increased, or if there is noise, the weight of the component may be reduced. Assuming that β (m) is a weight function unique to each frequency component m, a function H (m) indicating a correlation can be defined as follows.

[0066]

H (m) = G (m) β (m): (when F (m)> G (m)) or = F (m) β (m): (G (m) ≦ F ( In the case of m)) (Equation 5) According to the present embodiment, recognition processing is performed from the measured value of the state change, instead of measuring the state change and using the measurement result as a mere measured value. Accordingly, there is an effect that the movement / action of the subject can be automatically recognized from the state change.

Further, according to the present embodiment, a small number (at least one in this embodiment) attached to the body of the subject
There is an effect that the motion of the subject as a whole can be recognized from the measurement result measured by the measuring device of the state change of the subject, and the macro motion of the whole body can be estimated by a small number of measuring devices.

Furthermore, according to the present embodiment, there are fewer measuring points to which the measuring device is to be attached, so that there is an effect that the subject is not burdened.

Further, according to the present embodiment, when transmitting the state of the movement / action of the subject to another place, the measured value obtained by measuring the state change (measured value for each measurement cycle) is transmitted as it is. In comparison, after recognition, it is sufficient to transmit a recognition result for each operation / action cycle, which has the effect of reducing the amount of information.

In the present embodiment, an example has been described in which the feature extraction unit 5 performs a Fourier transform as a frequency analysis.
The frequency analysis method that can be used in the present invention is not limited to this. For example, another frequency analysis such as a time-frequency analysis including a wavelet transform may be performed, and the feature amount may be extracted from the result of the conversion.

An example using the wavelet transform will be described with reference to FIG. 210 is a change in acceleration measured in response to the motion of the subject shown at the top. When this waveform is subjected to wavelet transform, a characteristic value appears in Level (wavelet component). For example, in the case of the “walking” motion, a characteristic value 214 appears at 213 Level C. Similarly, in the case of “squatting”, as in the case of Level A 215 of 211, and in the case of “running”, the level B of 212 is performed.
And 213, a value such as 216 appears in Level C. In the case of the present example, these are feature amounts representing the respective actions, and the action / action can be recognized based on the feature quantities.

The way of obtaining the correlation when the wavelet component is used as the feature is the same as that when the frequency component is used as the feature. However, in the method of obtaining the correlation with the frequency component, the correlation is obtained with the value of the intensity of each frequency component. In the case of the wavelet component, the correlation of the intensity with respect to time for each level is obtained, and the correlation of the level is also obtained.

In the present embodiment, the feature obtained by frequency analysis using a digitized signal has been described. However, the feature that can be used in the present invention is not limited to this. In the present invention, the feature amount is
Other forms of feature quantity may be used as long as the action or action can be distinguished using the quantity. For example, measuring instruments 2 and 3
The frequency characteristic of the analog signal output from is extracted by a spectrum analyzer composed of a plurality of filter circuits, and the frequency component of the extracted analog value is used as a feature amount.
The configuration may be such that the correlation is obtained by comparing each component with an analog comparator. Alternatively, a configuration may be employed in which a function of a predetermined order is fitted to the output signal of the measuring instrument, and the feature amount is represented by a combination of coefficients included in the function.

In this embodiment, the operation / action is recognized by using the correlation. However, the same effect as that of this embodiment can be obtained by using a neural network instead of obtaining the correlation. . In this case, the signal processing unit 7 for recognizing the operation / action is configured by a neural network. Further, when creating the database 6 of the feature amount of the action / action, the feature quantity extracted in advance for the known action / action is memorized in the neural network as a teacher signal to the neural network. At the time of recognition, the feature amount is extracted from the observed measurement value by the feature amount extraction unit 5 and input to the signal processing unit 7 composed of a neural network for recognition.

Further, in the present embodiment, the case where the measurement result of one axis in the vertical direction is used for simplification of the explanation has been described as an example.
The feature amount may be extracted from the measurement of the acceleration change in the direction other than the vertical direction, and may be recognized based on the respective values or in a composite manner.

In this embodiment, an example is described in which an acceleration sensor that senses the acceleration of a human body in the height direction is taken as an example of a state change measuring device, and a feature amount is extracted from a change in acceleration. If a speed sensor or a position sensor is used instead of and the acceleration is calculated (first-order or second-order differentiation) from a change in speed or position, the same effect as described above can be achieved.

In the present embodiment, the feature amount of the acceleration change is extracted. However, the feature amount is extracted from the speed change, and the correlation with the feature amount of the database extracted and registered from the speed change is obtained. The present embodiment can be implemented in the same manner as the present embodiment. In this case, when an acceleration sensor is used, first-order integration is performed to calculate and execute the speed. When a position sensor is used, first-order differentiation is performed to calculate and execute the speed. Similarly,
The feature amount may be extracted from the position change, and the correlation with the feature amount of the database extracted and registered from the position change may be taken. In this case, when the acceleration sensor is used, the position is calculated by performing the second-order integration, and when the speed sensor is used, the position is calculated by performing the first-order integration.

Although the present embodiment has been described by taking a human as an example of a subject, it is needless to say that motion and behavior can be similarly recognized for non-human animals and articulated robots. .

In the present embodiment, "walk" and "run"
Although the recognition of the entire movement of the subject is described as an example, the state change of the specific place is measured by a state change measuring instrument attached to a specific place of the subject, and the state change is measured. It is also possible to recognize the movement and the action / behavior of the specific place from the feature amounts of the above in the same manner as in this embodiment.

For example, by using a state change measuring device attached to a wristwatch, a feature amount is extracted from a state change of a wristwatch that moves together with the hand, and the characteristic amount is measured from a wristwatch due to hand movement recorded in advance. By comparing with the feature amount, it is possible to recognize the movement or movement / action of the hand. Similarly, a state change measuring device attached to shoes or socks recognizes the movement or movement / action of a foot, and a state change measurement device attached to a hat or glasses measures head movement or movement / motion.
It is also possible to recognize actions.

In this embodiment, "walk" and "run"
As an example, the recognition of the movement of the subject is explained as an example.However, a state change measuring device is attached to the device moved by the subject, the state change of the device moved by the subject is measured, and the state change is measured. It is also possible to recognize the movement or operation of the device from the feature amount by the same method as in the present embodiment.

For example, if a state change measuring instrument is attached to a writing implement or an input device (for example, a mouse or the like) used in place of the writing implement in a computer system, the characteristic amount of the state change can be calculated from the state change of the writing implement. Extracted and featured when writing characters and figures, or
Character recognition, figure recognition, and signature recognition are also possible by comparing the feature amount when a signature is made.

Next, an embodiment of a recognition apparatus for realizing a recognition method that can be used when the movement speed of the subject changes will be described with reference to FIGS.

In general, when a human or an animal performs an action, for example, the same “walk” may be “walk slowly” or “walk fast”. Therefore, the length of time required to walk one step changes. When the time changes, the frequency changes, and when the feature amounts of the motions stored in advance are compared, they may not be recognized properly. The present embodiment solves this problem.

As shown in FIG. 4, the recognition apparatus according to the present embodiment has a configuration in which the operation time of the measurement target is set between the A / D converter 4 and the feature quantity extraction unit 5 in the apparatus shown in FIG. A normalization unit 221 for performing a measurement and a normalization process based on the operation time
Is provided.

In the present embodiment, the required operation time, which is the period of the operation (eg, walking) to be measured, is shown in FIG.
When defined as indicated by 222 in FIG. 5A and an observed waveform is cut out with a window function longer than the time required for one step and frequency analysis is performed, it corresponds to one step when walking as shown in FIG. Is obtained. Here, the fact that the operating speed changes means that this spectrum 22
4 moves left and right.

Therefore, in this embodiment, after the conversion processing by the A / D converter 4, the normalization unit 221 measures the time required for the operation. Specifically, if the motion is a periodic motion,
It is determined from the above-described fundamental frequency (operation required time 222).
If the motion is not a periodic motion such as “squatting”, the length of the edge portion of the acceleration change at 219 and 220 in FIG. 3 or a certain level (in this example, Level A) signal (217 and 218)
From the distance.

FIG. 5C shows a result obtained by normalizing the cutout of the observation waveform for the time required for this operation, and frequency-analyzing the result. Such a process is called a time normalization process, and can also be realized by using a function called a time expansion / contraction function.

In the spectrum after the normalization processing, FIG.
As shown in (c), the fundamental frequency spectrum 225 of the operation appears at the first harmonic. This is the same as the translation of the high-frequency components after 224 so that 224 becomes the first harmonic. Therefore, after obtaining the spectrum of FIG. The processing may be performed. As a result, the horizontal axis in FIG. 5C is the frequency based on one step. The above-described normalization processing is performed by
This is performed both in the case where the database 6 of the feature amount of the action is created and in the case where the observed waveform is recognized using the reference data of the database 6.

According to the present embodiment, since the feature amount is input to the signal processing unit 7 as a delimiter for each operation, independent of the operation time, it is possible to perform an accurate recognition process independent of the operation time. Can be. Further, by outputting the observed operation time, for example, "walking slowly"
Alternatively, more detailed movements such as "walking fast" can be distinguished.

Next, another embodiment in which the operating speed changes will be described with reference to FIG.

As shown in FIG. 6, the recognizing device of this embodiment is the expansion / contraction unit 231 for performing the expansion / contraction processing of the observed waveform between the A / D converter 4 and the feature quantity extraction unit 5 in the apparatus of FIG.
And a feature amount extraction unit 5 is provided between the A / D converter 4 and the feature amount database 6. here,
The construction of the feature amount database 6 is performed in the same manner as in FIG.

In the present embodiment, when the observed waveform can be recognized, the observed waveform is deformed by the expansion / contraction unit 231. Specifically, a process for expanding and contracting the time axis of the observed waveform is performed. Extending the time axis is equivalent to increasing (decreasing) the operation time, and contracting is decreasing (decreasing) the operation time. After performing this expansion / contraction processing with various predetermined expansion / contraction values, the characteristic amount is extracted by the characteristic amount extraction unit 5 for each processing result, and the recognition processing is performed by the signal processing unit 7.

In the recognition processing section 7, feature amounts corresponding to various waveforms obtained by expanding and contracting the observed waveform on the time axis are input, and an operation corresponding to the most correlated feature amount is output as a recognition result. By also outputting the expansion / contraction ratio in this case, it is possible to recognize the difference in the operation time from the operation stored in the feature amount database 6.

According to this embodiment, there is an effect that correct recognition can be performed even if the operation speed changes. Further, there is an effect that the operation time (operation speed) can be recognized.

In the present embodiment, the observed waveform is expanded or contracted along the time axis. However, in the present invention, other characteristics related to the observed waveform are expanded or contracted unless the characteristic amount unique to the operation to be identified is largely changed. No problem. For example, a configuration may be used in which the signal strength of the observed waveform is expanded and contracted, recognition processing is performed using the expanded and contracted observation waveform, and the expansion and contraction ratio used when recognition is successful is displayed.

Next, an embodiment in which a plurality of feature values are selected and recognition processing is performed based on a new feature value generated by superimposing them will be described with reference to FIG.

In the embodiments shown in FIGS. 1, 4 and 6, one feature amount is selected from each of the observed feature amounts and the database 6 stored in advance. The recognition process was performed by correlating with the above. On the other hand, in the present embodiment, recognition is performed by combining a plurality of feature amounts.

When a person performs an action, the action may be such as "waving hands while walking" or "walking in a train". However, in the embodiment shown in FIG. 1 and the like, “waving while walking”, “walking in a train” and the like are regarded as one operation, and the feature amount of the entire operation is extracted. It had to be stored in the database 6. However, "waving hands while walking" and "walking in a train" are, when considered in the frequency domain, the superposition of the characteristic amounts of "walking" and "waving hands". Similarly, “walking in a train” can be considered as a superposition of the feature amounts of “train vibration” and “walking”.

Therefore, in the present embodiment, in the operation in which a plurality of feature amounts are superimposed as described above, the operation / action is recognized using the processing shown in FIG.

First, a feature A301, a feature B302, and a feature C303 to be superimposed in the recognition process are selected. In the example described above, they correspond to the feature amounts of “walk”, “wave”, and “train vibration”.

Next, a superimposition process is performed by the feature amount combination unit 304 using these feature amounts A, B, and C. For example, if the feature is performed by frequency analysis, this processing is to superimpose each frequency spectrum.

Finally, the action / action is recognized based on the new feature amount (A + B + C) generated by the above-mentioned superimposition process and the feature amount 10 corresponding to the observation signal extracted by the feature amount extraction unit 5. The signal processing unit 7 performs a recognition process, and outputs the recognition result from the output unit 8.

In the above-described superimposition processing, if a newly generated feature is Q,

[0105]

Q = A + B + C (Equation 6) Each feature is given a weight,

[0106]

Q = αA + βB + γC (Equation 7) Here, α, β, and γ are weights of the respective feature amounts, and the weights (strengths) of the respective feature amounts can be controlled.

In this embodiment, since a plurality of feature values are combined, a new feature value is generated, and the recognition process can be performed, the recognition process can be performed even for a complicated action / action in which various feature values are combined. effective.

Next, an embodiment in which a plurality of state change measuring instruments are attached will be described with reference to FIG.

In the embodiment shown in FIG. 1, the place where the state change measuring device is attached is the position of the waist 31 in FIG. 8, and the measuring axis is 32 (the height direction of the body). In this case, it is possible to measure only a state change in the human height direction. It is said that the center of gravity of a human is almost at the waist position. Therefore, when the user raises his hand while standing, the waist position slightly moves in the direction opposite to the direction in which the hand is raised to maintain the center of gravity. In order to observe it, a measuring device for a state change in the axial direction of 34 is required. Similarly, in order to measure the state in which the neck is tilted forward or backward, a measuring instrument having 33 directional axes is required. Furthermore, in order to measure the rotation of the waist, it is necessary to measure the rotation direction 37 around the axial direction 32 and the like. If more detailed information is needed, the elbow 35
It is also necessary to measure the movement 36 and the like at the time.

Therefore, the rough motion / action of the subject, for example, “walk”, “run”, etc., is recognized in the axial direction 32, and the thinner additional motion is performed by the axial direction 33, 34 or the rotation direction 37 around the axial direction 32. For example, "raise right hand",
By recognizing "falling the neck", "twisting the waist", etc.,
It is possible to recognize complex actions / actions such as "raising your right hand while walking", "falling your neck while running", "twisting while standing". According to the present embodiment, by providing a plurality of state change measuring devices, it is possible to combine the results measured and recognized by each measuring device, and it is possible to recognize more detailed operations and actions. There is.

In this embodiment, the case where the spatial movement of each part of the body is detected has been described. However, in the present invention, the setting place of the measuring instrument and the type of the state change to be measured are not limited to these. For example, as a setting place of the measuring instrument, in addition to the above-described parts, a configuration may be adopted in which the movement of each part of the face accompanying a change in the expression is captured as the above-mentioned state change by attaching to each part of the face. Furthermore, instead of detecting the spatial movement of each part of the face or body, a measuring device that directly detects the state of the muscle causing the change may be used.

Next, the physical quantity other than the acceleration or the physical quantity or the biological information is used to comprehensively determine the characteristic quantity of the action / action.
An embodiment of a system for recognizing actions will be described with reference to FIG.

As shown in FIG. 9, the recognition system of this embodiment performs a recognition process on a recognition device 40 based on various information measured by measuring means attached to a subject.
1 to 404, a recognition device 405 that executes comprehensive recognition processing using the recognition results of these recognition devices, and an output unit 8 that outputs the recognition result.

The recognition device 401 is a device that executes a recognition process from acceleration information provided by an acceleration sensor attached to a subject, for example, as shown in the embodiment of FIG.

The recognition device 402 is a speed-based recognition device. For example, by detecting a change in the speed of a human being as a subject, it is determined whether the human is walking or riding on a vehicle.

The recognizing device 403 is a position-based recognizing device, and recognizes an action based on the position of the subject and what kind of trajectory the subject moves.

The recognizing device 404 is a recognizing device based on biological information. Biological information includes pulse, blood pressure, body temperature, blood glucose,
These include respiration, myoelectricity, electrocardiography, blood flow, and brain waves. For example, if the subject is a human, the above-described biological information changes with the action. The apparatus recognizes the state of the person based on the change. For example, exercise can increase your heart rate and your respiratory rate. Depending on the position where the blood pressure is measured, a difference in posture can be determined. The amount of movement, mental state, physical condition, etc. can be determined from changes in body temperature.
You can tell if you are asleep or awake by brain waves. Physical condition can be determined by blood sugar level. Exercise status can be determined by changes in myoelectricity.

In the recognition devices 402, 403, and 404, the observed various information is observed in the form of a waveform. A / D conversion of this waveform is performed, and the characteristic amount is also extracted in each of the recognition devices as in the case where the characteristic amount is obtained from the acceleration in the recognition device 401, and recognition processing is performed based on the extracted characteristic amount.

However, there is a case where the recognition result is ambiguous and accurate recognition processing cannot be performed by each of the above-described recognition devices alone. Therefore, in the present embodiment, the recognition device 405 uses the recognition results of the various recognition devices 401 to 404 to further comprehensively determine the motion / action of the subject, and outputs the recognition result to the output unit 8. I do.

According to the present embodiment, it is possible to make comprehensive judgments based on the results recognized by a plurality of sensing methods and obtain more accurate recognition results.

Next, an embodiment of an apparatus in which the recognition of the movement and behavior of the subject and the measurement of the position of the moving / behaving subject will be described with reference to FIGS. 10 and 11. FIG.
FIG. 10 is a configuration diagram of an apparatus according to the present embodiment, and FIG. 11 is an explanatory diagram of position measurement in the present embodiment.

As shown in FIG. 10, the recognition apparatus of the present embodiment is obtained by adding a position measuring unit 41 for measuring the position of the subject 1 to the apparatus of the embodiment shown in FIG. Here, the same components as those in the embodiment of FIG. 1 are denoted by the same reference numerals, and description thereof will be omitted. The position measurement unit 41 is realized by, for example, an apparatus using a cellular system as described below. In addition, as a device for measuring the position, a transmitter is attached to the subject 1, a method of obtaining the radio wave emitted from the subject by observing the direction of arrival of the radio wave from a plurality of receiving points, and a plurality of reference transmissions There are methods such as observing radio waves coming from a source and determining their own position based on them (GPS, Omega, Loran, Decca, etc.), and methods using lasers, magnetic fields, and ultrasonic waves. Either of them has the same effect as the present embodiment.

As shown in FIG. 11, a cellular system is, for example, an area 50 in which communication is performed by an antenna 51.
In order to perform communication in another area, the communication system switches communication by another antenna and communicates with another area adjacent to the area where communication has been performed so far. Here, the area 50 in which communication is possible with one antenna is called a cell or a zone. In this system, since the cells that can be communicated with are very narrow, it is possible to identify the area where the caller is located by identifying the antenna that is communicating. Each cell is 1-1 to 3-3, and the subjects are 52, 53,
54.

In the present embodiment, each subject is given
The measuring devices 2 and 3 and the processing units 4 to 7 in the apparatus shown in FIG. 10 are attached, and the recognition result from the signal processing unit 7 is
It is assumed that data is transmitted via a cellular system. That is, in the present embodiment, the position measuring unit 41 is realized by a transmitting unit that sends the recognition result to the antenna of the cell to which the subject 1 belongs, and a receiving unit that receives the recognition result sent via the antenna. .

The behavior of the subject 52 (in this example, “squatting”) is recognized by the method described in the embodiment of FIG. 1, and sent via the antenna in the cell 1-1.
Therefore, the subject 52 is recognized as “squatting” in the area of the cell 1-1. Similarly, subject 53 is cell 2
-2 is recognized as "running", and the subject 54 is recognized as "walking" in the cell 3-2.

As described above, according to the present embodiment, there is an effect that "where" and "what kind of operation / action" can be recognized.

Next, an embodiment of a system for estimating the behavior and work status of the subject or the environment and the position where the subject is placed based on the history of the recognition results will be described with reference to FIGS.
This will be described with reference to FIG.

As shown in FIG. 12, the system according to the present embodiment comprises a recognition device 61 for recognizing a single operation / action, a storage device 62 for storing a history of recognized operations / actions,
A storage device 63 in which an associative pattern database is stored in advance, and an associative processing device 64 that uses the data in the storage devices 62 and 63 to associate or guess the behavior and work status of the subject or the environment in which the subject is placed. And an output device 65 for outputting the association or estimation result.

The motion / behavior recognition device 61 has the same configuration as the recognition device shown in FIG. 1 or FIG. However, the output unit 8 of these devices can be omitted here. As described in the above embodiment, the recognition device 61 recognizes only a single action / action that occurs within a certain time. Therefore, it is not suitable for recognizing a work completed by stacking several actions / actions.

Therefore, in the present embodiment, the actions / actions recognized by the recognition device 61 are sequentially stored in the storage device 62 for storing the history of the actions / actions. The stored action / action history is arranged and recorded in chronological order, for example, as data 71 in FIG.

The stored history information 71 is monitored by the associative processing device 64 while referring to the storage pattern stored in the storage device 63 for the associative pattern. Here, the associative pattern refers to a pattern indicating a correspondence between a combination of a plurality of actions including an unrecognizable action and one task associated with the combination, as shown in FIG.

The operation of the present embodiment will be described by taking an inspection work of a plant supervisor as an example. In the case of inspection work by a plant supervisor, the content of the inspection work in a certain area (region) is considerably limited. Therefore, the work content can be estimated with a small combination of actions / actions. Reference numeral 81 in FIG. 14 shows an example of such a sequence of operations such as “climbing a ladder”, “walking”, and “unrecognizable”. Note that “unrecognizable” refers to an operation in which the recognition device 61 cannot identify the operation because the operation is performed but the operation is complicated.

Further, in this area, if the only operation to be performed on the ladder is "something", "operation of control panel" is assumed to be "unrecognizable" as "operation of control panel". Furthermore,
The environment in which the subject (plant supervisor) is said to be "at a high place" from the operation of "climbing the ladder" is also assumed. Therefore, the work associated and estimated from “climbing the ladder”, “walking”, and “unrecognizable” is recognized as “control panel operation at a high place”. Similarly, "go down the stairs", "walk",
In the case of the operation pattern 83 of “squatting” and “unrecognizable”, for example, it is associated with “valve operation on the floor surface”.

While referring to these association patterns, the association processing device 64 constantly monitors the storage device 62 for storing the history of operation / action. For example, section A (7
When the same operation pattern as the associative pattern 81 appears in the monitoring data as in 2), the recognition result indicating that the subject 1 is performing “control panel operation at a high place” in this section is used as the associative processing device 64. Will output. Similarly, section B (73)
Since the same operation pattern as the association pattern 83 appears, it is recognized that the subject is performing “valve operation on the floor” in this section.

Further, not only the work content but also the position can be specified. For example, when the recognition device shown in FIG. 10 is used as the recognition device 61 and the cellular method shown in FIG. 11 is used as the position measuring device 41, accurate position measurement is impossible, and the position measurement device 41 exists only in a certain area. This is a position measurement accuracy that can be understood. However, in combination with this embodiment, if there is only one place in the area where "control panel operation is performed at a high place",
If it is recognized as "control panel operation at a high place", that is, the place where the subject is located can be specified as "a place to operate the control panel at a high place".

According to the present embodiment, there is an effect that the behavior of the subject, the situation of the work, and the environment where the subject is placed can be recognized based on a plurality of operation histories and combination patterns.

Further, even if the operation cannot be recognized by the waveform recognition method due to the complicated operation, there is an effect that the operation content can be inferred from the operation history and the contents of the work to be performed.

Further, when the place where the work is to be performed is determined, there is also an effect that by recognizing the work of the subject, it is possible to use the recognition result to specify the exact place where the subject is performing the work. is there.

Next, an embodiment of a device to which the present invention is applied and which can present the difference between a standard operation and a measurement operation to a subject will be described with reference to FIGS.

The present embodiment can be used for rehabilitation when a foot or the like is damaged by an accident or the like. Here, the standard operation refers to a walking operation or the like before receiving an obstacle, and the measurement operation refers to a walking operation or the like of a current subject who is experiencing an obstacle.

As shown in FIG. 15, the apparatus according to the present embodiment comprises measuring instruments 2 and 3 attached to a subject 1 and an A / D converter 4 for digitizing the measurement results from the measuring instruments 2 and 3.
A feature amount extraction unit 5 for extracting feature amounts from digitized measurement results, and a standard operation feature amount database 9 for storing various standard operation and action feature amounts measured and extracted in advance.
1, a signal processing unit 92 for detecting a difference between an operation indicated by the feature amount extracted by the feature amount extraction unit 5 and a standard operation stored in the standard operation feature amount database 91, and a difference from the detected standard operation. And an output unit 93 that outputs

It is optimal to extract and register the feature values of the standard motion of the subject before the accident in the standard motion feature database 91, but if there is no such feature, the average of many subjects is averaged. A feature amount obtained from a simple operation may be used as a standard operation. The signal processing unit 92 compares the difference between the standard operation described above and the operation currently performed by the subject, for example, in FIG.
It is calculated as shown in FIG. Here, the standard operation is shown in FIG.
It is assumed that the frequency spectrum 101 is as shown in FIG.

In general, when the rehabilitation has not progressed, the operation cannot be performed smoothly. For this purpose, for example, as shown in the frequency spectrum 102 of FIG.
The spectral components 106 and 107 greatly differ from the corresponding standard operation spectrum components. The result (absolute value) of the difference between the frequency spectrum 101 and the frequency spectrum 102 is the frequency spectrum 104. This difference result shows the spectral components 108 and 109 because the operation cannot be performed smoothly. The subject can be presented with the difference result of the spectrum by the output unit 93 to visually show the difference from the standard operation.

The subject performs rehabilitation so that such prominent spectral components do not appear in the difference result. That is, the rehabilitation is continued so that the same spectrum as the standard frequency spectrum 101 can be obtained as much as the frequency spectrum 103 in FIG. When the rehabilitation is completed, the motion of the subject becomes almost equal to the standard motion, and as shown in the frequency spectrum 105,
No noticeable difference spectrum appears.

In the present embodiment, the rehabilitation of the patient has been described as an example. However, the same usage can be used for correcting the movement of the athlete.

According to the present embodiment, there is an effect that the progress of rehabilitation and the progress of motion correction can be automatically presented.

Next, FIG. 17 shows an embodiment of a recognition apparatus to which the present invention is applied and which can detect a difference between a standard operation and a measurement operation and can present the cause of the detected difference to a subject. It will be described using FIG.

As shown in FIG. 17, the recognizing device of this embodiment differs from the recognizing device shown in FIG. 15 in that a feature amount database 111 for abnormal operation and a signal processing unit (hereinafter, referred to as "recognizing") cause for abnormal operation. (Referred to as a second signal processing unit) 1
12 is added. Note that the “abnormal operation” here refers to an operation in which the way of raising the foot is different from the standard walking because the foot or the like is damaged by an accident or the like. The abnormal operation feature amount database 111 stores various abnormal operation feature amounts. For example, FIG.
If the frequency spectrum 104 of the right foot is raised because the way to raise the toe of the right foot is different from the standard operation, the difference results 109 and 108 of the spectral components are the feature amounts that appear because the “how to raise the toe of the right foot” is abnormal. Become. In this way, by observing various abnormal operations in advance and extracting their characteristic amounts, a characteristic amount database 111 for abnormal operations is constructed.

The feature amount database 111 of this abnormal operation
, The second signal processing unit 112 that recognizes the cause of the abnormal operation recognizes the cause of the abnormal operation that appears in the operation of the subject. The content of the process is the same as the above-described motion / behavior recognition method, and the difference between the measurement operation and the standard operation calculated by the signal processing unit 92 that calculates the difference between the measurement operation and the standard operation;
The correlation with the characteristic amount of the abnormal operation stored in the abnormal operation characteristic amount database 111 is obtained, and the abnormality cause indicated by the characteristic amount having the highest correlation is output as a recognition result using characters or graphics. Output from the unit 113.

In this embodiment, the rehabilitation of the patient has been described as an example. However, the same usage can be applied to the correction of the movement of the athlete.

According to the present embodiment, there is an effect that the cause of the abnormal operation can be automatically recognized.

Next, an embodiment of a system in which the motion and action recognition apparatus to which the present invention is applied is used for collecting work data of a workability evaluation system or a work content evaluation system will be described with reference to FIG. I do.

Work evaluation or work content evaluation is a field that has been actively studied in production management engineering and the like. Work evaluation /
In order to evaluate work contents, work data must be collected. Heretofore, collection of this work data had to rely on humans, and required a great deal of labor. In the present embodiment, the purpose is to automatically collect work data that had to rely on manual labor.

As shown in FIG. 18, for example, the system according to the present embodiment includes recognition devices 120, 121, and 122 attached to subjects A, B, and C, respectively, and a database that stores recognition results from the recognition devices. 123, 124,
125, and a workability evaluation system 126 and a work content evaluation system 127 that accept the history of the operation and behavior of each subject stored in each database.

The motion and behavior recognition devices 120, 121, 122 attached to the subjects A, B, and C
This is a recognition device to which the present invention is applied, which automatically recognizes the movement / action of each subject every moment. Specifically, the device shown in FIG. 1 or FIG. 10 can be used. The recognition results are transmitted via the transmission path 128 to the databases 123 and 124 that store the history of the operation and behavior of each subject.
Store in 125. The transmission path 128 does not need to be wired, and may be wireless. When the recognition result is transmitted by wireless, the effect that the subject can freely move around is produced.

The workability evaluation system 126 or the work content evaluation system 127 refers to the value stored in the database in which the history of the operation / action is stored, and is evaluated in each system.

According to the present embodiment, work data can be automatically collected, so that significant labor can be reduced.

In this embodiment, an apparatus to which the present invention is applied for collecting work data is used.
The specific contents of the workability evaluation system 126 and the work content evaluation system 127 that are actually used are not limited at all.

Next, FIG. 19 shows an embodiment of a system in which the operation and action recognition apparatus to which the present invention is applied is used for collecting work data of a proper staff distribution system or a proper work time distribution system. It will be described using FIG.

Here, the appropriate personnel distribution system is as follows.
It refers to a system that controls the number of personnel to be allocated to each work section, taking into account the work load of the workers. The appropriate work time distribution system refers to a system that controls work time according to the work load of a worker. In any system, it is necessary to grasp the work contents of many workers in real time. An object of the present embodiment is to provide a system that makes it easy to grasp the work contents of many workers in real time.

The system according to the present embodiment is, for example, as shown in FIG. 19, the recognition device 12 shown in FIG.
0, 121, 122 and databases 123, 124,
125, an appropriate personnel distribution system 134 and an appropriate work time distribution system 141 using the data collected in these databases, and the subjects A, B, and C
And instruction receiving devices 131, 135, and 138 that receive instructions from these distribution systems.

The motion and action recognition devices 120, 121, and 122 attached to each subject automatically recognize the motion / action of each subject at every moment, and store the history of the motion and action of each subject in the database 123. , 124, 125. The path for transmitting the recognition result may be wireless. In this case, the effect that the subject can freely move around is created.

The values stored in the database storing the operation / action history are stored in the appropriate personnel distribution system 13.
4 or the appropriate work time distribution system 141, and each system determines the work load situation of each worker by a predetermined method. For example, an appropriate personnel distribution system 13
When it is determined from the data that the work of a certain worker is excessively loaded, another worker selects an unloaded worker and calls the selected worker via the route 130. , And give instructions to support the work of the added worker.

Also, a proper work time distribution system 141
When it is detected that the cumulative load of a certain worker's work exceeds a certain value, the worker who exceeds the value is urged to take a rest, and a command to give the work to another worker is given via the path 142.
Each can be sent to the instruction receiving device.

According to the present embodiment, there is an effect that the work contents of many workers can be grasped in real time, and the workers can be optimally arranged according to the work situation.

Next, a description will be given of an embodiment of a system capable of notifying that a specific operation or action has been performed by using a recognition apparatus to which the present invention has been applied, with reference to FIG.

The system according to the present embodiment is for automatically reporting to a parent or an administrator when a vulnerable person or a worker who needs protection is in danger for some reason. is there. In the present embodiment, a part of the same configuration as that included in the apparatus in the embodiment of FIG. 12 is included. These same configurations are described in FIG.
The same reference numerals as in 2 denote the same parts, and a detailed description thereof will be omitted.

As shown in FIG. 20, the system according to the present embodiment includes an operation / action recognition device 61 and a recognized operation / action.
A storage device 62 for storing a history of actions, a storage device 145 for storing a specific operation pattern, and both storage devices 6
2. An associative processing device 64 for associating or estimating a specific action or behavior of a subject by using data stored in 2,145, and a notifying device for notifying a parent or guardian of a specific message in accordance with the result of associating or estimating. 146 and a receiving device 147 for receiving the message.

The action / action recognition device 61 corresponds to the above-described recognition device shown in FIG. This recognition device recognizes only a single action / action that occurs within a limited time, and is suitable for recognizing an action or task that is completed by stacking several actions / actions. Absent. Thus, the recognized operation / action is stored in the storage device 62 that stores the history of the operation / action every moment.

Further, the stored history information is stored in the storage device 14 of the specific operation pattern by the associative processing device 64.
5 is constantly monitored while referring to the storage pattern stored in the storage pattern 5. The specific operation pattern referred to here is a combination pattern of a plurality of operations for recognizing an action such as “suddenly falling” or “falling from a high place”. For example, the action of "suddenly falling" is recognized by the motion pattern of "walking or standing still" → "lying down in a short time" → "still lying down". In addition, the action of “falling from a high place” can be recognized by the action pattern of “climb up” → “fall” → “hit various things” → “arriving on the ground” → “do not move”.

The operation of “climbing up” can be recognized by detecting that an upward acceleration larger than, for example, the gravitational acceleration continues. The actions "fall", "collide variously", and "arrive on the ground" are respectively "zero acceleration in all directions (because of free fall)" and "strong acceleration in various directions in a short time". , It can be recognized by detecting that "substantial acceleration is received".

By recognizing the behavior of the subject using such a motion pattern, in the present embodiment, it is possible to eliminate a motion or a false recognition of the behavior. For example, consider a case where the sensor is removed from the body and placed roughly on the table. In this case, the sensor detects a fairly large acceleration, but a recognition device that recognizes a single motion does not know the history of the motion so far. The possibility of recognition increases. On the other hand, according to the associative processing device, the operation before and after the detection of the large acceleration can be checked, so that the situation of the subject can be grasped and more correct recognition can be obtained. For example, if an operation such as “dismounting the device” is recognized in advance by detecting an acceleration or the like that is impossible when the sensor is attached to the body, such a case occurs when only a single operation is recognized. Misunderstandings can be avoided.

When the associative processing device 64 finds these specific operation patterns from the history information stored in the storage device 62, the associative processing device 64 informs the parents via the notification device 146 to the parents. Issue a command to report the contents of The guardian can receive the notification by the notification receiving device 147 and grasp the situation where the subject is placed.

According to this embodiment, when a socially vulnerable person who needs protection or a single worker becomes in danger for some reason, the guardian or manager can be automatically notified of the danger. There is an effect that can be done.

Further, the recognition apparatus of this embodiment has
Position measuring means 41 for measuring the position of the subject described in FIG.
Alternatively, the position of the subject is specified from the operation history of the subject as in the associative processing device 64 in FIG. 12 described above. , And the effect that "what happened" can be grasped.

Further, by providing the position measuring means 41 as described above, even if a specific operation pattern is observed, it is possible to select whether or not to notify according to the observed position. Good. For example, when lying down in a consultation room in a hospital or a bed at home, an effect is produced that a false report is not issued as a subject falls down.

Next, an embodiment of a system in which the motion and action recognition apparatus according to the present invention is applied to action analysis, action tracking, and action monitoring will be described with reference to FIG.

In the present embodiment, the behavior analysis of an animal will be described as an example. In this field, there have been various researches on the trajectories of animals. However, in conventional research cases,
The movement / behavior of an animal is only a visual observation method of a human being, and the movement and behavior of an animal in a place where a human cannot observe is not clarified in many parts. The system according to the present embodiment can automatically track the operation / action.

The system according to the present embodiment has a configuration as shown in FIG. In this figure, 151 and 15
Reference numeral 2 denotes a “bird” to be subjected to behavior analysis, behavior tracking, and behavior monitoring. Each bird is provided with a measurement / communication device 150 that measures a change in the state of the bird and transmits the measurement result by wireless. Artificial satellite 15
Numeral 3 receives the state change measurement value sent from each bird and measures the position where the bird has transmitted data.

As a method of measuring the position, for example, a GPS receiver is built in the measurement / communication device 150, and when transmitting the activity data to the satellite 153, the GPS is transmitted together with the activity data.
There is a method of transmitting the position data indicating the current position of the bird measured by the S receiver together. In addition, a plurality of satellites 153
Then, a method of receiving a radio wave transmitted from a bird and detecting the position of the bird by triangulation may be employed.

The data collected and measured by the artificial satellite 153 is sent to, for example, the motion and behavior recognition device 154 shown in FIG. 1, and the motion / behavior of the bird within a short period is recognized.
For example, if the position of the bird does not move so much and does not move, it is recognized that the bird is sleeping, and if only the position of the bird changes significantly without changing the posture of the bird, it is recognized that it is flying, When the posture of the bird changes greatly and the position does not change much, it is recognized that the bird is preying.

In the system according to the present embodiment, information on the recognized movement and behavior of these birds is sent to the behavior analysis unit 155, the behavior tracking unit 156, and the behavior monitoring unit 157. In the behavior analysis unit 155, for example, the time of awakening, movement, predation, sleep, etc. of the bird to be observed is checked,
Analyze bird behavior patterns. The behavior tracking unit 156 analyzes the range of movement of the bird and the movement trajectory that moves for predation. The behavior monitoring unit 157 performs behavior monitoring by grasping the current position, behavior, and the like of each individual bird.

Finally, the analysis results obtained by the above-described devices are displayed in a list on a display device 158 such as a CRT. The figure shows an example in which the action performed by the bird is superimposed and displayed on the movement locus of a certain individual measured by the artificial satellite 153. According to this example, it is automatically known when, where, and what kind of action each bird individual has performed.

According to the present embodiment, there is an effect that the movement trajectory and the movement / behavior of the observation target can be automatically observed even in a place where humans cannot go.

Further, according to the present embodiment, it is not necessary for the observer to directly observe the object with the naked eye, so that there is no change in the animal behavior due to the human line of sight, and the movement and behavior of the natural figure can be observed. There is also.

Next, an embodiment of a gesture recognition apparatus using the action / action recognition method according to the present invention will be described.
This will be described with reference to FIG.

[0187] The gesture means means for transmitting a meaning to be conveyed by preliminarily determining an operation indicating a specific meaning and performing the predetermined operation. In this case, the action to be performed may be different from the general meaning and the meaning desired to be conveyed by gesture. For example, in the case of "pointing the right direction with a finger", this gesture is
If it is agreed that "meaning to the left", the person interpreting this gesture will proceed "to the left".

The apparatus of this embodiment is different from the apparatus of FIG. 1 in that a conversion table database 161 for associating the meaning of a gesture with the corresponding action / action, the action / action recognized by the signal processing unit 7 is used. A gesture recognition unit 162 for recognizing the meaning of the gesture of the subject 1 is added from the conversion table, and the output unit 163 outputs the meaning of the recognized gesture.

In the present embodiment, the gesture performed by the subject 1 is measured by the measuring units 2 and 3, and the measured signal is digitized by the A / D converter 4, and the characteristic amount extracting unit 5 extracts the digitized signal. The feature amount is extracted.
The feature amount corresponding to the gesture motion extracted in this manner is sent to the signal processing device 7 for recognizing the motion / action, and is recognized as a motion / action generally meaning. The result recognized here is the gesture recognition device 1
62.

The gesture recognizing device 162 includes a gesture conversion table database 16 in which the meaning of a predetermined action / action is stored as a table.
1, and the meaning of the gesture is recognized. The contents of this conversion table are, in the previous example, “pointed right with a finger” → “progress left”.
The meaning of the gesture recognized in this way is determined and output by the output unit 163.

According to the present embodiment, by performing a predetermined gesture, it is possible to transmit a specific meaning determined corresponding to the gesture.

Further, since the gesture is an action whose meaning is determined in advance, there is an effect that recognizing this gesture eliminates the ambiguity of the action.

Next, an embodiment of a device in which a wireless means is added to the operation and behavior recognition device to which the present invention is applied will be described with reference to FIG.
This will be described with reference to FIG.

In the apparatus of this embodiment, the operation and effect relating to recognition are the same as those of the apparatus of the above-described embodiment of FIG. 1, except that data relating to the detected state change of the subject 1 is transmitted by the added wireless function. Are different.

As shown in FIG. 23, the recognition device of the present embodiment comprises two parts: a subject side device 173 to be attached to the subject 1 and a measurer side device 177. The subject side device 173 includes a state change measuring unit 171 (corresponding to the measuring devices 2 and 3 in FIG. 1) attached to the subject 1 and a wireless transmission unit for transmitting a signal measured by the measuring unit 171. 172. For this reason, there are only two types of devices that the subject 1 must carry, the measuring unit 171 and the transmitting unit 172, and the device can be miniaturized, so that the subject can move around more freely.

The signal 178 transmitted by radio may be, for example, the signal itself (analog signal) measured by the state change measuring unit 171 or may be A / D converted and sent as a digital signal. Although the type of signal used in the present embodiment is not particularly limited, for example, radio waves, infrared rays, or the like can be used.

The measurement result sent to the measurer-side device 177 is received by the receiving unit 174, and is recognized by the operation and action recognition unit 175. The recognizing unit 175 is, for example, the A / D converter 4, the signal characteristic amount extracting unit 5, the operation / action characteristic amount database 6, and the signal processing unit for recognizing the operation / action in the embodiment of FIG. 7 is comprised.
The recognition result in the recognition unit 175 is sent to the output unit 176 and output.

According to the present embodiment, the state change measuring section 1
To transmit the result measured by 71 wirelessly,
The subject 1 can move around freely, and has the effect of not imposing a large burden on the subject 1.

Further, since the device attached to the subject is composed of only the measuring unit for state change and the transmitting unit for transmitting the signal, there is also an effect that the size and weight can be reduced.

Next, another embodiment of a device in which a radio means is added to an operation / action recognition device to which the present invention is applied is shown in FIG.
4 will be described.

In the present embodiment, the subject's device recognizes the motion or action of the subject and sends the recognition result to the subject's device by wireless communication. The same components as those of the apparatus of FIG. 23 are denoted by the same reference numerals, and description thereof will be omitted.

[0202] Measuring unit 1 for state change of subject side device 184
The signal measured by 71 is used as an operation / action recognition unit 17.
5. This recognition result is transmitted to the wireless transmission unit 181.
And sent to the measurer side device 185. The recognition result sent to the measurer-side device 185 is directly output by the output unit 176.

Here, the information 183 to be transmitted by radio is a result of recognition, and is considerably compressed compared to the measurement information. For this reason, the time occupied by the wireless band or transmission can be considerably reduced. In addition, subject 1
Can move around freely with the device.

According to the present embodiment, since the result measured by the state change measuring device is wirelessly transmitted, the subject can freely move around and there is an effect that the subject is not burdened.

Further, since the amount of information to be transmitted by radio is small, there is an effect that the radio band is narrowed, the occupation time of the radio channel is reduced, or the transmission speed is increased.

In the above-described embodiments of FIGS. 23 and 24, the device to be walked by the subject is reduced in size by providing a wireless function. However, a portable unit is used instead of the transmitting / receiving unit for realizing the wireless function. The data storage device and the data reading device may be provided on the subject side and the measurer side, respectively. According to this configuration, the measured motion data of the subject or the recognition data of the motion is temporarily stored, and even when the subject drops in on the observer side, the data reading device reads the data from the data storage device. Data can be read directly.

Next, an embodiment of a system in which a prepaid function is added to a motion / action recognition apparatus to which the present invention is applied will be described with reference to FIG.

As shown in FIG. 25, the system according to the present embodiment includes, as devices on the subject side, a recognition device 192 for recognizing the motion / action of the subject, a storage device 195 for a charge balance for realizing the prepaid function, and A charge balance calculation device 196. Further, the system according to the present embodiment has a receiving device 1 that receives a recognition result as a device on the observer side.
93, and a prepaid function control device 197 that outputs a charge balance reduction command or a charge balance confirmation command according to the received recognition result.

Here, as an application example of the embodiment, an example of a game in a game hall will be described. The motion / behavior recognition device 192 in the amusement arcade is used for gesture input, and is realized by, for example, the device shown in FIG. 1 or FIG.

In this example, when the player (subject) makes a certain gesture, the recognition device 192 recognizes the gesture, and the recognition result of the gesture becomes an input command to the gaming device (observer side device), People enjoy playing games. For example, a player may say "a gesture of a skill"
Then, the character displayed on the gaming device executes the technique and fights.

In the present embodiment, a fee is paid when playing a game, but in the present embodiment, devices 195 and 196 for realizing a prepaid function are built in the device on the player side.
In the present embodiment, when the test subject starts a game, the recognition device 192 detects this and sends the recognition result to the reception device 193 on the game device side. On the gaming device side, the reception device 193
The prepaid function control device 197 sends to the player a command such as a command to reduce the charge balance of the player or a command to confirm the charge balance in accordance with the type of the game corresponding to the recognition result received in the step. On the player side, the balance stored in the charge balance storage device 195 is reduced in response to the command sent by the charge balance calculation device 196.

Although not shown, the device on the subject side is provided with a warning means, and when the balance stored in the storage device 195 becomes lower than a certain value, a warning is issued to the player. It is configured to emit. Further, the player pays a predetermined fee to the gaming device before starting the game, and sets an initial value of the fee balance storage device 195 according to the paid amount.

According to the present embodiment, the gesture input function for playing a game and the prepaid function can be combined. Therefore, there is an effect that the player can enjoy the game freely without having to carry many devices.

Next, an embodiment of a system in which a credit function is added to an operation / action recognition device will be described with reference to FIG. In this embodiment, the same devices as those in FIG. 25 are denoted by the same reference numerals, and detailed description thereof will be omitted.

The system of this embodiment has an operation / action recognition device 192 and a fee payment command device 198 on the subject side, and a recognition result receiving device 193 and a payment command receiving device on the observer side. A receiving device 199 is provided. Here, as an example of the system, FIG.
An example of a game at a game hall described in the fifth embodiment will be described.

In the present embodiment, the device to be attached to the subject has a built-in credit card function.
That is, when the player plays a game, the recognition device 192 recognizes the type of the game and the like, and a fee payment command device 198 issues a fee payment command according to the recognition result. This payment instruction is received by the receiver 199 on the observer side, and processing such as notification of the payment fee and credit number to the corresponding credit company is performed by the credit processing device 200.
Is executed by

According to this embodiment, the gesture input function for playing a game and the credit function can be combined. Therefore, there is an effect that the player does not need to carry many devices.

Next, an embodiment of a system in which a deposit function is added to an operation / action recognition apparatus to which the present invention is applied, will be described.
This will be described with reference to FIG. In this embodiment, the same devices as those in the embodiment of FIG. 25 are denoted by the same reference numerals, and detailed description thereof will be omitted. The system of the present embodiment includes:
The device on the subject side includes an operation / action recognition device 192 and a deposit return instruction device 202, and the observer-side device includes a recognition result receiving device 193 and a deposit payment device 203. Here, as an example of the system, the example of the game in the game hall described in the embodiment of FIG. 25 will be described.

Here, it is assumed that the player borrows a gesture input device (recognition device 192) to play a game, and returns these after the game is over. It is known that when renting a part of a device used in a game, if the lent device is not returned, a penalty is applied to increase the collection rate. One of them is a deposit system. The system of the present embodiment supports this by paying a deposit when the player borrows the gesture input device 192, and the gesture input device 1
This is a system in which the deposit is returned when 92 is returned.

In the present system, the player borrows the gesture input device 192 and pays the deposit to the observer. When the gesture input device 192 is returned, a command is issued to the deposit payment device 203 on the observer side from the deposit return instruction device 202 attached as a device on the subject side together with the gesture input device 192. The deposit payment instruction device 203 receives the instruction and returns the deposit to the player.

According to the present embodiment, there is an effect that the collection rate of the motion / action recognition device is increased.

Note that the system of this embodiment stores the gesture input device, accepts a deposit of a predetermined amount, and pays out one of the deposits, and the borrowed gesture. The present invention can be applied to any automatic collection device of the gesture input device that accepts the input device and pays out the same amount of money as the deposit.

Next, an embodiment of a system in which a function for confirming intrusion or exit from an area is added to an operation / action recognition apparatus to which the present invention is applied will be described with reference to FIG. In the present embodiment, the same components as those in FIG. 25 are denoted by the same reference numerals, and detailed description thereof will be omitted.

The system of the present embodiment has a motion / action recognition device 192 and a response device 205 to a region intrusion or exit device as a device on the subject side, and receives a recognition result as a device on the observer side. It has a device 193, a region intrusion or exit confirmation device 206, and a notification device 207. Here, as an example of the system, the example of the game in the game hall described in the embodiment of FIG. 25 will be described.

[0225] Here, the player uses the gesture input device.
The game is performed by renting out the (subject-side device) and returning the game after the game is over. By the way, the recognition device 1
At the time of collection of 92, if a presentation for prompting the player to return is presented, there is an effect of increasing the collection rate. For example, there is a method of detecting a warning and issuing a warning if an attempt is made to leave the game input area while holding the gesture input device when trying to leave the area where the game is played. The purpose of this embodiment is to perform this automatically.

The gesture input device has a built-in recognition device 192 for recognizing a gesture and a reaction device 205 for a region intrusion or exit device. On the other hand, a radio wave reaching only an extremely close area is emitted from the observer side intrusion or exit confirmation device 206.

When the player tries to leave the area with the gesture input device attached, the response apparatus 205 receives a signal emitted from the area intrusion or exit confirmation apparatus 206 attached to the boundary of the area. Then, the response signal is returned to the confirmation device 206. By receiving the signal, the confirmation device 206 can detect that the subject is about to leave the game area while holding the gesture input device. Upon receiving this detection result, the notification device 207 issues a warning to the subject.

According to the present embodiment, there is an effect that the collection rate of the motion / action recognition device is increased.

Here, taking a periodic motion as an example, an example of recognition of a motion will be described in detail.

FIG. 47 is an explanatory diagram of a method of extracting a walking cycle, which is a basis for recognizing a periodic motion. The walking cycle is the elapsed time from the heel contact (the moment when the heel of the back foot touches the ground) to the heel contact of the same leg, and it is necessary to take two steps if the left and right legs are carried at the same speed. Time is the walking cycle. Observation of the frequency characteristics of the acceleration signal in the vertical direction of the body at various walking speeds and walking states (“walking”, “running”, etc.) on a flat ground shows that in the standard walking and running, FIG. It has been found that there is one very strong spectrum whose center frequency is the walking cycle such as 2001. Therefore, for a standard walking or running operation, the walking cycle can be easily obtained by extracting this spectrum by simple threshold processing. However, when walking extremely slowly with a walking cycle of about 1.5 seconds or more (average walking cycle of a healthy person is 1.03 seconds, walking cycle of an elderly person is 1.43 seconds), or a person drags his foot,
As shown in FIG. 47 (b), the frequency 2002 of the walking cycle
In addition, it has been found that a spectrum appears in some cases, and in an extreme case, the spectrum intensity becomes larger than the spectrum intensity in the walking cycle as in 2006. (FIG. 47 (a) and FIG. 47 (b)
Normalizes each spectrum excluding the DC component with the maximum spectrum intensity. However, fortunately, in the walking movement of 0.35 to 2.0 seconds (“running” to “extremely slow walking”) observed in the walking cycle, no strong spectrum exists at a frequency lower than the frequency of the running cycle. Therefore, the following processing can be performed. This will be described below with reference to FIG.

(1) The average value T of all the frequency spectra excluding the DC component (0th harmonic) is obtained.

[0232]

(Equation 8)

(2) To perform threshold processing that leaves a spectrum larger than the average value T (2003)

[0234]

## EQU9 ## A section (a _i , b _i ) that continuously satisfies the condition of n ≠ 0 and P (n)> T (Equation 9) is obtained.

Here, the threshold value is set to the average value T of the power spectrum. Since the spectrum intensity indicating the walking cycle and the other spectrum intensities vary depending on the type of operation, the walking cycle spectrum is extracted with the fixed threshold value. This is because it may not be possible.

In the present embodiment, the average value T is used as the threshold value. However, in the noise state of the measurement data, the determination process may be performed using a threshold value obtained by multiplying T by a certain coefficient.

(3) Among the sections satisfying (Equation 9), the maximum spectrum intensity P (j) 2005 and its frequency j2004 are obtained in the section of the lowest frequency spectrum group (i = 0).

[0238]

S = P (j) = max [P (a ₀ ), P (b ₀ )] (Equation 10) Here, S is a value representing the spectral intensity of the walking cycle. Also, since the frequency j is the number of steps per second, it is 1 in 2 steps.
Walking cycle G _C is a cycle,

[0239]

[Equation 11]

It can be obtained as follows.

In FIGS. 47A and 47B, 2001 and 2002 are extracted spectra showing the walking frequencies. The measurement using the stopwatch was performed in the walking cycle of 0.35 to 2.0 seconds in parallel with the extraction method of the present method, and the value almost coincided with the value measured by the stopwatch.

The difference between "walk" and "run" is the presence or absence of a section where both feet are separated from the ground. When the feet are separated and enter the free fall section, the acceleration measured by the acceleration sensor attached to the body will be observed as about 0 G, that is, almost zero gravity. Therefore, it was thought that the difference between "walking" and "running" can be recognized by judging whether or not this weightless state exists. Fig. 48 (a) and (b) show standard observation waveforms of "walking" and "running".
Shown in In the case of “walking” in FIG. 48 (a), an acceleration change with an amplitude of about 0.3G is shown around the gravitational acceleration 1.0G. Therefore, the minimum value of acceleration applied to the body is about 0.7.
G and no weightless state exists. FIG.
In “run” in (b), the amplitude increases to about 1.0 G, and a weightless state of 0.0 G exists. In order to recognize this weightless state, we used the recognition of the walking cycle,
The spectral intensity of the walking cycle was used. This is because if the acceleration change is a sine curve representing only the walking cycle, the obtained spectrum intensity of the walking cycle is the amplitude of the walking cycle. In reality, FFT
Sampling relationship between window function width and walking cycle, the effect of acceleration change deviating from the sine curve, and the effect of the inertia force of the minute weight in the acceleration sensor, etc., do not completely match and judge "run" Performs the walking cycle with an amplitude of 0.8G. However, focusing on the fact that the spectrum intensity of this walking cycle can be used to determine the transition of the walking state from the stationary state to “walk” and “run”, recognition based on the spectrum intensity was employed. Of course, the determination can be made by finding the minimum acceleration value of the measured waveform, but it is necessary to consider the problem of erroneous recognition due to noise.

As a result of the observation, it was found that the spectrum intensity of the recognized walking cycle became stronger as the movement changed from a stationary state to a violent movement from "walking" to "running". Therefore, instead of binary recognition such as "walking" and "running", an attempt was made to subdivide and recognize the walking state with reference to the spectral intensity of the walking cycle. FIG.
Fig. 7 shows how the accuracy of the recognition item is assigned to the spectrum intensity of the walking cycle. The recognition accuracy w _n of each recognition item is
Expressed as a function of the spectral intensity S of the walking cycle, and at a possible value S _U of the spectral intensity,

[0244]

## EQU12 ## It is assumed that an arbitrary value between 0 and 1 is taken as in w _n : S _U → <0,1> (Equation 12). Hereinafter, this is used as an evaluation function of the recognition accuracy. Two types of evaluation functions are used. One is a function that evaluates “walking” and “running.” If the spectrum intensity above a certain value becomes stronger, it is evaluated as a “running” state and a “running” state.

[0245]

(Equation 13)

The following function is used. The other function is a function that evaluates “relaxing” or “powerful”
Having a peak at a certain value,

[0247]

[Equation 14]

The following function was used. The values of the parameters are determined while observing the state of several healthy subjects. The horizontal axis is the spectrum intensity of the walking cycle, and the vertical axis is the recognition accuracy. Recognized actions are "still",
"Walk" and "run". "Relaxing" expresses the degree of the process from the "stationary" state to the standard "walking".
Furthermore, there are five types of “powerful” that express the degree of the process from the standard “walk” to “run”.

[0249] The recognition of "stillness" was made in a state where the spectral intensity of the walking cycle was 0.006G or less. This is because the maximum spectral intensity of the acceleration measured in the stationary state is 0.004.
G This is a threshold process for preventing noise with about G from being erroneously recognized as a walking cycle.

Recognition of "walking" increases the recognition accuracy from the threshold value in the stationary state, and the spectral intensity of the walking cycle is 0.0.
In the case of 3G or more, it is set to 1.0 (2101).
“Running” is defined as indicating all walking actions including “running”. Therefore, the recognition accuracy of "running" is always 0 or more when the motion is recognized as a periodic motion. This is because the walking intensity is 2.03 seconds, and the spectral intensity of the walking period is 0.03G in the case of walking while suppressing the vertical movement of the body as much as possible, and the stationary state (0.006G) And walking state
Such processing is performed in order to express an ambiguous section of (0.03G). Therefore, the evaluation function of “walking” is

[0251]

[Equation 15] w _walking (S: 0.006, 0.03) (Equation 15)

Recognition of "relaxation" is monotonically increased from the threshold value in the stationary state, the maximum accuracy is set to 1.0 at 0.1G, and thereafter monotonically reduced to 0.3G, and the accuracy is set to 0.0 at 0.3G. (2102). This was done because the spectral intensity when the subject walked leisurely was about 0.1 G, and when the subject walked normally was 0.3 G. The evaluation function is

[0253]

(16) w _leisurely (S: 0.006, 0.1, 0.3) (Formula 16)

Recognition of “strong” is based on the standard walking (0.
3G), increasing to 0.5G for maximum accuracy, and
The accuracy is set to 0.0 at 0.8 G (2103). This was also determined by observing the subject's "strong walking" and "running" as in the case of "relaxing". The evaluation function is

[0255]

(17) w is _strong (S: 0.3, 0.5, 0.8) (Equation 17)

"Run" is increased from 0.5G to 0.8G.
For G and above, the recognition accuracy is set to 1.0 (2104). When there is time when both feet do not touch the ground, the spectrum intensity observed when performing `` very quiet running '' is about 0,0.
This value was used because it was 8G. The evaluation function is

[0257]

[Expression 18] w _running (S: 0.5, 0.8) (Equation 18) was used.

For recognition, the accuracy of each recognition item is determined by using the above evaluation function based on the extracted spectrum intensity value of the walking cycle. For example, when the spectral intensity of the walking cycle is 0.6
G indicates that the accuracy of “walking” is 1.0 and the walking state includes “running”, and the accuracy of “strong” is 0.
67, it is recognized that the subject is performing a motion having a certainty of “run” of 0.33.

The standing position (“standing”) and the crawling position (“lying”) are recognized by the average value of the acceleration change observed by the acceleration sensor. The acceleration sensor observes a change in acceleration in one axial direction due to its structure. In the case of a tilt angle of 0 degree in which the sensitivity direction of the acceleration sensor and the gravity direction are matched, 1.0 G equal to the gravitational acceleration is observed. When the inclination is increased, the acceleration gradually decreases in almost proportion to the angle. When the inclination angle becomes 90 degrees where the sensitivity direction of the acceleration sensor and the gravity direction are orthogonal, the gravitational acceleration is not observed. Therefore, in a stationary state, the inclination of the acceleration sensor (= the inclination of the body) can be measured from the observed acceleration value. In addition, assuming that no acceleration other than gravity is applied to the body as an external force in a state in which the body is moved, the time average value of the observed acceleration is considered to be the gravitational acceleration. Therefore, the inclination of the body can be determined by calculating the average value of the acceleration change and comparing the average value with the gravitational acceleration. The average value of the change in acceleration is output as a DC component (0th harmonic) at the time of frequency analysis, and is used. The standing / crawl position is determined from the obtained body inclination.

The recognition method according to the present invention outputs the recognition accuracy of each recognition item such that the accuracy of “powerful” is 0.67 and the accuracy of “run” is 0.33. Therefore, it is difficult to understand what the subject is doing just by looking at the output result. Therefore, in the present embodiment, a method of displaying an action recognition result by animation using CG (Computer Graphics) instead of numerical values or words is described, and a display method that is intuitive and easy to understand is described.

FIG. 30 is a block diagram of a system for displaying the result recognized by the motion / action recognition device in an animation using computer graphics using the method of expressing the motion / action of the articulated structure.

This configuration diagram is an operation / action recognition device 1033 for recognizing an operation / action based on the measuring devices 2 and 3 attached to the subject 1 and the result measured by the measuring device, and an operation / action recognition device 1033. In order to display the recognition result as a two-dimensional or three-dimensional image based on the feature amount database 1030 of the action or action and the recognition result 1034, the action / action expression device 1035 of the articulated structure and the action / action are expressed. A database 1031 for storing feature values of actions / actions necessary for the operation and an output device 1032 for displaying an expressed result as an image.

First, the measuring instrument 2 attached to the subject 1
Alternatively, the measurement result sent from 3 is sent to the action / action recognition device 1033, and is compared with the feature amount stored in the action / action feature amount database 1030, and the recognition result is output (1034). For example, in this example, a “walking” operation is performed. This result is output to the motion / behavior expression device 1035 of the articulated structure.

The expression device 1035 retrieves feature data corresponding to the feature from the storage unit 1031 based on the feature of the motion as the recognition result of the recognition device 1033, and obtains the motion information of the articulated structure (human in this example). Is generated, and an animation is displayed on the display 1032.

A method of expressing a human motion corresponding to a feature using computer graphics is described in “Method of Generating Human Walking Motion with Emotion in Computer Animation” written by the inventors of the present invention. IEICE Transactions, D-II, Vol.J76-D-II, No.8, pp.1
822-1831, August 1993 or US Patent 5,4
No. 83,630. In this document, from various walking actions of a human, feature amount components representing features of each walking are extracted, and various extracted feature amount components are combined,
It describes a method of expressing various operations by computer animation by controlling the strength of a feature amount component.

In the present invention, an animation corresponding to the feature of the recognized motion is displayed using the technique described in the above-mentioned document.

The method disclosed in the above-mentioned document written by the inventors of the present invention can generate various periodic motions by controlling the feature amount of the motion. Hereinafter, an outline of the processing will be briefly described.

FIG. 31 is a schematic diagram showing a method of generating a human motion using the feature amount of the motion. FIG. 31 illustrates an example representing the movement 1044j (t) of the right knee joint. However, the movement of the other joints is generated by the same method to generate the movement of the joint of the entire body.

First, the bending angle of the joint when performing various walking motions is actually measured, and the frequency characteristics of the “standard walking” B (ω) and the joint motion J (ω) of the actually measured walking motion are measured. Extract the difference between

[0270]

G (ω) = B (ω) −J (ω) (Equation 19) Extracted as a feature amount G (ω) representing an actually measured operation. This feature amount is used as a feature amount database 1041 in FIG.
0 in the storage unit 1031. In this method, "walk",
In addition to verb expressions such as "run",
Adverbial expressions such as “depressed” and personality of walking can be extracted.

The motion is generated by creating a new feature by combining the features of the motion to be generated, and performing an inverse transformation process of generating a joint motion j (t) from the features.
At this time, as shown in the following equation (Equation 20), by superimposing a plurality of feature amounts or controlling the weight (strength) of each feature amount, an operation other than the actually measured operation can be generated. I can do it.

[0272]

(Equation 20)

For example, when “walking” and “vigorously” (adverbative features extracted from walking vigorously) are superimposed, actually measured “walking fine” is generated. Here, by controlling the weight of the feature quantity “energically”, for example,
If the weight is set to 1 or less, an intermediate action between “walking normally” and “walking well” is set to 1 or more. Expressions such as interpolation and extrapolation can be made. Further, a new operation that has not been actually measured can be generated. For example, "standard walking"
By adding the “run” action to the “run” action created by adding the “run” feature to the “run” action, an unmeasured action such as “run well” can be newly generated. The feature of this method is that various human actions can be easily generated by selecting and combining feature amounts and controlling the weights. The operation information is generated by the operation information generation device 1035 shown in FIG.

In the method of generating a human motion using the feature amount of the motion, the selection of the feature amount and the weight of the feature amount are controlled by using the recognition accuracy of each motion obtained by the motion recognition method using the acceleration sensor. By doing so, the recognition result can be displayed by CG animation. FIG. 32 is a schematic diagram showing a CG animation display of the recognition result. For example, as shown in FIG. 32 (1050), the recognition accuracy of _walking is w _walking , and the recognition accuracy of each recognition item R _n is w _n . First, a feature value G _n (ω) corresponding to the recognition item R _n is selected from the feature value database 1041. Based on these values, the feature values are superimposed as in the above equation.
By w _walking for recognition expression of _walking

[0275]

J (t) = F ^-1 [J (ω)] = F ^-1 [w _walking / {B (ω) + {w _n · G _n (ω)}]... Control of the entire feature amount is performed. With this control, when the recognition accuracy is smaller than 1, the movement of the entire body gradually decreases, and when the walking recognition accuracy becomes 0, all joint angles become 0 and an upright state is expressed.

Further, by selecting the frequency ω so as to move in the same walking cycle as the recognized walking cycle, a CG animation can be generated, and a CG character that walks in the same walking cycle as the subject can be generated. Thereby, the operation speed of the recognition result can also be expressed. FIGS. 33, 34, 35, and 36 show examples of display screens in which the operation is recognized using the recognition method of the present invention and the result is displayed by CG animation.
Shown in (A) of each figure is a measured waveform of the acceleration value measured from the acceleration sensor, (b) is an average value of the measured acceleration, (c) is a recognition result, (e) is a body motion spectrum,
(F) shows a walking cycle spectrum, and (g) shows an animation of a recognition result.

First, the example of FIG. 33 will be described. In the measurement waveform 1060 in (a), the horizontal axis represents time, and the vertical axis represents acceleration. In this example, the subject who is initially in a stationary state gradually starts walking and finally performs a strong walking motion. It can be seen that the amplitude increases as the person walks from a standstill state. (B)
It is an average value of the acceleration waveform of (a). In this example, since the subject is always standing, it always shows a value of approximately 1.0 G, which is the gravitational acceleration (1061). (E)
Represents a body motion spectrum obtained by frequency-analyzing the measured waveform. The horizontal axis is frequency and the vertical axis is spectrum intensity. The frequency 1072 having the highest spectral intensity is a spectrum representing the walking cycle, and (f) is a spectrum extracted from the spectrum recognized as the walking cycle. FIG.
(E) and (f) in the same spectrum are different only in the scale of the vertical axis. (E) is normalized so that the maximum spectral intensity is on the maximum scale of the graph. When the current time point is 1071 in the graph of (a), (e)
Are displayed so as to show the spectrum of the recognition result. In the graph of (e), the horizontal axis represents time, and the vertical axis represents recognition items. The recognition items are “walk” 107 in order from the bottom.
5, “powerful” 1074 and “relaxed” 1073. The recognition accuracy (weight) of each recognition is represented by the color density. When the color is white, the recognition accuracy is 0.0, and when the color is black, the recognition accuracy is 1.0. Nothing is displayed in the stationary section 1067 because there is no recognized recognition item. In the section 1068 at the beginning of walking, "walking"
It can be seen that the color of "Relaxed" gradually becomes black, and the recognition accuracy of each recognition item is increased. Furthermore, when the color reaches 1070 near the boundary between the start of walking and the energy, the color of “relaxing” gradually becomes white. This is due to approaching the standard walking, and indicates that the recognition accuracy of “relaxation” is gradually decreasing. It can be seen that the recognition accuracy of "powerful" gradually increases beyond the boundary, and that the player recognizes the "powerful walking" state. (G) shows an animation of the recognition result. This animation displays the recognition result at the present time (at the point 1071 in the graph (a)), and displays the action of “walking fine”.

Although not shown in detail in (h), a body motion spectrogram is displayed in which the horizontal axis represents time and the vertical axis represents frequency, and the spectral intensity is represented by the density of each pixel. FIG. 34 is an example in which “run” is recognized. It can be seen that the amplitude of the measured waveform (a) is considerably large.
The spectrum intensity 1080 of the walking period of the feature quantity (f)
Is considerably larger than that of FIG. The recognition result shows an animation of the "running" state as shown in (g). FIG. 35 is an example of recognition in the case of “walking leisurely”.

Since the state is “walking slowly”, that is, “walking slowly and quietly”, the amplitude of the measured waveform is not large as shown in FIG. The spectrum intensity of the fundamental frequency of the extracted feature amount is also considerably small as indicated by 1081 in FIG. As a result of the recognition, as shown in (g) of FIG. 35, “a leisurely walk” (“a bumpy walk”) is expressed by animation. This is expressed by overlapping the feature amount of “walking” with the recognition accuracy of recognizing the feature amount of “tobo-to-bo”. FIG. 36 is an example of recognition in the case of “falling down”. The point of the fall is indicated by a dotted line 1082. At the boundary of the dotted line 1082, the observed waveform sharply decreases to 0.0G, and the average value of the acceleration also gradually changes to 0.0G. This is because the state changes from the standing position to the lying position, the measurement axis of the acceleration sensor is orthogonal to the direction of gravity, and the gravitational acceleration cannot be observed. The recognition result displays the recumbent state as an animation as shown in FIG.

In the present embodiment, when the status of rehabilitation training is used for recognition (diagnosis) and its display, for example, the recognized result is not displayed as an animation as it is, but, for example, an action of dragging a foot is recognized. In this case, the recognition accuracy (weight) is emphasized and an animation is displayed, so that the action of dragging the foot can be emphasized and displayed. In this case, the defective part is emphasized and is easily understood. Of course, in any motion / action recognition other than rehabilitation, the emphasis processing may be performed to make the recognition result easier to understand.

As described above, according to this embodiment, it is possible to intuitively express a recognition result of an action or an action as an easily understandable animation, and it is possible to present the recognition result in an easy-to-understand manner to an observer.

Although the description has been given of the case where the multi-joint structure is displayed as an animation on the display device, a still image pose representing a feature of the operation may be used. For example, in the case of "walking", a walking person is photographed from the side, and the still image is used as a feature amount (feature image) in the case of "walking". When the recognition result is determined to be "walking", the image is read out. indicate.
In this case, since the amount of calculation is smaller than that of the animation display, there is an effect that the display can be performed even on a computer having a slow processing speed.

Next, a description will be given of an embodiment for displaying the result of recognizing the position and the motion or action of the subject. FIG. 37 is a structural diagram for displaying the result of recognizing the position and movement or action of the subject. Reference numeral 1100 denotes a motion / action recognition device. This recognizes the motion or action of a person, for example, by the method described with reference to FIG. 1
Reference numeral 101 denotes an expression processing unit for the motion or action of the multi-joint structure. This has a function of displaying an animation of the operation or action of the articulated structure based on the recognition result of the operation or action of the articulated structure recognized in 1100 by the method described with reference to FIG. 31, for example. 1102
Is a device for measuring the position of the subject. In this case, the position of the subject is specified by the method described with reference to FIG. A processing unit 1103 displays geographic information or information in a building. The geographical information referred to here is altitude data of mountains, road and track data, the position and shape data of buildings and structures, and the in-building information is information such as floor plans of buildings and arrangement of equipment. . These pieces of information are stored in the database 1104. Reference numeral 1105 denotes a display device. The geography or the information in the building around the position of the subject measured by 1102 is expressed in two dimensions or three dimensions by a method of displaying the geographic information or the information in the building of 1103,
An animation of the articulated structure is displayed at the position of the observer in the motion or action of the articulated structure 1101 at the position of the observer.

FIG. 38 is an example of a display screen displayed based on the result of recognizing the position and the movement / action of the subject.
This is an example in which the position and behavior of a patient or the like in a hospital are displayed.
In this example, the layout information inside the building is drawn,
If the subject goes outside, it can also represent outdoor roads and parks. Reference numeral 1105 denotes a display screen. Dining room (1106), lounge (1107), private room (1108)
Is created based on the geographical information or the in-building information database (1104) in FIG. Inside humans 1110, 1111, 1112, 1114, 111
5, 1109 is a measuring device (110
2) and a motion / behavior recognition device (1100), and the subject's position measuring device (1102 of 37) with reference to the strength of radio waves transmitted from the PHS base station of 1116 or 1113 The position is measured by Further, the motion or action is recognized by the motion / action recognition device (1100 of 37), and an animation is displayed by the motion / action expression unit of the articulated structure. In this example, the subject 1109 is walking fine in front of the private room 1, and the subject 1114 and the subject 1115 are sitting in the dining room.

As described above, according to this embodiment, it is possible to express in a very easy-to-understand manner where and what kind of motion or action the subject is performing, and to present the recognition result in an easy-to-understand manner to the observer. There is.

Further, according to the present embodiment, there is an effect that the influence of a blind spot or the like, which tends to occur in observation by a television camera or the like, hardly occurs, and it is possible to observe well in any place.

Further, according to the present embodiment, in the television camera, when the distance is large, the subject becomes small and it is difficult to observe the subject. However, if the present embodiment is used, any subject can be used as long as the observation result data can be transmitted. There is an effect that observation can be performed well in a place.

Further, according to the present embodiment, according to the observation by the television camera, by monitoring the image, it becomes clear that "somebody was there", and there is a possibility of privacy infringement. However, for example, FIG.
In the example of the subjects 1114 and 1115, the shapes of the animated characters of 1114 and 1115 and the other subjects are the same, and the motion or action is animated except for the individual habit of walking, so that the dining room can be displayed. It can be seen that two persons are sitting at the positions 1114 and 1115, but there is also an effect that it is not possible to know other personal information and protect privacy.

FIG. 3 shows a display example of an automatic report when an abnormal situation occurs.
9 will be described. An abnormal situation is when a patient "falls"
And "suffering". Subject 1112
However, it is in a state of falling down in front of the private room 4. In the above-described embodiment, a display method for hiding personal information from the viewpoint of privacy protection has been described. However, this example is an example in which personal information is positively disclosed when an abnormal situation occurs. The abnormal situation is recognized using the method described with reference to FIG. In the description of FIG. 20, a description has been given of recognition of a specific operation / action. However, the present embodiment exemplifies a hospital, and the specific operation / action refers to an abnormal situation of a patient. The method of FIG. 20 has already been described, and will not be described. When an abnormal situation occurs, the position and the motion or the behavior of the subject are displayed as an animation like 1112, and the name of the individual, the medical history, the physician, Displays the guardian and the like.

According to this embodiment, when an abnormal situation occurs, the observer can immediately grasp the situation and personal information of the person, and can observe the privacy of another person. While protecting,
There is an effect that personal information for the countermeasure can be immediately presented.

FIG. 40 shows an example of displaying the position, operation or action of the hospital staff. In the above embodiment,
When an abnormal situation occurs, it is necessary to select staff who can respond. Therefore, this is an example in which the position, operation, or behavior of the hospital staff is displayed. The hospital staff is 1118, and a hat indicating the hospital staff is attached to the animated character. In this example, staff 1118
Can be said to be walking and fully responsive.For example, if the result of motion recognition indicates surgery or the response of other patients, this screen can be viewed without directly checking each staff member. Meanwhile, it becomes possible to find other nearby staff.

According to the present embodiment, a person who can respond is
This has the effect that it is possible to judge without directly confirming.

In the above embodiment, the judgment is made by the observer. However, the position of the patient, the cause of the abnormal situation, the medical history, the position of the attending physician, and the position and work of the staff in the vicinity are described. It is also possible for the computer to automatically judge by various optimization algorithms on the basis of the state, specialty, etc., and to select the optimal staff.

FIG. 41 is a block diagram of a system for displaying a process until an abnormal situation occurs when an abnormal situation occurs. The method of FIG. 20 is used to recognize an abnormal situation. The configuration diagram of FIG. 41 is a configuration diagram for performing processing after recognition of an abnormal situation.

The storage device (1120) for recording the position / action / action of the subject has an action / action recognition device (110).
0), the type of action / action and position information measured by the measuring device (1102) for the position of the subject,
Further, the time at which recognition was performed is recorded. FIG. 42 shows an example in which information on the position / recognized action / action of the subject is stored in the storage device. The upper row (1122) indicates the item of the action / action, and the lower row (1123) indicates the time (t) and the position coordinates (x, y, z). These values are added as needed each time they are recognized. In this example, the last recognized stored position is 1125, and becomes older as it goes to the upper left.

Here, for example, it is assumed that the specific pattern of the situation is “fall down”, “lie down”, “lie down”. When this pattern appears, the abnormal situation recognition device (1
125) An expression / processing unit (1126) for displaying the operation / action of the multi-joint structure including movement for displaying the history of the abnormal situation.
To perform animation processing.

FIG. 43 shows a display example for displaying the history of occurrence of the abnormal situation, which is displayed using 1126. The point where the observer fell was 1112. Abnormal situation recognition device (112
When the specific operation pattern 1121 in FIG. 42 is recognized, the storage device (1) that records the position / operation / action of the subject
120), the animation is sequentially started toward a new time based on the positional information at that time (in this example, the state of “walking well” of 1124) and the movement / action of the subject. Therefore, in this example, in the section of 1130, "walk well", in the section 1131, "walk",
In section 1132, “stop”, section 1133
In this case, an animation can be generated in which the user “falls down” and the section 1134 “turns down” and does not move. By repeating the above process, it is also possible to repeatedly display the circumstances that led to the abnormal situation.

According to the present embodiment, the history of the abnormal situation can be displayed retrospectively, so that it is necessary to understand the abnormal situation in detail, which can be determined from the history of the abnormal situation, and to examine a method for dealing with the abnormal situation. This has the effect of making it easier to do.

Next, an embodiment of a system configuration for improving the accuracy of the measurement value obtained by the position measurement device based on the recognition result of the action / action will be described with reference to FIGS. 44 and 45. FIG. 44 is a system configuration diagram for improving the position measurement accuracy based on the recognition result of the action / action, and FIG.
FIG. 5 is an explanatory diagram for improving position measurement accuracy based on a recognition result of an action / action.

The description of the operation / action recognition apparatus 1100 is omitted. Subject position measuring device 1102
In, the distribution of the electric field strength of radio waves emitted from a plurality of base stations was obtained in advance by measurement or simulation, the radio waves of the plurality of base stations were received by the terminal of the subject, and the respective electric field strengths were measured and measured. The electric field intensity and the electric field intensity distribution obtained in advance are calculated as candidates for the comparison position. For example, in the example of FIG. 45, the terminal of the subject is the base station A (1
150) is received at an electric field strength of X dB, and the base station B (11
It is assumed that the radio wave of 51) is received in YdB. Base station A (115
The electric field intensity distribution of 0) and the electric field intensity distribution 1153 of the base station B (1151) are as shown by 1152 and 1153. Therefore, the portion satisfying the condition of the electric field strength from both stations is 11
There are 54 areas, which means that the subject exists somewhere in this area. However, due to the measurement accuracy of the electric field strength on the terminal side, the irregular reflection of radio waves, and the like, this region can be specified only at a relatively large and rough position. Accordingly, a method 1 for estimating a position from geographical information or information in a building
At 140, a detailed location is specified from the subject's presence area obtained at 1102, based on the subject's action / action obtained by the action / action recognition device 1100 and geographic information or building information 1104. For example, in the example of FIG.
If it is recognized that the operation of “climbing the stairs” is recognized, the subject 1156 searches for the place with the stairs from the geographic information or the in-building information 1104, and the subject 1156 finds the place with the stairs in the area 1154 (the place with the 1155). Can be specified. In addition, when the subject is in a place that is not affected by the terrain, such as 1157, it is apparent that the subject is not at least on the stairs, and the position is estimated from the time of finishing climbing the stairs and the time of performing the walking motion. You can do it.

This will be described in more detail with reference to FIG. FIG. 46 is an explanatory diagram illustrating a method of correcting a position based on a result of specifying a position based on a recognition result of an action and an action. In this figure, the direction is indicated by 1165. In this example, a device that measures an approximate absolute position by a radio wave emitted from a base station A (1150) of a cellular system, a device that measures the number of steps attached to a subject, and a device that measures a change in azimuth and traveling direction are used. Further, the position of the subject is measured by using the device for measuring the position and the device for specifying the position based on the recognition result of the operation and the action described above. As described above, the measurement accuracy is not very high in the cellular system. Therefore, if the condition of the electric field strength is within the same area, it cannot be measured even if the subject moves. Therefore, means for measuring the relative position is required. The device for measuring the relative position counts the number of steps, measures the moving distance from the average step length of the subject, and measures the moving direction of the subject using a direction sensor or a gyro attached to the subject. By integrating the measured moving distance and moving direction, the relative moving position of the subject can be measured. Thus, the movement of the subject can be measured even when the condition of the electric field strength is within the same area. The absolute position of the subject is obtained by integrating the relative value measured by a measuring device that measures the relative position with the absolute position measured by the cellular system as an initial value. However, the measurement accuracy of the measuring device that determines the initial position of the subject and measures the absolute position is poor. Therefore, the first error will affect later. Furthermore, in a device for measuring a relative position,
Errors accumulate due to sampling intervals, measurement errors, and integration errors. Therefore, the position of the subject is corrected using a device that specifies the position based on the recognition result of the action and the action.

In the example of FIG. 46, if the recognition result of the action and the action is recognized as “finished up the stairs”, the subject can be specified as the position of 1160 which is the upper end of the stairs. This is the measurement result 115 measured by the cellular system.
The position accuracy is much higher than 4. Using this as the initial value of the absolute position, the position of the subject is tracked by a measuring device that measures the relative position.

[0303] From the initial value 1160 of the absolute position, it is measured that the subject is traveling in the south direction by the subject's azimuth sensor or gyro. Further, the distance is calculated by the measuring device for the number of steps, and the movement locus of 1163 is obtained. In section 1165, the subject is turning west. This is measured by a direction sensor or a gyro, and the movement locus of the section 1165 is calculated from the number of steps. Thereafter, the trajectory is calculated in the same manner, and the current position of the subject 1161 is calculated.
Can be requested.

In the above embodiment, a cellular system is described as an example of the means for measuring the absolute position. However, other systems for measuring the absolute position, for example, G
The same applies to PS and the like.

In the above-described embodiment, the distance is calculated from the standard step length. However, for example, “relaxed walking” or “running” is recognized based on the recognition result of the motion and the action, and the recognition item and the recognition accuracy are used. It is also possible to calculate the step length in the motion and the action state and obtain the distance from the number of steps. in this case,
The accuracy is further improved than the distance obtained with the standard stride.

In the above embodiment, the distance is calculated from the standard stride length. However, the acceleration or speed in the traveling direction of the subject is measured by a sensor. By doing so, the distance may be calculated.

In the above embodiment, the position obtained by the device combining the measuring device for measuring the absolute position and the device for recognizing the operation and the behavior is set as the initial position, but the position is measured during the measurement by the relative position measuring device. When the subject performs an action or action that can specify the position, the position may be specified again with reference to the geographic information or the information in the building based on the action or action, and the value may be used as the initial value. in this case,
Accumulation of errors due to sampling intervals, measurement errors, and integration errors can be eliminated. This is particularly effective when measuring outside the service area of the cellular system.
Since there is no means for measuring the absolute position, the position is measured only by means for measuring the relative position, and the accumulation of errors becomes a problem.However, it is possible to eliminate the accumulation of errors by sequentially correcting using this method. it can.

In the present embodiment, a method of measuring a position using a cellular system having a plurality of base stations has been described. However, the present invention is to improve the position accuracy by combining with a GPS or other measuring device that performs positioning using a satellite. Is also possible.

In the drawing method, the position of the subject obtained by the present embodiment can be placed on the screen, and the drawing range can be drawn at any time in synchronization with the position of the subject as the subject moves.

Further, a system for navigating a subject based on the position of the subject determined according to the present embodiment is also possible by the same method.

According to the present embodiment, there is an effect that the measurement position measured by the conventional position measurement device can be corrected more accurately.

The movement or behavior recognition apparatus according to the above embodiment can be used for nursing care of an elderly person in a hospital, a nursing home, or at home by having the measuring device 2 worn by a sick person or an elderly person. In addition, if the baby wears the measuring device 2 and sounds an alarm at a specific action,
It can also be useful for monitoring infants at home.

[0313]

According to the present invention, instead of measuring the state change of a subject or a test object and using the measurement result as a mere measurement value, recognition processing is performed using the characteristic amount of the measurement value of the state change. Since the operation is performed automatically, the movement or action of the subject can be more correctly recognized.

[Brief description of the drawings]

FIG. 1 is a block diagram showing the configuration of an embodiment of a motion and behavior recognition device according to the present invention.

FIG. 2 is an explanatory diagram showing an output example from an acceleration sensor attached to the waist.

FIG. 3 is an explanatory diagram showing an example of a time-frequency analysis result using a wavelet transform.

FIG. 4 is a block diagram showing the configuration of another embodiment of the recognition device according to the present invention.

5A is a graph showing an example of an observed waveform in the embodiment of FIG. 4, FIG. 5B is a graph showing a spectrum analysis result of the data of FIG. 5A, and FIG. 7 is a graph showing a spectrum analysis result after processing.

FIG. 6 is a block diagram showing the configuration of another embodiment of the recognition device according to the present invention.

FIG. 7 is a block diagram showing a main configuration of another embodiment of the recognition device according to the present invention.

FIG. 8 is an explanatory diagram showing an example of measurement points and measurement axes of a state change.

FIG. 9 is a block diagram showing the configuration of an embodiment of a recognition system to which the present invention has been applied.

FIG. 10 is a block diagram showing the configuration of an embodiment of an operation and action recognition device combined with a position measurement device.

FIG. 11 is an explanatory view showing a position measuring method in the embodiment of FIG. 10;

FIG. 12 is a block diagram showing a configuration of an embodiment of a system for estimating and outputting a work situation or an environment where a subject is placed from a work history.

FIG. 13 is an explanatory view showing a history of actions or actions stored in the embodiment of FIG. 12;

FIG. 14 is an explanatory diagram showing an example of an associative pattern in the embodiment of FIG. 12;

FIG. 15 is a block diagram illustrating a configuration of an embodiment of a recognition device that displays a difference between a standard operation and a measurement operation.

16A is an explanatory diagram illustrating a calculation example of a difference between the measurement operation and the standard operation, and FIG. 16B is an explanatory diagram illustrating another calculation example of the difference between the measurement operation and the standard operation.

FIG. 17 is a block diagram illustrating a configuration of an embodiment of a recognition device that outputs a cause of a difference based on a difference between a standard operation and a measurement operation.

FIG. 18 is a block diagram showing a configuration of an embodiment of a system in which a recognition device according to the present invention is used for collecting work data of a workability evaluation system or a work content evaluation system.

FIG. 19 is a block diagram showing a configuration of an embodiment of a system in which the recognition device according to the present invention is used for collecting work data of a proper staff distribution system or a proper work time distribution system.

FIG. 20 is a block diagram showing the configuration of an embodiment of a recognition device that notifies a user when a specific operation or action is performed.

FIG. 21 is an explanatory diagram showing the configuration of an embodiment of a system in which the recognition device according to the present invention is applied to behavior analysis, behavior tracking, and behavior monitoring.

FIG. 22 is a block diagram showing a configuration of an embodiment of a gesture recognition device to which the present invention has been applied.

FIG. 23 is a block diagram showing a configuration of an embodiment in which a wireless device is added to the recognition device according to the present invention.

FIG. 24 is a block diagram showing a configuration of another embodiment in which a wireless device is added to the recognition device according to the present invention.

FIG. 25 is a block diagram showing a configuration example of a system in which a prepaid function is added to the recognition device according to the present invention.

FIG. 26 is a block diagram showing a configuration example of a system in which a credit function is added to the recognition device according to the present invention.

FIG. 27 is a block diagram showing a configuration example of a system in which a deposit function is added to the recognition device according to the present invention.

FIG. 28 is a block diagram showing a configuration example of a system in which a recognition device according to the present invention is provided with a function of confirming entry or exit from an area.

FIG. 29 is an explanatory diagram showing an example of a method for obtaining a correlation from a feature amount.

FIG. 30 is a configuration diagram of a system that displays a result recognized by the motion and action recognition device by using a method of expressing the motion and action of the articulated structure.

FIG. 31 is an explanatory diagram of a method of expressing the motion and behavior of the articulated structure.

FIG. 32 is an explanatory diagram of a method of expressing an action and an action when the action and the action are recognized as a superposition of a plurality of feature amounts.

FIG. 33 is a diagram showing an example of a display screen of a recognition result.

FIG. 34 is a diagram showing an example of a display screen of a recognition result.

FIG. 35 is a diagram showing an example of a display screen of a recognition result.

FIG. 36 is a diagram showing an example of a display screen of a recognition result.

FIG. 37 is a configuration diagram of a system for displaying a result obtained by recognizing a position and a motion or a behavior of a subject.

FIG. 38 is a diagram showing an example in which a display is made based on the position and motion of a subject or a result of recognition.

FIG. 39 is a diagram showing a display example of an automatic notification when an abnormal situation occurs.

FIG. 40 is a diagram showing a display example of staff arrangement.

FIG. 41 is a configuration diagram of a system for displaying a process until an abnormal situation occurs when an abnormal situation occurs.

FIG. 42 is a diagram showing an example of data stored in a storage device for storing the position of the subject and the recognized actions and actions.

FIG. 43 is a view showing a display example for displaying a history of occurrence of an abnormal situation.

FIG. 44 is a configuration diagram of a system that improves the position measurement accuracy based on the recognition results of actions and actions.

FIG. 45 is an explanatory diagram for improving the position measurement accuracy based on the recognition result of the action and the action.

FIG. 46 is an explanatory diagram illustrating a method of correcting a position based on a result of specifying a position based on a recognition result of an action and an action.

FIG. 47 is a view showing a spectrum intensity of an acceleration frequency of an action or an action.

FIG. 48 is a view showing a measured waveform of an acceleration of an action or action.

FIG. 49 is a diagram showing a relationship between spectrum intensity and recognition accuracy.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Subject, 2 ... Measuring device of state change (for waist), 3 ... Measuring device of state change (for arm), 4 ... A / D converter, 5 ... Signal feature extraction part, 6 ... Operation or behavior 7, a signal processing unit for recognizing an action or action; 8, an output unit for outputting a recognition result; 41, a position measuring unit of a subject; 62,
Storage device for storing the history of the operation or action; 63, a storage device for the association pattern; 64, an association processing device; 126, a workability evaluation system;
34: proper distribution system for personnel, 141: distribution system for proper work time, 146: reporting device to parents, 147
… Report receiving device, 162… gesture recognition device.

──────────────────────────────────────────────────の Continued on the front page (51) Int.Cl. ⁶ Identification symbol FI // G06T 15/70 G06F 15/62 340K (72) Inventor Satoshi Yashiki 4-6-1 Kanda Surugadai, Chiyoda-ku, Tokyo Hitachi, Ltd. Within the Systems Division (72) Inventor Shiro Nonaka 7-1-1, Omika-cho, Hitachi City, Ibaraki Prefecture Within Hitachi Research Laboratory, Hitachi, Ltd. (72) Inventor Shigeru Obo 7-1-1, Omika-cho, Hitachi City, Ibaraki Prefecture (72) Inventor Katsuo Tashiro 5-2-1 Omikacho, Hitachi City, Ibaraki Pref. Hitachi, Ltd. Omika Plant

Claims (38)

[Claims] 1. A recognition device for recognizing an operation or behavior of a subject, comprising: a measuring unit attached to the subject for observing a state change accompanying the operation or behavior of the subject; A feature amount extraction unit to be extracted; a storage unit that stores a feature amount of an action or an action to be recognized by the recognition device; and the feature amount extracted from the observation result and the stored feature amount, A recognition device comprising: recognition means for recognizing the operation or action of the above; and output means for outputting the recognition result. 2. The method according to claim 1, wherein the feature value extracting unit extracts, from the time series data of the observation result, one or more frequency components included in the data as the feature value. Recognition device. 3. The recognition apparatus according to claim 1, wherein the feature amount extracting unit extracts the feature amount by performing time-frequency conversion on the observation result. 4. The apparatus according to claim 3, wherein one of a Fourier transform and a wavelet transform is used for the time-frequency transform. 5. The operation time measuring means for obtaining an operation time characterizing a temporal characteristic of the operation or action from the observation result, and the observation result is displayed on the time axis by the obtained operation time. A recognition unit for extracting the feature amount by performing time-frequency conversion on the normalized observation result. 6. The operation time measuring means for obtaining an operation time characterizing a temporal characteristic of the action or action from the observation result, and the feature quantity extracted by the feature quantity extraction means is used as the operation time. And using the corrected result as a new feature amount. The recognition unit further includes a feature amount correction unit configured to obtain the new feature amount from the corrected feature amount and the stored feature amount. An action recognition device for recognizing an action or action. 7. The motion recognition apparatus according to claim 5, wherein the output unit outputs the operation time measured for the recognized motion or action, in addition to the recognition result. 8. The method according to claim 3, further comprising an expansion / contraction unit that expands / contracts the observation result along a time axis, wherein the characteristic amount extraction unit extracts a characteristic amount from the observation result expanded / contracted by the expansion / contraction unit. A recognition device characterized in that: 9. The method according to claim 3, further comprising an expansion / contraction means for expanding / contracting the observation result along the measurement intensity axis, wherein the feature amount extraction means extracts a characteristic amount from the observation result expanded / contracted by the expansion / contraction means. A recognition device characterized by extracting. 10. The recognition apparatus according to claim 8, wherein said output means outputs a value relating to an expansion / contraction ratio used in the processing of said expansion / contraction means. 11. The feature amount synthesizing unit according to claim 1, further comprising: selecting a plurality of feature amounts stored in the storage device, and superimposing the plurality of feature amounts to generate a new feature amount. Has, the recognition means, from the generated feature amount and the feature amount corresponding to the observation result, the operation or behavior of the subject,
A recognition apparatus characterized in that a plurality of actions or actions corresponding to the selected feature amount are recognized as an action or action obtained by combining them. 12. The recognition apparatus according to claim 11, wherein said feature amount combining means weights each feature amount when overlapping said plurality of selected feature amounts. 13. The recognition apparatus according to claim 12, wherein the output unit outputs a weight given to each of the feature amounts superimposed by the feature amount synthesizing unit. 14. The apparatus according to claim 1, wherein the recognizing means obtains a correlation between a feature amount corresponding to the observation result and a feature amount stored in the storage device when recognizing an action or an action. A recognition device that performs recognition in accordance with a calculated correlation height. 15. The recognition apparatus according to claim 1, wherein said recognition means is constituted by a neural network for recognizing said action or action. 16. The recognition apparatus according to claim 1, further comprising a position measuring means for measuring a position of the subject. 17. The state change observed by the measurement means according to claim 1, wherein the state change in one or more parts constituting the subject and a state change in the structure of each part other than the position change. A recognition device characterized by at least one of the following. 18. The method according to claim 1, wherein the state change observed by the measuring means is acceleration, angular acceleration, velocity, angular velocity, position, rotation at one or more points representing the motion or action of the subject. ,
And the bending angle, and the acceleration, angular acceleration, velocity, angular velocity, position, rotation, and bending angle of one or more parts constituting the subject, and the acceleration of one or more joints constituting the subject. , Angular acceleration, velocity, angular velocity, position, rotation, and bending angle, and pulse, blood pressure, body temperature, blood glucose, respiration, myoelectricity, electrocardiogram measured at one or more locations as the subject's biological information , A blood flow, and an electroencephalogram. 19. A recognition system for recognizing a behavior, a task, and a situation in which the subject is placed, the recognition system recognizing the motion or the behavior of the subject. A device, a first storage device for sequentially accepting the actions or actions recognized by the recognition device and storing the history thereof, an action to be recognized, a task and a situation, and a time-series pattern of the action or action corresponding to each. A storage device that stores the following: an estimating device for estimating the behavior, work, and situation of the subject from the history of the recognized operation or behavior and the stored time-series pattern. And an output device for outputting the estimation result. 20. A recognition device for detecting a difference between an operation or behavior of a subject and a standard operation or behavior corresponding thereto, wherein the recognition device is attached to the subject and observes a state change accompanying the operation or behavior of the subject. Measuring means; feature quantity extracting means for extracting feature quantities in the observation result; storage means for storing feature quantities for predetermined standard actions or actions; and storing the feature quantities extracted from the observation results. A difference detecting means for detecting a difference between the motion or action of the subject and a standard motion or action corresponding to the motion or action of the subject, and an output means for outputting the detection result. apparatus. 21. The method according to claim 20, wherein a second feature value extracting means for extracting a feature value in the difference detected by the difference detecting means, and A second storage unit that stores a feature amount of the difference to be recognized and a cause of the difference corresponding to the feature amount; a feature amount of the difference detected by the second feature amount extraction unit; The apparatus further comprises: cause identification means for identifying the cause of the difference based on the feature amount and the cause of occurrence stored in the second storage means. A recognition apparatus for outputting at least a cause of occurrence of a difference from an action or an action. 22. A recognition system for recognizing a work progressing by an action or action of at least one worker, comprising: a recognition device for recognizing the action or action of each worker; A recognition system, comprising: an evaluation device that uses a movement or action of each worker as input data and evaluates a predetermined evaluation item for the work based on the input data. 23. The apparatus according to claim 22, further comprising an instruction receiving device to be carried by each worker, wherein the evaluation device sends the instruction receiving device carried by the worker according to the evaluation result. A recognition system, comprising: an instruction transmission unit for transmitting a predetermined instruction. 24. The evaluation device according to claim 23, wherein the evaluation device further includes a work plan correction unit that corrects the work plan based on the evaluation result. A recognition system, wherein an instruction for each worker to perform a corresponding action or action is transmitted to an instruction receiving device of the worker. 25. A recognition system for monitoring the operation or behavior of a subject, comprising a subject-side device and a monitor-side device, wherein the subject-side device recognizes the operation or behavior of the subject. A recognition unit, a storage unit that stores reference data about a specific operation or action to be reported, and whether the operation or behavior of the subject is to be reported from the recognized operation or behavior and the reference data. A monitoring unit that transmits a report according to the determination result, wherein the monitor-side device receives a report transmitted from the reporting unit, and responds to the report. A recognition unit comprising: an output unit configured to perform an output. 26. The apparatus according to claim 25, wherein the subject-side device further comprises a history storage unit for storing a history of the recognized operation or action, wherein the storage unit is configured by a specific operation or action combination. The reference data is stored in advance for a specific behavior pattern to be reported, the determination unit, from the history and the reference data stored in the history storage unit, the combination of the operation or behavior of the subject A recognition system for determining whether a pattern is to be reported. 27. The object-side device according to claim 26, wherein the detection time is within a predetermined range even when the determination unit determines that the combination of the action or action to be reported is detected. In
A recognition system further comprising a report control unit that prohibits a report from the report unit. 28. The object-side device according to claim 26, wherein the subject-side device further includes a position measuring unit that measures a position of the subject, and the notification control unit is configured to determine the operation or behavior to be reported by the determination unit. Even if it is determined that a combination has been detected,
A condition that the time at which the detection is performed is within a predetermined first range, and a condition that the position of the subject at the time of the detection is within a predetermined second range Wherein at least one of the conditions is satisfied, the report from the notification unit is prohibited. 29. A recognition system for recognizing a motion or behavior of a subject, a recognition device for recognizing the motion or behavior of the subject, a position measuring device for measuring a position of the subject, and the recognition result. And a display device for displaying information indicating a locus of movement of the subject and an action or action performed in the course of the movement from the position measurement result. 30. A recognition device for recognizing a gesture performed by an operation of a subject, wherein the first storage means stores reference data predetermined for a gesture to be recognized; Recognizing means for recognizing a gesture corresponding to an action performed by the body; second storing means for preliminarily storing the correspondence between the gesture to be recognized and the meaning indicated by the gesture; and the stored correspondence And a means for specifying the meaning of the recognized gesture, and an output means for outputting the meaning of the specified gesture. 31. A measuring unit for measuring a change in state of a subject,
A recognition device that recognizes the operation or behavior of the subject based on the measurement data, and a recognition device including an output unit that outputs a recognition result, a subject-side portion attached to the subject, and the subject Is composed of two parts: a measurer-side part disposed at a distant place, and the subject-side part includes at least the measuring unit and a transmitting unit that transmits data by wireless communication. The observer-side part includes at least a receiving unit that receives the data and the output unit, and the data transmitted by radio is the same as the observer-side part when the recognition unit is included. A recognition device, which is measurement data of a measurement unit, and is data indicating a recognition result when the recognition unit is included in the subject side portion. 32. A subject-side device having at least a recognition unit for recognizing the motion or action of the subject, an observer having at least a receiving unit for receiving a recognition result by the recognition unit, and a device operating in accordance with the received recognition result. A recognition system, comprising: a storage unit that stores a charge balance; and a calculation unit that updates a charge balance stored in the storage unit in response to an input instruction. The recognition system further comprising: a control unit configured to output an instruction to cause the calculation unit to change a charge balance according to the recognition result received by the reception unit. 33. A subject-side device having at least a recognition unit for recognizing a motion or action of a subject, an observer having at least a receiving unit for receiving a recognition result by the recognition unit, and a device operating in accordance with the received recognition result. A recognition system including a device, wherein the subject-side device further includes a payment command unit that outputs a charge payment instruction according to a recognition result of the recognition unit, and the observer-side device includes the payment command. And a credit requesting unit for receiving a credit request for the payment amount for a credit account owned by the subject registered in advance. 34. A subject-side device having at least a recognizing unit for recognizing an action or behavior of a subject, an accepting unit for receiving a recognition result by the recognizing unit, and an observer side having at least a device operating in accordance with the received recognition result. A recognition system comprising a device, wherein the subject-side device further includes a return command unit that commands a return of a deposit deposited by the subject in advance, and the observer-side device includes the return command unit. A recognition system further comprising: a payment unit that outputs a command for causing the deposit to be paid after receiving a command from the company. 35. A subject side device including at least a recognizing unit for recognizing a motion or action of a subject, an observer side including at least a receiving unit for receiving a recognition result by the recognizing unit and a device that operates in accordance with the received recognition result. A recognition system comprising a device, wherein the subject-side device further has a response unit that generates a predetermined response signal, wherein the observer-side device has an entrance to an area where entry or exit is restricted. A recognition system further comprising an intrusion confirmation unit, which is disposed in the area and confirms intrusion or egress into the area by receiving the response signal. 36. A recognition method for recognizing a motion or a behavior of a subject, wherein a measurement unit attached to the subject observes a state change accompanying the motion or the behavior of the subject, and outputs data of the measurement unit. Extracting the feature amount included in the object, and recognizing the motion or action of the subject from the extracted feature amount and reference data predetermined for the action or action to be recognized. Method. 37. A recognition device for recognizing a motion or action of a person, comprising: an acceleration sensor attached to the person for measuring a change in acceleration accompanying the motion or action of the person; and a time series of the acceleration measured by the acceleration sensor. One or more frequency components included in the data are extracted from the data, and recognition means for recognizing a motion or action of a person who has attached the acceleration sensor based on the extracted frequency components, and outputs the recognition result. A recognizing device comprising: 38. A recognizing device for recognizing a motion or a behavior of a subject, comprising: a measuring unit attached to the subject for observing a change in the acceleration of the subject accompanying the motion or the behavior of the subject; A feature amount extracting unit that extracts a feature amount of acceleration specific to the motion or action of the subject from a result of the acceleration; and a processing unit that generates a motion of the computer graphics model of the subject based on the extracted feature amount. And a display processing unit for displaying an animation of the generated motion.