JP2005250708A - Tool motion recognition device and tool motion recognition method - Google Patents

Abstract

An object of the present invention is to recognize which tool is used from an operation scene in which a person operates various tools by hand under a situation where occlusion and discontinuity exist.
An image input unit 11 inputs a frame of an operation scene in which a person operates various tools, and an image storage unit 12 stores the input frame. The speed estimation unit 13 estimates a speed vector from the operation scene, and the symbol time series generation unit 14 generates a symbol time series from the estimated speed vector using the symbol conversion table 141. The learning unit 15 calculates and learns the HMM model parameters using the generated symbol time series. The recognition unit 16 identifies which tool operation is being performed using the learning result of the learning unit 15 based on the speed vector estimated by the speed estimation unit 13 for the newly input motion scene, 17 outputs the identification result by the recognition unit 16.
[Selection] Figure 1

Description

  The present invention relates to tool motion recognition technology, and more particularly to pattern learning / processing of a tool operated by a person without using shape knowledge about the tool in a time-series scene input from a WEB camera or a general camera. The present invention relates to a tool motion recognition device and a tool motion recognition method.

  Research is being actively conducted to recognize human gestures and movements from imaged camera scenes obtained from WEB and various observation sources. In addition, the construction of surveillance systems and networks at remote locations using cameras and multipoints is progressing, and technology that can recognize even complex operations in complex environments is desired. However, there has been little research on natural motion recognition without constraints. In the example of conventional gesture research, it lacks naturalness because the command system corresponding to the motion is limited in advance.

  When using the knowledge of tools, for example, there are pencils and erasers around us, and there are various actions such as writing and erasing, and the number of combinations becomes enormous due to the variety of tools themselves and the way they are operated. When a human being recognizes the movement of a person who is far away, it is unlikely that all combinations are learned to recognize the movement of a far person. In addition, there are many types of basic movements, such as cutting (material), sweeping (room), wiping (desk), etc., depending on the object and tool, but we humans, by occlusion of hands and arms, Even if the tool itself is not completely visible, it can be easily recognized.

  However, for example, recognition from the robot's eyes is not easy. This is because it is necessary for the robot to sufficiently learn occlusion, tools, and movements in order to identify movements under actual environment changes and lighting changes. In particular, when writing a character with a pen and when turning a screw with a screwdriver, the recognition becomes even more difficult. If such a recognition technology is established, the robot will be able to smoothly perform tasks such as supporting people from visual judgment.

  In addition, by automatically recognizing human actions in detail through remote monitoring, a wide range of applications such as unnatural movement recognition and crime prediction using tools can be considered. Because of this background, recognition of gestures such as human gestures and hand gestures and motion recognition using tools have been actively studied as important themes.

  Starner et al. Have proposed that 40 types of ASL hand gestures are recognized by HMM (Hidden Morkov Model) based on color glove or skin color feature (see Non-Patent Document 1).

  Fels et al. Used a data glove and learned by a multi-layer NN (Neural Network) based on hand movement information sensed therefrom (see Non-Patent Document 2).

  Siskind et al. Conducted an experiment on the assumption that human motion perception was not an object recognition but a visual trajectory separate from the object (see Non-Patent Document 3). In the experiment, a color model and an HMM (Hidden Markov Model) were used to analyze six movements of pick up, put down, push, pull, drop, and throw.

  Bobick et al. Recognized DTW (Dynamic Time Warping) based on the main curvature of the hand trajectory and the hand's unique image using a data glove with a magnetic sensor (see Non-Patent Document 4). They extended to learning and recognition of trajectories by HMM (see Non-Patent Document 5).

  Lee et al. Applied a method of learning and recognizing 10 types of gestures using HMM using hand skin color to basic commands used in presentations such as start, first, and next (see Non-Patent Document 6).

  Yang et al. Proposed a method for learning and recognizing TDNN (Time Delayed Neural Network) by generating hand trajectories extracted based on skin color and motion segmentation without using data gloves in order to improve versatility. (Refer nonpatent literature 7). Experiments with 40 types of ASL gestures have resulted in high recognition rates.

  In the method described above, there are problems that the operation recognition with the tool is difficult to apply due to the occlusion problem, and that the method based on the background difference cannot cope with the actual environment change and the illumination change.

  For the lighting change problem, Bobick et al. Proposed a method for recognizing motion by creating a cumulative image of difference images from moving images such as arm swings (see Non-Patent Document 8). It has been shown that it can be recognized at high speed when the posture changes greatly as in aerobics. However, in a detailed operation such as screwing and drilling within a certain range as will be described later in the present invention, an image with a high degree of similarity is obtained, and recognition is difficult.

  Yamato et al. Have proposed the application of HMM to identify four forms (volley, etc.) during tennis play as an example of motion recognition when holding a tool (see Non-Patent Document 9). Background differences are calculated for each input image, and learning is performed with 36 symbol patterns after silhouette image generation, binarization, and vector quantization. It is conceivable that the posture including the player's limbs and the position of the racket contributed to the recognition. The motion feature quantity is not used. Since this method generates silhouette images, there remains a problem that is easily affected by sunlight and environmental changes. Associating images with symbols, which have the greatest effect on the recognition rate, requires a lot of time for experience and representative pattern selection.

  Duric et al. Try to analyze the function of the tool as seen from the camera line of sight when the carpenter tool is operated by hand (see Non-Patent Document 10). In the method described in this Non-Patent Document 10, four types of carpenter tools (shovel, monkey, knife, spanner) held in the hand are picked up, and from the motion flow of the contour line, the pseudo three-dimensional optical flow from the monocular camera. Estimated. The motion parameters were obtained from the normal flow and the motion of each tool was analyzed. Even with the same tool, the parameters show different transitions in time series, such as tapping the monkey as a tonchi from the original use of the screwdriver and cutting the knife back and forth. In this Non-Patent Document 10, the functionality of each tool is clarified and classified, but the effects of hand and arm movements are not considered in the model. Also, we did not conduct any time series pattern recognition experiments.

In addition, about the technique regarding the optical flow relevant to embodiment of this invention, it describes in the following nonpatent literature 11-nonpatent literature 14.
TE.Starner, J. Weaver, and A. Pentland, "Rea1-time american signlanguage recognition using desk and wearable computer based video", IEEE Trans. PAMI, vo1.20, no.12, pp.1371-1375, 1998. SSFels and GE Hinton, "Glove-talk: a neural network interface which maps gestures to parallel format speech synthesizer controls", IEEE Trans.Neural Network, vol.9, no.1, pp.205-212, 1997. JMSiskind and Q. Morris, "A maximum-likelihood approach to visual event classification", Proc. Fourth European Conf. Computer Vision, pp. 347-360, 1996. AFBobick and AD Wilson, "A state-based approach to the representation and recognition of gesture", IEEE Trans. PAMI, vol. 19, no. 12, pp. 1325-1337, 1997. ADWilson and AF Bobick, "Parametric Hidden Markov Models for gesture recognition", IEEE Trans. PAMI, vol. 21, no. 9, pp. 884-900, 1999. HKLee and JHKim, "An HMM-based threshold model approach for gesture recognition", IEEE Trans. PAMI, vol.21, no.10, pp, 961-973, 1999. MHYang, N. Ahuja, and M. Tabb, "Extraction of 2D motion trajectories and its application to hand gesture recognition", IEEE Trans. PAMI, vol. 24, no. 8, pp. 1061-1074, 2002. AFBobick and JWDavis, "The Recognition of Human Movement Using Temporal Templates", IEEE Trans. PAMI, vol. 23, no. 3, pp. 257-267, 2001. J. Yamato, J. Ohya, and K. Ishii, "Recognizing human action in time-sequential images using Hidden Markov Mode1", Proc. Computer Vision and Pattern Recognition, pp. 379-385, 1992. Z. Duric, JAFayman, and E. Rivlin, "Function from motion", IEEE PAMI, vol. 18, no. 6, pp. 579-591, 1996. BDLucas and T. Kanade, "An iterative image registration technique with an application in stereo vision", IJCAI-81, pp.674-679. A. Bab-Hadiashar and D. Suter, "Robust optic flow computation, International Journal of Computer Vision, 29, 1, pp. 59-77, 1998. EPOng and M. Spann, "Robust optical flow computation based on least-median-of-squares regression", International Journal of Computer Vision, 31, 1, pp. 51-82, 1999. N. Cornelius and T. Kanade, "Adapting optical flow to measure object motion in reflectance and X-ray image sequences", ACM SIGGRAPH / SIGART Interdisciplinary Workshop on Motion; Representation and Perception, Toronto, Canada, 1983.

The conventional techniques described in Non-Patent Documents 1 to 10 have the following problems.
(1) Most of the researches are gestures that do not have tools in hand, and use of data gloves and color marking on fingers are unnatural.
(2) Sufficient research has not been done on the recognition when the driver is finely operated with the hand. In other words, the study of occlusion and discontinuity problems with tools using hands and arms is insufficient.
(3) In the HMM using an image pattern, the performance evaluation is not sufficiently performed for the number of input symbols, the number of output symbols, and the recognition rate.
(4) Little research has been done on the output symbol conversion method from the image pattern.
(5) In motion estimation from a moving image, an optical flow method resistant to noise and illumination fluctuation is not applied.

  The problems of the above prior art will be described more specifically. FIG. 11 shows an image when a person operates with a tool in his hand. FIG. 11A shows the operation of turning the driver, FIG. 11B shows the operation of turning the drill, FIG. 11C shows the operation of tapping the torch, and FIG. 11D shows the operation of pulling the saw.

  At first glance, it seems easy to identify the motion by each of the four tools, but the problem is complicated by the fact that there is similarity in shape and similarity in motion. First, regarding the shape of a tool, for example, even if the shape of each tool is stored in advance and an attempt is made to refer to it, the tool is concealed by hands or arms, so that identification by matching becomes difficult.

  In addition, when trying to understand from the movement, the operation of turning the driver hardly produces a clean periodic component even for the same operator, and varies, so it is difficult to cope with the movement. Furthermore, drills and drivers show similar movements centered on rotational movement. Tonkachi and saw have similarities such as a mixture of forward and backward movement and rotational movement around the wrist or cutting object. In addition, since each tool has a clear edge, the optical flow near the edge becomes discontinuous, which reduces the estimation accuracy.

  As described above, conventionally, there has not been established a technique for accurately estimating the fine movement (optical flow) of each operation in the presence of occlusion and discontinuity. In an actual environment scene, there are environmental changes such as noise and luminance fluctuations.

  The present invention solves the above-mentioned problems of the prior art, and it is possible to determine which tool is used from an operation scene in which a person operates various tools by hand under a situation where occlusion and discontinuity exist. It aims at providing the tool movement recognition technology which recognizes.

  An outline of typical ones of the present invention will be described below.

  (1) The present invention is a tool that learns actions when a person is using various tools from time series images (scenes) obtained from WEB and various observation sources and recognizes them using the learning results. In a motion recognition system, a time-series image of various image information is input in an image input unit, stored in an image storage unit, a velocity vector indicating motion in a scene is estimated by an optical flow in a velocity estimation unit, and a learning unit The HMM model parameters are calculated in various tool operation scenes, a new scene is input, and the tool operation being performed in the recognition unit is classified and identified, and the result is presented in the output unit.

  (2) Further, according to the present invention, in the speed estimation unit (1), an objective function is set in the optical flow framework, and the speed component and the luminance fluctuation component are estimated as unknowns by the least square method.

  (3) Further, the present invention estimates each unknown by minimizing the non-linear function through the Lorentz function, the bi-weight function, etc. as the non-linear robust function in the objective function of (2).

  (4) In the present invention, the steepest descent method, neural network, Levenberg-McCart method, or the like is used for the (3) non-linear function minimization calculation.

  (5) Further, in the present invention, the variance value included in the Lorentz function or the like of (3) is varied from a large value to a small value stepwise in the minimization process.

  (6) Further, according to the present invention, in the learning unit of (1), the velocity vector estimated by the velocity estimation unit is converted into an HMM model parameter using a conversion correspondence diagram for converting preset velocity vector information into a symbol. It is converted into input symbols required for the calculation of and used for learning.

  (7) Further, according to the present invention, a concentric pattern is used for the conversion correspondence diagram in the learning unit of (6).

  Specifically, according to the present invention, in a method of recognizing a motion when a person is using a tool by a computer, a time-series image obtained from an observation source is input, and a frame of continuous motion scenes in the time-series image is input. The motion vector is estimated by the optical flow method, and the average velocity vector is obtained by removing the disturbed vector. This velocity vector is applied to, for example, a concentric conversion correspondence diagram, and each tool to be recognized is learned by the HMM. After learning, a new unknown scene is input, calculation is performed by the HMM, and the operation with the smallest likelihood is taken as the recognition result.

  According to the present invention, it is possible to accurately recognize which tool is being used from an operation scene in which a person is operating various tools by hand under a situation where occlusion and discontinuity exist.

  Hereinafter, embodiments of the present invention will be described in detail. FIG. 1 is a system configuration diagram showing the entire processing of the present invention. In the tool motion recognition apparatus 1, 11 is an image input unit for inputting time-series images (frames) of an operation scene in which a person is operating various tools by hand, 12 is an image storage unit for storing input frames, 13 Is a speed estimation unit that estimates a speed vector from the motion scene, 14 is a symbol time series generation unit that generates a symbol time series from the speed vector, and 15 is a learning method that calculates HMM model parameters for each tool from the generated symbol time series. A learning unit 16 for recognizing which tool operation is being performed based on a learning result of the learning unit 15 for a newly input motion scene, and 17 an output for outputting a recognition result by the recognition unit 16 , 100 is an operation database (DB) in which HMM model parameters for each tool are stored, and 141 is a variable for converting a velocity vector into a symbol. A symbol conversion table which stores information of a corresponding view a table to.

  A motion scene of known motion when a human tool is held by hand is input from the image input unit 11 as an information source from a WEB camera or a general video camera. The image is stored in the image storage unit 12. The speed estimation unit 13 estimates a speed vector from the operation scene of the time-series image stored in the image storage unit 12, the symbol time-series generation unit 14 converts the speed vector into a symbol using the symbol conversion table 141, and the learning unit In step 15, HMM model parameters are calculated from the converted symbols, and learning is performed using the HMM. As described above, learning is performed with the HMM for each tool to be recognized, and the same HMM as the number of tool types is created and stored in the operation DB 100.

  For a scene whose motion newly input by the recognition unit 16 is unknown, the symbol time series is input to the previously learned HMM in the motion DB 100, and the motion corresponding to the HMM having the lowest likelihood is input. The output unit 17 outputs the result as a result.

  As described above, it is necessary to estimate the detailed movement (optical flow) of each operation with high accuracy in the presence of occlusion and discontinuity, and the environment changes such as noise and luminance fluctuations in the actual environment scene. It is necessary to cope with. Therefore, in the present invention, an optical flow method based on a luminance variation model and a robust estimation method is introduced to estimate a velocity vector.

  As for the recognition method of movements, as each tool continues to be used, the conditions such as the tightening of screws, the degree of cutting of objects to be cut, the resistance to drilling, etc. change from moment to moment. The feature is that it doesn't move smoothly. For this reason, a time-series pattern identification method based on HMM, which has a proven record in the field of speech due to its resistance to time-axis elasticity, is adopted.

  The derivation method for optical flow is described below. Many methods have been proposed for optical flow. Among them, a method such as Lucas described in Non-Patent Document 11 referred to as a region method is widely applied because of its accuracy and stability. However, since it is a model with constant luminance variation, it does not respond to illumination changes. In addition, since the solution is based on the method of least squares, the accuracy decreases due to the influence of outliers and discontinuous components.

  Therefore, as described in Non-Patent Document 12 and Non-Patent Document 13, a robust estimation method is applied to improve accuracy. However, since the methods described in Non-Patent Document 12 and Non-Patent Document 13 are both models with constant luminance fluctuations, if the luminance fluctuation between frames is large, the robust estimation method alone is less effective. .

  Therefore, in the present invention, a solution based on a robust estimation method is adopted based on a model that allows the luminance to change linearly between frames proposed by Cornelius et al. The robust Lorentz type was selected for the robust function. Therefore, unknown estimation becomes a nonlinear least squares problem.

For simplicity of explanation, a linear luminance change model is assumed where the sampling time is 1.0, the two-dimensional position vector in the nth frame is X, the velocity vector is U, and the intensity value is I (X, n). ceremony,
I (X + U, n + 1) = I (X, n) + b (X, n) (1)
Is described. The component of the position vector X is X = (x, y), and the component of the velocity vector U is U = (u, v).

  Subsequently, when Taylor expansion approximation is performed around the vector X for equation (1),

となる。

It becomes.

The two velocity components and coefficients are handled on a discrete grid point. In the time direction, it is divided into n, and spatially, the image (window) M × N is divided into
h _x = 1.0, h _y = 1.0,
Divide by and calculate. Here, i and j are integers, and 0 ≦ i ≦ M and 0 ≦ j ≦ N. The position vector is
X _{i, j} ⁿ = (ih _x , jh _y ) ⁿ ,
The velocity vector is
U _{i, j} ⁿ = (u _{i, j} , v _{i, j} ) ⁿ ,
Are displayed discretely. In order to obtain the first derivative terms such as each space term and time term, the discretization approximation is performed on the pixel points by the difference method as follows. The image intensity and each coefficient b are also expressed on each pixel at time n and position (i, j).

式（２）の誤差を窓内で最小化するための目的関数を，
Ｅ＝Σρ（ｅｒｒ） （３）
と定義する。ここで非線形ロバスト関数をρとする。この式が最小値をもつための条件式として，３つの未知数についての１次微分がゼロとなればよい，即ち，

The objective function for minimizing the error in equation (2) within the window is
E = Σρ (err) (3)
It is defined as Here, the nonlinear robust function is represented by ρ. As a conditional expression for this expression to have a minimum value, the first derivative with respect to three unknowns should be zero, that is,

となるように，解を求めればよい。

Find the solution so that

  Here, the steepest descent method is applied, and three unknowns are estimated from several to a dozen or more pixels (sub-block), one set for each pixel. The following equation (5) may be iteratively calculated for three unknowns u, v, b (represented as w). Equation (5) can be expressed by assuming the number of iterations p and the adjustment parameter μ:

で表される。調整パラメータμは経験的に決定される。

It is represented by The adjustment parameter μ is determined empirically.

  The three first derivative values required in Equation (5) are based on chain-rule,

である。ただし，非線形ロバスト関数が１次導関数をもつとすれば，

It is. However, if the nonlinear robust function has a first derivative,

とおくと，それぞれの１次微分は，

The first derivative of each is

となる。

It becomes.

  FIG. 2 is a diagram for explaining the derivation of the average velocity vector, and shows a process of inputting two continuous motion images, detecting a motion vector, and obtaining an average velocity vector. As shown in FIG. 2 (A), two continuous motion scene frames are input, and as shown in FIG. 2 (B), a motion vector is estimated by the optical flow method, as shown in FIG. 2 (C). In this way, an average velocity vector as shown in FIG. 2D is obtained by removing a disturbed vector by SVD (Singular Value Decomposition).

FIG. 3 shows an example of a 2-state 25-output HMM. Here, a left-to-right type HMM, which is widely used, is used, and two models of 2 states 17 outputs and 25 outputs are selected. The difference in performance between the two will be shown in experiments as will be described later. In FIG. 3, s1 and s2 are states, a _ij is a state transition probability, and b _ij is a symbol output probability.

  FIG. 4 shows an example of a conversion correspondence diagram for converting an average velocity vector into a symbol. Here, two methods, rectangular and concentric, are shown. The symbol conversion table 141 has correspondence information between the average velocity vector and the symbol corresponding to this conversion correspondence diagram. Using the symbol conversion table 141, an average velocity vector is converted into a symbol, a symbol time series is generated from a motion pattern at the time of operation, and HMM model parameters (for example, see Non-Patent Document 5) are calculated and learned for each tool. . The symbol conversion table 141 is similarly used for recognition.

  Although several methods have been proposed for symbol conversion from an average velocity vector, what kind of feature value is selected and how it is converted has not been fully studied. Here, two-dimensional information of each frame was converted into one-dimensional information as follows, and a comparative experiment was performed.

  That is, in this embodiment, the conversion correspondence diagram as shown in FIG. 4 is used for the magnitude and direction of the average velocity vector obtained from each frame. In the conversion correspondence diagram, it was found from the results of preliminary experiments that it is desirable to increase the density near the center. Each frame corresponds to one symbol, but the number of divisions and the shape are, for example, a rectangular 25 output symbol as shown in FIG. 4A or a circular 17 as shown in FIG. 4B. Designed to use output symbols. The advantages and disadvantages of the two output symbol patterns shown in FIG. 4 were verified by recognition experiments. 4A shows an example in which the symbol 17 is assigned to the average velocity vector, and FIG. 4B shows an example in which the symbol 7 is assigned.

  FIG. 5 shows an example in which a symbol time series is generated from a time series image of a certain tonkachi operation. The velocity vector obtained through the optical flow shown in FIG. 5 (B) from the time-series image of nailing with a tonkachi as shown in FIG. 5 (A) is converted into the symbol conversion of the conversion correspondence diagram shown in FIG. 5 (C). Conversion is performed using the table 141. As shown in FIG. 5C, the direction and magnitude of the velocity vector changed until the torch reached the nail. As a result, the output symbol string is (7, 7, 7, 8, 1, 4) as shown in FIG.

  FIG. 6 is a diagram illustrating an example of HMM learning for four types of tools. For example, the driver operation is learned, and the BW (Baum-Welch) algorithm is obtained from the driver symbol time series (14, 14, 12, 7, 6, 6, 6...) Shown in FIG. Thus, an HMM model parameter composed of a transition probability matrix A and an output probability matrix B as shown in FIG. This HMM model parameter is recorded in the operation DB 100 for each tool.

  By performing such HMM learning for each of the four types of tools, as shown in FIG. 6C, four types (categories) of HMMs such as a driver, a tonker, a drill, and a saw are generated in the operation DB 100.

  FIG. 7 is a diagram showing a flow of motion recognition based on a time series pattern of symbols. For example, when an unknown symbol time series (14, 2, 4, 12, 7, 2, 12) as shown in FIG. 7A is input, as shown in FIG. The degree is calculated with four types of HMMs stored in the operation DB 100. Then, the tool corresponding to the calculated logarithmic likelihood is recognized as the tool operated in the operation scene of the input symbol time series pattern. In this example, the log likelihood is calculated from 219.675661 to 819.264267, but the driver with the smallest log likelihood (219.675661) is operated in the operation scene of the input symbol time series pattern. Will be recognized.

  FIG. 8 is a diagram showing a learning process by the BW algorithm when one operator operates each of the four tools. The vertical axis represents log likelihood, and the horizontal axis represents the number of iterations. In either case, the number of iterations was 50 times or more, and the convergence was approximately in the range of -330 to -430. The log likelihood in the case of Kiri converged to the smallest value. The same tendency was observed in other subjects. The learning time was 20-30 seconds per person, per category, and the recognition time was 0.02 seconds in four categories.

  FIG. 9 is a diagram showing an average recognition rate of each person's movement when using an HMM learned by three persons (A, B, C). Using HMMs learned by three people, experiments were performed on the recognition rate of each person's movement, and the differences from HMMs learned by one person were examined. FIG. 9 shows the average recognition rate of the experimental results when the number of output symbols is 17 and 25, respectively. In FIG. 9, the parentheses indicate the improvement rate from the recognition rate when learning by one person.

  When the number of output symbols is 17, the driver and tonkerchie improved by 10% or more, and when the number of output symbols is 25, the driver, tonkerchid and drill improved by 7% to 12%. The average recognition rate of the four tools was about 3% higher when the number of output symbols was 17.

  FIG. 10 is a diagram illustrating a change in recognition rate when the number of input symbols is changed. As shown in FIG. 10, when the number of input symbols decreased from 50 to 5, the recognition rate decreased for any tool. This is natural because time-series feature values decrease.

  As described above, according to the present invention, it can be confirmed that which tool is used from a moving image scene in which a person manually operates various carpenter tools in an actual environment based on the optical flow and the HMM. did it.

  The problem of motion recognition from a monocular camera is that it is difficult to fit the tool shape due to the occlusion of the operator's fingers, back and arms, the periodicity of movement is weak, and the speed of operation is different. . Therefore, in the present invention, the operation is modeled without separating the integral movement of the hand and the tool, and in the experiment for verifying the effect, motion recognition is performed with the four tools as categories from these movements. The optical flow method via a nonlinear robust function suppresses estimation errors caused by discontinuous motion components, and realizes learning and recognition by applying an HMM that is strong in time-axis elasticity. The symbol time series was generated by mapping the average velocity vector of the optical flow to the number of output symbols using a conversion correspondence diagram designed in advance.

  Learning was performed by one to three people, and recognition experiments were conducted on differences in the number of input symbols and the number of output symbols. As a result, an average recognition rate of up to 100% was obtained for learning and recognition by the same person, and a recognition rate of up to 88.6% was obtained even when different persons operated for learning and recognition. . Even in the case of short data with 5 input symbols (0.2 seconds) that are difficult to recognize, a high recognition rate of an average of 79.4% or more was obtained, indicating the robustness and effectiveness of the present invention. ing.

  In HMM learning, a high recognition rate was obtained for recognition of the same person. It was also found that features that can distinguish four categories are included in a very short symbol time series of 0.2 seconds. Regarding learning and recognition with different persons, we obtained a recognition rate above a certain level, and at the same time, it was confirmed that the overall recognition rate was improved by learning with multiple people.

  From this, it is understood that when the number of input symbols is small, the number of learning data (persons) may be increased, while when the number of input symbols is large, the number of learning data (persons) may be small. From a practical point of view, increasing the number of learning data requires learning time. However, if the number of input symbols is small, recognition calculation (maximum likelihood) is possible by that much.

It is a system configuration figure showing the whole processing of the present invention. It is a figure explaining derivation of an average velocity vector. It is a figure which shows the example of HMM of 2 states 25 output. It is a figure which shows the example of the conversion corresponding | compatible figure which converts an average speed vector into a symbol. It is a figure which shows the example of the symbol time series production | generation from a tonkachi operation. It is a figure which shows learning of HMM with respect to four types of tools. It is a figure which shows to the flow of the operation | movement recognition from the time-sequential pattern of a symbol. It is a figure which shows the mode of the learning process of one operator. It is a figure which shows the improvement of the recognition rate when learning by one person when learning by three persons. It is a figure which shows the change of the recognition rate at the time of changing the number of input symbols. It is a figure which shows the example of the operation | movement scene which is operating four types of tools.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Tool motion recognition apparatus 11 Image input part 12 Image storage part 13 Speed estimation part 14 Symbol time series generation part 15 Learning part 16 Recognition part 17 Output part 100 Motion DB
141 Symbol conversion table

Claims (8)

A device that recognizes movements when people are using various tools from the movement scenes of time-series images.
Image input means for inputting time-series images;
Image storage means for storing input time-series images;
Speed estimation means for estimating the speed vector of the motion scene in the time series image by optical flow;
Learning means for calculating and learning HMM model parameters for each tool operation of a plurality of types of tools to be recognized based on the estimated velocity vector;
With respect to an unknown motion scene of a time-series image newly input to the image input means, a speed vector is estimated by the speed estimation means, and which tool operation is performed based on an HMM of a learning result by the learning means A recognition means for identifying
A tool motion recognition device comprising: output means for outputting a tool operation identification result. The tool movement recognition device according to claim 1,
The speed estimation means includes
A tool motion recognition device characterized in that an objective function is set in an optical flow framework, and the velocity vector is estimated by estimating each unknown by a least square method with the velocity component and the luminance fluctuation component as unknowns. The tool movement recognition device according to claim 2,
The speed estimation means includes
A tool motion recognition apparatus, wherein a nonlinear robust function based on a Lorentz function or a bi-weight function is set as the objective function, and the unknown is estimated by minimizing the set nonlinear robust function. The tool movement recognition device according to claim 3,
The speed estimation means includes
A tool motion recognition device using a steepest descent method, a neural network or a Levenberg-McCart method for minimizing the nonlinear robust function. The tool movement recognition device according to claim 3,
The speed estimation means includes
A tool motion recognition apparatus, wherein when the nonlinear robust function is minimized, a variance value included in the Lorentz function or by weight function is changed stepwise from a large value to a small value. In the tool movement recognition device according to any one of claims 1 to 5,
The learning means includes
Converting the estimated speed vector information into a symbol using a conversion correspondence diagram for converting predetermined speed vector information into a symbol, and calculating the HMM model parameter using the converted symbol. Tool motion recognition device. The tool movement recognition device according to claim 6,
The learning means includes
The tool operation characterized in that the estimated velocity vector information is converted into a symbol using the conversion correspondence diagram in which the arrangement of symbols is a concentric pattern according to the magnitude and direction of the velocity vector as the conversion correspondence diagram. Recognition device. A method for recognizing movements when people are using various tools from movement scenes of time-series images.
An image input step for inputting a time-series image;
An image accumulation step for accumulating input time-series images;
A speed estimation step of estimating a speed vector of an operation scene in the time-series image by an optical flow;
A learning step of calculating and learning HMM model parameters for each tool operation of a plurality of types of tools to be recognized based on the estimated velocity vector;
A recognition step for estimating a speed vector for the unknown motion scene of the newly input time-series image by the speed estimation step and identifying which tool operation is being performed based on the HMM of the learning result by the learning step When,
An output step for outputting a result of the type of tool operation.

Images