JP2021163217A - Motion detection device, motion detection method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an operation detection device, an operation detection method, and a program capable of improving the detection accuracy even if the operation is ambiguous. SOLUTION: An motion detection device has an image acquisition unit that acquires an image of an object for which an motion is detected, an acoustic signal acquisition unit that acquires an acoustic signal of an environment, and movements of characteristic feature points in the acquired image. A detection unit that detects the amount of change, detects the acoustic feature information of the acquired acoustic signal, inputs the detected change amount and the acoustic feature information to the trained neural network, and detects a predetermined operation. To prepare for. [Selection diagram] Fig. 1

Description

The present invention relates to a motion detection device, a motion detection method, and a program.

Communication between humans and robots is important for robots and humans to cooperate in the actual living environment. Robots need to understand not only linguistic expressions but also non-verbal expressions such as nods and gestures.
As a method for detecting a human motion, a device that recognizes the motion from a moving image of the human motion has been proposed (see, for example, Patent Document 1 and Patent Document 2).

JP-A-2007-94619 Japanese Unexamined Patent Publication No. 5-67213

However, in the prior art, when the operation is ambiguous, it is difficult to detect the operation.

The present invention has been made in view of the above problems, and provides an operation detection device, an operation detection method, and a program capable of improving the detection accuracy even if the operation is ambiguous. With the goal.

(1) In order to achieve the above object, the motion detection device according to one aspect of the present invention includes an image acquisition unit that acquires an image of an object for which motion is detected, an acoustic signal acquisition unit that acquires an environmental acoustic signal, and an acoustic signal acquisition unit. The movement of the characteristic feature point in the acquired image is detected as a change amount, the acoustic feature information of the acquired acoustic signal is detected, and the detected change amount and the acoustic feature information are used in a trained neural network. It is provided with a detection unit for detecting a predetermined operation by inputting to.

(2) Further, in the motion detection device according to one aspect of the present invention, the acoustic feature information includes the strength and weakness of the acoustic signal based on the total power of the acoustic signal for a predetermined time, and the acoustic signal having a predetermined magnitude or more. The amount of change is the position of the feature point detected from the captured image, the position of the feature point detected at the first time, and the feature point detected at the second time. It may be the difference from the position of.

(3) Further, in the motion detection device according to one aspect of the present invention, the predetermined time may be a length for adjusting to the frame rate of the image.

(4) Further, in the motion detection device according to one aspect of the present invention, the detection unit may connect the change amount and the acoustic feature information before inputting to the neural network.

(5) Further, in the motion detection device according to one aspect of the present invention, the target for detecting the motion may be a person, and the motion may be a nod.

(6) In order to achieve the above object, in the motion detection method according to one aspect of the present invention, the image acquisition unit acquires the image of the target for which the motion is detected, and the acoustic signal acquisition unit acquires the acoustic signal of the environment. Then, the detection unit detects the movement of the characteristic feature point in the acquired image as a change amount, detects the acoustic feature information of the acquired acoustic signal, and detects the change amount and the acoustic feature information. , Input to the trained neural network to detect a predetermined motion.

(7) In order to achieve the above object, the program according to one aspect of the present invention causes a computer to acquire an image of a target for detecting an operation, and an acoustic signal acquisition unit acquires an environmental acoustic signal and acquires the image. The movement of characteristic feature points in the image is detected as a change amount, the acoustic feature information of the acquired acoustic signal is detected, and the detected change amount and the acoustic feature information are transmitted to a trained neural network. Input to detect a predetermined operation.

According to the above-mentioned (1) to (6), the detection accuracy can be improved as compared with the conventional case even if the operation is ambiguous.
According to (2) described above, information on the movement of the detection target can be acquired, and information on the acoustic signal of the environment can be acquired.
According to (3) described above, the acoustic feature information can be combined with the amount of change based on the image.
According to (4) described above, the detection accuracy can be improved as compared with the conventional case even if the operation is ambiguous.
According to (5) described above, nodding, which is an ambiguous behavior, can be detected.

It is a block diagram which shows the structural example of the motion detection system which concerns on embodiment. It is a figure which shows the example which visualized the speaker's utterance intonation and the listener's nod. It is a figure for demonstrating the detection of the key point of a face using Dlib. It is a figure for demonstrating the configuration example and the processing example of the detection part which concerns on embodiment. It is a flowchart of the processing procedure example performed by the motion detection system which concerns on embodiment. It is a figure which shows the accuracy example when the nod is judged only based on the voice information of the 1st comparative example. It is a figure which shows the accuracy example when the nod is judged only based on the change amount of the facial organ point of the 2nd comparative example. It is a figure which shows the accuracy example at the time of determining the nod in an embodiment. It is a figure which shows the 1st modification of the neural network which concerns on this embodiment. It is a figure which shows the 2nd modification of the neural network which concerns on this embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each member is appropriately changed in order to make each member recognizable.

FIG. 1 is a block diagram showing a configuration example of the motion detection system 1 according to the present embodiment. As shown in FIG. 1, the motion detection system 1 includes a sound collecting section 2 (acoustic signal acquisition section), a photographing section 3 (image acquisition section), and a motion detection device 4.
The motion detection device 4 includes a voice processing unit 41 (acoustic signal acquisition unit), an image processing unit 42 (image acquisition unit), a detection unit 43, and an output unit 44.

The sound collecting unit 2 is a microphone, collects an acoustic signal, and outputs the collected acoustic signal to the motion detection device 4. The sound collecting unit 2 may be a microphone array including a plurality of microphones. Further, the sound collecting unit 2 may convert the collected acoustic signal from an analog signal to a digital signal and output the sound signal of the converted digital signal to the motion detection device 4.

The photographing unit 3 is, for example, a CMOS (Complementary MOS) image sensor or a CCD (Charge Coupled Device) image sensor. The photographing unit 3 outputs the captured image to the motion detection device 4. The image to be taken is. It may be a still image at predetermined time intervals or a moving image.

The motion detection device 4 estimates the probability that the listener nods to the speaker's utterance based on the acoustic signal (voice signal) and the image information.

The voice processing unit 41 acquires the acoustic signal output by the sound collecting unit 2. The audio processing unit 41 still converts the acoustic signal output by the sound collecting unit 2 from an analog signal to a digital signal (for example, 16-bit discrete data). The voice processing unit 41 detects the presence or absence of voice (whether or not the sound signal has a predetermined magnitude or more) and voice intonation (strength or weakness of the sound signal) with respect to the acquired sound signal. The detection method will be described later. The voice processing unit 41 outputs the detected utterance presence / absence information and utterance inflection information to the detection unit 43 as utterance information (acoustic feature information). The utterance information is also time-series information on the system side when the speaker is a system such as a robot.

The image processing unit 42 acquires an image taken by the photographing unit 3. When the image processing unit 42 detects a nod as an ambiguous motion of the listener, the image processing unit 42 performs image processing on the acquired image, extracts, for example, a region of the listener's face, and features an amount from the image corresponding to the extracted face. Is extracted. The image processing unit 42 deletes information unnecessary for nod recognition. The image processing unit 42 detects the key points of the speaker's face included in the image. The image processing unit 42 calculates the amount of change in the facial organ points of each of the face key points indicating the detected key points of the face, and outputs the calculated amount of change in the facial organ points to the detection unit 43. The method of detecting the key points of the face and the method of calculating the amount of change in the facial organ points will be described later.

The detection unit 43 acquires the utterance information output by the voice processing unit 41 and the face key point information output by the image processing unit. The detection unit 43 integrates the acquired nod information and the face key point information, and detects the movement of the listener by using the learned model. A configuration example of the detection unit 43 and an operation example of the detection unit 43 will be described later. The detection unit 43 outputs operation information indicating the detected operation of the listener to the output unit 44.

The output unit 44 is, for example, an image display device, a printing device, or the like. The output unit 44 outputs the operation information output by the detection unit 43.

In the following embodiment, nodding will be described as an example of the listener's ambiguous movement, but the listener's ambiguous movement is not limited to this. The listener's ambiguous movements may be, for example, blinking, waving gestures, or turning the face sideways.

[Speaker's speech intonation and listener's nod]
Here, an example of the speaker's utterance intonation and the listener's nod in the utterance will be described.
FIG. 2 is a diagram showing an example of visualizing the speaker's utterance intonation and the listener's nod. In FIG. 2, the horizontal axis is the frame number, and the vertical axis is the utterance intonation (change in amplitude). In FIG. 2, the speaker is the speaker, and the listener is performing the nodding action. In addition, the nodding section g11 was visually confirmed.

As shown in FIG. 2, nodding is likely to occur during the speaker's utterance. Further, as shown in FIG. 2, when a speaker speaks one sentence, the shape of the utterance intonation tends to be mountainous. Since the listener's nod is caused by the speaker's voice, the present embodiment uses voice information as an additional clue to the listener's nod timing.
The acoustic signal and the image of the listener's motion used in the present embodiment may be, for example, a recorded video chat or the like, or may be acquired in real time, for example, during the video chat.

[Processing of voice processing unit]
Next, a processing example performed by the voice processing unit 41 will be described.
The voice processing unit 41 detects the presence or absence of voice by comparing, for example, the binary value of the digitized voice signal with the threshold value of the presence or absence of voice. For example, the voice processing unit 41 sets the entire utterance section to 1 and the other sections to 0. Alternatively, the voice processing unit 41 may detect the presence or absence of voice based on the voice power for each frame.

Further, the voice processing unit 41 detects voice intonation based on the amplitude of the acoustic signal. However, since the voice signal has a negative value, the voice processing unit 41 calculates the voice power from the waveform data in the preprocessing. In addition, the sampling frequency is different for the video image sequence and the acoustic signal.

Therefore, in the present embodiment, as shown in the following equation (1), the voice processing unit 41 uses the sum of the voice powers of 10 samples before and after the time t as the feature S (t) of the voice information. .. Thereby, according to the present embodiment, the acoustic feature information can be matched with the amount of change based on the image.

In the equation (2), s (t) is the voice power at time t, s (t + i) ² is the voice power, and i is the number of samples. In this embodiment, an example in which the number of samples is 10 samples before and after the time t will be described, but the number of samples is an example and is not limited to this.

In general, there are two types of nods. Nods in response to other people's speeches (represented by the label "you") and nods in your own speech (represented by the "me" label). In this embodiment, only the "you" sample is used for training and evaluation because the focus is on nodding in response to the speeches of others.

[Processing of image processing unit]
Next, a processing example performed by the image processing unit 42 will be described.
In this embodiment, the difference in the position of the key point of the face between the current frame and the previous frame is used as motion information.
When the features of the entire face image are extracted, the features of the appearance are formed by this. As a result, information unnecessary for nodding recognition can be acquired. Therefore, in the present embodiment, the movement information is captured and unnecessary appearance information is deleted by using the difference in the movement of the key points of the face.

The image processing unit 42 performs face detection and face key point detection by, for example, Dlib (see Reference 1).

Reference 1; Davis E. King, “Dlib-ml: A machine learning toolkit.”, Journal of Machine Learning Research, Vol.10, pp.1755-1758, 2009.

FIG. 3 is a diagram for explaining the detection of facial key points using Dlib. Reference numeral g21 is a region of the detected face, and reference numeral g22 is an example of the key points of the detected face. As shown in FIG. 3, Dlib detects, for example, 68 facial key points.

Movement amount _{d x} in the x direction at time t keypoint i _face, the movement amount _{d y i} and _{y-direction, i} is given by the following equation (2). The image processing unit 42 outputs the time shift amount d _x in the x direction in _t, the movement amount d _{y i} and y _{directions, i} to the detection unit 43 as a variation of the face organ point.

In the formula (2), i is the corresponding number of the facial organ point, and d _{x, i} (t) and d _{x, i} (t) are the feature quantities at the time t (first time). Further, x _i (t) and y _i (t) are facial organ point positions at time t, and x _i (t-5) and y _i (t-5) are faces 5 frames before (second time). Organ point position. In the present embodiment, an example in which the difference between the position 5 frames before and the position at time t is used as the feature quantity has been described, but the number of frames used for comparison is not limited to 5 frames before, but may be other frames. May be good.
Since the total number of key points on the face is 68, the feature vector of the motion information becomes 136 dimensions when _{d x, i} (t) and d _{x, i (t) are combined.}
Further, the frame whose face could not be detected by Dlib may be interpolated by using the average of the front and back face detection.

[Configuration example and processing example of detection unit]
Next, a configuration example and a processing example of the model included in the detection unit 43 will be described.
FIG. 4 is a diagram for explaining a configuration example and a processing example of the detection unit 43 according to the present embodiment. As shown in FIG. 4, the detection unit 43 includes a connecting unit 431 (concat), a fully connected layer 432 (FC; Fully Connected), an LSTM433 (Long Short Term Memory), an LSTM434, and a fully connected layer 435. As shown in FIG. 4, the detection unit 43 is a network model having a structure having two layers of RNNs.

The amount of change in the 136-dimensional facial organ points and the one-dimensional utterance information are input to the detection unit 43.
The connecting unit 431 normalizes the amount of change in the 136-dimensional facial organ points and each input value of the one-dimensional utterance information to 0 to 1, and connects them. The connecting unit 431 applies weights to these functional values during connection.
The activation function of the fully connected layer 432 is, for example, ReLU (Rectifier Liner Unit, see Reference 2).
The SSTM433 has, for example, 256 units, a dropout of 0.5, and an activation function of ReLU.
The SSTM434 has, for example, 256 units, a dropout of 0.5, and an activation function of ReLU.
The activation function of the fully connected layer 435 is, for example, ReLU.

Reference 2; Vinod Nair, Geoffrey E. Hinton, "Rectied linear units improve restricted boltzmann machines", Proceedings of the 27th International Conference on Machine Learning, pp. 807-814 (2010).

The detection unit 43 uses, for example, RMSprop for optimization, and trains a model included in the detection unit 43 with a batch size of 32 and 300 epochs. RMSprop is one of the gradient methods in deep learning, and is an algorithm improved from AdaGrad. Further, the detection result indicates, for example, that "1" is speaking and "0" is not speaking.

Further, the activation function is not limited to ReLU, and may be another activation function such as sigmoid or tanh.
Further, the method is not limited to LSTM, and may be a method capable of handling current data and past data in time series.

[Processing procedure]
Next, an example of the processing procedure will be described.
FIG. 5 is a flowchart of an example of a processing procedure performed by the motion detection system 1 according to the present embodiment. The model (neural network) included in the detection unit 43 is trained using teacher data such as a video conference, and then performs the following processing.

(Step S1) The sound collecting unit 2 collects the acoustic signal of the speaker. The motion detection device 4 acquires the acoustic signal of the speaker.

(Step S2) The photographing unit 3 photographs an image including the face of the listener. The motion detection device 4 acquires an image of the listener.

(Step S3) The voice processing unit 41 detects utterance information (presence or absence of utterance, utterance intonation) using an acoustic signal.

(Step S4) The image processing unit 42 detects a face region and a face key point from the image. Subsequently, the image processing unit 42 calculates the amount of movement of the key points of the face.

(Step S5) The detection unit 43 normalizes the amount of change in the 136-dimensional facial organ point and each input value of the one-dimensional utterance information to 0 to 1, and concatenates them.

(Step S6) The detection unit 43 inputs the concatenated information into the model and detects whether or not it is a nod.

(Step S7) The output unit 44 outputs the result determined by the detection unit 43.

[Evaluation results]
Next, an example of the evaluation result will be described.
FIG. 6 is a diagram showing an accuracy example when the nod is determined based only on the voice information of the first comparative example. The network structure of the detection unit used in the first comparative example includes a first LSTM, a second LSTM, and a fully connected layer, and is connected in the order of the first LSTM, the second LSTM, and the fully connected layer. In the first comparative example, one-dimensional utterance information is input to the first LSTM, and a nodding judgment result is output.
As shown in FIG. 6, it is not possible to judge a nod only by the presence or absence of a speech as an input.
The speech intonation information achieves a recognition accuracy of about 42.0%. The average recognition rate is about 63.6%.
As a result, it is shown that the information of the speaker's speech intonation is important for determining whether the listener is nodding.

FIG. 7 is a diagram showing an accuracy example when the nod is determined based only on the amount of change in the facial organ points of the second comparative example. The network structure of the detection unit used in the second comparative example also includes a first LSTM, a second LSTM, and a fully connected layer, and is connected in the order of the first LSTM, the second LSTM, and the fully connected layer. In the second comparative example, the amount of change in the facial organ points of 136 dimensions is input to the first LSTM, and the nodding judgment result is output.
As described above, the amount of movement of the key points of the face is the difference between the positions of the key points of the current frame and the previous frame.

In the example shown in FIG. 7, the effect of the frame interval used for calculating the movement amount of the key points of the face is evaluated. As shown in FIG. 7, when the interval of 5 frames was used, the recognition accuracy of nodding of about 83.1% was achieved. When the 1-frame interval was used, the nodding recognition accuracy was improved compared to 5 frames, but more false positives (recognizing the nod when it did not exist) occurred than when the 5-frame interval was used. Therefore, in the present embodiment, the frame interval used for calculating the movement amount of the key points of the face is set to 5 frames.

FIG. 8 is a diagram showing an accuracy example when the nod in the present embodiment is determined.
The model was trained simultaneously on both the speech detector and the amount of movement of key points on the face. As shown in FIG. 8, when the two pieces of information were used simultaneously, an overall accuracy of 84.4% was achieved and it was determined whether or not there was a nod. This result is more accurate than the results of the first and second comparative examples achieved using only one piece of information.

[Modification example]
The configuration example of the model (neural network) included in the detection unit 43 shown in FIG. 4 is an example, and is not limited to this. For example, the number of RSTM units may be less than 256 or more than 256. Further, the number of RSTMs is not limited to two, and may be one or three or more.
Further, the connection position of the connecting portion 431 may be in front of the fully connected layer 435. In this case, the model may include, for example, two layers of LSTMs, one for the amount of change in facial organ points and the other for one-dimensional utterance information.

FIG. 9 is a diagram showing a first modification of the neural network according to the present embodiment. In the example of FIG. 9, the fully connected layer is one example, and the connecting portion, the LSTM, the LSTM, and the fully connected layer are connected in this order. Even with such a neural network configuration, the nodding recognition rate (correct answer rate) can be improved as compared with the conventional case.

FIG. 10 is a diagram showing a second modification of the neural network according to the present embodiment. In the example of FIG. 10, the LTSM layer is one example, and the connecting portion, the fully connected layer, the LSTM, and the fully connected layer are connected in this order. Even with such a neural network configuration, the nodding recognition rate (correct answer rate) can be improved as compared with the conventional case.

Further, the speaker may be an object that generates sound during operation, for example, a robot or the like. For example, the sound collecting unit 2 may collect the environmental sound, and the photographing unit 3 may take an image of a person or an object for detecting an operation existing in the environment. Alternatively, for example, the sound collecting unit 2 may collect the utterance of the speaker, and the photographing unit 3 may take an image of a person or an object for detecting an operation existing in the environment.
When the motion detection target is an object, the motion detection device 4 extracts an operating area from the captured image of the object, and uses the characteristic position of the image of the extracted area as a key point. The amount of movement of the key point between frames may be used instead of the amount of change in facial organ points.

As described above, in the present embodiment, the acoustic feature information is detected from the acoustic signal, and the amount of change is detected from the image. Further, in the present embodiment, the acoustic feature information is determined by the strength of the acoustic signal based on the total power of the acoustic signals for a predetermined time and whether or not the acoustic signal has a predetermined magnitude or more, and the amount of change is changed. , The position of the feature point was detected from the captured image, and the difference between the position of the detected feature point at the first time and the position of the detected feature point at the second time was set. Further, in the present embodiment, in addition to the moving image of the person, the voice power of the speaker is input to the trained neural network to judge the nod.

As a result, according to the present embodiment, it is possible to acquire information on the movement of the detection target, and it is possible to acquire information on the acoustic signal of the environment. As a result, according to the present embodiment, it is possible to detect a specific movement such as nodding.

A program for realizing all or a part of the functions of the motion detection device 4 in the present invention is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read by the computer system and executed. By doing so, all or part of the processing performed by the motion detection device 4 may be performed. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. Further, the "computer system" shall also include a WWW system provided with a homepage providing environment (or display environment). Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, or a storage device such as a hard disk built in a computer system. Furthermore, a "computer-readable recording medium" is a volatile memory (RAM) inside a computer system that serves as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, it shall include those that hold the program for a certain period of time.

Further, the program may be transmitted from a computer system in which this program is stored in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the "transmission medium" for transmitting a program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Further, the above program may be for realizing a part of the above-mentioned functions. Further, it may be a so-called difference file (difference program) that can realize the above-mentioned function in combination with a program already recorded in the computer system.

Although the embodiments for carrying out the present invention have been described above using the embodiments, the present invention is not limited to these embodiments, and various modifications and substitutions are made without departing from the gist of the present invention. Can be added.

1 ... Motion detection system, 2 ... Sound collection unit, 3 ... Shooting unit, 4 ... Motion detection device, 41 ... Voice processing unit, 42 ... Image processing unit, 43 ... Detection unit, 44 ... Output unit

Claims (7)

An image acquisition unit that acquires an image of the target for which motion is detected, and an image acquisition unit
An acoustic signal acquisition unit that acquires the acoustic signal of the environment,
The movement of the characteristic feature point in the acquired image is detected as a change amount, the acoustic feature information of the acquired acoustic signal is detected, and the detected change amount and the acoustic feature information are used in a trained neural network. A detector that detects a predetermined operation by inputting to
Motion detection device. The acoustic feature information is the strength of the acoustic signal based on the total power of the acoustic signal for a predetermined time, and whether or not the acoustic signal has a predetermined magnitude or more.
The amount of change is the difference between the position of the feature point detected at the first time and the position of the feature point detected at the second time by detecting the position of the feature point from the captured image.
The motion detection device according to claim 1. The predetermined time is a length for adjusting to the frame rate of the image.
The motion detection device according to claim 2. The detection unit
The change amount and the acoustic feature information are connected before the input of the neural network.
The motion detection device according to any one of claims 1 to 3. The target for detecting the above motion is a person.
The action is nodding,
The motion detection device according to any one of claims 1 to 4. The image acquisition unit acquires the image of the target for which the operation is detected,
The acoustic signal acquisition unit acquires the acoustic signal of the environment and
The detection unit detects the movement of the characteristic feature point in the acquired image as a change amount, detects the acoustic feature information of the acquired acoustic signal, and learns the detected change amount and the acoustic feature information. Input to a completed neural network to detect a predetermined motion,
Motion detection method. On the computer
Get the image of the target to detect the movement,
The acoustic signal acquisition unit acquires the acoustic signal of the environment,
The movement of the characteristic feature points in the acquired image is detected as the amount of change.
Detects the acoustic feature information of the acquired acoustic signal and
The detected change amount and the acoustic feature information are input to the trained neural network to detect a predetermined motion.
program.

Images