KR101030603B1 - Determination of starting point of multi-modal command and acquisition of control authority using clap sound and gesture image - Google Patents

Abstract

The present invention provides a method of detecting the starting point of an efficient command using multi-modality in a multi-modal application system, but the system command misrecognition due to misunderstanding or unintentional gesture behavior expressed in the user's consciousness / unconsciousness. There is a purpose and effect to minimize.

In the present invention, when inputting a multi-modality command required for operation of a multi-modal system, a hand-held sound and a hand gesture image generated by performing one or n lines of hand gestures simultaneously occurring at the same time, a microphone and a camera, etc. Recognizes that the system operation command or content control command is started by using individual or multi-modality based on the recognized time point, and identifies the user who wants to input the current command among a plurality of users. In addition to granting control, it is configured to improve control efficiency and accuracy for various sound and gesture commands.

Clap, gesture video, multi-mortal, control

Description

A Starting Point Decision and Control Right Acquisition Method of Multi-Mode Commands Using Clap and Gesture Image}

The present invention provides a microphone for the sound of a hand gesture and a gesture of a hand gesture generated by performing one or n lines of a touch that simultaneously occurs a sound and a gesture in inputting a multi-modality command necessary for operation of a multi-modal (multi-mode) system. The system operation command or the content control command is started to the system by using the individual or multi-modality based on the recognition time, and the current command among a plurality of users The present invention relates to a method for determining a starting point of a multimodal command and obtaining a control right, which can identify and grant a control right to a user and improve control efficiency and accuracy of various sound, voice, and gesture commands.

Korean Patent Publication No. 10-0873470, which is related to the present invention, discloses a configuration of controlling a system using a multi-modal command to be used for starting and ending recognition, but showing a sound and gesture image using a modal command. Since the signals are not used at the same time, there is a problem that misunderstanding of system commands may occur due to misunderstanding or unintentional gestures expressed in the user's consciousness / unconsciousness.

Korean Patent Publication No. 10-0860407, another prior art related to the present invention, discloses a technical configuration for performing a corresponding command by inferring an action, but there is a possibility of misrecognition of a system command by performing a command only for the action. There is a problem that increases.

Another related art related to the present invention is similar in that it can generate and output a command based on a user's situation in Korean Patent Publication No. 10-0651729, but the user's consciousness is generated by using one kind of output. There is a problem in that misunderstanding of a system command occurs due to misunderstanding or unintentional gesture behavior expressed in unconsciousness.

The problem to be solved by the present invention is to propose a method for detecting the starting point of an efficient command using multi-modality in a multi-modal application system, the clasp that can simultaneously acquire the audio modality (such as the sound of a gesture) and gesture modality information By performing one or n times, recognizing these modality information, and identifying and granting control to users so that system errors due to misunderstanding or unintentional gestures expressed during user's consciousness or unconsciousness I want to minimize it.

Another problem to be solved by the present invention is to obtain the audio modality and gesture modality to determine the same input / recognition for the individual recognition result through a simple decision algorithm such as AND operation, and to input a command By identifying the user and granting the system control authority, it is intended to minimize the misunderstanding of the system command due to the misunderstanding or unintentional gesture action expressed in the user's consciousness or unconsciousness.

Another problem to be solved by the present invention is the end point detection method in the time domain using the short-term energy and zero crossing rate, the end point detection method using the frequency spectrum, the end point detection method based on the MFB (Mel-frequency Filter Bank), voice Subdividing the band into individual bands and detecting end point intervals such as the method of using the spectrum subtraction method and the band energy for each divided band, and using the end point detection method using the FFT spectrum and Mel-spectrum based entropy. It aims to minimize the misunderstanding of system commands due to misunderstanding or unintentional gestures expressed during consciousness / unconsciousness.

The solution of the present invention is that the user performs the clapping action once or n or more times, to obtain a sound generated when the clapping with a microphone or a stereo microphone, and to make a clasp by using a camera or a stereo camera, etc. Photographing a gesture image for extracting a gesture signal from a signal detected by a microphone, and extracting a feature of a user image from a gesture image signal detected by the camera. Through the steps, the finger recognition unit for comparing and recognizing the characteristics of the detected hand signal and the handheld sound stored in the server or the memory and the characteristic of the detected hand gesture video signal and the hand gesture video signal stored in the server or the memory Through the gesture recognition unit to compare and recognize And a step of recognizing a gesture video signal, and determining a command starting point by recognizing a sound and gesture video signal through a multi-modal method through the hand gesture recognition unit and the hand gesture recognition unit, in a multi-user or single user environment. The present invention provides a method of determining a starting point of a multi-modal command and obtaining a control right using a clap sound and a gesture image, which includes acquiring a system control right by identifying a user who has performed the clap and a hand gesture.

In another aspect of the present invention, there is provided a method for detecting an end point in a time domain using short-term energy and a zero crossing rate, and a frequency spectrum, in order to remove the background noise included in the beep and improve the end point detection performance of the beep. End point detection method and FFT spectrum and method such as end point detection method using MFB (Mel-frequency Filter Bank) based end point detection method, subdividing sound band into individual bands and using spectral subtraction method and band energy for each divided band The present invention provides a method for determining a starting point of a multi-modal instruction and obtaining a control authority using a step of removing background noise by using an endpoint detection method using mel-spectrum-based entropy.

According to another aspect of the present invention, in order to recognize a hand gesture video signal at the time of a gesture of a hand, the method is divided into a step of comparing and recognizing the shape of the hand in a multi-dimensional manner and comparing and analyzing the motion shapes of the hand and the arm. In order to implement a more accurate method for determining the starting point of a multi-modal command and obtaining control authority by tracing the components of a user's hand or arm through the steps of tracking and analyzing the multidimensional shape movement of the arm by reconstruction of the multidimensional human model. have.

Another solution of the present invention is to recognize a hand shape that is robust to angle and self occlusion using the multidimensional hand shape (image) estimation, and to store the motion of the hand image in a server or a memory. The present invention provides a method for determining a starting point of a multi-modal command and acquiring control authority by performing a comparative analysis with a gesture image signal.

The present invention proposes a method for detecting an effective starting point of an instruction using multi-modality in a multi-modal application system, wherein a hand-wound stroke capable of simultaneously acquiring audio modality (such as a clapping sound) and gesture modality information is performed once or n. By performing the circuit, recognizing the modality information, and identifying the user and granting the authority to control the system, the system command misrecognition due to misunderstanding or unintentional gestures expressed by the user during consciousness or unconsciousness There is an effect that can be minimized.

Another effect of the present invention is to acquire the audio modality and gesture modality, such as the sound of a hand, and to identify the user who wants to input a command through a simple decision algorithm such as AND operation to give the system control authority during the consciousness of the user / It aims to minimize the misunderstanding of system commands due to misunderstanding or unintentional gestures expressed in the unconscious.

Another effect of the present invention is to detect the end-point performance of the hand-capped sound in order to recognize the noise-resistant hand-capped sound in the time domain using the short-term energy and zero crossing rate, the end point detection method using the frequency spectrum, By using the endpoint detection method based on the Mel-frequency Filter Bank (MFB) and the endpoint detection method using the FFT spectrum and Mel-spectrum based entropy, it is possible to minimize the system command misrecognition due to misleading or unintended gesture behavior. It is to make it possible.

Hereinafter, the present invention will be described in detail. According to the present invention, a method for determining a starting point of a multi-modal command and obtaining control authority using a clap gesture and a clap gesture image includes performing a clap gesture at least once or n times to generate a microphone or stereo sound. And detecting gestures using a microphone or the like and acquiring a gesture video signal generated to generate a beeping sound using a camera or a stereo camera, and performing end point detection on the obtained sounding signal. Through the extracting and extracting the feature from the photographed gesture image signal, the hand recognition unit and the photographing unit for recognizing and comparing the detected hand signal feature with the user's hand feature previously stored in a server or a memory, etc. Features of hand gestures and video signals Comprising a hand gesture recognition unit for comparing and analyzing the characteristics of the user gesture gesture image signal stored in the step, the step of acquiring the control point of the multi-modal command consisting of the step of acquiring the system control authority by the corresponding touch gesture is configured to acquire the control authority It is.

In addition, the present invention is an end point detection method in the time domain using the short-term energy and the zero crossing rate to improve the end point detection performance of the clasping sound in order to remove the background noise included in the clasping signal to be resistant to the noise, Endpoint detection method and FFT such as endpoint detection method using frequency spectrum, endpoint detection method based on MFB (Mel-frequency Filter Bank), subdividing sound band into individual bands and using spectrum subtraction method and band energy for each divided band Removing the background noise using an end point detection method using spectral and mel-spectrum based entropy.

Typically, clapping hands are accompanied by both clapping and clapping gestures, and each person's physique, weight, hand size, and hand shape are different, and the hands, arms, and body of the clapping action that occur accordingly This moving shape also has certain characteristics.

Features of the clasp and gesture image signal according to the present invention are generally configured as a recognition / reference model and stored in a database of a server or a memory of a device for determining a starting point of a multi-modal command and obtaining control authority.

Hereinafter, the configuration and operation of the embodiments of the present invention with reference to the accompanying drawings, the configuration and operation of the present invention shown and described in the drawings will be described by at least one embodiment, whereby the present invention described above The technical idea and its core composition and operation are not limited.

Look at the drawings to facilitate understanding of the present invention. Figure 1 shows an example of the overall system structure according to the present invention, Figure 2 shows an example of the entire system flow diagram for a method for determining the starting point of the multi-modal command using the hand gesture and gestures and obtaining control authority. 3 illustrates an example of a signal waveform, an analysis result of a frequency spectrum, and an end point detection result of a clapping sound according to the present invention, and FIG. 4 shows acquisition of audio and gesture modality information that can be simultaneously acquired from a clapping according to the present invention. An example is shown. A specific embodiment according to the present invention will be described.

<Example>

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A preferred embodiment of the present invention will now be described with reference to the drawings. 1 shows a block diagram of an entire system according to the invention. Looking at the method of recognizing the clasp according to the present invention through Figure 1, the user performs a clap, the step of detecting the sound generated when clapping with a microphone or a stereo microphone, such as a microphone or a stereo microphone Extracting the feature of the user's sound signal from the signal obtained from the detected gesture, and passing the gesture recognition unit for comparing the feature of the extracted finger signal stored in a server or a memory. Comparing the characteristics of the handwriting sound detected through the handwriting recognition unit with the handwriting sound stored in the memory, the start point determination method of the multi-modal command consisting of the step of determining the command starting point and the step of determining the starting point The stage where the actor gains control of the system A judgment is made as a starting point and a control method of obtaining multimodal commands.

Referring to the method of recognizing a gesture video signal according to the present invention with reference to Figure 1, when the user performs a gesture of clapping, using the camera or a stereo camera to take a variety of gestures to take a gesture image taken Detecting a feature from a gesture video signal captured by the camera or a stereo camera, extracting features of various gesture video signals taken at the time of clapping, and storing the features of the extracted chess image signal and the memory A gesture gesture for comparing and recognizing features of a gesture gesture image signal is performed, and a gesture start point is determined through the gesture gesture recognition unit, and a gesture gesture actor acquires system permission.

Typically, clapping is accompanied by a clapping gesture and a clapping gesture at the same time, and each person has a different physique, weight, hand size, and hand shape, and thus the hands, arms, and body of the clapping action that occur. This moving shape also has certain characteristics.

Features of the clasp and gesture image signal according to the present invention are generally configured as a recognition / reference model and stored in a database of a server or a memory of a device for determining a starting point of a multi-modal command and obtaining control authority.

End point in the time domain using short-term energy and zero crossing rate to remove the background noise included in the handheld sound detected by the microphone or stereo microphone to improve the end point detection performance of the handwriting sound, and to increase the accuracy of the judgment in recognition. Detection method, endpoint detection method using frequency spectrum, endpoint detection method based on MFB (Mel-frequency Filter Bank), spectral subtraction method and spectrum energy method to subdivide sound band into individual bands and analyze the spectrum by divided bands It is configured to remove background noise by using the same end point detection method, the end point detection method using the FFT spectrum and mel-spectrum based entropy.

In order to recognize the hand gesture image at the time of the clap gesture, basically, a multi-dimensional image of a hand-shaped image signal is recognized from a video signal photographed by a camera or the like and a step of comparing and analyzing the movements of the hand and arm is performed. And extracting the hand image signal from the image signal photographed by the camera, and tracking and analyzing the pattern motion of the multidimensional shape of the arm by reconstruction of the multidimensional human model.

Recognizing a hand shape pattern that is robust to angle and self occlusion using the multi-dimensional hand pose estimation, and recognizing a corresponding gesture by comparing and analyzing the movement of the shape pattern of the hand.

The clasp recognition technology according to the present invention includes a process of removing background noise included in a sound, detecting an effective clapping sound section, pre-emphasis, and windowing as a preprocessing step for an input sound signal. . The detection of the effective clapping sound section uses time-base features such as energy of the short section of the signal and ZCR (Zero Crossing Rate). Perform the vector extraction process. In the present invention, the feature vector may use LPC (Linear Predictive Coefficient), LPCC (Linear Predictive Cepstral Coefficient), MFCC (Mel Frequency Cepstral Coefficient), PLP (Perceptual Linear Prediction), Rasta (Relative Spectral Transform) -PLP. The feature vector acquired in the feature extraction step is subjected to the similarity measurement and the step of recognizing the pattern of the clapping sound.

Techniques for clasp recognition include rule-based recognition, HMM-based recognition, and Gaussian Mixture Model (GMM), which is a mixture of Gaussian density. In the present invention, the clapping sound recognition includes an overall process using a microphone and a stereo microphone. In order to realize the noise recognition system that is resistant to the noise environment, it is possible to improve the end point detection performance of the noise by removing the background noise included in the noise at the preprocessing stage, and use it to minimize the performance degradation of the recognition system. . Accordingly, the present invention subdivids the end point detection method in the time domain using the short-term energy and the zero crossing rate, the end point detection method using the frequency spectrum, the end point detection method based on the MFB (Mel-frequency Filter Bank), and the voice band into individual bands. The spectral subtraction method for analyzing the spectrum for each divided band, a method using band energy, an FFT spectrum method, and an endpoint detection method using mel-spectrum-based entropy may be applied. In order to apply this, FIG. 3 shows an example of applying an analysis result of a signal waveform and a frequency spectrum of a clapping sound and an end point detection result using mel-spectrum-based entropy.

Gesture image recognition technology according to the present invention basically recognizes the multi-dimensional hand shape in order to recognize the hand gesture when moving the hand to generate hand gestures, and analyzes and recognizes the shape of the movement of the hand, arm In order to extract the hand gesture video signal generated when clapping, and to track and analyze the multidimensional motion of the arm, it is necessary to reconstruct the multidimensional human model. That is, by tracking the body components obtained through the multidimensional human model reconstruction, it is possible to track the motion image of the hand and arm, and extract the hand region. Recognizing the shape of the hand robust to angle and self occlusion using the estimation of the multi-dimensional hand motion image, and comparing the movement of the hand image signal to recognize the gesture image signal.

As a method for recognizing a gesture video signal, a method of recognizing motion by attaching an optical marker to an important part of the body and tracking the traces of the markers from an image signal input by a camera, and extracting feature points of the body The method recognizes the movement of the feature point by the pattern of time, expresses the human body shape model, and the whole body appearance using various parameters. You can use the method of recognizing the movement by interpreting the change. In general, in order to stabilize the recognition result, temporal continuity must be considered. Motion energy image (MEI), which is a binary image representing a change in temporal position of motion, and motion history image (MHI), which expresses temporal lapse information of motion, etc. Using the HMM (Hidden Markov Model) recognition method that expresses the continuous posture as a state change by using.

In addition, a single erosion and morphology calculation is performed to remove noise included in a video signal input through a camera or a stereo camera. dilation) or the like using a morphology operation.

The method of effectively extracting and recognizing body poses and motions by extracting feature values from a gesture video signal input through a camera or a stereo camera is a method of extracting features from global motion information and the body, such as head, hands and feet. It can be divided into a method of extracting a feature from partial motion information meaningful to a specific part. The gesture or motion feature extraction method may use the Motion Energy Image (MEI), Motion History Image (MHI), Ross Cutler's method, a cardboard model, and the like.

As the recognition technology of the gesture video signal is applicable to the overall application of the image-based gesture recognition technology including a 2D or 3D gesture video signal, the gesture image recognition technology in the present invention is a single / multi camera or a stereo camera. And the overall process of conventional image recognition techniques using the same. 4 is a diagram illustrating a recognition time point of a gesture image of a hand gesture. A gesture image recognition technology includes a technical configuration of detecting a moment when two hands touch each other using recognition of the various image signals, and interpreting and recognizing it in a multi-dimensional manner. have.

According to the present invention, when the user performs one or n or more touches, the sound and video signals generated by each touch can be used for system command recognition in a multi-modal manner, thereby increasing the accuracy of recognition.

The present invention provides a method for detecting the starting point of an efficient instruction using multi-modality in a multi-modal application system, and performs one or n lines of clapping, which can simultaneously acquire audio modality and gesture modality information. By recognizing these modality information and granting control by identifying users, system command misrecognition due to misunderstanding or unintentional gestures expressed in user's consciousness / unconsciousness can be minimized. Application is possible.

1 shows an example of the overall system structure according to the present invention.

2 is an exemplary diagram of an overall system flow for a method of determining a starting point of a multimodal command using a clap and a gesture and obtaining control authority

3 is an exemplary view of a signal waveform, a frequency spectrum analysis result, and an endpoint detection result for a clapping sound according to the present invention.

4 is an example of acquiring audio and gesture modality information that can be simultaneously acquired from a clasp according to the present invention.

Claims (7)

In the method of determining the starting point of a multi-modal instruction, A user performing a clap gesture one or more times, detecting a sound generated by the clap with a microphone or a stereo microphone, and capturing a clap gesture image using a camera or a stereo camera; Extracting a feature of a sound signal possessed by a user from a sound signal obtained from a hand-held sound detected by a microphone or a stereo microphone, and extracting a feature of the gesture video signal from the captured gesture image; Comparing and recognizing the characteristics of the extracted gesture signal and the characteristics of the extracted gesture image signal and the characteristics of the extracted gesture image signal and the characteristics of the gesture image signal stored in the server or memory Recognizing a sound and gesture video signal through a hand gesture recognition unit; And A method for determining a starting point of a multi-modal command using the clapping sound and the gesture image, comprising: determining a command starting point through the clapping sound recognition unit and the clapping gesture recognizing unit. In the method of determining the starting point of the multi-modal instruction and obtaining the control authority, A user performing a clap gesture one or more times, detecting a sound generated by the clap with a microphone or a stereo microphone, and capturing a clap gesture image using a camera or a stereo camera; Extracting a feature of a sound signal possessed by a user from a sound signal obtained from a hand gesture detected by a microphone or a stereo microphone, and extracting a feature of the gesture image signal from the captured gesture image; Comparing and recognizing the characteristics of the extracted gesture signal and the characteristics of the extracted gesture image signal and the characteristics of the extracted gesture image signal and the characteristics of the gesture image signal stored in the server or memory Recognizing a sound and gesture video signal through a hand gesture recognition unit; Determining a starting point of a command through the hand gesture recognition unit and the hand gesture recognition unit; And Determining the starting point of the command, and comparing and analyzing the characteristics of the hand gesture and the gesture image signal stored in the memory, the user acquires the authority to control the system. Way. The method according to claim 2, In order to remove the background noise included in the handheld sound detected by the microphone or the stereo microphone, the end point detection method in the time domain using the short-term energy and the zero crossing rate, the end point detection method using the frequency spectrum, MFB-based end point detection method, subdivided sound band into individual bands, spectrum subtraction method to analyze the spectrum by divided band, band energy method, FFT spectrum and Mel-spectrum based end point detection method Method for acquiring a starting point of a multi-modal command and acquiring control authority by using a clap and gesture image, the method including removing background noise. The method according to claim 2, Extracting the feature of the gesture video signal is performed using a MEI, MHI, Los Cutler method and a cardboard model. The method according to claim 2, In the step of recognizing the gesture image signal, an optical marker is attached to an important part of the body, and a method of recognizing motion by tracking the trajectories of the markers from the gesture image signal inputted by the camera, extracting the feature points of the body It recognizes the motion of the human body by the image signal with respect to time, uses the shape model of the human body, and expresses the overall shape of the human body using various parameters, and analyzes the change to recognize the motion. Method for acquiring a starting point of a multi-modal command and acquiring control authority by using a clapping sound and a gesture image. The method according to claim 2, In the recognizing the hand gesture image signal, the step of comparing and recognizing the hand image signal is basically multi-dimensional in order to recognize the hand gesture image during the hand gesture; Comparing and analyzing the shape of the movement of the hand and the arm; And And extracting hand image signals and tracking and analyzing multi-dimensional shape movements of the arm by reconstruction of the multi-dimensional human model. The method according to claim 2, In the step of extracting the feature from the clasp and clasp gesture video signals, a sound and a video signal generated when each clap is performed when the user performs a clasp one or more times n are used for recognition in a multi-modal manner. Method for acquiring a starting point of a multi-modal command and acquiring control authority by using a clapping sound and a gesture image.

Images