CN107241643A - A kind of multimedia volume adjusting method and system - Google Patents

Abstract

The invention discloses a multimedia volume adjustment method and system. The method includes: acquiring a depth image of a user gesture; performing image segmentation of a hand target area on the depth image to obtain a segmented target area image; performing gesture edge contour detection using a Sobel operator according to the target area image, Extract edge parameter features; utilize DAG-SVMs classifier to obtain gesture classification results according to the edge parameter features; adjust the volume level of multimedia according to the gesture classification results, and different gesture classifications correspond to different volume levels. With the method or system of the present invention, it is only necessary to connect a multimedia device to a peripheral device to acquire a depth image, and then the volume of the multimedia itself can be controlled through gesture operations. The control process uses common digital gestures in daily life, and the entire control process The external equipment is simple, the operability is strong, and the convenience is high.

Description

Method and system for adjusting multimedia volume

technical field

The invention relates to the field of intelligent control, in particular to a multimedia volume adjustment method and system.

Background technique

Multimedia is a combination of various media, generally including text, sound and image and other media forms. Currently, there are two commonly used methods for controlling the volume of multimedia: one is manual adjustment through the volume button of the multimedia device; the other is remote adjustment through a remote control device. Compared with the way of manually adjusting the volume, the remote adjustment improves the convenience to a certain extent. However, different multimedia devices require specific remote control devices to operate, and not all multimedia devices can use remote control devices to remotely adjust the volume. In the PowerPoint demonstration in the classroom, the remote control device can be used to perform remote control of the page, but the volume of the video or other multimedia can only be adjusted by the mouse. Therefore, the existing multimedia volume adjustment method has low convenience.

Contents of the invention

The object of the present invention is to provide a multimedia volume adjustment method and system to improve the convenience of multimedia volume adjustment.

To achieve the above object, the present invention provides the following scheme:

A method for adjusting multimedia volume, said method comprising:

Obtaining a depth image of the user gesture, the depth image including the spatial coordinates of the hand joints in the user gesture;

Carrying out image segmentation of the hand target area on the depth image to obtain a segmented target area image;

According to described target area image, utilize Sobel operator to carry out gesture edge contour detection, extract edge parameter feature;

Utilize DAG-SVMs classifier to obtain gesture classification result according to described edge parameter feature;

The volume level of the multimedia is adjusted according to the gesture classification result, and different gesture classifications correspond to different volume levels.

Optionally, the acquiring the depth image of the user gesture specifically includes:

According to the user's gesture, a Kinect camera is used to obtain a depth image of the user's gesture.

Optionally, the image segmentation of the hand target area on the depth image specifically includes:

Draw an image depth histogram according to the depth image;

Determining a plurality of undetermined threshold intervals according to the depth value trough in the histogram;

Use Kinect bone tracking technology to determine the final threshold interval;

The image segmentation of the target region of the hand is performed according to the final threshold interval to obtain the segmented image of the target region.

Optionally, before performing the image segmentation of the hand target region on the depth image, it also includes:

Perform binarization processing on the depth image to obtain a processed depth image.

Optionally, the edge parameter features include edge parameter feature Hu moments and contour edge parameter feature length moments, wherein,

The gesture edge features of Hu moments and length moments are used as the features of the digital gesture image, and the edge parameter feature Hu moment and the contour edge parameter feature length moment are extracted.

A multimedia volume adjustment system, said system comprising:

A depth image acquisition module, configured to acquire a depth image of a user gesture, the depth image including the spatial coordinates of the hand joints in the user gesture;

An image segmentation module, configured to perform image segmentation of the hand target area on the depth image to obtain a segmented target area image;

Edge parameter feature extraction module, for according to described target area image, utilize Sobel operator to carry out gesture edge profile detection, extract edge parameter feature;

Gesture classification result acquisition module, for utilizing DAG-SVMs classifier to obtain gesture classification result according to the edge parameter feature;

The volume adjustment module is configured to adjust the volume level of the multimedia according to the gesture classification result, and different gesture classifications correspond to different volume levels.

Optionally, the depth image acquisition module is a Kinect camera, configured to acquire the depth image of the user's gesture according to the user's gesture in front of the camera.

Optionally, the image segmentation module specifically includes:

a histogram drawing unit, configured to draw an image depth histogram according to the depth image;

An undetermined threshold interval determination unit, configured to determine a plurality of undetermined threshold intervals according to the depth value trough in the histogram;

The final threshold interval determination unit is used to determine the final threshold interval by utilizing the Kinect skeleton tracking technology;

The graphic segmentation unit is configured to perform image segmentation of the target region of the hand according to the final threshold interval, and obtain a segmented image of the target region.

Optionally, the system also includes:

The binarization processing module is configured to perform binarization processing on the depth image before performing image segmentation of the hand target region on the depth image to obtain a processed depth image.

Optionally, the edge parameter feature includes an edge parameter feature Hu moment and a contour edge parameter feature length moment, and the edge parameter feature extraction module is used to use the gesture edge feature of the Hu moment and the length moment as a feature of a digital gesture image to extract Edge parameter characteristic Hu moment and contour edge parameter characteristic length moment.

According to the specific embodiments provided by the invention, the invention discloses the following technical effects:

The entire adjustment process only requires the multimedia device to be connected to a peripheral device to collect depth images, and the volume of the multimedia itself can be controlled through gesture operations. The control process uses digital gestures that are common in life, so the external devices are simple in the entire control process. Strong operability and high convenience. And it has changed the previous mechanical interaction mode that the volume of multimedia can only be communicated through manual contact with the mouse and keyboard, and realized the medium and long-distance non-contact control of multimedia by human gestures.

In the specific implementation, the gesture recognition process is based on the depth information of the bone depth carried by Kinect, which avoids the interference of light intensity to the color information in the recognition process based on color information, so that even in the case of low brightness and even dark conditions It can also recognize gestures and reduce the external environment restrictions in the gesture recognition process; in the digital gesture algorithm, by improving the strategy structure of the DAG-SVMs classifier, the recognition algorithm is optimized, the recognition accuracy of the algorithm is improved, and the Recognition stability of the entire interactive system.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required in the embodiments. Obviously, the accompanying drawings in the following description are only some of the present invention. Embodiments, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without paying creative labor.

Fig. 1 is the flow chart of the embodiment 1 of the multimedia volume adjustment method of the present invention;

Fig. 2 is the depth image acquired in Embodiment 1 of the multimedia volume adjustment method of the present invention;

Fig. 3 is the histogram drawn in embodiment 1 of the multimedia volume adjustment method of the present invention;

Fig. 4 is the segmented target area image in Embodiment 1 of the multimedia volume adjustment method of the present invention;

Figures 5a-5b are comparison diagrams before and after detection using the Sobel operator in Embodiment 1 of the multimedia volume adjustment method of the present invention; wherein Figure 5a is an image before detection, and Figure 5b is an image after detection;

6 is a diagram of gesture classification results in Embodiment 1 of the multimedia volume adjustment method of the present invention;

7a-7e are depth images of all gestures in Embodiment 1 of the multimedia volume adjustment method of the present invention;

8a-8e are segmentation diagrams corresponding to the depth images of all gestures in Embodiment 1 of the multimedia volume adjustment method of the present invention;

9 is a characteristic diagram of edge parameters of all gestures in Embodiment 1 of the multimedia volume adjustment method of the present invention;

10a-10e are diagrams of classification results corresponding to all gestures in Embodiment 1 of the multimedia volume adjustment method of the present invention;

FIG. 11 is a correspondence diagram between volume level and gesture classification in Embodiment 1 of the multimedia volume adjustment method of the present invention;

12 is a flow chart of Embodiment 2 of the multimedia volume adjustment method of the present invention;

FIG. 13 is a classification result diagram of Embodiment 2 of the multimedia volume adjustment method of the present invention;

14 is a schematic diagram of volume adjustment in Embodiment 2 of the multimedia volume adjustment method of the present invention;

Fig. 15 is a structural diagram of the multimedia volume adjustment system of the present invention.

detailed description

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

In order to make the above objects, features and advantages of the present invention more comprehensible, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

Example 1:

FIG. 1 is a flow chart of Embodiment 1 of the multimedia volume adjustment method of the present invention. As shown in Figure 1, the method includes:

Step 101: Obtain a depth image of a user's gesture. The depth image includes the spatial coordinates of the hand joints in the user gesture, as shown in FIG. 2 , which is the depth image obtained in Embodiment 1 of the multimedia volume adjustment method of the present invention;

For example, a Kinect camera may be used to acquire depth images of user gestures. Kinect is a 3D somatosensory camera, which is an XBOX360 somatosensory peripheral peripheral officially announced by Microsoft at the E3 exhibition on June 2, 2009. It is a 3D somatosensory camera (developed code-named "ProjectNatal"), and it introduces functions such as real-time motion capture, image recognition, microphone input, voice recognition, and community interaction. The Kinect camera is used as the device for collecting depth gesture images to effectively recognize depth digital gesture images, taking into account the gesture segmentation based on depth information in the case of poor lighting conditions, and overcoming the problem based on ordinary RGB (red, green, blue) color images The environmental constraints imposed by the capture device on gesture recognition. In the process of gesture image acquisition, firstly, it is set that the plane of the hand is always parallel to the plane of the sensor (camera), and is located at the front of the body, and the acquisition of depth images is effective.

Step 102: Carry out image segmentation of the hand target area on the depth image. Carrying out image segmentation of the hand target area on the depth image to obtain the segmented target area image, the specific process is:

drawing an image depth histogram according to the depth image; the image depth histogram is a histogram drawn according to the image depth, including the distribution of peaks and troughs of depth pixel values in the depth image. As shown in Figure 3, Figure 3 is a histogram drawn in Embodiment 1 of the multimedia volume adjustment method of the present invention;

Determining a plurality of undetermined threshold intervals according to the depth pixel value valleys in the histogram;

Use Kinect bone tracking technology to determine the final threshold interval, which is the ideal depth threshold interval for approaching the target;

The image segmentation of the target region of the hand is performed according to the final threshold interval to obtain the segmented image of the target region. As shown in FIG. 4, FIG. 4 is an image of the target area after segmentation in Embodiment 1 of the multimedia volume adjustment method of the present invention;

Before image segmentation, binarization processing may also be performed on the depth image to obtain a processed depth image. The pixels of the hand target area are processed by image binarization. The entire segmentation process is based on depth information, which effectively avoids the interference caused by light intensity and ensures that gesture images can be recognized even when the external brightness is low.

Step 103: Extract edge parameter features. According to the segmented image of the target area, the Sobel operator is used to detect the gesture edge contour and extract the edge parameter features. Sobel operator is Sobel operator: an important processing method in the field of computer vision. It is mainly used to obtain the first-order gradient of digital images, and is commonly used in edge detection. The Sobel operator is used to extract the entire gesture contour, and the typical edge parameter characteristic Hu moment and contour edge parameter characteristic length moment are used as the recognition features of digital gestures, as shown in Figure 5, Figures 5a-5b are the multimedia volume adjustment of the present invention In the method embodiment 1, the Sobel operator is used to detect the comparison diagram before and after; wherein Fig. 5a is the image before the detection, and Fig. 5b is the image after the detection; the Hu moment is to describe the whole image with a set of simple data (image description amount) , is a set of parameter features for image recognition.

Step 104: Obtain a gesture classification result. Utilize DAG-SVMs (Directed Acyclic Graph Support Vector Machine) classifier according to the edge parameter feature of extraction to obtain gesture classification result, as shown in Figure 6, Fig. 6 is the gesture classification result figure in the multimedia volume adjustment method embodiment 1 of the present invention . DAG-SVMs (Database Availability Group-SupportVectorMchine) is a directed acyclic graph support vector machine, also known as a callable database support vector machine, which is a structural strategy graph with decision-oriented and no closed-loop structure.

In order to use DAG-SVMs as a classifier for classification, it is first necessary to construct a classifier that includes multiple gesture classifications. The specific construction process is as follows:

Obtain the depth images of all gestures, as shown in Figure 7, Figures 7a-7e are the depth images of all gestures in Embodiment 1 of the multimedia volume adjustment method of the present invention;

Obtain target area segmentation diagrams of all gestures, as shown in Figure 8, Figures 8a-8e are the segmentation diagrams corresponding to the depth images of all gestures in Embodiment 1 of the multimedia volume adjustment method of the present invention; wherein Figure 8a is the corresponding segmentation diagram of Figure 7a , FIG. 8b is a segmentation diagram corresponding to FIG. 7b, FIG. 8c is a segmentation diagram corresponding to FIG. 7c, and FIG. 8d is a segmentation diagram corresponding to FIG. 7d.

Utilize the sobel operator to obtain the edge parameter feature of all gestures, as shown in Figure 9, Figure 9 is the edge parameter feature map of all gestures in the embodiment 1 of the multimedia volume adjustment method of the present invention; adopt the Sobel operator to extract the whole gesture profile, As a template for classifier classification training, the typical edge parameter feature Hu moment and the length moment parameter of the contour edge are used as the recognition features of digital gestures, and the classifier DAGSVM (hierarchical classifier) is trained to construct a classification that meets the requirements of five categories device. As shown in FIG. 10, FIGS. 10a-10e are diagrams of classification results corresponding to all gestures in Embodiment 1 of the multimedia volume adjustment method of the present invention.

Step 105: Adjust the multimedia volume level. The volume level of the multimedia is adjusted according to the gesture classification result, and different gesture classifications correspond to different volume levels. As shown in FIG. 11 , FIG. 11 is a correspondence diagram between volume level and gesture classification in Embodiment 1 of the multimedia volume adjustment method of the present invention. Wherein, different digital gestures correspond to different volume levels, and each volume level corresponds to a specific volume value, and the volume value corresponding to each volume level can also be reset according to actual needs.

Gesture interaction is a very important interaction method in human-computer interaction. It mainly detects gestures from computer video images, tracks and recognizes them, so as to understand people's intentions. Gesture recognition is an accurate interpretation of human gestures by computers. However, the reason why the current gesture recognition system is not popular enough is that the real-time and anti-interference performance of the recognition algorithm cannot be guaranteed. This is because the gesture recognition system based on computer vision image processing will be restricted by factors such as illumination, shading, and shadows in actual operation, and changes in each factor will affect the final recognition accuracy.

With the popularity of somatosensory games, the gesture recognition technology based on the somatosensory peripheral 3D camera Kinect is no longer unfamiliar to the public, and now multimedia has become a very important display in the form of reports, conferences, teaching and other activities Methods: Using Kinect to control the multimedia volume, including the volume control of the video content added in the PPT, can simplify the form of multimedia presentation to the greatest extent, make the entire multimedia control process more convenient, and improve the efficiency of the display operator in imparting knowledge.

By studying the gesture recognition technology of depth data, different gestures can be accurately judged, and the impact of light intensity on recognition accuracy can be reduced during the recognition process. The optimized algorithm is used to improve the accuracy of gesture recognition, ensure the real-time performance of gesture recognition, and realize the precise short-range control of the volume of multimedia players by digital gestures.

Example 2:

FIG. 12 is a flow chart of Embodiment 2 of the multimedia volume adjustment method of the present invention; as shown in FIG. 12 , FIG. 12 is a specific flow chart executed according to the multimedia volume adjustment method of the present invention. The output classification result figure is shown in Figure 13, and Figure 13 is the classification result figure of embodiment 2 of the multimedia volume adjustment method of the present invention; it is shown in the figure that the classification result is the first type of gesture, corresponding to the level of adjusting the multimedia volume according to the gesture classification result It is level 1, and the volume level corresponding to the first level volume level in this embodiment is 20; as shown in Figure 14, Figure 14 is a schematic diagram of volume adjustment in Embodiment 2 of the multimedia volume adjustment method of the present invention.

Fig. 15 is a structural diagram of the multimedia volume adjustment system of the present invention. As shown in Figure 15, the system includes:

The depth image acquisition module 1501 is configured to acquire the depth image of the user gesture, the depth image includes the spatial coordinates of the hand joints in the user gesture; the depth image acquisition module 1505 can specifically use a Kinect camera, according to the user in front of the camera Gesture, acquire the depth image of the user's gesture.

The image segmentation module 1502 is used to perform image segmentation of the hand target area on the depth image to obtain the segmented target area image; the image segmentation module 1502 specifically includes:

a histogram drawing unit, configured to draw an image depth histogram according to the depth image;

An undetermined threshold interval determination unit, configured to determine a plurality of undetermined threshold intervals according to the depth value trough in the histogram;

The final threshold interval determination unit is used to determine the final threshold interval by utilizing the Kinect skeleton tracking technology;

The graphic segmentation unit is configured to perform image segmentation of the target region of the hand according to the final threshold interval, and obtain a segmented image of the target region.

Edge parameter feature extraction module 1503, for according to described target region image, utilize Sobel operator to carry out gesture edge profile detection, extract edge parameter feature; Edge parameter feature comprises edge parameter feature Hu moment and profile edge parameter feature length moment, edge parameter The feature extraction module 1503 uses the gesture edge features of Hu moments and length moments as the features of the digital gesture image to extract edge parameter feature Hu moments and contour edge parameter feature length moments.

The gesture classification result acquisition module 1504 is used to obtain the gesture classification result by using the DAG-SVMs classifier according to the edge parameter characteristics;

The volume adjustment module 1505 is configured to adjust the volume level of the multimedia according to the gesture classification result, and different gesture classifications correspond to different volume levels.

The system may further include: a binarization processing module, configured to perform binarization processing on the depth image before performing image segmentation of the hand target region on the depth image to obtain a processed depth image.

Each embodiment in this specification is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to each other. As for the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and for the related information, please refer to the description of the method part.

In this paper, specific examples have been used to illustrate the principle and implementation of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; meanwhile, for those of ordinary skill in the art, according to the present invention Thoughts, there will be changes in specific implementation methods and application ranges. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims (10)

1. a multimedia volume adjustment method, is characterized in that, described method comprises: Obtaining a depth image of the user gesture, the depth image including the spatial coordinates of the hand joints in the user gesture; Carrying out image segmentation of the hand target area on the depth image to obtain a segmented target area image; According to described target area image, utilize Sobel operator to carry out gesture edge contour detection, extract edge parameter feature; Utilize DAG-SVMs classifier to obtain gesture classification result according to described edge parameter feature; The volume level of the multimedia is adjusted according to the gesture classification result, and different gesture classifications correspond to different volume levels. 2. The method according to claim 1, wherein the acquiring the depth image of the user gesture specifically comprises: According to the user's gesture, a Kinect camera is used to obtain a depth image of the user's gesture. 3. The method according to claim 1, wherein the image segmentation of the hand target region to the depth image specifically comprises: Draw an image depth histogram according to the depth image; Determining a plurality of undetermined threshold intervals according to the depth value trough in the histogram; Use Kinect bone tracking technology to determine the final threshold interval; The image segmentation of the target region of the hand is performed according to the final threshold interval to obtain the segmented image of the target region. 4. The method according to claim 1, wherein, before performing the image segmentation of the hand target region on the depth image, further comprising: Perform binarization processing on the depth image to obtain a processed depth image. 5. method according to claim 1, is characterized in that, described edge parameter characteristic comprises edge parameter characteristic Hu moment and contour edge parameter characteristic length moment, wherein, The gesture edge features of Hu moments and length moments are used as the features of the digital gesture image, and the edge parameter feature Hu moment and the contour edge parameter feature length moment are extracted. 6. A multimedia volume adjustment system, characterized in that the system comprises: A depth image acquisition module, configured to acquire a depth image of a user gesture, the depth image including the spatial coordinates of the hand joints in the user gesture; An image segmentation module, configured to perform image segmentation of the hand target area on the depth image to obtain a segmented target area image; Edge parameter feature extraction module, for according to described target area image, utilize Sobel operator to carry out gesture edge profile detection, extract edge parameter feature; Gesture classification result acquisition module, for utilizing DAG-SVMs classifier to obtain gesture classification result according to the edge parameter feature; The volume adjustment module is configured to adjust the volume level of the multimedia according to the gesture classification result, and different gesture classifications correspond to different volume levels. 7. The system according to claim 6, wherein the depth image acquisition module is a Kinect camera, configured to acquire the depth image of the user gesture according to the user gesture in front of the lens. 8. The system according to claim 6, wherein the image segmentation module specifically comprises: a histogram drawing unit, configured to draw an image depth histogram according to the depth image; An undetermined threshold interval determination unit, configured to determine a plurality of undetermined threshold intervals according to the depth value trough in the histogram; The final threshold interval determination unit is used to determine the final threshold interval by utilizing the Kinect skeleton tracking technology; The graphic segmentation unit is configured to perform image segmentation of the target region of the hand according to the final threshold interval, and obtain a segmented image of the target region. 9. The system according to claim 6, further comprising: The binarization processing module is configured to perform binarization processing on the depth image before performing image segmentation of the hand target region on the depth image to obtain a processed depth image. 10. system according to claim 6, is characterized in that, described edge parameter feature comprises edge parameter feature Hu moment and contour edge parameter feature length moment, and described edge parameter feature extraction module is used for utilizing Hu moment and length moment The gesture edge feature is used as the feature of the digital gesture image, and the edge parameter feature Hu moment and the contour edge parameter feature length moment are extracted.