CN114840086A - Control method, electronic device and computer storage medium - Google Patents

Abstract

The embodiment of the application discloses a control method, which is applied to electronic equipment and comprises the following steps: when the non-contact touch control function of the electronic equipment is in an open state, a video sequence corresponding to non-contact touch operation is obtained, touch identification is carried out on the video sequence to obtain a target object and a touch type, a first distance between the target object and the electronic equipment is determined, a touch operation parameter at the first distance is determined, and the electronic equipment is controlled to execute a target function corresponding to the touch type according to the touch operation parameter. The embodiment of the application also provides the electronic equipment and a computer storage medium.

Description

A control method, electronic device and computer storage medium

technical field

The present application relates to a non-contact touch control technology in an electronic device, and in particular, to a control method, an electronic device and a computer storage medium.

Background technique

At present, with the rapid development of smartphones in recent years, people use mobile phones more and more times and scenes. At present, the mainstream human-computer interaction method is mainly touch operation. On this basis, gesture interaction as a new With the continuous development of applications in driving, dining and other scenarios, gesture interaction can realize various controls on the page, such as sliding up and down the page, turning pages, taking pictures, taking screenshots, ending recording, etc.

However, in the control of electronic equipment through changes in gesture types or changes in other parts of the body, only the electronic equipment can be controlled to achieve its own functions, resulting in insufficient control. It can be seen that the existing The non-contact touch control method is not refined enough.

SUMMARY OF THE INVENTION

Embodiments of the present application provide a control method, an electronic device, and a computer storage medium, which can improve the refinement of non-contact touch control performed by the electronic device.

The technical solution of the present application is realized as follows:

An embodiment of the present application provides a control method, and the method is applied to an electronic device, including:

When the non-contact touch control function of the electronic device is in an on state, acquiring a video sequence corresponding to the non-contact touch operation;

Perform touch recognition on the video sequence to obtain the target object and touch type;

determining a first distance between the target object and the electronic device;

determining touch operation parameters at the first distance;

According to the touch operation parameters, the electronic device is controlled to execute the target function corresponding to the touch type.

The embodiment of the present application provides an electronic device, including:

an acquisition module, configured to acquire a video sequence corresponding to a non-contact touch operation when the non-contact touch control function of the electronic device is on;

a processing module for performing touch recognition on the video sequence to obtain a target object and a touch type;

a first determining module, configured to determine a first distance between the target object and the electronic device;

a second determining module, configured to determine touch operation parameters at the first distance;

The control module is configured to control the electronic device to execute the target function corresponding to the touch type according to the touch operation parameter.

The embodiment of the present application provides an electronic device, including:

A processor and a storage medium storing instructions executable by the processor, the storage medium relying on the processor to perform operations through a communication bus, and when the instructions are executed by the processor, perform one or more implementations above The control method described in the example.

Embodiments of the present application provide a computer storage medium storing executable instructions. When the executable instructions are executed by one or more processors, the processors execute the control described in one or more embodiments. method.

Embodiments of the present application provide a control method, an electronic device, and a computer storage medium, including: when a non-contact touch control function of the electronic device is in an enabled state, acquiring a video sequence corresponding to a non-contact touch operation, The touch recognition is performed in sequence, the target object and the touch type are obtained, the first distance between the target object and the electronic device is determined, the touch operation parameters at the first distance are determined, and the electronic device is controlled to execute according to the touch operation parameters. The target function corresponding to the touch type; that is, in the embodiment of the present application, in the realization of the non-contact touch control function of the electronic device, the first distance between the target object and the electronic device is determined, and then the first distance is determined. Touch operation parameters at a distance, and then control the electronic device to execute the target function corresponding to the touch type according to the touch operation parameters. In this way, for the same touch type, different first distances can determine different touch operation parameters. Then, the electronic device can realize the response to the touch type according to the distance between the target object and the electronic device. Compared with the existing control of the target function corresponding to the touch type, the same touch type is used to utilize different distances. The touch operation parameters under the control are different, which makes the non-contact touch control more refined.

Description of drawings

1 is a schematic flowchart of an optional control method provided by an embodiment of the present application;

2 is a schematic flowchart of a page control method in the related art;

3a is a schematic diagram of example 1 of an optional gesture frame provided by an embodiment of the present application;

3b is a schematic diagram of Example 2 of an optional gesture frame provided by an embodiment of the present application;

3c is a schematic diagram of Example 3 of an optional gesture frame provided by an embodiment of the present application;

3d is a schematic diagram of Example 4 of an optional gesture frame provided by an embodiment of the present application;

4 is a schematic diagram of Example 1 of an optional control method provided by an embodiment of the present application;

5 is a schematic diagram of Example 2 of an optional control method provided by an embodiment of the present application;

6a is a schematic diagram of Example 5 of an optional gesture frame provided by an embodiment of the present application;

6b is a schematic diagram of Example 6 of an optional gesture frame provided by an embodiment of the present application;

6c is a schematic diagram of Example 7 of an optional gesture frame provided by an embodiment of the present application;

FIG. 7a is a schematic diagram of the arrangement of example 1 of an optional screen provided by an embodiment of the present application;

FIG. 7b is a schematic diagram of the arrangement of example 2 of an optional screen provided by an embodiment of the present application;

FIG. 7c is a schematic diagram of the arrangement of Example 3 of an optional screen provided by an embodiment of the present application;

FIG. 8 is a schematic structural diagram of an optional electronic device provided by an embodiment of the present application;

FIG. 9 is a schematic structural diagram of another optional electronic device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application.

An embodiment of the present application provides a control method, which is applied to an electronic device. FIG. 1 is a schematic flowchart of an optional control method provided by an embodiment of the present application. As shown in FIG. 1 , the control method can be include:

S101: When the non-contact touch control function of the electronic device is in an enabled state, acquire a video sequence corresponding to the non-contact touch operation;

FIG. 2 is a schematic flowchart of a page control method in the related art. As shown in FIG. 2 , taking gestures to control a page as an example, the page control method may include:

S201: Turn on the front camera;

S202: Snap a picture;

S203: continuous gesture detection;

S204: gesture judgment;

S205: an operation in response to a gesture.

The electronic device turns on the front camera, uses the front camera to capture a picture containing gestures, and after capturing the picture, continuously detects the gesture in the picture, obtains the gesture and judges the gesture, and obtains the corresponding gesture of the gesture after the judgment. The operation, for example, the upward movement operation, and finally, the operation in response to the gesture, however, in this way, only the electronic device can be controlled to realize its own function, resulting in an insufficiently refined control.

In order to improve the refinement of the non-contact touch control, the embodiment of the present application provides an optional control method. First, when the non-contact touch control function of the electronic device is in an on state, that is, the electronic device is turned on Non-contact touch control function. At this time, the front camera of the electronic device is turned on, which is used to capture the front of the screen of the electronic device. When the target object is detected through the captured picture, a video sequence is obtained to obtain a non-contact A video sequence corresponding to the touch operation, wherein the video sequence includes a plurality of consecutive image frames.

The above touch operations may include a slide-up operation, a slide-down operation, a long-press operation, a page-turning operation, a page-turning operation, and a click operation, which are not specifically limited in this embodiment of the present application.

S102: Perform touch recognition on the video sequence to obtain the target object and touch type;

When a video sequence of a non-contact touch operation is acquired, touch recognition needs to be performed on the video sequence, so that the target object and touch type included in the video sequence can be obtained.

The above-mentioned target object may be a body part of a human body, such as a hand, a head, an eye, etc., which is not specifically limited in this embodiment of the present application.

For the target object, the user can use gestures to control the electronic device to perform the target function corresponding to the touch type corresponding to the gesture, or use the change of the head, for example, use the left and right shaking of the head to control the electronic device to perform the movement of the head. The target function corresponding to the touch type corresponding to the shaking left and right can also be used, and the change of the eyes can also be used. For example, the blink action is used to control the electronic device to execute the target function corresponding to the touch type corresponding to the blink action. Specific restrictions.

Based on the above description of the touch operation, it can be seen that the touch type can include sliding type, page turning type, click type, and long press type, among which, the up-swipe operation and the slide-down operation belong to the slide type, the long-press operation is a long-press type, and the up-swipe operation is a long-press type. The page turning operation and the page turning down operation belong to the page turning type, and the click operation belongs to the click type, which is not specifically limited in this embodiment of the present application.

In an optional embodiment, S102 may include:

Perform touch recognition on the video sequence to obtain the target object, the bounding box of the target object and the confidence value of the touch operation of the target object;

The touch type is determined based on the confidence value of the touch operation of the target object.

In the touch recognition of video sequences, the target object, the bounding box of the target object and the confidence value of the touch operation of the target object can be obtained, wherein the bounding box represents the boundary information of the target object, and the boundary information of the target object refers to is the contour information of the target object, wherein the contour information may include the shape of the contour, the area of the contour, and the size information of the contour line; the above-mentioned bounding box is a rectangular box wrapping the above-mentioned target object, and the bounding box includes the height of the border and the width of the border.

That is to say, through the touch recognition of the video sequence, not only the target object, but also the boundary information of the target object can be obtained. At the same time, by matching the gesture category of the target object, the touch operation of the target object can be obtained by matching, here , it should be noted that the touch operation of one or more target objects can be matched, and the confidence value of the touch operation of the target object will also be obtained in the touch recognition, that is, the confidence degree of the touch operation of the target object value, in order to determine the touch type, the touch type can be determined based on the confidence value of the touch operation of the target object.

It is understandable that when only one touch operation of a target object is obtained by matching, the type of the touch operation of the matched target object is directly determined as the touch type; for the touch operation of more than one target object obtained by matching , in an optional embodiment, determining the touch type based on the confidence value of the touch operation of the target object, including:

The type of the touch operation corresponding to the maximum value of the confidence value of the touch operation of the target object belongs to is determined as the touch type.

Among them, the confidence value of each touch operation of the target object can reflect the credibility of each touch operation. The larger the confidence value, the higher the credibility of the touch operation. Therefore, here, the target object is The maximum value of the confidence value of the touch operation is selected, and the type of the touch operation corresponding to the maximum value is determined as the touch type. In this way, the touch type can be determined more accurately by using the confidence value. The touch type can be selected, so as to realize the precise control of electronic equipment.

S103: Determine a first distance between the target object and the electronic device;

Here, after the target object is obtained by touch recognition, the first distance between the target object and the electronic device needs to be further determined, wherein, it should be noted that the first distance here may be between the target object and the screen of the electronic device The first distance can also be the first distance between the target object and the backplane of the electronic device, or the first distance between the target object and the image sensor of the electronic device. Specific restrictions.

Understandably, if each image frame in the video sequence obtained by using the front camera also corresponds to a depth image, then the first distance between the target object and the electronic device can be determined according to the depth image, and the first distance between the target object and the electronic device can also be determined according to the depth image. The boundary information of the object is used to determine the first distance between the target object and the electronic device, which is not specifically limited in this embodiment of the present application.

In order to determine the first distance between the target object and the electronic device through the boundary information of the target object, since each image frame in the video sequence corresponds to a boundary information, here, according to the target object in each image frame The first distance between the target object and the electronic device can also be determined according to the boundary information of a target object in the boundary information of the target object in each image frame to determine the first distance between the target object and the electronic device. The distance, here, is not specifically limited in this embodiment of the present application.

In order to determine the first distance between the target object and the electronic device according to the boundary information of the target object in each image frame, in an optional embodiment, S103 may include:

Select the boundary information of a target object from the boundary information of the target object in each image frame of the video sequence;

The first distance between the target object and the electronic device is determined according to the boundary information of the selected target object.

Here, after the boundary information of the target object in each image frame of the video sequence is obtained through touch recognition, the boundary information of a target object is selected from the boundary information of the target object in each image frame, and the boundary information of a target object is selected according to the selected boundary information. The boundary information of the target object is used to determine the first distance between the target object and the electronic device.

In an optional embodiment, when the boundary information is represented by a bounding box, the first distance between the target object and the electronic device is determined according to the selected boundary information of the target object, including:

determining a second distance between each boundary of the bounding box and a corresponding edge of the screen of the electronic device;

determining the target boundary according to the second distance between each boundary and the corresponding edge of the screen;

According to the length value of the target boundary, the relationship between the preset boundary length value and the distance between the target object and the electronic device is used to determine the first distance between the target object and the electronic device.

Understandably, when the boundary information is represented by a bounding box, according to the selected bounding box of the target object, in determining the first distance between the target object and the electronic device, first determine the difference between each boundary of the bounding box and the corresponding edge of the screen. Whether the bounding box locks the complete target object, that is, whether the target object is cut off by the screen, can be determined by judging the second distance between each boundary and the corresponding edge of the screen.

After the second distance between each boundary of the bounding box and the corresponding edge of the screen is determined, it can be determined whether the target object is cut off by the screen according to the second distance between each boundary and the corresponding edge of the screen. Based on this, from A boundary selected from each boundary is the target boundary, and the length value of the target boundary is the width or height value available for the target boundary when the target object is not truncated or partially truncated. A first distance between the target object and the electronic device is determined.

It should be noted that two boundaries of the target boundary can be selected, for example, both the width value and the height value are selected, and the width and height values of the bounding box are used to determine the first distance between the target object and the electronic device.

In this way, when the boundary is filtered out, the target boundary can be used to determine the first distance between the target object and the electronic device, and then the touch operation parameters at the first distance can be determined.

In order to select the target object that is not truncated or partially truncated to obtain the available target boundary, so as to improve the accuracy of the first distance between the target object and the electronic device, in an optional embodiment, according to The second distance between each boundary of the bounding box and the corresponding edge of the screen of the electronic device, determining the target boundary, including:

When the second distance between each boundary and the corresponding edge of the screen is greater than the preset threshold, select a boundary from each boundary;

When only one of the second distances between each boundary and the corresponding edge of the screen has a second distance between the boundary and the corresponding edge of the screen that is less than or equal to the preset threshold, select the second distance from each boundary to the corresponding edge of the screen The boundary is less than or equal to the preset threshold;

The selected boundary is determined as the target boundary.

That is to say, the second distance between each boundary and the corresponding edge of the screen is compared with the preset threshold. If both are greater than the preset threshold, it means that the target object in the bounding box is not cut off by the screen. Therefore, One boundary can be selected from each boundary as the target boundary; if there is only one boundary and the second distance between the corresponding edge of the screen is less than or equal to the preset threshold, it means that the target object in the bounding box is truncated by the screen, and although The target object is truncated, but the boundary corresponding to the boundary where the second distance between the corresponding edges of the screen is less than or equal to the preset threshold can reflect the size of the target object. Therefore, the second distance between the corresponding edges of the screen can be selected. The boundary whose distance is less than or equal to the preset threshold is used as the target boundary, and the length value of the target boundary is used to determine the first distance between the target object and the electronic device.

In addition, if both are less than or equal to the preset threshold, it means that the target object in the bounding box is cut off by the screen, and the height and width values in the bounding box cannot reflect the size of the target object, so the bounding box cannot be used. The length value and/or the width value are used to calculate the first distance between the target object and the electronic device. At this time, prompt information can be generated for prompting that the target object does not conform to the control range of the electronic device.

The following describes the bounding box by taking the target object as the hand and the bounding box as the gesture box as an example. FIG. 3a is a schematic diagram of Example 1 of an optional gesture box provided by the embodiment of the present application, as shown in FIG. 3a , The second distance between each boundary of the gesture frame and the corresponding edge of the screen is greater than the preset threshold, so the gesture is not cut off by the screen, and any boundary in the gesture frame can be selected as the target boundary to determine the hand and the electronic device. The first distance between the Among the distances, there is only one width boundary and the second distance between the corresponding edge of the screen is less than or equal to the preset threshold. Therefore, the hand is cut off by the screen, but the width boundary in the gesture frame can be selected as the target boundary to determine the distance between the hand and the electronic device. Figure 3c is a schematic diagram of a third example of an optional gesture frame provided by the embodiment of the application, as shown in Figure 3c, the second distance between each boundary of the gesture frame and the corresponding edge of the screen There is only one height boundary and the second distance between the corresponding edge of the screen is less than or equal to the preset threshold, so the hand is cut off by the screen, but the height boundary in the gesture frame can be selected as the target boundary to determine the distance between the hand and the electronic device 3d is a schematic diagram of Example 4 of an optional gesture frame provided by the embodiment of the application. As shown in Figure 3d, in the second distance between each boundary of the gesture frame and the corresponding edge of the screen The second distance between two borders and the corresponding edge of the screen is less than or equal to the preset threshold, which are the height border and the width border respectively. Therefore, the hand is cut off by the screen. In this case, the border in the gesture frame cannot be selected as the target border. The first distance between the hand and the electronic device is determined, and prompt information can be generated for prompting that the target object does not conform to the control range of the electronic device.

In an optional embodiment, according to the length value of the target boundary, the first distance between the target object and the electronic device is calculated by using the relationship between the preset boundary length value and the distance between the target object and the electronic device ,include:

When the target boundary is a width boundary, the first distance between the target object and the electronic device is calculated by using the relationship between the preset boundary width value and the distance between the target object and the electronic device according to the length value of the target boundary.

Wherein, the relationship between the boundary width value and the distance between the target object and the electronic device is pre-stored in the electronic device. Then, after the width value of the target boundary is known, the preset boundary width can be used according to the target boundary width value. The relationship between the value and the distance between the target object and the electronic device is used to calculate the first distance between the target object and the electronic device.

Understandably, there is a certain relationship between the boundary width value and the distance between the target object and the electronic device. Using this relationship, the first distance between the target object and the electronic device can be calculated, taking the target object as the hand and the bounding box as Taking the gesture frame as an example, the relationship between the border width value and the distance between the target object and the electronic device can be determined as follows:

When the distance between the user's hand and the electronic device is recorded as D1, the width of the gesture frame is W1; when the distance between the user's hand and the electronic device is recorded as D2, the width of the gesture frame is W2; therefore, when the distance is D, the current gesture frame W can be calculated as:

W=W1+(D-D1)(W2-W1)/(D2-D1) (1)

It can be determined that the relationship between the border width value and the distance from the target object to the electronic device can be:

D=(W-W1)(D2-D1)/(W2-W1)+D1 (2)

Wherein, the above W1 and W2 are both the width values of the gesture frame when the hand is not cut off by the screen.

In an optional embodiment, according to the length value of the target boundary, the first distance between the target object and the electronic device is calculated by using the relationship between the preset boundary length value and the distance between the target object and the electronic device ,include:

When the target boundary is a height boundary, the first distance between the target object and the electronic device is obtained by calculating the relationship between the preset boundary height value and the distance between the target object and the electronic device according to the length value of the target boundary.

Wherein, the relationship between the boundary height value and the distance between the target object and the electronic device is pre-stored in the electronic device. Then, after the height value of the target boundary is known, the preset boundary height value can be used according to the boundary height value. The first distance between the target object and the electronic device is obtained by calculating the relationship with the distance between the target object and the electronic device.

It can be understood that there is a certain relationship between the boundary height value and the distance between the target object and the electronic device. Using this relationship, the first distance between the target object and the electronic device can be calculated, taking the target object as the hand and the bounding box as Taking a gesture frame as an example, the relationship between the height of the border and the distance between the target object and the electronic device can be determined as follows:

When the distance between the user's hand and the electronic device is recorded as D1, the width of the gesture frame is H1; when the distance between the user's hand and the electronic device is recorded as D2, the width of the gesture frame is H2; therefore, when the distance is D, the current gesture frame H2 can be calculated as:

H=H1+(D-D1)(H2-H1)/(D2-D1) (3)

Thus, it can be determined that the relationship between the border width value and the distance from the target object to the electronic device can be:

D=(H-H1)(D2-D1)/(H2-H1)+D1 (4)

Wherein, the above H1 and H2 are both height values of the gesture frame when the hand is not cut off by the screen.

In addition to using the boundary information of a target object selected above to determine the first distance between the target object and the electronic device, the distance between the target object and the electronic device can also be determined according to the boundary information of the target object in each image frame. In an optional embodiment, after acquiring the boundary information of the target object in each image frame of the video sequence, the above method further includes:

Determine the distance between the target object in each image frame and the electronic device according to the boundary information in the target object in each image frame;

The average value of the distances between the target object and the electronic device in each image frame is determined as the first distance between the target object and the electronic device.

Understandably, the boundary information of the target object in each image frame is used to determine the distance between the target object in each image frame and the electronic device, and specifically the distance between the target object and the electronic device is determined according to the boundary information of the selected target object. The implementation manner of the first distance between electronic devices is the same, and details are not repeated here.

After determining the distance between the target object and the electronic device in each image frame, an average value algorithm can be used to calculate the average value of the distance between the target object and the electronic device in each image frame, and the average value It is determined as the first distance between the target object and the electronic device.

Further, in order to determine the accurate first distance, in an optional embodiment, the average value of the distance between the target object and the electronic device in each image frame is determined as the distance between the target object and the electronic device. The first distance between, including:

When the absolute value of the difference between any two of the distances between the target object and the electronic device in each image frame is less than or equal to the preset error threshold, the difference between the target object and the electronic device in each image frame is calculated. The average value of the distance is determined as the first distance between the target object and the electronic device.

Here, before calculating the average value, first determine whether the absolute value of the difference between any two of the distances between the target object and the electronic device in each image frame is less than or equal to the preset error threshold, if it is less than or equal to, then It means that the distance between the target object and the electronic device vibrates within the preset error threshold and does not change much. Therefore, the average value can be used to determine the first distance between the target object and the electronic device.

In addition, in an optional embodiment, the above method further includes:

When the absolute value of the difference between any two of the distances between the target object and the electronic device in each image frame is greater than the preset error threshold, according to the preset touch at the standard distance control operation parameters, and control the electronic device to perform the target function corresponding to the touch type.

That is to say, by judging that the absolute value of the difference between any two of the distances between each target object and the electronic device has an absolute value that is greater than the preset error threshold, it means that there is a difference between the target object and the electronic device. The distance jitters and changes greatly, so the first distance between the target object and the electronic device cannot be determined. Therefore, the electronic device can be controlled directly according to the preset touch operation parameters at the standard distance. to perform the target function corresponding to the touch type.

S104: Determine touch operation parameters at the first distance;

After the first distance is determined, the touch operation parameters at the first distance can be determined. Exemplarily, for each touch operation or touch type in the electronic device, the distance and the touch operation parameter are stored. Here, the corresponding relationship can be used to determine the touch operation parameters at the first distance, and the preset parameter calculation formula can also be used to calculate the touch operation parameters at the first distance. This is not specifically limited in the application examples.

In an optional embodiment, the touch operation parameters include any one of the following: the sliding distance of the sliding operation, the sliding speed of the sliding operation, the page turning speed of the page turning operation, the long pressing time of the long press operation, the click operation click frequency.

That is to say, the touch operation of the target object may be a sliding operation, a long press operation or a click operation, etc. The operation parameters for the sliding operation may include the sliding distance and the sliding speed, and the operation parameters for the long pressing operation may include the long pressing time. The operation parameter of the click operation may include the click frequency, which is not specifically limited in this embodiment of the present application.

In order to determine the touch operation parameters at the first distance, in an optional embodiment, S104 may include:

Calculate the sensitivity coefficient at the first distance based on the preset correspondence between the distance and the sensitivity coefficient;

The touch operation parameters at the first distance are determined according to the sensitivity coefficient at the first distance.

Here, after the first distance is determined, the corresponding relationship between the first distance and the sensitivity coefficient is pre-stored in the electronic device, then, based on the corresponding relationship, the sensitivity coefficient at the first distance can be calculated, and then based on the first distance The sensitivity coefficient at a distance determines the touch operation parameters at the first distance.

In an optional embodiment, the touch operation parameters at the first distance are determined according to the sensitivity coefficient at the first distance, including:

Using the sensitivity coefficient at the first distance and the preset touch operation parameters at the standard distance, the touch operation parameters at the first distance are calculated.

It can be understood that the touch operation parameters corresponding to each touch operation are stored in the electronic device. Therefore, in determining the touch operation parameters at the first distance according to the sensitivity coefficient at the first distance, exemplarily , and the product of the sensitivity coefficient at the first distance and the preset touch operation parameter at the standard distance is determined as the touch operation parameter at the first distance.

That is to say, using the sensitivity coefficient at the first distance to determine the touch operation parameters at the first distance, so that the target objects with different distances from the electronic device have different touch operation parameters corresponding to the same touch operation. Therefore, the functions of the non-contact touch control are expanded, so that the user's non-contact touch operation controls the electronic device more refined, and the user experience is improved.

S105: Control the electronic device to execute the target function corresponding to the touch type according to the touch operation parameters.

After the touch operation parameters are determined, the electronic device executes the target function corresponding to the touch type according to the determined touch operation parameters. Among them, the electronic device can control the page of the screen to perform the target function corresponding to the touch type according to the touch operation parameters, and the electronic device can also perform data processing according to the touch operation parameters, such as the processing function of the image, here, this application implements This example is not specifically limited.

For controlling the screen of the electronic device, in an optional embodiment, S105 may include:

According to the touch operation parameters, the controlled object of the screen of the electronic device is controlled to execute the target function corresponding to the touch type;

The controlled object includes any of the following: a page, an interface, and a control; that is, the above control method can not only control the page on the screen, but also control the interface, or realize the control on the screen. The control of a certain control is not specifically limited in this embodiment of the present application.

Taking page sliding as an example, in an optional embodiment, the controlled object on the screen of the electronic device is controlled according to the touch operation parameters to execute the target function corresponding to the touch type, including:

According to the sliding distance of the sliding operation, control the page of the screen to perform the sliding function.

Exemplarily, when the user slides the page through the palm, the electronic device responds to the sliding action of the palm, determines that the operation corresponding to the sliding action is a sliding operation, determines the distance between the palm and the screen, and then determines the sliding operation of the sliding operation at the distance. Distance, the electronic device control page executes the sliding function corresponding to the sliding operation according to the sliding distance of the sliding operation under this distance, so that when the palms with different distances from the screen make a sliding motion, the same sliding motion corresponds to different sliding operations. The sliding distance makes the user's control over the page more refined and improves the user's experience.

The following examples are used to describe the control method in one or more of the above embodiments.

The above control method is described below by taking the target object as the hand and the bounding box as the gesture frame as an example. FIG. 4 is a schematic diagram of Example 1 of an optional control method provided by the embodiment of the present application, as shown in FIG. 4 , The control method may include:

S401: Snap a gesture map;

In S401, the gesture interaction application will turn on the front camera, and capture the gesture map through the front camera.

S402: gesture detection;

S403: Get the gesture frame;

In S401-S402, after the picture is captured, the picture is detected to obtain the gesture category, the gesture confidence and the gesture frame; in the gesture detection, FIG. 5 is an optional control provided by the embodiment of the present application The schematic diagram of the second example of the method, as shown in FIG. 5 , the gesture detection method may include:

S501: Obtain a picture containing gestures;

S502: perform model inference on the picture;

S503: model post-processing;

S504: non-maximum value suppression;

S505: Get the gesture frame.

Here, after obtaining a picture containing gestures, the detection of the picture mainly includes model inference, model post-processing, non-maximum suppression and other operations, so that gestures in a picture can be detected, and each detection result Contains gesture frame, gesture category (referring to the category corresponding to the gesture change of the hand, which corresponds to the touch type) and gesture confidence (referring to the confidence of the touch type corresponding to the gesture category, for example, 0.9), If there are detection results of multiple gesture categories in the picture, the gesture category with the highest confidence is taken as the final detection result.

S404: Calculate the distance between the hand and the screen;

Here, according to the gesture frame, the distance between the user's hand and the screen of the mobile phone is calculated, that is, the distance between the hand and the screen. Among them, the distance between the hand and the screen can be calculated in the following ways:

First, record the width and height of the gesture frame when the distance between a group of hands and the screen is D1. FIG. 6a is a schematic diagram of Example 5 of an optional gesture frame provided by the embodiment of the application. As shown in FIG. 6a, record the user When the distance between the hand and the screen is D1, the width W1 and height H1 of the gesture frame;

Then, record the width and height of the gesture frame when the distance between a group of hands and the screen is D2. FIG. 6b is a schematic diagram of Example 6 of an optional gesture frame provided by the embodiment of the present application. As shown in FIG. 6b, record When the distance between the user's hand and the screen is D2, the width W2 and height H2 of the gesture box;

Finally, FIG. 6c is a schematic diagram of Example 7 of an optional gesture frame provided by the embodiment of the present application. As shown in FIG. 6c, when the distance is D, the calculation method of the current gesture frame W and H is formula (1) and formula (3).

In this way, the reverse can be obtained. When the width and height of the gesture frame are W and H, the distance D between the current hand and the screen can be obtained by formula (2) and formula (4).

It should be noted that the distance between the hand and the screen can be calculated according to the width or height of the gesture frame. When the hand is at the edge of the screen, the hand may be cut off by the screen. In this case, you can calculate the four sides of the gesture frame to the edge of the screen. To determine whether the width or height of the gesture frame is available when the hand is cut off, you need to select an available frame to calculate the distance between the hand and the screen. If neither width nor height are available, the distance between the hand and the screen cannot be calculated.

S405: Update the sensitivity coefficient of the touch type corresponding to the gesture;

In S405, the sensitivity coefficient of the touch type corresponding to the gesture is adjusted according to the distance between the user's hand and the screen of the mobile phone.

S406: Control the sliding distance of the page.

In S406, the sliding distance of the page is calculated according to the sensitivity coefficient of the touch type corresponding to the current gesture.

After the distance between the hand and the screen is calculated by S403, the sensitivity coefficient of the touch type corresponding to the gesture can be adjusted. Assuming that the distance between the hand and the screen is 25cm, the sensitivity coefficient is 1, then the distance is D (10cm<D<60 cm), the sensitivity coefficient K of the touch type corresponding to the updated gesture is:

K=1-0.01*(D-25) (5)

That is to say, when the distance between the hand and the screen is less than 25cm, the sensitivity coefficient of the touch type corresponding to the gesture is improved; when the distance between the hand and the screen is greater than 25cm, the sensitivity of the touch type corresponding to the gesture is reduced.

In calculating the sliding distance on the mobile phone page according to the sensitivity coefficient of the touch type corresponding to the current gesture, taking the touch type corresponding to the gesture as the sliding type as an example, when the preset distance between the hand and the screen is 25cm, every time the hand sends out For sliding operation, the sliding distance of the mobile page is M0, then when the distance between the hand and the screen is D, every time the gesture sends out a sliding operation, the sliding distance M of the mobile page is:

M=M0*K=1-0.01*(D-25) (6)

Finally, the distance D between the hand and the screen can be calculated by substituting the formula (2) or the formula (4), and then the sliding function corresponding to the sliding operation is performed on the control page according to D.

Fig. 7a is a schematic diagram of the arrangement of example 1 of an optional screen provided by an embodiment of the present application. As shown in Fig. 7a, a list of communication records is displayed on the screen. Swing up and down to control the electronic device to slide the screen page; Figure 7b is a schematic diagram of the arrangement of the second example of an optional screen provided by the embodiment of the application. As shown in Figure 7b, when the distance between the user's hand and the screen is At 25cm, each time the four fingers swing up and down, the page slides the distance of one communication record, so the first communication record displayed on the page is "4-17 sister access"; A schematic diagram of the layout of Example 3 of an optional screen, as shown in Figure 7c, when the distance between the user's hand and the screen is 26cm, each time the four fingers swing up and down, the page slides through two communication records. Therefore, the first communication record displayed on the page is "4-16 Li Si called out"; in this way, the distance between the hand and the screen is used to determine the sliding distance of the page corresponding to the up and down swing of each four fingers. Different distances correspond to different sliding distances.

In this example, the sliding distance is intelligently adjusted based on the gesture sensitivity coefficient of gesture detection, the current distance between the hand and the screen can be calculated based on the gesture detection result, the current gesture sensitivity coefficient is adjusted according to the distance between the hand and the screen, and acts on the user to operate the mobile phone The sliding distance control of the page. When the user's hand is far from the screen, a lower sensitivity coefficient can be obtained, and when the user's hand is closer to the screen, a higher sensitivity coefficient can be obtained.

In this way, more refined page control is achieved, and the user can select a comfortable distance to control the page by gestures according to his own usage habits, which improves the user experience of page control by gestures.

In addition, it should be noted that this example can be used not only to adjust the sensitivity coefficient of the sliding distance in gesture interaction, but also to adjust the sliding speed of the sliding operation, the clicking frequency of the clicking operation, the page turning speed of the page turning operation, and the long pressing operation. long-press time, the response time of returning to the previous level operation, the moving distance of controlling the movement operation of the characters in the game, etc.), and can also be applied to the control of the sensitivity coefficient of new interactions such as face recognition, body recognition, and human eye gaze. , to provide a more comfortable user experience for the above scenarios.

An embodiment of the present application provides a control method, including: when a non-contact touch control function of an electronic device is in an on state, acquiring a video sequence corresponding to a non-contact touch operation, performing touch recognition on the video sequence, and obtaining The target object and the touch type, determine the first distance between the target object and the electronic device, determine the touch operation parameters under the first distance, and control the electronic device to execute the target corresponding to the touch type according to the touch operation parameters That is to say, in the embodiment of the present application, in the realization of the non-contact touch control function of the electronic device, the touch operation at the first distance is determined by determining the first distance between the target object and the electronic device parameters, and then control the controlled object of the electronic device to perform the target function corresponding to the touch type according to the touch operation parameters. In this way, for the same touch type, different first distances can determine different touch operation parameters. Then, the electronic The device can respond to the touch type according to the distance between the target object and the electronic device. Compared with the existing control of only the target function corresponding to the touch type, the same touch type is used to utilize the touch at different distances. The control operation parameters are different, which makes the non-contact touch control more refined.

Based on the same inventive concept as the foregoing embodiments, an embodiment of the present application provides an electronic device, and FIG. 8 is a schematic structural diagram of an optional electronic device provided by an embodiment of the present application. As shown in FIG. 8 , the electronic device includes:

an acquisition module 81, configured to acquire a video sequence corresponding to a non-contact touch operation when the non-contact touch control function of the electronic device is in an on state;

The processing module 82 is used to perform touch recognition on the video sequence to obtain the target object and the touch type;

a first determining module 83, configured to determine a first distance between the target object and the electronic device;

The second determination module 84 is configured to determine touch operation parameters at the first distance;

The control module 85 is configured to control the electronic device to execute the target function corresponding to the touch type according to the touch operation parameters.

In an optional embodiment, the first determining module 83 is specifically configured to:

Select the boundary information of a target object from the boundary information of the target object in each image frame of the video sequence;

According to the boundary information of the selected target object, the first distance between the target object and the screen of the electronic device is determined.

In an optional embodiment, when the boundary information is represented by a bounding box, the first determining module 83 determines the first distance between the target object and the electronic device according to the selected boundary information of the target object, including:

determining a second distance between each boundary of the bounding box and a corresponding edge of the screen of the electronic device;

determining the target boundary according to the second distance between each boundary and the corresponding edge of the screen;

According to the length value of the target boundary, the relationship between the preset boundary length value and the distance between the target object and the electronic device is used to determine the first distance between the target object and the electronic device.

In an optional embodiment, the first determining module 83 determines the target boundary according to the second distance between each boundary of the bounding box and the corresponding edge of the screen, including:

When the second distance between each boundary and the corresponding edge of the screen is greater than the preset threshold, select a boundary from each boundary;

When only one of the second distances between each boundary and the corresponding edge of the screen has a second distance between the boundary and the corresponding edge of the screen that is less than or equal to the preset threshold, select the second distance from each boundary to the corresponding edge of the screen The boundary is less than or equal to the preset threshold;

The selected boundary is determined as the target boundary.

In an optional embodiment, the first determining module 83 calculates the distance between the target object and the electronic device by using the relationship between the preset boundary length value and the distance between the target object and the electronic device according to the length value of the target boundary. In the first distance between, including:

When the target boundary is a width boundary, the first distance between the target object and the electronic device is calculated by using the relationship between the preset boundary width value and the distance between the target object and the electronic device according to the length value of the target boundary.

In an optional embodiment, the first determining module 83 calculates the distance between the target object and the electronic device by using the relationship between the preset boundary length value and the distance between the target object and the electronic device according to the length value of the target boundary. The first distance between, including:

When the target boundary is a height boundary, the first distance between the target object and the electronic device is obtained by calculating the relationship between the preset boundary height value and the distance between the target object and the electronic device according to the length value of the target boundary.

In an optional embodiment, the first determining module 83 is specifically configured to:

Determine the distance between the target object in each image frame and the electronic device according to the boundary information in the target object in each image frame in the video sequence;

The average value of the distances between the target object and the electronic device in each image frame is determined as the first distance between the target object and the electronic device.

In an optional embodiment, the first determining module 83 determines the average value of the distance between the target object and the electronic device in each image frame as the first distance between the target object and the electronic device, include:

When the absolute value of the difference between any two of the distances between the target object and the electronic device in each image frame is less than or equal to the preset error threshold, the difference between the target object and the electronic device in each image frame is calculated. The average value of the distance is determined as the first distance between the target object and the electronic device.

In an optional embodiment, the electronic device is further used for:

When the absolute value of the difference between any two of the distances between the target object and the electronic device in each image frame is greater than the preset error threshold, according to the preset touch at the standard distance control operation parameters, and control the electronic device to perform the target function corresponding to the touch type.

In an optional embodiment, the second determining module 84 is specifically configured to:

Calculate the sensitivity coefficient at the first distance based on the preset correspondence between the distance and the sensitivity coefficient;

The touch operation parameters at the first distance are determined according to the sensitivity coefficient at the first distance.

In an optional embodiment, the second determination module 84 determines the touch operation parameters at the first distance according to the sensitivity coefficient at the first distance, including:

Using the sensitivity coefficient at the first distance and the preset touch operation parameters at the standard distance, the touch operation parameters at the first distance are calculated.

In an optional embodiment, the processing module 82 performs touch recognition on the video sequence, and obtains the target object and touch type, including:

Perform touch recognition on the video sequence to obtain the target object, the bounding box of the target object, and the confidence value of the touch operation of the target object; wherein, the bounding box represents boundary information;

The touch type is determined based on the confidence value of the touch operation of the target object.

In an optional embodiment, the processing module 82 determines the touch type based on the confidence value of the touch operation of the target object, including:

The type of the touch operation corresponding to the maximum value of the confidence value of the touch operation of the target object belongs to is determined as the touch type.

In an optional embodiment, the touch operation parameters include any of the following: sliding distance of the sliding operation, sliding speed of the sliding operation, page turning speed of the page turning operation, long pressing time of the long pressing operation and clicking operation click frequency.

In an optional embodiment, the control module 85 is specifically configured to:

According to the touch operation parameters, the controlled object on the screen of the electronic device is controlled to execute the target function corresponding to the touch type; wherein, the controlled object includes any one of the following: a page, an interface and a control.

In an optional embodiment, the control module 85 controls the controlled object on the screen of the electronic device according to the touch operation parameters to execute the target function corresponding to the touch type, including:

According to the sliding distance of the sliding operation, control the page of the screen to perform the sliding function.

In practical applications, the above acquisition module 81 , processing module 82 , first determination module 83 , second determination module 84 and control module 85 can be implemented by a processor located on the electronic device, specifically a central processing unit (CPU, Central Processing Unit) , Microprocessor (MPU, Microprocessor Unit), Digital Signal Processor (DSP, Digital Signal Processing) or Field Programmable Gate Array (FPGA, Field Programmable Gate Array) etc.

FIG. 9 is a schematic structural diagram of another optional electronic device provided by an embodiment of the present application. As shown in FIG. 9 , an embodiment of the present application provides an electronic device 900, including:

The processor 91 and the storage medium 92 storing the executable instructions of the processor 91. The storage medium 92 relies on the processor 91 to perform operations through the communication bus 93. When the instructions are executed by the processor 91, The control method performed in one or more of the above embodiments is performed.

It should be noted that, in practical application, various components in the terminal are coupled together through the communication bus 93 . It can be understood that the communication bus 93 is used to realize the connection communication between these components. In addition to the data bus, the communication bus 93 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the various buses are labeled as communication bus 93 in FIG. 9 .

An embodiment of the present application provides a computer storage medium, which stores executable instructions. When the executable instructions are executed by one or more processors, the processors execute the steps as described in the first step in one or more of the foregoing embodiments. The control method performed by an electronic device.

Wherein, the computer-readable storage medium may be a magnetic random access memory (ferromagnetic random access memory, FRAM), a read only memory (Read Only Memory, ROM), a programmable read only memory (Programmable Read-Only Memory, PROM), an erasable memory In addition to programmable read-only memory (ErasableProgrammable Read-Only Memory, EPROM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), flash memory (Flash Memory), magnetic surface memory, optical disks, Or memory such as Compact Disc Read-Only Memory (CD-ROM).

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the application may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media having computer-usable program code embodied therein, including but not limited to disk storage, optical storage, and the like.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

The above descriptions are only preferred embodiments of the present application, and are not intended to limit the protection scope of the present application.

Claims (19)

1. a control method, is characterized in that, described method is applied in electronic equipment, comprises: When the non-contact touch control function of the electronic device is in an on state, acquiring a video sequence corresponding to the non-contact touch operation; Perform touch recognition on the video sequence to obtain the target object and touch type; determining a first distance between the target object and the electronic device; determining touch operation parameters at the first distance; According to the touch operation parameters, the electronic device is controlled to execute the target function corresponding to the touch type. 2. The method according to claim 1, wherein the determining the first distance between the target object and the electronic device comprises: Select the boundary information of a target object from the boundary information of the target object in each image frame of the video sequence; The first distance between the target object and the electronic device is determined according to the boundary information of the selected target object. 3 . The method according to claim 2 , wherein, when the boundary information is represented by a bounding box, the first step between the target object and the electronic device is determined according to the selected boundary information of the target object. 4 . A distance, including: determining a second distance between each boundary of the bounding box and a corresponding edge of the screen of the electronic device; determining a target boundary according to a second distance between each boundary and a corresponding edge of the screen; According to the length value of the target boundary, the first distance between the target object and the electronic device is determined by using the relationship between the preset boundary length value and the distance between the target object and the electronic device. 4. The method according to claim 3, wherein determining the target boundary according to the second distance between each boundary of the bounding box and the corresponding edge of the screen, comprising: When the second distances between the borders and the corresponding edges of the screen are all greater than a preset threshold, selecting a border from the borders; When only one of the second distances between the borders and the corresponding edge of the screen exists, the second distance between the border and the corresponding edge of the screen is less than or equal to a preset threshold, selecting the border from the borders and the corresponding edge of the screen. The second distance of the corresponding edge of the screen is less than or equal to the boundary of the preset threshold; The selected boundary is determined as the target boundary. 5 . The method according to claim 3 , wherein, according to the length value of the target boundary, using the relationship between the preset boundary length value and the distance from the target object to the electronic device, 6 . Calculating the first distance between the target object and the electronic device, including: When the target boundary is a width boundary, according to the length value of the target boundary, the relationship between the preset boundary width value and the distance between the target object and the electronic device is used to calculate the distance between the target object and the electronic device. a first distance between the electronic devices. 6 . The method according to claim 3 , wherein, according to the length value of the target boundary, using the relationship between the preset boundary length value and the distance from the target object to the electronic device, 6 . Calculating the first distance between the target object and the electronic device, including: When the target boundary is a height boundary, according to the length value of the target boundary, the relationship between the preset boundary height value and the distance between the target object and the electronic device is used to calculate the distance between the target object and the electronic device. a first distance between the electronic devices. 7. The method according to claim 1, wherein the determining the first distance between the target object and the electronic device comprises: determining the distance between the target object in each image frame and the electronic device according to the boundary information in the target object in each image frame in the video sequence; The average value of the distances between the target object and the electronic device in each image frame is determined as the first distance between the target object and the electronic device. 8 . The method according to claim 7 , wherein the average value of the distance between the target object in each image frame and the electronic device is determined as the target object and the electronic device. 9 . The first distance between electronic devices, including: When the absolute value of the difference between any two of the distances between the target object and the electronic device in each image frame is less than or equal to a preset error threshold, the target object in each image frame is The average value of the distances from the electronic device is determined as the first distance between the target object and the electronic device. 9. The method according to claim 8, wherein the method further comprises: When the absolute value of the difference between any two of the distances between the target object and the electronic device in each image frame has an absolute value that is greater than the preset error threshold, according to the preset standard The touch operation parameters under the distance control the electronic device to execute the target function corresponding to the touch type. 10 . The method according to claim 1 , wherein the determining the touch operation parameters at the first distance comprises: 10 . Calculate the sensitivity coefficient at the first distance based on the preset correspondence between the distance and the sensitivity coefficient; The touch operation parameters at the first distance are determined according to the sensitivity coefficient at the first distance. 11 . The method according to claim 10 , wherein the determining the touch operation parameters at the first distance according to the sensitivity coefficient at the first distance comprises: 11 . Using the sensitivity coefficient at the first distance and the preset touch operation parameters at the standard distance, the touch operation parameters at the first distance are calculated and obtained. 12 . The method according to claim 1 , wherein the performing touch recognition on the video sequence to obtain a target object and a touch type, comprising: 12 . Perform touch recognition on the video sequence to obtain the target object, the bounding box of the target object, and the confidence value of the touch operation of the target object; wherein, the bounding box represents boundary information; The touch type is determined based on the confidence value of the touch operation of the target object. 13 . The method according to claim 12 , wherein the determining the touch type based on the confidence value of the touch operation of the target object comprises: 13 . The type of the touch operation corresponding to the maximum value of the confidence value of the touch operation of the target object is determined as the touch type. 14. The method according to any one of claims 1 to 13, wherein the touch operation parameters comprise any one of the following: The sliding distance of the sliding operation, the sliding speed of the sliding operation, the page-turning speed of the page-turning operation, the long-pressing time of the long-pressing operation and the clicking frequency of the clicking operation. 15. The method according to any one of claims 1 to 13, wherein the controlling the electronic device to execute the target function corresponding to the touch type according to the touch operation parameter comprises: Controlling the controlled object on the screen of the electronic device according to the touch operation parameter to execute the target function corresponding to the touch type; Wherein, the controlled object includes any one of the following: a page, an interface and a control. 16 . The method according to claim 15 , wherein the controlled object on the screen of the electronic device is controlled according to the touch operation parameter to execute a target function corresponding to the touch type, 16 . include: According to the sliding distance of the sliding operation, the page of the screen is controlled to perform the sliding function. 17. An electronic device, comprising: an acquisition module, configured to acquire a video sequence corresponding to a non-contact touch operation when the non-contact touch control function of the electronic device is turned on; a processing module for performing touch recognition on the video sequence to obtain a target object and a touch type; a first determining module, configured to determine a first distance between the target object and the electronic device; a second determination module, configured to determine touch operation parameters at the first distance; The control module is configured to control the electronic device to execute the target function corresponding to the touch type according to the touch operation parameter. 18. An electronic device, characterized in that, comprising: A processor and a storage medium storing instructions executable by the processor, the storage medium relying on the processor to perform operations through a communication bus, and when the instructions are executed by the processor, the above claims 1 to 1 are executed. The control method of any one of 16. 19. A computer storage medium, characterized in that it stores executable instructions, and when the executable instructions are executed by one or more processors, the processors perform the execution according to any one of claims 1 to 16. the described control method.

Images