CN111432131A - Photographing frame selection method and device, electronic equipment and storage medium - Google Patents

Abstract

The embodiment of the invention relates to the technical field of intelligent equipment, and discloses a method and a device for photo frame selection, electronic equipment and a storage medium. The method comprises the following steps: when a frame selection instruction is received, acquiring position coordinates of an operating body on a bearing body and a watching area of eyeballs of an operator on the bearing body by using a camera; performing a first operation when there is only one of the position coordinates within the gazing zone; the first operation is to photograph the content on the bearer based on the position coordinates and a preset rule. By implementing the embodiment of the invention, the content on the frame selection bearing body is realized by combining the operation body positioning and the eyeball tracking technology, the situation that the specific frame selection content cannot be identified due to multiple operation bodies can be abandoned, the frame question accuracy rate is improved, and the frame question efficiency is improved.

Description

A method, device, electronic device and storage medium for frame selection

technical field

The present invention relates to the technical field of intelligent devices, and in particular, to a method, device, electronic device and storage medium for frame selection of photos.

Background technique

At present, smart devices such as learning machines and tutoring machines have the function of framing questions. The framing methods are roughly divided into two categories. One is to take pictures of the content that needs to be selected on the carrier (such as a book) through the rear camera of the smart device; the other One is mainly completed by the front camera of the smart device. The camera recognizes the position of the operating body (such as a finger) on the carrier, and then takes a picture of the content on the carrier based on the position. Framed questions are mainly used to search for questions, or to store knowledge points or wrong questions for future review.

The first type of framing method is relatively cumbersome, and it also depends on the operator's photographing level, which is too vague to carry out subsequent operations; the second type of framing method is simple to use, but there are also the following problems: In practice, it is found that due to the load The body may be wrinkled, curled, etc. During the process of framing the question, the operator often habitually presses one side of the carrier with one hand, and the other hand is used as the operating body to locate the content selected by the box. In this case In this case, the camera will recognize multiple operating bodies, and the smart device cannot make a preparation judgment for this situation. As a result, the specific content of the frame selection cannot be recognized, or the frame selection content is inconsistent with the user's needs.

SUMMARY OF THE INVENTION

In view of the above-mentioned defects, the embodiments of the present invention disclose a method, an apparatus, an electronic device and a storage medium for frame selection by taking pictures, which can improve the accuracy of the frame selection.

A first aspect of the embodiments of the present invention discloses a method for frame selection by taking pictures, and the method includes:

When receiving the frame selection instruction, use the camera to obtain the position coordinates of the operating body on the carrier and the gaze area of the operator's eyeballs on the carrier;

When there is only one of the position coordinates in the gaze area, a first operation is performed; the first operation is to take a picture of the content on the carrier based on the position coordinates and a preset rule.

As an optional implementation manner, in the first aspect of the embodiment of the present invention, the use of a camera to obtain the position coordinates of the operating body on the carrier body includes:

Start the first camera to obtain a preview image of the carrier with the operating body;

Determine the image coordinates of the operator in the preview image;

The position coordinates of the operating body are determined using the image coordinates.

As an optional implementation manner, in the first aspect of the embodiment of the present invention, the determining the image coordinates of the operating body in the preview image includes:

Identify the operating body using the color difference between the operating body and the content on the carrier;

Determine the image coordinates of the operating body in the preview image.

As an optional implementation manner, in the first aspect of the embodiment of the present invention, the using a camera to obtain the gaze area of the operator's eyeball on the carrier includes:

Start the second camera to obtain the operator's face image;

determining the position of the center of the pupil in the face image and the offset of the position of the center of the pupil relative to the reference point;

determining the gaze direction of the eyeball and the gaze point on the carrier based on the offset;

With the gaze point as the center, the gaze area is determined.

As an optional implementation manner, in the first aspect of the embodiment of the present invention, determining the position of the center of the pupil in the face image and the offset of the position of the center of the pupil relative to the reference point includes:

Inputting the face image into a previously trained convolutional neural network to determine the feature points of the pupil;

Use the feature points of the pupil to determine the position of the center of the pupil;

The offset of the position of the pupil center is determined according to the position of the pupil center and the reference point.

As an optional implementation manner, in the first aspect of the embodiment of the present invention, determining the offset of the position of the pupil center according to the position of the pupil center and the reference point, including:

According to the feature points of the pupil, the appearance of the eye is constructed, and the position of the pupil center when looking directly is used as the reference point;

According to the position of the pupil center and the position of the reference point, the offset of the position of the pupil center relative to the reference point is calculated.

As an optional implementation manner, in the first aspect of the embodiment of the present invention, the method further includes:

The second operation is performed when the gaze area is not on the carrier, or the position coordinates are not detected, or there are no position coordinates or there are two or more position coordinates in the gaze area.

A second aspect of the embodiments of the present invention discloses a frame selection method for taking pictures, including:

When receiving the frame selection instruction, use the camera to obtain the position coordinates of the operating body on the carrier and the gaze area of the operator's eyeballs on the carrier;

When there are a plurality of the position coordinates in the gaze area, determine whether the plurality of position coordinates are continuous points;

If the multiple position coordinates are continuous points, detect the edge of the carrier, and obtain the included angle between the continuous point and any edge of the carrier;

If the included angle between the continuous point and one of the edges of the carrier is less than or equal to the preset angle, a first operation is performed; the first operation is to perform an operation on the content on the carrier based on the continuous point and a preset rule Photograph.

As an optional implementation manner, in the second aspect of the embodiment of the present invention, the method further includes:

When the gaze area is not on the carrier, or the location coordinates are not detected, or there are no location coordinates in the gaze area, or there are multiple location coordinates in the gaze area and the multiple location coordinates are not consecutive points or not all If there are multiple position coordinates in the continuous point or the gaze area and the multiple position coordinates are continuous points but the included angle between the continuous point and all edges of the carrier is greater than the preset included angle, the second operation is performed.

A third aspect of the embodiments of the present invention discloses a photo frame selection device, which includes:

an acquisition unit, configured to use a camera to acquire the position coordinates of the operating body on the carrier and the gaze area of the operator's eyeballs on the carrier when receiving the frame selection instruction;

an execution unit, configured to execute a first operation when there is only one position coordinate in the gaze area; the first operation is to take a picture of the content on the carrier based on the position coordinate and a preset rule .

A fourth aspect of the embodiments of the present invention discloses an electronic device, comprising: a memory storing executable program codes; a processor coupled to the memory; the processor calling the executable program codes stored in the memory , which is used to execute the method for frame selection for taking pictures disclosed in the first aspect or the second aspect of the embodiments of the present invention.

A fifth aspect of the embodiments of the present invention discloses a computer-readable storage medium, which stores a computer program, wherein the computer program causes a computer to execute the method for taking a picture frame selection disclosed in the first aspect or the second aspect of the embodiments of the present invention .

A sixth aspect of the embodiments of the present invention discloses a computer program product, which, when the computer program product runs on a computer, causes the computer to execute a photo frame selection method disclosed in the first aspect or the second aspect of the embodiments of the present invention method.

A seventh aspect of the embodiments of the present invention discloses an application publishing platform, and the application publishing platform is used for publishing a computer program product, wherein when the computer program product runs on a computer, the computer is made to execute the first embodiment of the present invention. In one aspect or the second aspect, a method for frame selection is disclosed.

Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

In the embodiment of the present invention, when a frame selection instruction is received, the camera is used to obtain the position coordinates of the operating body on the carrier and the gaze area of the operator's eyeballs on the carrier; there exists in the gaze area and only When there is one of the position coordinates, a first operation is performed; the first operation is to take a picture of the content on the carrier based on the position coordinates and a preset rule. It can be seen that, by implementing the embodiment of the present invention, the content on the frame selection carrier is realized by the combination of operator positioning and eye tracking technology, which can eliminate the inability to identify the specific frame selection content caused by multiple operators, improve the accuracy of frame questions, and improve the frame selection. question efficiency.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present invention more clearly, the following briefly introduces the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

FIG. 1 is a schematic flowchart of a method for frame selection by taking a picture disclosed in an embodiment of the present invention;

2 is a schematic flowchart of obtaining the position coordinates of an operating body on a carrier disclosed by an embodiment of the present invention;

3 is a schematic flowchart of obtaining the gaze area of an operator's eyeball on a carrier disclosed in an embodiment of the present invention;

4 is a schematic diagram of an operation of using a finger to point to a frame selection area disclosed in an embodiment of the present invention;

FIG. 5 is a schematic flowchart of another method for frame selection according to an embodiment of the present invention;

6 is a schematic diagram of an operation of using a pencil to point to a frame selection area disclosed in an embodiment of the present invention;

7 is a schematic diagram of an operation of using two rulers to point to a frame selection area disclosed in an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of a frame selection device for taking pictures disclosed in an embodiment of the present invention;

FIG. 9 is a schematic structural diagram of another device for frame selection for taking pictures disclosed in an embodiment of the present invention;

FIG. 10 is a schematic structural diagram of an electronic device disclosed in an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

It should be noted that the terms "first", "second", "third", "fourth", etc. in the description and claims of the present invention are used to distinguish different objects, rather than to describe specific order. The terms "comprising" and "having" and any variations thereof in the embodiments of the present invention are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to Those steps or elements that are expressly listed may instead include other steps or elements that are not expressly listed or are inherent to the process, method, product or apparatus.

The embodiment of the present invention discloses a method, device, electronic device and storage medium for frame selection by taking a picture. By combining the positioning of the operator and the eye tracking technology, the frame problem can be achieved, the frame problem can be more accurate, and the frame problem efficiency can be improved. A detailed description is given below in conjunction with the accompanying drawings.

Example 1

Please refer to FIG. 1 . FIG. 1 is a schematic flowchart of a method for frame selection for a photo frame disclosed in an embodiment of the present invention. The methods described in the embodiments of the present invention are applicable to smart devices such as learning machines, tutoring machines, point readers, tablet computers, or mobile phones with front-facing cameras. The frame selection refers to selecting the corresponding content on the carrier to take a picture of the positioning position of the operating body according to the preset rules, and the frame title is the frame selection content. As shown in Figure 1, the method for frame selection of a photo includes the following steps:

110. When a frame selection instruction is received, use a camera to acquire the position coordinates of the operating body on the carrier and the gaze area of the operator's eyeballs on the carrier.

Framed questions are used in learning, and their purpose can be to search for questions, store wrong questions and knowledge points, etc. The frame selection instruction is initiated by the user, that is, the operator. It can be a voice command, or a pulse command triggered by the user through a touch screen or a mechanical button, or a user through a specific gesture, which is not limited here. Before receiving the frame selection instruction, the camera and most of the components of the smart device are in a sleep state, which can save power, and wake up the two through the frame selection instruction.

The camera is a front camera of the smart device, of course, it can also be a camera set externally and communicate with the smart device in a wired or wireless manner. The carrier is the carrier of the content to be framed, which can be a book, exercise book, test paper, etc. The operating body is used to operate on the carrier, and then realize the positioning of the frame selection content according to the position of the positioning operating body. The operating body can be a finger, a stylus, a pencil, etc., or even a small stick, which is not limited here.

Taking the front camera as an example, the carrier is placed on the front side of the smart device, and the smart device and the carrier present a certain angle, so that the front camera can capture all or most of the contents of the carrier. Obtaining the position coordinates of the operating body on the carrier and the gaze area of the operator's eyeballs on the carrier can be achieved by a camera, such as a wide-angle camera.

In a preferred embodiment of the present invention, different cameras are used to determine the positioning of the operating body and the eye tracking information.

Referring to Figure 2, using the camera to obtain the position coordinates of the operating body on the carrier may include:

1101. Acquire a preview image of a carrier with an operating body through a first camera.

In the embodiment of the present invention, there is at least one operating body that operates on the carrier, and at least one operating body among the one or more operating bodies is in point contact with the carrier, and the final photographing content is the point of contact. The frame selection of coordinates is not considered in the embodiment of the present invention if all the operating bodies are placed in the carrier body.

The preview image does not need to be photographed and saved, and the preview image may or may not be displayed on the display interface of the smart device. The preview image is first preprocessed by the smart device, and the preprocessing includes but is not limited to: denoising, contrast enhancement, and shape correction.

1102. Determine the image coordinates of the operating body in the preview image.

The image coordinates of the operating body in the preview image are pixel coordinates. If one or more of the operating bodies and the carrier have multiple contact points, the image coordinates of the operating body have multiple consecutive points.

There are many ways to obtain the image coordinates of the operating body. Exemplarily, the image sample of the carrier with the operating body can be trained by means of machine learning, and the features of the operating body in the target image can be recognized through training. Then, the image coordinates of the operating body are obtained based on the features of the recognized operating body. In actual operation of this method, it may be necessary to establish a neural network model for training on different operators.

In a preferred embodiment of the present invention, the operating body is identified by the color difference between the operating body and the content on the carrier.

Color difference recognition can be realized by color conversion. Exemplarily, grayscale processing is performed on the preview image, all contents in the preview image are grayscaled, and then based on the grayscale value of the operating body and the content on the carrier body The difference in gray value distinguishes the operator. Generally speaking, the pixel occupied by the operator in the preview image is much less than the content on the carrier, that is to say, a certain gray value in the preview image occupies the least pixel. The object is the operator.

Before performing color conversion, the background and foreground can be distinguished to extract the foreground image, so as to prevent the influence of the background image on the recognition of the operator. Exemplarily, an adaptive background template can be obtained through the background Gaussian modeling method, and then extracted. foreground image. Of course, in general, the background color is single and the gray value tends to be 255. Therefore, an image with a preset value smaller than a certain gray value (for example, 200) can also be extracted as a foreground image.

If the color of the operating body is more distinguishable from the color of the content on the carrier, the operating body can be identified through the binarized image, that is, a gray value reference point is set. Exemplarily, the content of the carrier generally tends to be black. For example, 100 is set to 0 for an image whose gray value is less than a certain value, and 1 is set for an image whose gray value is greater than or equal to a certain value.

If the color of the operating body is too close to the color of the content on the carrier, exemplarily, the color space conversion matrix provided by the multi-color space CN (Color Names) algorithm can be used to convert the RGB image of the preview image into an 11-dimensional multi-color space. The color space is further distinguished according to the refined multi-color space to identify the operator.

For the operating body in the form of a stylus, a pencil, a small stick, etc., when it is used in the embodiment of the present invention, point contact is generally used directly, that is, as long as the position coordinates of the end of such an operating body are detected, here the The end position refers to the end of the hand-held part of the operating body, that is, the three parts of the operating body, the palm and the content of the carrier are detected to distinguish, and then the position coordinates of the end of the operating body far from the palm are determined. is the position coordinate of the operator.

For the operating body realized by the finger, in this embodiment of the present invention, it may be a point contact, or the entire finger may be placed on the carrier as shown in FIG. 4 . In this case, the detection fingertip and the carrier Then the determined fingertip coordinates are regarded as the position coordinates of the operating body. Therefore, when using fingers, there may be an operation mode in which both hands are placed on the carrier, for example, one hand is used to press the carrier to smooth the carrier, the other hand is used to point to the frame selection content, and the other During the guidance process of one hand, multiple fingers may all be in contact with the carrier, so that multiple fingers appear in the preview image.

1103. Determine the position coordinates of the operating body by using the image coordinates.

There are many ways to convert pixel coordinates to world coordinates.

Exemplarily, it can be performed through a coordinate transformation algorithm, and the coordinate transformation algorithm is shown in formula (1):

Among them, s is the scale coefficient, (u, v) is the image coordinate, (x, y, z) is the position coordinate, M and P are the camera's internal parameter matrix and external parameter matrix, respectively. In the case of determining the placement method of the smart device , both the internal parameter matrix and the external parameter matrix are fixed values.

It is also possible to establish a mapping relationship between the image coordinates and the position coordinates (the z-axis direction coordinates can be ignored) in the case of determining the placement method of the smart device.

Please refer to FIG. 3 , use the camera to obtain the gaze area of the operator's eyeball on the carrier, including:

1111. Start the second camera to acquire a face image of the operator.

The position of the second camera is not limited, and is based on being able to capture all or most of the features of the human face.

1112. Determine the position of the center of the pupil in the face image.

There are several ways to determine the center of the pupil. Exemplarily, the feature of the face image is extracted by means of machine learning, such as a convolutional neural network, so as to obtain the position of the center of the pupil in the face image. It is also possible to train a cascade classifier for human eye recognition based on the Adaboost algorithm, and at the same time combine the ASM algorithm to track the feature points of the human eye to determine the center position of the pupil.

1113. Determine the offset of the position of the pupil center relative to the reference point.

The offset here is relative to the position of the center of the pupil when the operator is looking straight in the horizontal direction. The position of the center of the pupil when the operator is looking straight in the horizontal direction is used as the reference point. Offset, based on the direct viewing direction and the offset position, the gaze direction of the offset position can be determined.

Exemplarily, to determine the offset position, the appearance of the eye can be constructed by the feature points of the pupil, and the position of the pupil center when looking directly is determined, which is denoted as the first position, and the position of the pupil center obtained according to step 1111 is denoted as the second position. The change of the second position relative to the first position is the offset.

1114. Determine the line-of-sight direction of the eyeball and the gaze point on the carrier based on the offset.

Exemplarily, the mapping relationship between the offset and the gaze direction and the gaze point can be directly established according to the measurement data, the gaze direction is determined based on the offset, and the gaze point is determined based on the gaze direction.

Exemplarily, training data may be established in advance, and the classifier may be trained by using the training samples, and the mapping relationship between the offset and the gaze direction and the gaze point may be established. After that, the classifier can be used for direct classification to obtain the gaze direction and gaze point. It is also possible to use the training samples to correspond to the classifier for training to establish a mapping relationship between the offset and the gaze area. After that, the classifier can be used for direct classification to directly obtain the gaze area.

Exemplarily, the gaze point may also be determined with the aid of an auxiliary light source, and step 1113 may be omitted in this manner. For example, if the human eye is illuminated by multiple auxiliary light sources such as near-infrared light sources, there will be Purkins spots of these auxiliary light sources in the face image. The mapping relationship between the carriers, and thus the gaze point of the eyeball on the carrier.

After the fixation point is obtained, the fixation point position can also be corrected by the face orientation of the face image, and the correction may be to add the feature of the face orientation to the above mapping relationship, so as to obtain more accurate fixation point information.

1115. Taking the gaze point as a center, determine a gaze area.

After the fixation point position is obtained, the fixation area can be determined with the fixation point as the center. The gaze area can be regular or irregular in shape, and its size is set as desired. For example, a circular or square gaze area can be set.

120. When there is only one position coordinate in the gaze area, perform a first operation; the first operation is to take a picture of the content on the carrier based on the position coordinate and a preset rule.

Based on the description of step 110, under the premise of realizing the frame problem with point contact, when there is a position coordinate in the gaze area, it means that the eye movement is the same as the guide of the operating body, then the frame problem operation can be performed. If there is an operating body, it will not be considered. Of course, in some cases, if the operating body outside the gaze area (the operating body here can not be regarded as the operating body, it is defined as the auxiliary body) affects the preset rules. When taking pictures of the content of the body, the smart device can issue an interactive instruction, for example, it can send out a voice: "Please move the auxiliary body covering the subject"; or perform an operation of removing the auxiliary body from the content with the auxiliary body.

4 shows a schematic diagram of the operation of using a finger to point to the frame selection area, wherein the hand 210 is used to press the carrier 240, and the fingers 221 of the hand 220 are used to locate the frame selection area. If the gaze area 232 formed by the gaze point 231 of the eyeball 230 Covering the fingertip part of the finger 221, there is one and only one coordinate position 2211 in the gaze area shown in FIG. 4, then the coordinate position of the fingertip of the finger 221 is the positioning point of the frame selection area. The content 241 on the carrier is frame-selected by the coordinate position and the preset rule, that is, taking a picture.

There are various ways of photographing the content on the carrier based on the position coordinates and preset rules. Exemplarily, the preset rule is to perform a frame selection on a question on the upper side of the operating body, then based on the above-mentioned preview image, a first horizontal line is set according to the target position coordinates (a single position coordinate in the gaze area), the first horizontal line. Be parallel to the upper or lower edge of the carrier, then find the question number, set a second horizontal line on the upper side of the question number, the second horizontal line is parallel to the first horizontal line, select the area between the two parallel lines in the preview image Take a photo. The second horizontal line can also be set based on the spacing between the two questions (the question spacing is generally greater than the line spacing).

As an optional implementation manner, in addition to step 120, if the gaze area is not on the carrier, or the position coordinates are not detected, or there are no position coordinates in the gaze area, or there are two or more When the position coordinates, perform the second operation.

The second operation may be to do nothing, or to send a warning reminder or a voice command. Exemplarily, the second operation may be a voice instruction that cannot recognize the frame selection content, or a voice mutual assistance instruction. For example, if the gaze area is not on the carrier, the smart device on the carrier"; if the position coordinates are not detected, the smart device can make a voice: "please confirm the selection with your finger"; if there are no position coordinates in the gaze area, the smart device can voice: "please look On the operating body" or "Please put the operating body within the gaze range"; if there are two or more position coordinates in the gaze area, the smart device can make a voice: "Please remove the auxiliary body".

By implementing the embodiments of the present invention, in the case of multiple touch-type operating bodies, the frame selection area can be obtained by combining the operating body positioning and eye tracking technology, and the existing phenomenon that multiple operating bodies cannot be identified can be discarded. Improve the accuracy of frame questions and improve the efficiency of frame questions. On this basis, the user experience can obviously be improved.

Embodiment 2

When the operator uses an operating body such as a ruler to place it on the carrier, the position coordinates of multiple operating bodies may appear, and some or all of the position coordinates of these multiple operating bodies may fall within the gaze area. Please Referring to Fig. 5, in this case, the method for frame selection by taking pictures includes:

310. When a frame selection instruction is received, use a camera to acquire the position coordinates of the operating body on the carrier and the gaze area of the operator's eyeballs on the carrier.

Step 310 is substantially the same as step 110 described in the first embodiment, and details are not repeated here.

320. When there are multiple location coordinates in the gaze area, determine whether the multiple location coordinates are continuous points.

For the presentation form in which the operating body is a finger, a plurality of the position coordinates existing in the gaze area are formed by multiple fingers side by side, that is, multiple fingertips can be detected, and there are at least two of the multiple fingertips located in the gaze area. If the distance between multiple fingertips in the gaze area is less than the preset distance threshold, they are continuous points, and the line segment formed by the connection between the continuous points is called the fingertip judgment line segment.

For the presentation forms in which the operating body is a stylus, a pencil, a stick, a ruler, etc., a plurality of the position coordinates existing in the gaze area satisfies that the hand-held part is not located at the end of the operating body, and the gaze area is not located at the end of the operating body. At the end, a plurality of the position coordinates must form a continuous point, and the line segment formed by the continuous point is located in the length direction of the corresponding operating body, and there is one or more line segments in the gaze area, then the one line segment or multiple line segments constitute The whole line segment of , is regarded as a flat judgment line segment.

330. If the multiple position coordinates are continuous points, detect the edge of the carrier, and obtain the angle between the continuous point and any edge of the carrier.

Recognizing the edge of the carrier is realized by performing edge detection on the preview image, and the edge detection is also performed after the preview image is preprocessed.

Exemplarily, the angle between the continuous point and any edge of the carrier can be calculated by a slope calculation method, for example, two points in the continuous point and two points on the edge can be selected for calculation.

For a fingertip judgment line segment or a flat judgment line segment formed by one line segment, there is only one included angle; while for a flat judgment line segment composed of multiple line segments as a whole, there may be multiple included angles. Therefore, in a preferred embodiment of the present invention, in order to facilitate the calculation, the flat judging line segment composed of a plurality of line segments as a whole is excluded before the included angle is obtained.

Exemplarily, the exclusion method may be: if it is detected that there are more than two endpoints in the gaze area, the included angle is not calculated, and the second operation is directly performed.

340. If the included angle between the continuous point and one of the edges of the carrier is less than or equal to the preset angle, perform a first operation; the first operation is to perform the first operation on the carrier based on the continuous point and a preset rule. content to be photographed.

When there is only one included angle, the included angle is compared with the preset included angle, and the preset included angle is set as required, for example, it may be 5 degrees. That is, when the angle is less than or equal to the preset angle, it can be determined that the line segment is approximately parallel to an edge of the carrier, and the frame selection area can be determined according to the continuous points and preset rules, and the content in the frame selection area can be photographed.

The preset rule is different from the first embodiment, and the frame selection area of the preset rule here is generally a part or all of the area between a continuous point and an edge parallel or approximately parallel to the continuous point. Exemplarily, taking the straight line where the continuous point is located as the first straight line, for the first straight line that is parallel or approximately parallel to the upper or lower edge of the carrier, the first straight line is defined as a horizontal line, and the frame selection area is Part or all of the area on the upper or lower side of the horizontal line; for the first straight line that is parallel or approximately parallel to the left or right edge of the carrier, the first straight line is defined as a vertical line, and the frame selection area is the Part or all of the area to the left or right of the vertical line.

FIG. 6 shows a schematic diagram of using a pencil as an operating body and placing it on a carrier to perform frame selection. Among them, the pencil 410 is used to locate the frame selection area, if there are multiple position coordinates of the pencil 410 in the gaze area 432 formed by the gaze point 431 of the eyeball 430, and the multiple position coordinates constitute the continuous point 420; and in FIG. 6, If the continuous point 420 is parallel to the upper or lower edge of the carrier 440, according to step 320, a frame selection area can be determined for the content 441 on the carrier according to the continuous point and the preset rule, and the content of the frame selection area can be photographed.

FIG. 7 shows a schematic diagram of using two rulers as operating bodies and placing them on a carrier for frame selection. The ruler 510 and the ruler 520 are placed on the carrier 560 in an intersecting manner, and there are multiple position coordinates of the ruler 510 and the ruler 520 in the gaze area 532 formed by the gaze point 531 of the eyeball 530, and the multiple position coordinates respectively form a continuous point 540 and continuous point 550; since the continuous point 540 and the continuous point 550 have four endpoints, the second operation is directly performed.

As an optional implementation manner, in addition to the case of step 320, when the gaze area is not on the carrier, or the position coordinates are not detected, or there are no position coordinates in the gaze area, or there are multiple locations in the gaze area Coordinates and the plurality of position coordinates are not continuous points or not all continuous points, or there are multiple position coordinates in the gaze area and the plurality of position coordinates are continuous points but the continuous points are sandwiched with all edges of the carrier. If the angles are all greater than the preset included angle, the second operation is performed.

The second operation may be to do nothing, or to send a warning reminder or a voice command. Exemplarily, the second operation may be a voice instruction that cannot recognize the content of the frame selection, or may be a voice mutual assistance instruction similar to the first embodiment.

By implementing the embodiments of the present invention, the frame selection area can be obtained by combining the positioning of the operating body and the eye-tracking technology for the situation where there are multiple line-touched operating bodies, and the existing phenomenon that multiple operating bodies cannot be identified can be discarded. Improve the accuracy of frame questions and improve the efficiency of frame questions. On this basis, the user experience can obviously be improved.

Embodiment 3

Please refer to FIG. 8 . FIG. 8 is a schematic structural diagram of a frame selection device for photographing disclosed in an embodiment of the present invention. As shown in FIG. 8 , the device for frame selection of photos may include:

The acquisition unit 610 is configured to use the camera to acquire the position coordinates of the operating body on the carrier and the gaze area of the operator's eyeballs on the carrier when the frame selection instruction is received;

The execution unit 620 is configured to execute a first operation when there is only one of the position coordinates in the gaze area; the first operation is to perform the content on the carrier based on the position coordinates and preset rules. Photograph.

As an optional implementation manner, the acquisition unit 610 may include a first unit 611 and a second unit 612, wherein the first unit 611 is used to obtain the position coordinates of the operating body on the carrier by using a camera; the second unit 612 is used for The camera is used to obtain the gaze area of the operator's eyeball on the carrier.

As an optional implementation manner, the first unit 611 may include:

The first subunit 6111 is used to activate the first camera to obtain a preview image of the carrier with the operating body;

The second subunit 6112 is used to determine the image coordinates of the operating body in the preview image;

The third subunit 6113 is configured to use the image coordinates to determine the position coordinates of the operating body.

As an optional implementation manner, the second subunit 6112 may include: identifying the operating body by using the color difference between the operating body and the content on the carrier; determining the image of the operating body in the preview image coordinate.

As an optional implementation manner, the second unit 612 may include:

The fourth subunit 6121 is used to start the second camera to obtain the face image of the operator;

The fifth subunit 6122 is used to determine the position of the center of the pupil in the face image and the offset of the position of the center of the pupil relative to the reference point;

The sixth subunit 6123 is used to determine the line of sight direction of the eyeball and the gazing point on the carrier based on the offset;

The seventh subunit 6124 is used to determine the gaze area with the gaze point as the center.

As an optional implementation manner, the fifth subunit 6122 may include: inputting the face image into a previously trained convolutional neural network to determine the feature points of the pupil; using the feature points of the pupil to determine the position of the center of the pupil ; According to the feature points of the pupil, the appearance of the eye is constructed, and the position of the pupil center when looking directly is used as the reference point; According to the position of the pupil center and the position of the reference point, calculate the position of the pupil center relative to the reference point The offset of the point.

As an optional implementation manner, the system may further include: when the gaze area is not on the carrier, or the position coordinates are not detected, or there are no position coordinates in the gaze area, or there are two or more When the position coordinates of , perform the second operation.

The photo frame selection device shown in FIG. 8 can obtain the frame selection area by combining the positioning of the operation body and the eye tracking technology for the situation where there are multiple touch-type operating bodies. Identify the phenomenon, improve the accuracy of frame questions, and improve the efficiency of frame questions. On this basis, the user experience can obviously be improved.

Embodiment 4

Please refer to FIG. 9 . FIG. 9 is a schematic structural diagram of a frame selection device for taking pictures disclosed in an embodiment of the present invention. As shown in FIG. 9, the photo frame selection device may include:

A collection unit 710, configured to use a camera to acquire the position coordinates of the operating body on the carrier and the gaze area of the operator's eyeballs on the carrier when receiving the frame selection instruction;

Judging unit 720, configured to judge whether the plurality of position coordinates are continuous points when there are a plurality of the position coordinates in the gaze area;

an obtaining unit 730, configured to detect the edge of the carrier if the multiple position coordinates are continuous points, and obtain the included angle between the continuous point and any edge of the carrier;

Executing unit 740, configured to execute a first operation if the included angle between the continuous point and one of the edges of the carrier is less than or equal to a preset angle; the first operation is to perform a pair based on the continuous point and a preset rule. Take pictures of the content on the carrier.

As an optional implementation manner, the system may further include: when the gaze area is not on the carrier, or the position coordinates are not detected, or there are no position coordinates in the gaze area, or there are multiple position coordinates and the plurality of position coordinates are not continuous points or not all continuous points, or there are multiple position coordinates in the gaze area and the plurality of position coordinates are continuous points but the continuous points and all edges of the carrier If the included angle is greater than the preset included angle, the second operation is performed.

The photo frame selection device shown in FIG. 9 can obtain the frame selection area by combining the positioning of the operation body and the eye tracking technology for the situation where there are multiple touch-type operating bodies. Identify the phenomenon, improve the accuracy of frame questions, and improve the efficiency of frame questions. On this basis, the user experience can obviously be improved.

Embodiment 5

Please refer to FIG. 10. FIG. 10 is a schematic structural diagram of an electronic device disclosed in an embodiment of the present invention. Electronic devices can be smart devices such as learning machines, tutoring machines, point readers, tablet computers or mobile phones. As shown in Figure 10, the electronic device may include:

a memory 810 storing executable program code;

a processor 820 coupled to the memory 810;

Wherein, the processor 820 invokes the executable program code stored in the memory 810 to execute some or all of the steps in any of the methods for frame selection in Embodiment 1 or Embodiment 2.

An embodiment of the present invention discloses a computer-readable storage medium, which stores a computer program, wherein the computer program enables a computer to execute part or all of the steps in any one of the first or second embodiments of the method for frame selection.

The embodiment of the present invention also discloses a computer program product, wherein when the computer program product runs on the computer, the computer is made to execute some or all of the steps in any of the first or second embodiments of the method for frame selection.

The embodiment of the present invention also discloses an application publishing platform, wherein the application publishing platform is used for publishing a computer program product, wherein when the computer program product runs on a computer, the computer is made to execute any one of the first embodiment or the second embodiment to take a picture Some or all of the steps in the method to be framed.

In various embodiments of the present invention, it should be understood that the size of the sequence numbers of the described procedures does not imply a necessary order of execution, and the execution order of each procedure should be determined by its functions and internal logic, and does not deal with the present invention. The implementation of the embodiments constitutes no limitation.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be object units, and may be located in one place or distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The integrated unit may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated unit, if implemented as a software functional unit and sold or used as a stand-alone product, may be stored in a computer-accessible memory. Based on such understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a memory , including several requests to cause a computer device (which may be a personal computer, a server, or a network device, etc., specifically a processor in the computer device) to execute some or all of the steps of the methods described in the various embodiments of the present invention.

In the embodiments provided by the present invention, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean that B is only determined according to A, and B may also be determined according to A and/or other information.

Those of ordinary skill in the art can understand that some or all of the steps in the various methods of the embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes only Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-only Memory (PROM), Erasable Programmable Read-Only Memory, EPROM), One-time Programmable Read-Only Memory (OTPROM), Electronically Erasable Programmable Read-Only Memory (EEPROM), CD-ROM ( CompactDisc Read-Only Memory, CD-ROM) or other optical disk storage, magnetic disk storage, tape storage, or any other computer-readable medium that can be used to carry or store data.

The method, device, electronic device, and storage medium for taking a picture frame selection disclosed in the embodiments of the present invention are described above in detail. The principles and implementations of the present invention are described with specific examples in this paper. The description of the above embodiments It is only used to help understand the method of the present invention and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific embodiments and application scope. In summary, The contents of this specification should not be construed as limiting the present invention.

Claims (12)

1. a method for taking a picture frame selection, is characterized in that, comprises: When receiving the frame selection instruction, use the camera to obtain the position coordinates of the operating body on the carrier and the gaze area of the operator's eyeballs on the carrier; When there is only one of the position coordinates in the gaze area, a first operation is performed; the first operation is to take a picture of the content on the carrier based on the position coordinates and a preset rule. 2. The method according to claim 1, wherein the obtaining the position coordinates of the operating body on the carrier by using a camera comprises: Start the first camera to obtain a preview image of the carrier with the operating body; Determine the image coordinates of the operator in the preview image; The position coordinates of the operating body are determined using the image coordinates. 3. The method according to claim 2, wherein the determining the image coordinates of the operating body in the preview image comprises: Identify the operating body using the color difference between the operating body and the content on the carrier; Determine the image coordinates of the operating body in the preview image. 4. The method according to any one of claims 1 to 3, wherein the using a camera to acquire the gaze area of the operator's eyeball on the carrier comprises: Start the second camera to obtain the operator's face image; determining the position of the center of the pupil in the face image and the offset of the position of the center of the pupil relative to the reference point; determining the gaze direction of the eyeball and the gaze point on the carrier based on the offset; With the gaze point as the center, the gaze area is determined. 5. The method according to claim 4, wherein determining the position of the center of the pupil in the face image and the offset of the position of the center of the pupil relative to the reference point, comprising: Inputting the face image into a previously trained convolutional neural network to determine the feature points of the pupil; Use the feature points of the pupil to determine the position of the center of the pupil; The offset of the position of the pupil center is determined according to the position of the pupil center and the reference point. 6. The method according to claim 5, wherein determining the offset of the position of the pupil center according to the position of the pupil center and a reference point, comprising: According to the feature points of the pupil, the appearance of the eye is constructed, and the position of the pupil center when looking directly is used as the reference point; According to the position of the pupil center and the position of the reference point, the offset of the position of the pupil center relative to the reference point is calculated. 7. The method of claim 1, wherein the method further comprises: The second operation is performed when the gaze area is not on the carrier, or the position coordinates are not detected, or there are no position coordinates or there are two or more position coordinates in the gaze area. 8. A photo frame selection device, characterized in that it comprises: an acquisition unit, configured to use a camera to acquire the position coordinates of the operating body on the carrier and the gaze area of the operator's eyeballs on the carrier when receiving the frame selection instruction; an execution unit, configured to execute a first operation when there is only one position coordinate in the gaze area; the first operation is to take a picture of the content on the carrier based on the position coordinate and a preset rule . 9. An electronic device, comprising: a memory storing executable program codes; a processor coupled to the memory; the processor calling the executable program codes stored in the memory to use A method for frame selection of a photo frame according to any one of claims 1 to 7 is performed. 10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, wherein the computer program enables a computer to execute a photo frame selection according to any one of claims 1 to 7 Methods. 11. A method for frame selection, comprising: When receiving the frame selection instruction, use the camera to obtain the position coordinates of the operating body on the carrier and the gaze area of the operator's eyeballs on the carrier; When there are a plurality of the position coordinates in the gaze area, determine whether the plurality of position coordinates are continuous points; If the multiple position coordinates are continuous points, detect the edge of the carrier, and obtain the included angle between the continuous point and any edge of the carrier; If the included angle between the continuous point and one of the edges of the carrier is less than or equal to the preset angle, a first operation is performed; the first operation is to perform an operation on the content on the carrier based on the continuous point and a preset rule Photograph. 12. The method of claim 11, wherein the method further comprises: When the gaze area is not on the carrier, or the location coordinates are not detected, or there are no location coordinates in the gaze area, or there are multiple location coordinates in the gaze area and the multiple location coordinates are not consecutive points or not all If there are multiple position coordinates in the continuous point or the gaze area and the multiple position coordinates are continuous points but the included angle between the continuous point and all edges of the carrier is greater than the preset included angle, the second operation is performed.

Images