CN116777955A - Image processing method, device, equipment and medium - Google Patents
Image processing method, device, equipment and medium Download PDFInfo
- Publication number
- CN116777955A CN116777955A CN202310779515.5A CN202310779515A CN116777955A CN 116777955 A CN116777955 A CN 116777955A CN 202310779515 A CN202310779515 A CN 202310779515A CN 116777955 A CN116777955 A CN 116777955A
- Authority
- CN
- China
- Prior art keywords
- target
- frame image
- mapping relation
- plane area
- gray
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 20
- 238000013507 mapping Methods 0.000 claims abstract description 201
- 238000000034 method Methods 0.000 claims abstract description 35
- 238000012545 processing Methods 0.000 claims abstract description 23
- 238000012937 correction Methods 0.000 claims description 28
- 238000004590 computer program Methods 0.000 claims description 19
- 239000011159 matrix material Substances 0.000 description 13
- 230000000694 effects Effects 0.000 description 9
- 230000008569 process Effects 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000009471 action Effects 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 4
- 238000001514 detection method Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 238000013475 authorization Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
- G06T7/248—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving reference images or patches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The embodiment of the disclosure relates to an image processing method, an image processing device and a medium, wherein the method comprises the following steps: acquiring a first target frame image and a second target frame image in a target video; wherein the first target frame image is a previous frame image of the second target frame image; acquiring gray information of a target point of a first target plane area in the first target frame image; determining a target mapping relation between the second target frame image and the first target frame image according to the gray information of the target point, so as to determine a second target plane area in the second target frame image based on the target mapping relation; wherein the second target plane area matches the first target plane area. The embodiment of the disclosure can effectively ensure the accuracy of plane tracking on the basis of low consumption power consumption.
Description
Technical Field
The present disclosure relates to the field of image processing technologies, and in particular, to an image processing method, apparatus, device, and medium.
Background
In the field of image processing, in order to be able to perform functional processing such as edge lighting, content replacement, and the like on a planar area (such as a billboard, a photo frame, a screen, and the like) specified in a video, it is necessary to track the position of the planar area in a video frame image. However, the planar tracking method provided by the related art has the problems of high processing loss or inaccurate tracking result.
Disclosure of Invention
In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides an image processing method, apparatus, device, and medium.
The embodiment of the disclosure provides an image processing method, which comprises the following steps: acquiring a first target frame image and a second target frame image in a target video; wherein the first target frame image is a previous frame image of the second target frame image; acquiring gray information of a target point of a first target plane area in the first target frame image; determining a target mapping relation between the second target frame image and the first target frame image according to the gray information of the target point, so as to determine a second target plane area in the second target frame image based on the target mapping relation; wherein the second target plane area matches the first target plane area.
Optionally, the acquiring the first target frame image and the second target frame image in the target video includes: performing feature point matching on adjacent frame images in the target video to obtain a feature point matching result; and taking the adjacent frame images with the matching errors larger than a preset error threshold corresponding to the characteristic point matching results as a first target frame image and a second target frame image.
Optionally, the target point is a point in the first target plane area where the gray gradient is greater than a preset gradient threshold.
Optionally, the gray information includes gray and gray gradient; the determining, according to the gray information of the target point, a target mapping relationship between the second target frame image and the first target frame image includes: determining a corresponding point of the target point in the second target frame image based on an initial mapping relation preset between the second target frame image and the first target frame image, and acquiring the gray level of the corresponding point; according to the gray gradient of the target point and the difference between the gray of the target point and the gray of the corresponding point, performing adjustment operation on the initial mapping relation until a target mapping relation is obtained; wherein the adjusting operation is used to reduce the difference.
Optionally, the performing an adjustment operation on the initial mapping relationship until a target mapping relationship is obtained includes: and executing the adjustment operation of the designated times based on the initial mapping relation so as to determine the target mapping relation according to the mapping relation obtained after the adjustment of the designated times.
Optionally, the determining, based on the target mapping relationship, a second target plane area in the second target frame image includes: acquiring a first mapping relation between a reference frame image of the target video and the first target frame image; acquiring a second mapping relation between the reference frame image and the second target frame image according to the target mapping relation and the first mapping relation; determining a second target plane area in the second target frame image according to the reference target plane area in the reference frame image and the second mapping relation; the reference target plane area, the first target plane area and the second target plane area all correspond to the same plane.
Optionally, the obtaining, according to the target mapping relationship and the first mapping relationship, a second mapping relationship between the reference frame image and the second target frame image includes: determining a pre-estimated mapping relation between the reference frame image and the second target frame image according to the target mapping relation and the first mapping relation; based on the estimated mapping relation, a projection frame image obtained by re-projecting the second target frame image to the reference frame image is obtained; acquiring correction information of the estimated mapping relation according to the projection frame image and the reference frame image; and correcting the estimated mapping relation by using the correction information to obtain a second mapping relation between the reference frame image and the second target frame image.
Optionally, the obtaining the correction information of the estimated mapping relationship according to the projection frame image and the reference frame image includes: adjusting the brightness of the projection frame image so that the adjusted brightness matches the brightness of the reference frame image; and obtaining correction information of the estimated mapping relation based on the projection frame image and the reference frame image after brightness adjustment.
Optionally, the obtaining the correction information of the estimated mapping relationship based on the projection frame image and the reference frame image after the brightness adjustment includes: determining a mapping relation between the projection frame image and the reference frame image after brightness adjustment according to the gray scale and gray scale gradient of a target point of a reference target plane area in the reference frame image; and obtaining correction information of the estimated mapping relation based on the mapping relation between the projection frame image and the reference frame image after brightness adjustment.
The embodiment of the disclosure also provides an image processing apparatus, including: the frame image acquisition module is used for acquiring a first target frame image and a second target frame image in the target video; wherein the first target frame image is a previous frame image of the second target frame image; the gray information acquisition module is used for acquiring gray information of a target point of a first target plane area in the first target frame image; a plane area determining module, configured to determine a target mapping relationship between the second target frame image and the first target frame image according to gray information of the target point, so as to determine a second target plane area in the second target frame image based on the target mapping relationship; wherein the second target plane area matches the first target plane area.
The embodiment of the disclosure also provides an electronic device, which comprises: a storage device having a computer program stored thereon; processing means for executing the computer program in the storage means to implement the steps of any one of the image processing methods described above.
The present disclosure also provides a computer-readable storage medium storing a computer program for executing the image processing method as provided by the embodiments of the present disclosure.
According to the technical scheme provided by the embodiment of the disclosure, gray information of the target point of the first target plane area in the first target frame image (i.e., the previous frame) can be obtained, and the target mapping relationship between the second target frame image (i.e., the subsequent frame) and the first target frame image is determined according to the gray information of the target point, so as to determine the second target plane area matched with the first target plane area in the second target frame image based on the target mapping relationship. Compared with the traditional plane tracking mode adopting a deep learning algorithm, the mode has the advantages that the consumption power consumption is small, and the processing loss is low; the method for carrying out plane tracking by using the gray information has better effect, the tracking result is more accurate, and the plane registration accuracy of the image frames with characteristics such as blurring or weak textures can be better ensured.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure.
In order to more clearly illustrate the embodiments of the present disclosure or the solutions in the prior art, the drawings that are required for the description of the embodiments or the prior art will be briefly described below, and it will be obvious to those skilled in the art that other drawings can be obtained from these drawings without inventive effort.
Fig. 1 is a schematic flow chart of an image processing method according to an embodiment of the disclosure;
fig. 2 is a schematic flow chart of an image processing method according to an embodiment of the disclosure;
fig. 3 is a schematic diagram of an image processing flow provided in an embodiment of the disclosure;
fig. 4 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present disclosure;
Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.
Detailed Description
In order that the above objects, features and advantages of the present disclosure may be more clearly understood, a further description of aspects of the present disclosure will be provided below. It should be noted that, without conflict, the embodiments of the present disclosure and features in the embodiments may be combined with each other.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present disclosure may be practiced otherwise than as described herein; it will be apparent that the embodiments in the specification are only some, but not all, embodiments of the disclosure.
Fig. 1 is a flowchart of an image processing method according to an embodiment of the present disclosure, where the method may be performed by an image processing apparatus, and the apparatus may be implemented by software and/or hardware, and may be generally integrated in an electronic device. As shown in fig. 1, the method mainly includes the following steps S102 to S106:
step S102, a first target frame image and a second target frame image in a target video are obtained; wherein the first target frame image is a previous frame image of the second target frame image.
The embodiments of the present disclosure do not limit the video content of the target video, and illustratively, the target video may include a planar area therein. The first target frame image and the second target frame image are two frame images adjacent to each other in front and back in the target video, in some embodiments, any two adjacent frame images in the target video can be respectively used as the first target frame image and the second target frame image, in other embodiments, adjacent frame images meeting the specified condition can be used as the first target frame image and the second target frame image, for example, a blurred frame image or a frame image with weaker texture in the target video can be used as the second target frame image, and a previous frame image can be used as the first target frame image; in addition, after the target plane area of the special frame image is determined in the manner provided by the embodiments of the present disclosure, the special frame image may continue to be the first target frame image, so as to be used to determine the target plane area of the subsequent frame image of the special frame image. In addition, other modes such as a feature point matching algorithm can be adopted to determine the target plane area of each frame image in the target video, and then adjacent frame images with poor plane area registration effect can be selected as the first target frame image and the second target frame image. The selection modes of the first target frame image and the second target frame image can be flexibly set according to requirements, and the selection modes are not limited.
Step S104, gray information of a target point of a first target plane area in the first target frame image is acquired. Illustratively, the gray information may include gray and gray gradient.
The shape of the target plane area is not limited in the embodiment of the disclosure, and the target plane area can be a quadrilateral plane such as a common screen, a tablet and the like, or can be other polygonal planes such as a triangle and the like appearing in life or a special-shaped plane. The first target plane area in the first target frame image is a plane to be tracked in the target video, and the number of the first target plane area may be one or more, which is not limited herein. It will be appreciated that the first target planar region in the first target frame image is known, and that embodiments of the present disclosure primarily require tracking of the corresponding planar region in the second target frame image based on the location of the first target planar region in the first target frame image. The embodiments of the present disclosure do not limit the target point, for example, the target point may be all points, a designated plurality of points, a random plurality of points, or a plurality of points distributed in a preset manner in the first target plane area, and is not limited to the feature point, so that it may be well applied to a blurred frame image or a frame image with weaker texture. In order to better ensure the plane tracking accuracy, in some specific examples, the target point may be a point with a higher gray gradient, for example, the target point is a point with a gray gradient greater than a preset gradient threshold in the first target plane area, and the preset gradient threshold may be flexibly set according to requirements. In practical applications, a manner such as a canny operator may be adopted to determine a target point with a stronger gradient in the first target plane area, and gray information (gray scale and gray scale gradient) of the target point is recorded, which can be understood that even a blurred image or an image with weaker texture generally contains more points with a certain gray scale gradient, and compared with a manner of performing plane tracking by only using feature points, the manner of performing plane tracking based on the gray scale information of the points provided by the embodiment of the disclosure has stronger universality.
Step S106, determining a target mapping relation between the second target frame image and the first target frame image according to the gray information of the target point, so as to determine a second target plane area in the second target frame image based on the target mapping relation; wherein the second target plane area matches the first target plane area.
In practical application, an initial mapping relationship between two adjacent frame images may be preset first, and then multiple adjustment iterations are performed on the initial mapping relationship until a target mapping relationship is obtained, where a second target plane area determined based on the target mapping relationship is matched with a first target plane area, that is, corresponds to the same plane, and the plane is a designated plane to be tracked in the target video.
Compared with the traditional plane tracking mode adopting a deep learning algorithm, the mode has the advantages that the consumption power consumption is small, and the processing loss is low; the method for carrying out plane tracking by using the gray information has better effect, the tracking result is more accurate, and the plane registration accuracy of the image frames with characteristics such as blurring or weak textures can be better ensured.
In order to further reduce power consumption and reduce requirements for device performance, embodiments of the present disclosure provide an implementation example for acquiring a first target frame image and a second target frame image in a target video, including: performing feature point matching on adjacent frame images in the target video to obtain feature point matching results; and taking the adjacent frame images with the matching errors larger than a preset error threshold corresponding to the characteristic point matching results as a first target frame image and a second target frame image. The embodiment of the disclosure does not limit the feature point matching manner, and for example, feature point matching can be performed on adjacent frame images in a target video based on a corner detection algorithm and a sparse optical flow algorithm, in some specific implementation examples, the corner detection algorithm may be a Shi-Tomasi corner detection algorithm, the sparse optical flow algorithm may be a KLT sparse optical flow algorithm, and in practical applications, other corner detection algorithms and/or other sparse optical flow algorithms may also be adopted, where no limitation is made.
It can be understood that a better registration effect can be obtained by adopting a characteristic point matching mode for most image frames, namely, target plane matching can be performed more accurately, but for special frame images with fewer or less obvious characteristic points, such as a blurred frame image or a weak texture frame image, the plane matching error obtained by adopting the characteristic point matching mode is larger, the target plane of the following frame obtained by tracking is not matched with the target plane of the preceding frame, namely, the position of the target plane of the following frame determined based on the characteristic points of the preceding frame is not accurate. In order to give consideration to plane tracking efficiency and precision, the embodiment of the present disclosure may perform plane tracking on adjacent frame images in a target video by adopting a feature point matching manner, thereby improving plane tracking efficiency, reducing required power consumption and reducing requirements on device performance, and then screening out frame images needing to be matched again based on the magnitude of a matching error of a feature point matching result, so as to perform region matching again based on gray information. On the basis that the gray information includes gray and gray gradient, the embodiment of the present disclosure provides a specific implementation example of determining the target mapping relationship between the second target frame image and the first target frame image according to the gray information of the target point, and may be performed with reference to the following steps (1) and (2):
And (1) determining a corresponding point of the target point in the second target frame image based on an initial mapping relation preset between the second target frame image and the first target frame image, and acquiring the gray level of the corresponding point. In practical application, the mapping relationship can be represented in a matrix form, and specifically, an identity three-dimensional matrix can be preset as an initial mapping relationship. Under the condition that the initial mapping relation between the two frame images and the target point of the first target frame image are known, the corresponding point of the target point in the second target frame image can be directly obtained. It should be noted that, the initial mapping relationship is only preset, and therefore, is not accurate, and at this time, the corresponding point determined in the second target frame image is also not accurate, but the initial mapping relationship may be adjusted in multiple iterations later, that is, the positions of the corresponding points may be adjusted multiple times until a more accurate corresponding point may be found.
Step (2), according to the gray gradient of the target point and the difference between the gray of the target point and the gray of the corresponding point, performing adjustment operation on the initial mapping relation until a target mapping relation is obtained; wherein the adjustment operation is used to reduce the difference. That is, the adjustment may be performed from the initial mapping relationship in a direction of decreasing the difference value, and each adjustment is performed once, the difference value between the gray scale of the corresponding point and the gray scale of the corresponding point is redetermined based on the adjusted mapping relationship, and then the mapping relationship is readjusted based on the newly determined difference value until the target mapping relationship meeting the requirement is obtained.
In some implementations, the adjustment operation is performed a specified number of times based on the initial mapping to determine the target mapping from the mapping that is adjusted the specified number of times. For example, the mapping relationship obtained after the specified number of times of adjustment may be directly used as the target mapping relationship. The specified times can be set according to experience values, and the mapping relationship obtained after the adjustment operation of the specified times can approximate to the accurate mapping relationship between the first target frame image and the second target frame image.
For example, knowing the target point P with a strong gradient in the first target plane area in the first target frame image and acquiring the gray value gradient of the target point P, the mapping relationship H between the first target frame image and the second target frame image may be initialized first, and may be represented by using an identity three-dimensional matrix. And then the iterative mapping relation H is updated for K times in a circulating way so as to gradually reduce the gray level difference between the target point P of the first target frame image and the corresponding point P' of the second target frame image. In particular, the corresponding point P ' =h×p of the second target frame image, assuming that the gray level of P ' in the second target frame image is I ', the gray level of P in the first target frame image is I, The gray differences d=i ' -I of P ' and P may be updated according to the mapping relationship, in some embodiments, the formula H ' =h+α×d T * And (P') updating the mapping relation H between the iterative first target frame image and the second target frame image. For each adjustment operation, H is a mapping relationship before update, H 'is a mapping relationship after update, α is a learning rate, and may be set to 0.1, d (P') represents a differential of P ', specifically, the mapping relationship may be represented by a homography matrix, and the differential of P' represents an offset generated when each parameter of the 8 parameters slightly changes by a small value (change amount). The change amount can be calculated, for example, h= [ H1, H2, ]. H8,1]Assuming that the coordinates of P are (x, y), the coordinates of P' are: (h1 x+h2 x+h3)/(h7 x+h8 x+y+1), (h4 x+h5 x+h6)/(h7 x+h8 x y+1). H1, H2 and h3., H8 are derived through the above formula, and the H is further iterated for a plurality of times by using the above updated formula until the latest H obtained by the loop K times is the target mapping relationship between the first target frame image and the second target frame image.
After the target mapping relationship is determined in the above manner, the second target plane area in the second target frame image can be determined based on the target mapping relationship. In some embodiments, the target mapping relationship may be directly used as a mapping relationship between the first target frame image and the second target frame image, and the second target plane area in the second target frame image may be determined based on the first target plane area in the first target frame image and the target mapping relationship. In other embodiments, considering that there may be a problem of accumulated error in the plane tracking process, a reference frame image of the target video may be further introduced, and a mapping relationship between the second target frame image and the reference frame image is determined based on the target mapping relationship, so as to more accurately determine the second target plane area in the second target frame image. Illustratively, the steps a through C may be performed with reference to the following steps:
and step A, acquiring a first mapping relation between a reference frame image of the target video and a first target frame image. Illustratively, the reference frame image may be a first frame image of the target video. In practical application, the first mapping relationship can be directly obtained on the basis of knowing the first target plane area of the first target frame image and the reference target plane area of the reference frame image, wherein the first target plane area of the first target frame image is also obtained by tracking the target plane area of the previous frame image, and so on, in other words, the first mapping relationship between the reference frame image and the first target frame image of the target video can be accurately obtained by a recurrence mode of the mapping relationship between the adjacent frame images in the target video.
And B, acquiring a second mapping relation between the reference frame image and the second target frame image according to the target mapping relation and the first mapping relation. Because the target mapping relationship is the mapping relationship between the first target frame image and the second target frame image, and the first mapping relationship is the mapping relationship between the first target frame image and the reference frame image, the second mapping relationship between the reference frame image and the second target frame image can be reasonably and reliably obtained based on the target mapping relationship and the first mapping relationship. In order to sufficiently secure the accuracy of the second mapping relationship, it may be exemplarily performed with reference to the following steps B1 to B4:
and B1, determining a pre-estimated mapping relation between the reference frame image and the second target frame image according to the target mapping relation and the first mapping relation. It can be appreciated that, in the embodiments of the present disclosure, the problem that there may be an accumulated error and the like is fully considered, so that there may be a certain error in the obtained target mapping relationship between the first target frame image and the second target frame image, which may not be very precise, so that the mapping relationship between the reference frame image and the second target frame image determined based on the target mapping relationship and the first mapping relationship is also a predicted value, which may not be very precise, and further adjustment is required.
And B2, acquiring a projection frame image obtained by re-projecting the second target frame image to the reference frame image based on the estimated mapping relation. The projection frame image obtained by the re-projection may also be referred to as a second target frame image after the re-projection.
And step B3, acquiring correction information of the estimated mapping relation according to the projection frame image and the reference frame image. In practical application, the projection frame image and the reference frame image can be compared, so that correction information of the estimated mapping relation can be determined. In some specific implementation examples, the following steps B3.1 to B3.2 may be referred to:
and step B3.1, adjusting the brightness of the projection frame image so that the adjusted brightness is matched with the brightness of the reference frame image. That is, the difference between the luminance of the projection frame image after adjustment and the luminance of the reference frame image is smaller than the preset luminance threshold, that is, the difference between the luminance of the projection frame image and the luminance of the reference frame image is made smaller. For example, the brightness of the projection frame image may be adjusted to substantially coincide with the brightness of the reference frame image in a manner such as histogram matching. Embodiments of the present disclosure fully take into account that there may be differences in luminance from frame image to frame image, especially for frame images that are not adjacent, the luminance differences may be large. Therefore, the embodiment of the disclosure can firstly adjust the brightness of the projection frame image, and adjust the brightness of the reference frame of the projection frame image to be basically consistent so as to effectively eliminate the influence of the brightness on the gray scale.
And B3.2, obtaining correction information of the estimated mapping relation based on the projection frame image and the reference frame image after brightness adjustment. In practical application, the mapping relationship between the projection frame image after brightness adjustment and the reference frame image can be obtained, the mapping relationship can be used as correction information of the pre-estimated mapping relationship, and can be specifically expressed in a matrix form, and can also be called as a correction matrix, and the mapping relationship between the projection frame image after brightness adjustment and the reference frame image can be determined by referring to the mode based on gray information, so that the accuracy of the mapping relationship between the projection frame image after brightness adjustment and the reference frame image is ensured. That is, in some specific implementation examples, the mapping relationship between the projection frame image and the reference frame image after the brightness adjustment may be determined according to the gray scale and the gray scale gradient of the target point of the reference target plane area in the reference frame image; and then, based on the mapping relation between the projection frame image and the reference frame image after brightness adjustment, obtaining correction information of the estimated mapping relation. In other words, the reference frame image and the projection frame image with adjusted brightness are aligned by the gray information, so as to obtain a relatively accurate mapping relationship between the reference frame image and the projection frame image.
And step B4, correcting the estimated mapping relation by utilizing the correction information to obtain a second mapping relation between the reference frame image and the second target frame image. The correction information is represented in the form of a correction matrix, the estimated mapping relation is represented in the form of an estimated mapping matrix, the correction matrix and the estimated mapping matrix are multiplied to obtain a result matrix after correction of the estimated mapping matrix, and the result matrix can represent a second mapping relation between the reference frame image and the second target frame image. By the method, the second mapping relation between the reference frame image and the second target frame image can be obtained more accurately and reliably on the basis of eliminating the accumulated error.
Step C, determining a second target plane area in a second target frame image according to the reference target plane area in the reference frame image and a second mapping relation; the reference target plane area, the first target plane area and the second target plane area all correspond to the same plane, for example, all correspond to the same plane in a target scene, and the target scene is a scene presented by the target video.
Because the second mapping relation is more accurate, the accumulated error can be effectively eliminated, and the position of the second target plane area obtained based on the accumulated error is also more accurate, so that an accurate and reliable plane tracking effect is achieved.
Further, on the basis of the foregoing, the embodiment of the present disclosure further provides a flowchart of an image processing method as shown in fig. 2, which mainly includes the following steps S202 to S214:
step S202, a first target frame image and a second target frame image in a target video are obtained; wherein the first target frame image is a previous frame image of the second target frame image.
Step S204, the gray scale and gray scale gradient of the target point of the first target plane area in the first target frame image are obtained.
Step S206, based on the preset initial mapping relation between the second target frame image and the first target frame image, determining the corresponding point of the target point in the second target frame image, and obtaining the gray level of the corresponding point.
Step S208, according to the gray gradient of the target point and the difference between the gray of the target point and the gray of the corresponding point, an adjustment operation of the designated times is performed on the initial mapping relationship, and the mapping relationship obtained after the designated times is adjusted is used as the target mapping relationship.
Step S210, a first mapping relationship between the reference frame image and the first target frame image of the target video is obtained.
Step S212, obtaining a second mapping relation between the reference frame image and the second target frame image according to the target mapping relation and the first mapping relation.
Step S214, determining a second target plane area in the second target frame image according to the reference target plane area in the reference frame image and the second mapping relation.
The specific implementation of the above steps may refer to the foregoing related matters, and will not be described herein. Compared with the traditional deep learning mode, the method has the advantages that the required power consumption is small, the equipment performance requirement is low, the mapping relation between the adjacent frame images can be reasonably determined based on gray information, the reference frame image can be introduced on the basis, and the second mapping relation between the reference frame image and the second target frame image can be finally determined, so that a more accurate plane tracking effect is realized.
For easy understanding, reference may also be made to an image processing flow chart shown in fig. 3, in which a first target frame image is represented by an n-1 th frame, a second target frame image is represented by an n-1 th frame, first, a target mapping relationship H (n) between the n-1 th frame and the n-1 th frame is determined by means of inter-frame registration, then a first mapping relationship H x (n-1) between a reference frame (for example, a first frame of a video) and the n-1 th frame is obtained, the target mapping relationship H (n) is multiplied by the first mapping relationship H x (n-1), and a predicted mapping relationship H x (n) 'between the reference frame and the n-th frame is obtained, and then, the predicted mapping relationship H x (n)' is secondarily corrected by using the reference frame image (specifically, refer to the step B2-step B4).
Through the mode, the accurate inter-frame registration effect can be achieved, even if the precise registration can be achieved for the blurred frame image or the weak texture frame image, the mode of secondary correction by introducing the reference frame image on the basis can effectively improve the problem of accumulated errors, and the better plane tracking effect is comprehensively achieved.
Corresponding to the foregoing image processing method, fig. 4 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present disclosure, where the apparatus may be implemented by software and/or hardware, and may be generally integrated in an electronic device, as shown in fig. 4, and includes:
a frame image acquisition module 402, configured to acquire a first target frame image and a second target frame image in a target video; wherein the first target frame image is a previous frame image of the second target frame image;
a gray information obtaining module 404, configured to obtain gray information of a target point of the first target plane area in the first target frame image;
a plane area determining module 406, configured to determine a target mapping relationship between the second target frame image and the first target frame image according to the gray information of the target point, so as to determine a second target plane area in the second target frame image based on the target mapping relationship; wherein the second target plane area matches the first target plane area.
Compared with the traditional plane tracking mode adopting a deep learning algorithm, the device has the advantages of small power consumption, low processing loss and low requirement on equipment performance; the method for carrying out plane tracking by using the gray information has better effect, the tracking result is more accurate, and the plane registration accuracy of the image frames with characteristics such as blurring or weak textures can be better ensured.
In some embodiments, the frame image acquisition module 402 is specifically configured to: performing feature point matching on adjacent frame images in the target video to obtain a feature point matching result; and taking the adjacent frame images with the matching errors larger than a preset error threshold corresponding to the characteristic point matching results as a first target frame image and a second target frame image.
In some embodiments, the target point is a point in the first target plane region where the gray gradient is greater than a preset gradient threshold.
In some embodiments, the gray information includes gray and gray gradient; the plane area determining module 406 is specifically configured to: determining a corresponding point of the target point in the second target frame image based on an initial mapping relation preset between the second target frame image and the first target frame image, and acquiring the gray level of the corresponding point; according to the gray gradient of the target point and the difference between the gray of the target point and the gray of the corresponding point, performing adjustment operation on the initial mapping relation until a target mapping relation is obtained; wherein the adjusting operation is used to reduce the difference.
In some embodiments, the planar area determination module 406 is specifically configured to: and executing the adjustment operation of the designated times based on the initial mapping relation so as to determine the target mapping relation according to the mapping relation obtained after the adjustment of the designated times.
In some embodiments, the planar area determination module 406 is specifically configured to: acquiring a first mapping relation between a reference frame image of the target video and the first target frame image; acquiring a second mapping relation between the reference frame image and the second target frame image according to the target mapping relation and the first mapping relation; determining a second target plane area in the second target frame image according to the reference target plane area in the reference frame image and the second mapping relation; the reference target plane area, the first target plane area and the second target plane area all correspond to the same plane.
In some embodiments, the planar area determination module 406 is specifically configured to: determining a pre-estimated mapping relation between the reference frame image and the second target frame image according to the target mapping relation and the first mapping relation; based on the estimated mapping relation, a projection frame image obtained by re-projecting the second target frame image to the reference frame image is obtained; acquiring correction information of the estimated mapping relation according to the projection frame image and the reference frame image; and correcting the estimated mapping relation by using the correction information to obtain a second mapping relation between the reference frame image and the second target frame image.
In some embodiments, the planar area determination module 406 is specifically configured to: adjusting the brightness of the projection frame image so that the adjusted brightness matches the brightness of the reference frame image; and obtaining correction information of the estimated mapping relation based on the projection frame image and the reference frame image after brightness adjustment.
In some embodiments, the planar area determination module 406 is specifically configured to: determining a mapping relation between the projection frame image and the reference frame image after brightness adjustment according to the gray scale and gray scale gradient of a target point of a reference target plane area in the reference frame image; and obtaining correction information of the estimated mapping relation based on the mapping relation between the projection frame image and the reference frame image after brightness adjustment.
The image processing device provided by the embodiment of the disclosure can execute the image processing method provided by any embodiment of the disclosure, and has the corresponding functional modules and beneficial effects of the execution method.
It will be clearly understood by those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described apparatus embodiments may refer to corresponding procedures in the method embodiments, which are not described herein again.
The embodiment of the disclosure provides an electronic device, which includes: a storage device having a computer program stored thereon; processing means for executing the computer program in the storage means to implement the steps of any of the methods of the present disclosure.
Referring now to fig. 5, a schematic diagram of an electronic device 500 suitable for use in implementing embodiments of the present disclosure is shown. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and the like, and stationary terminals such as digital TVs, desktop computers, and the like. The electronic device shown in fig. 5 is merely an example and should not be construed to limit the functionality and scope of use of the disclosed embodiments.
As shown in fig. 5, the electronic device 500 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 501, which may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 502 or a program loaded from a storage means 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic apparatus 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
In general, the following devices may be connected to the I/O interface 505: input devices 506 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 507 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 508 including, for example, magnetic tape, hard disk, etc.; and communication means 509. The communication means 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. While fig. 5 shows an electronic device 500 having various means, it is to be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a non-transitory computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication means 509, or from the storage means 508, or from the ROM 502. The above-described functions defined in the methods of the embodiments of the present disclosure are performed when the computer program is executed by the processing device 501.
In addition to the methods and apparatus described above, embodiments of the present disclosure may also be computer program products comprising computer program instructions which, when executed by a processor, cause the processor to perform the image processing methods provided by the embodiments of the present disclosure.
The computer program product may write program code for performing the operations of embodiments of the present disclosure in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server.
Further, embodiments of the present disclosure may also be a computer-readable storage medium having stored thereon computer program instructions which, when executed by a processor, cause the processor to perform the image processing method provided by the embodiments of the present disclosure.
The computer readable storage medium may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The disclosed embodiments also provide a computer program product comprising a computer program/instruction which, when executed by a processor, implements the image processing method in the disclosed embodiments.
It will be appreciated that prior to using the technical solutions disclosed in the embodiments of the present disclosure, the user should be informed and authorized of the type, usage range, usage scenario, etc. of the personal information related to the present disclosure in an appropriate manner according to the relevant legal regulations.
For example, in response to receiving an active request from a user, a prompt is sent to the user to explicitly prompt the user that the operation it is requesting to perform will require personal information to be obtained and used with the user. Thus, the user can autonomously select whether to provide personal information to software or hardware such as an electronic device, an application program, a server or a storage medium for executing the operation of the technical scheme of the present disclosure according to the prompt information.
As an alternative but non-limiting implementation, in response to receiving an active request from a user, the manner in which the prompt information is sent to the user may be, for example, a popup, in which the prompt information may be presented in a text manner. In addition, a selection control for the user to select to provide personal information to the electronic device in a 'consent' or 'disagreement' manner can be carried in the popup window.
It will be appreciated that the above-described notification and user authorization process is merely illustrative, and not limiting of the implementations of the present disclosure, and that other ways of satisfying relevant legal regulations may be applied to the implementations of the present disclosure.
It should be noted that in this document, relational terms such as "first" and "second" and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The foregoing is merely a specific embodiment of the disclosure to enable one skilled in the art to understand or practice the disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown and described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (12)
1. An image processing method, comprising:
acquiring a first target frame image and a second target frame image in a target video; wherein the first target frame image is a previous frame image of the second target frame image;
acquiring gray information of a target point of a first target plane area in the first target frame image;
determining a target mapping relation between the second target frame image and the first target frame image according to the gray information of the target point, so as to determine a second target plane area in the second target frame image based on the target mapping relation; wherein the second target plane area matches the first target plane area.
2. The method of claim 1, wherein the acquiring the first target frame image and the second target frame image in the target video comprises:
performing feature point matching on adjacent frame images in the target video to obtain a feature point matching result;
and taking the adjacent frame images with the matching errors larger than a preset error threshold corresponding to the characteristic point matching results as a first target frame image and a second target frame image.
3. The method according to claim 1, wherein the target point is a point in the first target plane area where a gray gradient is greater than a preset gradient threshold.
4. The method of claim 1, wherein the gray information comprises gray and gray gradient; the determining, according to the gray information of the target point, a target mapping relationship between the second target frame image and the first target frame image includes:
determining a corresponding point of the target point in the second target frame image based on an initial mapping relation preset between the second target frame image and the first target frame image, and acquiring the gray level of the corresponding point;
according to the gray gradient of the target point and the difference between the gray of the target point and the gray of the corresponding point, performing adjustment operation on the initial mapping relation until a target mapping relation is obtained; wherein the adjusting operation is used to reduce the difference.
5. The method of claim 4, wherein performing an adjustment operation on the initial mapping until a target mapping is obtained comprises:
and executing the adjustment operation of the designated times based on the initial mapping relation so as to determine the target mapping relation according to the mapping relation obtained after the adjustment of the designated times.
6. The method according to any one of claims 1 to 5, wherein the determining a second target plane area in the second target frame image based on the target mapping relation comprises:
acquiring a first mapping relation between a reference frame image of the target video and the first target frame image;
acquiring a second mapping relation between the reference frame image and the second target frame image according to the target mapping relation and the first mapping relation;
determining a second target plane area in the second target frame image according to the reference target plane area in the reference frame image and the second mapping relation; the reference target plane area, the first target plane area and the second target plane area all correspond to the same plane.
7. The method of claim 6, wherein the obtaining a second mapping between the reference frame image and the second target frame image according to the target mapping and the first mapping comprises:
determining a pre-estimated mapping relation between the reference frame image and the second target frame image according to the target mapping relation and the first mapping relation;
based on the estimated mapping relation, a projection frame image obtained by re-projecting the second target frame image to the reference frame image is obtained;
acquiring correction information of the estimated mapping relation according to the projection frame image and the reference frame image;
and correcting the estimated mapping relation by using the correction information to obtain a second mapping relation between the reference frame image and the second target frame image.
8. The method of claim 7, wherein the obtaining correction information of the estimated mapping relationship according to the projection frame image and the reference frame image comprises:
adjusting the brightness of the projection frame image so that the adjusted brightness matches the brightness of the reference frame image;
And obtaining correction information of the estimated mapping relation based on the projection frame image and the reference frame image after brightness adjustment.
9. The method according to claim 8, wherein the obtaining the correction information of the estimated mapping relationship based on the luminance-adjusted projection frame image and the reference frame image includes:
determining a mapping relation between the projection frame image and the reference frame image after brightness adjustment according to the gray scale and gray scale gradient of a target point of a reference target plane area in the reference frame image;
and obtaining correction information of the estimated mapping relation based on the mapping relation between the projection frame image and the reference frame image after brightness adjustment.
10. An image processing apparatus, comprising:
the frame image acquisition module is used for acquiring a first target frame image and a second target frame image in the target video; wherein the first target frame image is a previous frame image of the second target frame image;
the gray information acquisition module is used for acquiring gray information of a target point of a first target plane area in the first target frame image;
a plane area determining module, configured to determine a target mapping relationship between the second target frame image and the first target frame image according to gray information of the target point, so as to determine a second target plane area in the second target frame image based on the target mapping relationship; wherein the second target plane area matches the first target plane area.
11. An electronic device, the electronic device comprising:
a storage device having a computer program stored thereon;
processing means for executing said computer program in said storage means to carry out the steps of the image processing method according to any one of claims 1-9.
12. A computer-readable storage medium, characterized in that the storage medium stores a computer program for executing the image processing method according to any one of the preceding claims 1-9.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310779515.5A CN116777955A (en) | 2023-06-28 | 2023-06-28 | Image processing method, device, equipment and medium |
| PCT/CN2024/102133 WO2025002288A1 (en) | 2023-06-28 | 2024-06-27 | Image processing method and apparatus, device, and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310779515.5A CN116777955A (en) | 2023-06-28 | 2023-06-28 | Image processing method, device, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116777955A true CN116777955A (en) | 2023-09-19 |
Family
ID=88007867
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310779515.5A Pending CN116777955A (en) | 2023-06-28 | 2023-06-28 | Image processing method, device, equipment and medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN116777955A (en) |
| WO (1) | WO2025002288A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2025002288A1 (en) * | 2023-06-28 | 2025-01-02 | 北京字跳网络技术有限公司 | Image processing method and apparatus, device, and medium |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110881117A (en) * | 2018-09-06 | 2020-03-13 | 杭州海康威视数字技术股份有限公司 | Inter-picture area mapping method and device and multi-camera observation system |
| CN109461174B (en) * | 2018-10-25 | 2021-01-29 | 北京陌上花科技有限公司 | Video target area tracking method and video plane advertisement implanting method and system |
| CN111179315A (en) * | 2019-12-31 | 2020-05-19 | 湖南快乐阳光互动娱乐传媒有限公司 | Video target area tracking method and video plane advertisement implanting method |
| CN114565872B (en) * | 2022-02-25 | 2025-03-28 | 腾讯科技（深圳）有限公司 | Video data processing method, device, equipment and computer readable storage medium |
| CN116777955A (en) * | 2023-06-28 | 2023-09-19 | 北京字跳网络技术有限公司 | Image processing method, device, equipment and medium |
-
2023
- 2023-06-28 CN CN202310779515.5A patent/CN116777955A/en active Pending
-
2024
- 2024-06-27 WO PCT/CN2024/102133 patent/WO2025002288A1/en unknown
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2025002288A1 (en) * | 2023-06-28 | 2025-01-02 | 北京字跳网络技术有限公司 | Image processing method and apparatus, device, and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2025002288A1 (en) | 2025-01-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9489722B2 (en) | Method and apparatus for implementing image denoising | |
| US10187546B2 (en) | Method and device for correcting document image captured by image pick-up device | |
| AU2017232186A1 (en) | Fast and robust image alignment for burst mode | |
| CN112306301B (en) | Touch data processing method, device, equipment and storage medium | |
| CN104318259A (en) | Target picture identifying device and method for and computing device | |
| CN109118447B (en) | A picture processing method, picture processing device and terminal equipment | |
| CN115209031B (en) | Video anti-shake processing method and device, electronic equipment and storage medium | |
| CN109886864B (en) | Privacy mask processing method and device | |
| KR20200017185A (en) | Apparatus and method for applying noise pattern to image processed bokeh | |
| US8984424B2 (en) | Layout of user interface elements | |
| US20150253880A1 (en) | Image segmentation device and image segmentation method | |
| CN116777955A (en) | Image processing method, device, equipment and medium | |
| CN111275622B (en) | Image stitching method, device and terminal equipment | |
| CN113099101B (en) | Camera shooting parameter adjusting method and device and electronic equipment | |
| CN109360167B (en) | Infrared image correction method, device and storage medium | |
| CN111966473B (en) | Operation method and device of linear regression task and electronic equipment | |
| CN104978731A (en) | Information processing method and electronic equipment | |
| CN117409174B (en) | Speckle image temperature compensation method and device, readable medium and electronic equipment | |
| CN109559319B (en) | Normal map processing method and terminal | |
| CN116485645A (en) | Image stitching method, device, equipment and storage medium | |
| CN116071280A (en) | Video complement method, device, medium and electronic equipment | |
| US20240289929A1 (en) | Methods and apparatus to perform mask-based depth enhancement for multi-view systems | |
| CN110209851B (en) | Model training method and device, electronic equipment and storage medium | |
| CN112819693A (en) | Sliding verification code generation method and device, electronic equipment and computer readable medium | |
| CN110827313A (en) | Fast optical flow tracking method and related equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |