JP2008046749A - Image processing method and device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable an operator to easily confirm that an index has been correctly installed when an imaging apparatus is positioned using an image obtained by photographing the index. <P>SOLUTION: This image processing method includes: an index detection step for detecting an index from a photographed image; a position posture calculation step for calculating the position posture of the detected index; and a holding step for holding the calculated position posture of the index, and the image processing method for updating the held position posture of the index by applying the detection step and the position posture calculation step to the acquired image. This image processing method is characterized in that it includes: a determination step of determining the index by which a projected image should be superimposed on the image; a generation step of generating the projected image of the index which is determined that the projected image should be superimposed on the image based on the held position posture of the index; and a superimposition step of superimposing the generated projected image on the image. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a method for calculating the position and orientation of an index from a captured image.

  One of the important technologies for providing high-quality mixed reality is alignment processing between the real space and the virtual space. Various methods have been proposed as the alignment processing. Among them, there is a method for calculating the relative position and orientation of the index and the imaging device (camera) from the image (index image or marker image) obtained by capturing the index (marker). Widely used in terms of cost and usability (see Non-Patent Document 1). In this specification, an image in which a marker is photographed is expressed as a marker image.

  Using a three-dimensional graphics (3DCG) technique, it looks as if another marker whose relative position and orientation with the marker in the marker image is known from the relative position and orientation between the marker in the photographed marker image and the camera. Can be calculated and drawn. In this specification, the drawn marker image is expressed as a projection marker. An actual marker is expressed as a real marker, and an image area of the real marker in the marker image is expressed as an image marker.

  By superimposing and displaying the projection marker on the marker image, it is possible to visualize the displacement between the position and orientation of the image marker and the projection marker. The operator can grasp the operation confirmation of the alignment processing, the accuracy, the error in the marker installation position and posture, the marker installation forgotten, and the like from the image on which the projection marker is superimposed.

  On the other hand, the alignment process is based on the premise that the positions and orientations of all the real markers are known, and requires an operation of measuring and inputting the marker positions and orientations in advance. Depending on the number of markers and the installation position / posture, it is unrealistic for a human to perform measurement manually, and means for efficiently and highly accurately measuring the position / posture of the marker is required.

In recent years, a method related to marker calibration for calculating a relative position and orientation between markers from a marker image has been proposed (see Non-Patent Document 2). After obtaining the relative positions and orientations between all markers by marker calibration, the positions and orientations of all markers in the world coordinate system can be obtained by determining the positions and orientations of any marker in the world coordinate system.
S. Uchiyama, K .; Takemoto, K .; Satoh, H .; Yamamoto, and H.K. Tamura: “MR Platform: A basic body on which mixed reality applications are built,” Proc. IEEE / ACM Int'l Symp. on Mixed and Augmented Reality (ISMAR 2002), pp. 246-253, 2002. Kotake, Uchiyama, Yamamoto: Marker calibration method using pioneering knowledge about marker placement, Journal of the Virtual Reality Society of Japan, vol. 10, no. 3, pp. 401-410, 2005.

  By superimposing the projected marker on the marker image and comparing the projected marker with the image marker, the operator can grasp the operation of the alignment process, accuracy, incorrect marker installation position and orientation, marker forgotten installation, etc. Can do.

  At this time, if the projection markers of all the markers whose positions and orientations are known are drawn, there is a possibility that the operator who sees the image on which the projection markers are superimposed will be confused. For example, when the projected marker of a partition installed on the back side of a partition or a column when viewed from the operator is superimposed, whether the marker is forgotten to be installed on the front side of the partition or column viewed from the operator, or the position and orientation of the marker on the back side Cannot be easily determined.

  In addition, when positioning a movable object having six degrees of freedom from an image, a marker must be placed on the object surface viewed from the camera regardless of the value of the relative posture with the camera. When marker calibration is performed by setting a marker on the object surface so as to satisfy this condition, a projection marker corresponding to the marker on the front side of the object viewed from the camera during the marker calibration and a projection marker corresponding to the marker on the back side Are displayed overlapping. For this reason, the operator is more likely to be confused.

  The present invention has been made in view of the above problems, and an object of the present invention is to prevent an operator from being confused by superimposing a projected image of an index existing in an image on the image.

  To achieve the above object, the present invention provides an index detection step for detecting an index from a captured image, a position and orientation calculation step for calculating a position and orientation of the detected index, and a position and orientation of the calculated index. An image processing method for updating the position and orientation of the held index by performing the detection step and the position and orientation calculation step on the acquired image. And a determination step of determining an index on which the projection image is to be superimposed on the image, and a projection image of the index on which the projection image is determined to be superimposed on the image based on the position and orientation of the held index And a superimposing step of superimposing the generated projection image on the image.

  According to the present invention, it is possible to reduce the confusion of an operator who has seen an image on which a projection marker is superimposed. Therefore, for example, it is possible to improve the efficiency of confirmation of the operator alignment processing and processing in marker calibration.

(Example 1)
The first embodiment is an example in which the relative position and orientation between the camera and the marker are calculated from the marker image and the projected marker is superimposed on the marker image in the marker calibration.

  FIG. 1 shows a marker format in this embodiment, and FIG. 2 shows an example of an actual marker. The marker is one large square composed of 16 small squares. The small square is filled with white or black and represents 1-bit information. If it is painted white, it represents 0, and if it is black, it represents 1. A marker ID is expressed by a combination of small squares in black and white.

  In this embodiment, the length of one side of the marker (the length of one side of the large square) is the same for all markers, and is a predetermined value determined in advance.

  The posture of the marker is determined by how the marker coordinate system defined on the marker is rotated with respect to the world coordinate system. FIG. 3 shows how the marker coordinate system is defined.

  The direction of each axis of the marker coordinate system is determined by four small squares at the corners of the large square. Three of the four small squares are always black (blk1,2,3), one is always white (wht), the direction of the line connecting blk1,2 is the positive direction of the x coordinate axis, and blk3,2 The direction of the connecting line segment is the positive direction of the y coordinate axis. The normal direction (marker direction) on the marker front side coincides with the positive direction of the z coordinate axis. The position of the marker is determined by the position of the origin of the marker coordinate system in the world coordinate system. The origin of the marker coordinate system is the center of the large square as shown in FIG.

  The marker ID is determined by eight small squares of information bits (x1 to x8). The range of the marker ID is 8 bits, that is, 0 to 255. The other four small squares are inspection bits (c1 to c4), and are used for correction when an information bit is erroneously recognized by image processing.

The value of the check bit can be obtained from the following equation.
c1 = not (x1 + x2 + x3 + x5 + x6)
c2 = not (x2 + x3 + x4 + x6 + x7)
c3 = not (x3 + x4 + x5 + x7 + x8)
c4 = not (x1 + x4 + x5 + x6 + x8)
The relative position and orientation between the camera and the marker can be obtained from the shape of the quadrilateral in the marker image, the length of the side of the quadrilateral, and the center position of the quadrilateral in the marker image.

  The marker used in the present embodiment has been described above.

  FIG. 4 is a block diagram illustrating a hardware configuration of the image processing apparatus 100 according to the present embodiment.

  The input unit 110 is an imaging device (camera) such as a video camera, and captures a marker to be calibrated.

  The acquisition unit 120 acquires a still image (snapshot) of the video captured by the input unit 110 and transmits the still image (snapshot) to the marker detection unit 130, or the video captured by the input unit 110 for the marker detection unit 130 and the synthesis unit 170. Or send to. The snapshot is acquired at an arbitrary time according to an instruction from the operator. Further, it is assumed that the video is continuously shot except for taking a snapshot.

  The marker detection unit 130 detects a marker from the video and snapshot taken by the input unit 110.

  When a snapshot is sent from the acquisition unit 120, marker detection is performed from the snapshot. On the other hand, when an image is sent, marker detection processing is performed on each frame constituting the image, and the result is transmitted to the position and orientation calculation unit 140.

  The position and orientation calculation unit 140 calculates the relative position and orientation of the detected marker and the video camera and the relative position and orientation of the markers from the marker detection result sent from the marker detection unit 130.

  When the marker detection unit 130 receives information about the marker detected from the snapshot, the relative position and orientation of the detected marker and the camera are calculated. Subsequently, the relative position and orientation of the markers existing in the same snapshot are calculated. The relative position and orientation of the markers are stored in the storage unit 150. Then, all combinations of the relative positions and orientations of the markers stored in the storage unit 150 are read, and the relative positions and orientations of all the markers are calculated and stored in the storage unit 150.

  When marker information detected from each frame of the video is sent from the marker detection unit 130, the relative position and orientation of the marker detected from each frame and the video camera at the time of frame shooting are calculated. Subsequently, the relative position and orientation of all the markers and the camera are calculated from the relative position and orientation information of the marker read from the storage unit 150 and the calculated relative position and orientation of the camera, and are stored in the storage unit 150.

  As shown in FIG. 5, the storage unit 150 stores a marker definition information table associated with marker IDs, relative positions and orientations, detection marker flags, and reference marker flags.

  The reference marker is a marker that serves as a reference for the relative position and orientation. In this embodiment, the first marker detected from the snapshot is set as the reference marker. The position and orientation are relative positions and orientations with respect to the reference marker. Here, it is assumed that the expression by the axis and the rotation angle is used as the posture, but the expression is not limited to this, and the expression by Euler angle may be used.

  The detection marker flag is a flag indicating whether or not the target marker exists in the snapshot or frame sent from the marker detection unit 130. The detection marker flag is used for drawing in the generation unit 160 described later.

  The marker definition information table is updated every time a marker detection processing result in a snapshot or a frame is sent from the marker detection unit 130, the position and orientation of all markers are recalculated, and detection flags are updated. Is retained.

  Further, the marker stored in the marker definition information table is not limited to the marker detected by the marker detection unit 130, and may be a manually added marker. For example, it is possible to operate the image processing apparatus of the present embodiment by manually defining all the markers without acquiring any snapshot.

  The generation unit 160 generates an image of the projection marker in the frame where the marker is detected from the relative position and orientation between the camera and the marker, which is the calculation result of the position and orientation calculation unit 140. At this time, the drawing process is changed for a marker for which a detection flag in the marker definition information table is set and a marker for which it is not.

  The synthesis unit 170 superimposes the projection marker image generated by the generation unit 160 on the video frame received from the acquisition unit 120. Note that the frames transmitted from the acquisition unit 120 are synchronized so as to be the same as the frames for which the marker detection processing has been performed by the marker detection unit 130. The display unit 180 displays the video frame synthesized by the synthesis unit 170.

  Heretofore, the hardware configuration and the operation outline of the present embodiment have been described.

  The procedure for marker calibration of the relative position and orientation of the marker attached to the surface of the object and the flow of processing will be described using the flowchart of FIG.

  Here, a rectangular parallelepiped in which markers having the same marker ID as the dice eyes are attached to each surface as shown in FIGS. 7 to 8 will be described as an example of an object. 7 shows an object viewed from the front, FIG. 8 shows the object viewed from the back, and FIG. 9 shows the object viewed from the right. Reference numerals 201 to 206 denote markers attached to a rectangular parallelepiped, which correspond to the marker IDs 1 to 6, respectively.

  Further, it is assumed that the coordinate system of the camera that performs shooting is defined as shown in FIG. The shooting direction of the camera is the negative direction of the z coordinate axis, the right hand direction is the positive direction of the x coordinate axis, and the upward direction is the positive direction of the y coordinate axis.

  In response to the operator's snapshot acquisition instruction, the image processing apparatus acquires a snapshot (S100), and detects a marker in the snapshot (S102). Subsequently, the relative positions and orientations of all the detected markers and the camera are calculated (S104). If the detected marker is not described in the marker definition information table, it is added, and all the relative positions and orientations of the markers described in the marker definition information table are recalculated and updated. Then, the detection flag in the marker definition information table of the detected marker is updated (S106). At this time, if the reference marker has not been determined, it is determined, and the reference marker flag of the marker that has become the reference marker in the marker definition information table is set. The calculation result is held as a marker definition information table and the process returns to S100.

  For example, when taking a snapshot of the rectangular parallelepiped 200 to which the marker is attached as shown in FIG. 7, the markers ID1 (201), ID2 (202), and ID4 (204) are detected, and the relative position and orientation are calculated. The At this time, the marker of ID1 detected first automatically is determined as the reference marker. Markers of ID1, ID2, and ID4 are registered in the marker definition information table, and a reference marker flag of ID1 is set. The position and orientation of ID2 and ID4 are updated with the relative position and orientation with the marker of ID1. ID1, ID2, and ID4 marker detection flags are also set.

  Subsequently, when taking a snapshot of the rectangular parallelepiped 200 to which the marker is attached as shown in FIG. 9, markers ID2 (202), ID4 (204), and ID6 (206) are detected. Markers of ID2 and ID6 are added to the marker definition information table. The relative position and orientation from the ID2 and ID6 reference markers are calculated via the relative position and orientation of the ID4 marker, and are described in the marker definition information table. At the same time, the detection flags for all markers are temporarily lowered, and only the detection flags for the ID2, ID4, and ID6 markers are set.

  In this way, the relative positions and orientations of ID1, ID2, ID4, and ID6 are calculated.

  If there is no snapshot acquisition instruction, the image processing apparatus continues to shoot video. Then, marker detection is performed from the frame of the captured video (S110). The relative positions and orientations of all the markers detected from the frame and the camera are calculated (S112). If the detected marker is not described in the marker definition information table, it is added and all the relative positions and orientations of the markers described in the marker definition information table are recalculated and updated. Then, the detection flag in the marker definition information table of the detected marker is updated (S114).

  Of the markers described in the marker definition information table, projection markers are drawn for markers for which a detection flag is set (S116, S118). Projection markers are not drawn for markers for which no detection flag is set. The projection marker drawing result calculated from this frame is superimposed on the video frame (S120) and displayed (S122).

  For example, when the snapshot is taken from the direction in which the rectangular solid with the marker attached is seen as shown in FIGS. 7 and 9, the relative positions and orientations of ID1, ID2, ID4, and ID6 are known. At this time, when the rectangle 200 is again photographed from the direction that looks like FIG. 7 and all the registered projection markers are drawn, the result is as shown in FIG. In addition to the image markers 201, 202, and 204 of ID1, ID2, and ID4 in the video frame, projection markers 301, 302, 304, and 306 of ID1, ID2, ID4, and ID6 are displayed. That is, the projection marker 306 of ID6 that does not exist in the video frame is displayed as shown in FIG. Therefore, in the present embodiment, only the markers detected from the video are drawn from the markers described in the marker definition information table using the detection flag. Thereby, as shown in FIG. 12, the projection marker of the marker which is not detected is not drawn.

  As described above, how the projected marker is displayed in the procedure and processing for performing the marker calibration of the relative position and orientation of the marker attached to the surface of the object has been described.

  According to the present embodiment, when performing the marker calibration of the relative position and orientation of the marker attached to the surface of the movable object having 6 degrees of freedom, the projection marker of the marker that cannot be seen from the camera is not drawn. According to the present invention, it is possible to prevent the projection marker of the marker that cannot be seen from the camera from being drawn, so that it is possible to avoid confusion when viewing the match between the projection marker and the image marker on the screen.

  In particular, in the case of a relatively small 6-DOF movable object that can be held and moved freely by hand, if all projection markers are drawn, the object is covered with the projection markers. According to the present embodiment, it is possible to prevent the object from being covered with the projection marker.

  Also, in this embodiment, projection markers that are invisible to the camera are not drawn, but they are not drawn at all, but they are drawn in another color or a dull and light color with low lightness so as not to cause confusion. It may be. Furthermore, the operator may select a method of drawing / not drawing / drawing with another color. In this embodiment, the object on which the marker is installed is movable, but it may be fixedly installed. Furthermore, instead of marker calibration, the same processing may be performed in the alignment processing.

(Example 2)
Example 2 is an example in which the inner product of the imaging direction of the imaging apparatus and the normal direction of the index is used as means for determining whether or not an image marker exists in the marker image. The basic configuration and operation are the same as in the first embodiment.

  The flowchart in FIG. 13 explains the procedure and process flow for calibrating the relative position and orientation of the marker attached to the surface of the object. Except for S215 and S216, the processing is the same as in the first embodiment. For the same processing as in Example 1, the last two digits of the steps are the same as in FIG.

  When there is no snapshot acquisition instruction, the image processing apparatus continues to shoot video. Then, marker detection is performed from the frame of the captured video (S210). Subsequently, the relative positions and orientations of all the detected markers and the camera are calculated (S212).

  If the detected marker is not described in the marker definition information table, it is added, and all the relative positions and orientations of the markers described in the marker definition information table are recalculated and updated. (S214).

  Finally, the normal product of the markers described in all the marker definition information tables and the inner product in the imaging direction of the camera are calculated (S215). A marker whose inner product is smaller than 0 draws a projected marker and is greater than or equal to zero. The marker does not draw a projection marker (S216).

  After the above processing is performed for the markers described in the marker definition information table, the projection marker drawing result calculated from this frame is superimposed on the frame of the video (S220) and displayed (S222).

  In this embodiment, the inner product of the imaging direction of the imaging device and the normal direction of the index is used as means for determining whether an image marker exists in the marker image.

  Humans can recognize that a marker is reflected, but if the marker cannot be recognized by image processing, for example, if only a half of the marker is reflected at the edge of the image as shown in FIG. The projection marker 312 of the marker 212 is not drawn.

  On the other hand, in the processing method according to the second embodiment, the projection marker 312 of the marker 212 can be drawn.

  Furthermore, by making it possible to select a projection marker drawing method for the unobservable marker from methods prepared in advance, it becomes possible to use the marker flexibly according to the color of the object on which the marker is installed and the marker installation density.

(Other embodiments)
Also, an object of the present invention is to supply a recording medium (or storage medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded to a system or apparatus, and a computer (or CPU or CPU) of the system or apparatus. Needless to say, this can also be achieved when the MPU) reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Furthermore, after the program code read from the recording medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

It is a figure which shows the format of a marker. It is a figure which shows the example of an actual marker. It is a figure which shows the coordinate system of a marker. 1 is a configuration diagram of an image processing apparatus 100 in Embodiment 1. FIG. It is an example of the marker definition information table in Example 1. 4 is a series of operations and processing for marker calibration in the first embodiment. It is an example of observation of the rectangular parallelepiped 200 with which the marker was affixed. It is an example of observation of the rectangular parallelepiped 200 with which the marker was affixed. It is an example of observation of the rectangular parallelepiped 200 with which the marker was affixed. It is a figure which shows the coordinate system of a camera. It is an example at the time of drawing all the projection markers. It is an example of the image on which the projection marker in Example 1 was superimposed. 7 is a series of operations and processing for marker calibration in the second embodiment. It is an example of the image on which the projection marker in Example 2 was superimposed.

Claims (5)

An index detection step of detecting an index from the captured image;
A position and orientation calculation step of calculating a position and orientation of the detected index;
Holding the calculated position and orientation of the index,
It is an image processing method for updating the position and orientation of the held index by performing the detection step and the position and orientation calculation step on the acquired image,
A determination step of determining an index on which the projection image should be superimposed on the image;
Generating a projected image of the index determined to be superimposed on the image based on the held position and orientation of the index;
An image processing method comprising: a superimposing step of superimposing the generated projection image on the image.   The image processing method according to claim 1, wherein the determination step determines an index on which the projection image is to be superimposed on the image based on a detection result of the index detection step.   2. The image processing according to claim 1, wherein the determining step determines an index on which the projection image is to be superimposed on the image based on a shooting direction of an imaging apparatus that has shot the image and a normal direction of the index. Method.   A program recorded so as to be readable by a computer in order to realize the image processing method according to any one of claims 1 to 3. Index detecting means for detecting the index from the captured image;
Position and orientation calculation means for calculating the position and orientation of the detected index;
Holding means for holding the calculated position and orientation of the index,
An image processing apparatus that updates the position and orientation of the held index by performing detection by the detection unit and calculation of a position and orientation by the position and orientation calculation unit with respect to the acquired image.
Determining means for determining an index on which a projection image should be superimposed on the image;
Generating means for generating a projected image of the index determined to be superimposed on the image based on the position and orientation of the index held;
An image processing apparatus comprising: a superimposing unit that superimposes the generated projection image on the image.

Images