JP4167482B2 - Image conversion method and image conversion apparatus - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a method and apparatus for imaging an imaging object such as an inspection object from an oblique direction, and converting the captured image data into image data captured from a direction facing the imaging object, and these method and apparatus. The present invention relates to a used semiconductor manufacturing apparatus and manufacturing inspection line.
[0002]
[Prior art]
When measuring the position of an object from the process of making a map from an aerial photograph or the position on the photograph, the position on the object in real space coordinates (hereinafter sometimes referred to as “world coordinates”) and the object It is necessary to know the correspondence between the photographed image (hereinafter also referred to as “image”) and the position on the screen coordinates. In addition, there is a demand for finding a pattern having a specific shape from the image of the target object and projecting the target image on the imaging surface (hereinafter also referred to as “image search”). There is also. In this case, the image of the image being captured is taken into account by taking into account the external parameters of the imaging device such as the imaging direction that is the direction of the object viewed from the imaging device and the internal parameters such as the optical characteristics of the imaging system of the imaging device. It is necessary to perform processing such as image search in consideration of distortion.
[0003]
Conventionally, a reference point is provided in an object, the world coordinate value of this reference point is associated with a screen coordinate value, and external and internal parameters of the image pickup device are obtained with high accuracy. There is a method for performing calibration (see, for example, Patent Document 1). According to this method, if the imaging device is calibrated regardless of the direction of imaging or the distance between the object and the imaging device, specifying an unknown point on the image allows the world coordinates of the unknown point to be specified. Can be obtained. Therefore, since the world coordinates of the unknown point on the object can be obtained using the image obtained by imaging the object, this is a useful technique for measuring the position of the object from the position on the photograph.
[0004]
In addition, in a semiconductor manufacturing apparatus such as a stepper, there is a wafer orientation flat detection apparatus that detects an orientation flat (hereinafter, abbreviated as “orientation flat”) of a semiconductor wafer (see, for example, Patent Document 2 and Patent Document 3). ). In this orientation flat detection device for a wafer, an imaging device for imaging a wafer (imaging target) having an orientation flat is installed at a position facing the wafer so that the imaging target is not distorted and imaged.
[0005]
[Patent Document 1]
JP 2001-264037
[Patent Document 2]
JP-A-9-14918
[Patent Document 3]
JP-A-9-14938
[0006]
[Problems to be solved by the invention]
However, the conventional method using the imaging device calibration is to specify the unknown point on the captured image plane and obtain the world coordinates of the target unknown point. That is, since the conventional method only associates the position in the image (screen coordinates) and the position in space (world coordinates) based on the image, the image is obtained from an image obtained by imaging the object from an oblique direction. When a screen search is performed, there is a problem that an object is distorted. That is, as a model pattern that is a search purpose, a normal direction of the pattern, for example, a shape viewed from directly above is given, but from an image captured from an oblique direction based on it, This is because the model pattern is searched. This means that it is difficult to perform a screen search unless an image captured from an oblique direction is converted into an image captured from a facing direction by some method.
[0007]
In addition, in the prior art wafer orientation flat detection device (for example, see Patent Document 2 and Patent Document 3), the image pickup device for picking up an image of the wafer (object to be picked up) is arranged so that the wafer is not distorted and picked up. It is installed in the position perpendicular to. However, since the semiconductor manufacturing equipment is installed in a clean room, it is necessary to make the size as small as possible, and it is difficult to attach the imaging device so that the wafer can be imaged from the direction perpendicular to the upper surface of the wafer. There is.
[0008]
Further, the screen search technique is an important technique in addition to the detection of the orientation flat of the semiconductor manufacturing apparatus, for example, in the monitoring or inspection of articles in a manufacturing line including a plurality of manufacturing processes. Therefore, a technique for converting an image captured from an oblique direction into an image captured from a facing direction is also required in monitoring and inspection of articles on these production lines.
[0009]
[Means for Solving the Problems]
Therefore, the image conversion method and the image conversion apparatus of the present invention have the following steps or means.
[0010]
That is, the method and apparatus according to the present invention captures an imaging object from an oblique direction by an imaging device including an imaging optical system and a light receiving plane, and also defines an oblique light receiving plane (hereinafter referred to as “first light receiving plane”). The first image data obtained in (1)) is obtained on a directly-facing light receiving plane (hereinafter also referred to as a “second light receiving plane”) when the imaging apparatus temporarily captures an image of the object to be imaged. An image conversion method and apparatus for converting to second image data.
[0011]
First, coordinate conversion is performed to determine the relationship between the first plane coordinate for determining the pixel position on the oblique light receiving plane and the second plane coordinate for determining the pixel position on the opposite light receiving plane. Next, the light intensity of one pixel constituting the directly-facing light receiving plane is obtained as described below. One pixel constituting the directly-facing light receiving plane is received across a plurality of pixels constituting the oblique light receiving plane when projected onto the oblique light receiving plane. Therefore, by calculating how much area this one pixel is projected to which pixel constituting the oblique light receiving plane, the amount of light received by the pixels constituting the oblique light receiving plane is calculated for each of the plurality of pixels. Weighting in proportion to the area of the projected portion is performed to obtain a weighted average, and this value is used as the light intensity of one pixel constituting the directly-facing light receiving plane, and re-sampling is performed.
[0012]
In carrying out the present invention, it is preferable to perform the following procedure in order to perform coordinate conversion for determining the relationship between the first plane coordinates and the second plane coordinates.
(A) Multiple points selected to include at least 4 points selected so that 3 or more points do not line up on a straight line, or at least 2 points outside this line, including 3 points on a straight line (These points are hereinafter referred to as “a plurality of characteristic points”.) An imaging object having a mark thereon, and the coordinate value of the mark is known by a coordinate system provided on the plane where the mark exists. Using the imaging object as a test pattern, the imaging device captures the test pattern.
(B) The coordinate values of the first plane coordinates are read for marks on a plurality of characteristic points in the test pattern that are imaged on the first light receiving plane when imaged from an oblique direction.
(C) An arbitrary position in the test pattern is defined as the origin, and based on the image captured on the first light receiving plane of the test pattern captured from an oblique direction, the test pattern has the origin at the second light reception. When the image is taken from the opposite direction projected on the central portion of the plane, the position of the central portion of the second light receiving plane on which the origin is projected and the test pattern image taken from the oblique direction are geometrically The scale for similarity transformation is virtually set.
(D) Second light reception when each mark in the test pattern is imaged from a virtually opposite direction with respect to marks on a plurality of characteristic points imaged from an oblique direction Projecting on a plane, the position of the mark in the second plane coordinates is calculated.
(E) the first coordinate of each of the marks on the plurality of characteristic points in the test pattern projected onto the first light receiving plane when imaged from an oblique direction, and a virtually opposite direction set Based on the second coordinates of the marks on the plurality of characteristic points in the calculated test pattern on the second light receiving plane when imaged from the coordinate conversion parameter b_i(Where i = 1, 2, 3, 4, 5, 6, 7, 8).
[0013]
In order to perform the calibration step of virtually setting the center position and scale in the second plane coordinate system when imaging from the directly facing direction, it is preferable to perform the following.
(C ′) The scale is determined so that the image dimensions are equal at the position of the central portion of the second light receiving plane that is virtually set to the position of the central portion of the first light receiving plane, and The position of the central portion is virtually set when the imaging object is imaged from the facing direction so that the positions of the central portions coincide.
[0014]
  In implementing the present invention, the resampling is performed by calculating the light intensity Img2 (I, J) of the pixel [I, J] of the second image data by the following formula.(1)It is preferable to carry out by calculating according to
[0015]
  Img2 (I, J) = Σ [W (k, m) x Img1 (k, m)] / ΣW (k, m)(1)
  However, Img1 (k, m) is the light intensity of the pixel [k, m] in the k-th row and the m-th column of the first image data, where I, J, k, and m are integers. W (k, m) indicates that the pixel [I, J] in the I-th row and the J-th column of the second image data projected onto the first light receiving plane is the first light receivingflatIt represents the area projected on the pixel [k, m] on the surface.
[0016]
  Also, the above formula(1)The range for obtaining the sum of the right sides of the second image data pixels [I, J] projected on the first light receiving plane is the first light receivingflatIf even a part of the image is projected onto the pixel [k, m] on the surface, it is assumed to cover all the projected pixels [k, m].
[0017]
According to the image conversion method or the image conversion apparatus of the present invention, the first image data (hereinafter also referred to as “oblique image data”) obtained by imaging the imaging object from the oblique direction is directly above the imaging object. It is converted into second image data (hereinafter also referred to as “directly-up image data”) obtained by imaging from the direction. Since the image of the image data directly above is similar to the object, an existing program library or the like that realizes the following various image processing functions can be used effectively.
[0018]
That is, identification of an object, detection of a difference from an object standard, position measurement of the object, shape measurement of the object, and the like. These pre-built program libraries are created on the premise that they are applied to an image obtained by imaging an object from directly above, so that they are applied to image data obtained by imaging the object from obliquely above. In this case, it is indispensable to convert the captured oblique image data into the directly above image data.
[0019]
Further, the screen search technique can effectively perform a screen search according to the image conversion method of the present invention in the detection of orientation flat in a semiconductor manufacturing apparatus and in the monitoring and inspection in a manufacturing line composed of a plurality of manufacturing processes.
[0020]
A semiconductor manufacturing apparatus comprising a wafer cassette for stocking a plurality of wafers and a transfer means for taking out the wafers one by one from the wafer cassette and transporting them to a stage, comprising the image conversion apparatus of the present invention. Is preferred. According to the semiconductor manufacturing apparatus provided with the image conversion apparatus of the present invention, since the oblique image of the wafer acquired by the imaging device installed in the oblique direction with respect to the upper surface of the wafer is converted into the facing image, the facing image The orientation flat detection can be performed using a well-known image search technique for the target. For this reason, it is not always necessary to install the imaging device at a position facing the wafer upper surface.
[0021]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, an embodiment of the present invention will be described with reference to FIGS. Each drawing merely schematically shows the shape, size, and arrangement relationship of each component to the extent that the present invention can be understood, and the present invention is not limited to the illustrated examples.
[0022]
With reference to FIG. 1, a description will be given of a direction in which an object to be imaged is directly facing, for example, when imaging is performed from directly above and when imaging is performed from obliquely above. In the following description, the case of imaging from the facing direction only assumes a virtual model, and does not actually include an imaging device or the like. Furthermore, for convenience of explanation, a planar grid pattern 14 is used as an imaging target, but the technical contents described below are true regardless of whether the target is a grid pattern. .
[0023]
The imaging device includes an imaging optical system and a light receiving plane. A position where the imaging device is directly opposed to the imaging object 14 and imaged is defined as a directly facing position B, and a position where imaging is performed from an oblique direction is defined as an oblique position A. In the description of this configuration example, when the imaging device is arranged at the oblique position A, this imaging device is displayed as the first imaging device 10, its imaging optical system is indicated as 10a, and its light receiving plane is indicated as the oblique light receiving plane 10b. To do. In addition, when an apparatus equivalent to the imaging apparatus disposed at the oblique position A is virtually disposed at the directly facing position B, this imaging apparatus is the second imaging apparatus 12, its imaging optical system is 12a, and Each is shown as a light receiving plane 12b.
[0024]
Taking an image of the object to be imaged from the direction facing directly means that the imaging device 12 is installed in the normal direction of the plane on which the grid pattern 14 is present and the image is taken. Therefore, the optical axis of the imaging optical system 12a constituting the imaging device 12 is parallel to the normal direction, and the light receiving plane is perpendicular to the normal direction. On the other hand, imaging the imaging object 14 from an oblique direction means imaging by installing the imaging device 10 in a direction deviating from the normal direction of the plane on which the grid pattern exists.
[0025]
In the configuration example shown in FIG. 1, the directly-facing position B is a position directly above the imaging object 14 (facing direction), and the oblique position A is obliquely above the imaging object 14. Position. For the sake of brevity, the x ′ axis is taken in the direction parallel to the left and right with respect to the grid pattern 14 regularly arranged in a matrix form shown in FIG. A case where the imaging device 10 is installed in a direction rotated clockwise about the y ′ axis will be described.
[0026]
<Coordinate transformation>
In this way, it is assumed that the first image pickup device 10 and the second image pickup device 12 pick up images of the grid pattern, respectively. As shown in FIGS. 2A, 2B, and 2C, images 18a and 18b of a lattice pattern are captured on the light receiving planes of the first and second imaging devices 10 and 12, respectively. In FIGS. 2A and 2B, the lattice pattern indicated by the thin line indicates the image 18a projected on the light receiving plane 10b of the first imaging device 10, and the lattice pattern indicated by the thick line is the second. An image 18b projected on the light receiving plane 12b of the imaging device 12 is shown, and each of them is shown in an overlapping manner in FIG.
[0027]
On the light receiving plane, there are uniformly minute light receivers (hereinafter, each light receiver may be referred to as a “pixel”) that receives light and generates a voltage or current proportional to the intensity of the light. They are arranged in a matrix. A set of sets of received light intensity corresponding to each pixel arranged in order on the light receiving plane may be hereinafter referred to as “image data”. In order to identify individual pixels in order on the light receiving plane, an orthogonal coordinate axis is provided on the light receiving plane, and along this orthogonal coordinate axis, the position is i-th in the horizontal axis direction and the j-th position in the vertical axis. A pixel is referred to as a pixel [i, j]. That is, the pixel [i, j] is a pixel of i rows and j columns. Details will be described later.
[0028]
When the first image data captured by the first image capturing device 10 is converted into second image data captured by the second image capturing device 12 having the same structure as the first image capturing device 10, the first image is obtained. It is important to convert the image quality of the second image data so as not to deteriorate as much as possible with respect to the image quality of the data.
[0029]
In order to prevent image quality deterioration as much as possible when image conversion is performed from an oblique image to a face-to-face image, in the central portion of the light receiving plane 10b of the first imaging device 10 and the central portion of the light receiving plane 12b of the second imaging device 12 The scale at the time of coordinate conversion from the xy coordinate system to the XY coordinate system may be determined so that the dimensions of the images are equal. Here, a description will be given using a grid pattern as the imaging object. 2A to 2C, a grid pattern 18b indicated by a thick line is a front image, and a grid pattern 18a indicated by a thin line is an oblique image. That is, in the figure, the scale is set so that the interval between the grid points arranged on the thick straight line indicated by the symbol a in the image 18b is equal to the interval between the grid points arranged on the thin line indicated by the symbol B in the image 18a. Confirm.
[0030]
By the way, the thick straight line indicated by the symbol a is a straight line connecting lattice points arranged in the y′-axis direction in the central portion of the light receiving plane 12b of the second imaging device 12, and this straight line and the lattice in the lateral direction (x′-axis direction). The intersection point with the thick straight line connecting the points is the position of the lattice point imaged on the light receiving plane 12b of the second imaging device 12. A thin straight line indicated by symbol B is a straight line connecting lattices arranged in the y′-axis direction in the central portion of the light receiving plane 10b of the first imaging device 10. Therefore, in the central portion of the light receiving plane 10b, the lattice shape indicated by the thick line imaged by the second imaging device 12 so that the intervals of the lattice points arranged in a straight line along the straight lines indicated by symbol A and symbol B are equal. If the scale of the pattern image 18b is determined, the image quality of the face-to-face image that has been subjected to the image conversion is minimally degraded with respect to the entire screen.
[0031]
  In order to define the positional relationship between the oblique light receiving plane 10b of the first image pickup device 10 and the directly-facing light receiving plane 12b of the second image pickup device 12, (x, y) and (X, Y) Each coordinate system is provided. As is well known between the two coordinate systems,(2) as well as (3)There is a relationship given in The following formula(2) as well as (3)The conversion from the (x, y) coordinate system of the oblique light receiving plane 10b to the (X, Y) coordinate system of the directly facing light receiving plane 12b based on the relationship given below is hereinafter referred to as “coordinate conversion”.
[0032]
  X = (b1x + b2y + b3) / (b7x + b8y + 1) (2)
  Y = (b4x + b5y + b6) / (b7x + b8y + 1) (Three)
  Here, bi (where i = 1, 2, 3, 4, 5, 6, 7, 8) is a parameter for the coordinate transformation. Since there are a total of eight, if the relationship between the corresponding four or more points on the light receiving planes 10b and 12b of the first imaging device 10 and the second imaging device 12 is known, the parameter bi (where i = 1, 2) , 3, 4, 5, 6, 7, 8).
[0033]
That is, as shown in FIG. 2C, points corresponding to points p, q, r, and s on the oblique light receiving plane 10b of the first image pickup device 10 are on the opposite light receiving plane 12b of the second image pickup device 12. As corresponding to points P, Q, R, and S, the coordinates of these eight points are respectively (x_p, Y_p), (X_q, Y_q), (X_r, Y_r), (X_s, Y_s), (X_P, Y_P), (X_Q, Y_Q), (X_R, Y_R), (X_S, Y_S). And (x_p, Y_p) And (X_P, Y_P), (X_q, Y_q) And (X_Q, Y_Q), (X_r, Y_r) And (X_R, Y_R), (X_s, Y_s) And (X_S, Y_S), The parameter b can be obtained by a method known analytically._i(However, i = 1, 2, 3, 4, 5, 6, 7, 8).
[0034]
Here, a method for obtaining the correspondence between the coordinate positions of the oblique light receiving plane 10b and the directly facing light receiving plane 12b will be described. In order to obtain this correspondence, a grid-like test pattern is provided as the imaging object 14. As shown in FIG. 1, four points p ', q', r 'and s' are defined in this test pattern. The test pattern is imaged by the first imaging device 10 to obtain a first image (image projected on the first light receiving plane).
[0035]
In the second imaging device 12, the coordinates that can be placed on the second light receiving plane of the points corresponding to the points p ′, q ′, r ′ and s ′ on the test pattern are assumed to be a virtual model. Calculate by calculation. First, the origin is determined for points p ', q', r 'and s' on the test pattern. The second imaging device 12 is installed in the normal direction with respect to the surface of the test pattern. Therefore, on the light receiving plane 12b of the second imaging device 12, the positional relationship between the origin of the test pattern and the points p ', q', r 'and s' is similar. When the X-axis direction and Y-axis direction of the second imaging device 12 are made to coincide with the x ′ direction and y ′ direction of the grating on the test pattern, respectively, and the similarity ratio is determined, the points P, Q, R on the second light receiving plane are determined. The position of S is fixed. That is, the coordinate values of the points P, Q, R, and S in FIG.
[0036]
The coordinates of the points p, q, r, and s in the first image thus obtained are read by the first imaging device 10 and stored in the memory (storage device), and the second image (second light receiving plane) The coordinates of the calculated points P, Q, R, and S in the image projected onto (memory) are stored in a memory (storage device). As described above, the first and second imaging devices 10 and 12 are one imaging device having an equivalent light receiving plane, that is, an imaging surface. The (X, Y) coordinate system of the plane is an equivalent plane orthogonal coordinate system (that is, coordinate conversion is mutually possible).
[0037]
  Therefore, the points p ′, q ′, r ′ and s ′ in the test pattern are imaged at two coordinate positions in the same coordinate system. That is, p 'is imaged at points p and P, q' is q and Q, r 'is r and R, and s' is imaged at points s and S. Since these (p, q, r, s) and (P, Q, R, S) are in the same coordinate system, the correspondence between p and P, q and Q, r and R, and s and S, respectively. Was mentioned aboveformula (2)as well asformula (3)It is defined by 8 parameters.
[0038]
  When converting the image of the first image data into the image of the second image data, it is important that the image quality is not degraded as much as possible. For this purpose, the position of the second imaging device 12 is determined and the scale is adjusted so that the central portion of the first image data in the x direction matches the central portion of the second image data in the X direction. The scale adjusting method will be described below. At the lattice points shown in FIG. 1, three points o ′, t ′ and u ′ are newly selected in the central portion. The three points o, t, u and O 1, T, U in the first and second images shown in FIG. 2 (C) corresponding to these three newly selected points respectively correspond. When the first imaging device 10 captures the test pattern, the position of the test pattern is adjusted so that o ′ is at the center of the first image. Thereafter, the first imaging device 10 reads the positions of seven points o ′, p ′, q ′, r ′, s ′, t ′, and u ′ on the test pattern. The origin of the second imaging device 12 is set so as to coincide with the coordinates of the point o in the read first image (the position of the point O in the second image is determined). Then, the position of the T point in the second image is determined so that the interval between the o point and the t point in the first image and the interval between the O point and the T point in the second image are the same. At this time, the X coordinate of the T point is set to be the same as the O point, and only the Y coordinate is changed. Thus, when the position of the T point in the second image is determined, the scale of the second image data is determined. That is, the distance from the test pattern of the second imaging device 12 is determined. Since the second imaging device 12 is installed in the direction facing the point o ′ of the test pattern, the points p ′, q ′, r ′, s ′, t ′ and u ′ with respect to the point o ′ on the test pattern In the second image, the positional relationship is the position of each point with respect to point O with the same similarity ratio. That is, the coordinates of the points P, Q, R, S, T and U in the second image are calculated by performing similarity transformation on the O point. Since the seven points on the test pattern obtained in this way are in a correspondence relationship on the light receiving planes of the first and second imaging devices, there are a large number of equations related to equations (2) and (3). So using the least squares methodformula (2)as well asformula (3)Find the 8 parameters.
[0039]
  If the parameter bi (where i = 1, 2, 3, 4, 5, 6, 7, 8) is stored in the storage device, an arbitrary point on the light receiving plane 10b of the first imaging device 10 is stored. Isformula (2)as well asformula (3)Is used to uniquely determine which point on the light receiving plane 12b of the second imaging device 12 corresponds to. That is, the central processing unit (CPU) stores the coordinate values and parameters stored in the storage device andformula (2)as well asformula (3)Is read out and calculated, the position on the light receiving plane 10b can be converted into a position on the light receiving plane 12b to correspond.
[0040]
In the above-described example, seven points are selected on each of the light receiving planes 10b and 12b of the first imaging device 10 and the second imaging device 12, and the coordinate conversion parameters are obtained from the corresponding relationships. It is not necessary to select 7 points.
[0041]
When the number of selected points is selected so that three or more points do not line up on a straight line, the number is a minimum of four points. In addition, when selecting so as to include three points on a straight line and to include at least two or more points outside the straight line, the number of selected points is at least five. That is, a plurality of points that satisfy the above-described conditions may be selected. Of course, the more selection points, the higher the calculation accuracy of the required coordinate conversion parameters.
[0042]
Coordinate transformation parameter b for the coordinate transformation described above_i(However, i = 1, 2, 3, 4, 5, 6, 7, 8) The procedure (step) for obtaining is summarized as follows with reference to the flowchart shown in FIG.
[0043]
  (a) Test pattern imaging process (ST1) for imaging a test pattern with the first imaging device 10
  (b) A coordinate reading step (ST2) for reading the coordinates on the light receiving plane of each point where a plurality of characteristic points in the test pattern are imaged on the light receiving plane 10b of the first imaging device 10
  (c) A mark on a plurality of characteristic points imaged in an oblique direction to set the origin at an arbitrary position in the test pattern and to set the scale of the image captured by the second imaging device 12 A calibration process (ST3) that determines the center position and scale of the second imager that geometrically transforms the position relative to the origin
  (d) the above process (c) Is used to calculate the coordinates on the light receiving plane of each point selected in step (a) in the test pattern on the light receiving plane 12b of the second imaging device 12, using the center position and scale set in (2). Coordinate calculation process (ST4)
  (e) The coordinates of a plurality of characteristic points in the test pattern imaged on the light-receiving plane 10b of the first imaging device 10 and the plurality of points in the test pattern calculated on the light-receiving plane 12b of the second imaging device 12 Based on the coordinates of, the coordinate conversion parameter bi (where i = 1, 2, 3, 4, 5, 6, 7, 8) is calculated by solving the simultaneous equations or by the least square method Process (ST5)
[0044]
Moreover, it is preferable that the following steps are included in the steps (a) and (c).
(a ′) In order to make the center point of the light receiving plane 10b of the first image pickup device 10 coincide with the center point of the light receiving plane 12b of the second image pickup device 12, a test pattern is placed on the center point of the light receiving plane 10b of the first image pickup device 10. Test pattern position adjustment process (ST1-1) for adjusting the test pattern to adjust the test pattern placement position so that the center grid point is imaged
(c ′) The center point of the light receiving plane 12b of the second image pickup device 12 is matched with the coordinate value read as the center point of the light receiving plane 10b of the first image pickup device 10, and the center of the light receiving plane 10b of the first image pickup device 10 Calibration step (ST3-1) for setting the scale of the second imaging device so that the dimensions of the image are equal between the portion and the central portion of the light receiving plane 12b of the second imaging device 12
[0045]
Note that the coordinate conversion parameter b of the coordinate conversion described above is used._iAn apparatus for realizing the procedure (process) for obtaining (where i = 1, 2, 3, 4, 5, 6, 7, 8) will be described later.
[0046]
<Resampling>
With reference to FIG. 4, the coordinates provided on the light receiving plane of the imaging apparatus and the index for designating the pixels constituting the light receiving plane will be described. Here, the pixels are arranged in a matrix, for example, and the shape of each pixel is assumed to be a square for the sake of simplicity, and the boundary area between adjacent pixels has only a negligible width. Since each pixel has light receiving sensitivity inside the square and the boundary area between adjacent pixels has only a negligible width, it will be described as having light receiving sensitivity everywhere on the light receiving plane.
[0047]
The dimension of one side of the pixel is L, and the center of the pixel is indicated by a black circle in FIG. 4 and the vertices at the four corners of the pixel are indicated by x. As shown in FIG. 4, an xy coordinate system is taken, and coordinate values are shown as (x, y). On the other hand, an index for designating each pixel is shown as [i, j]. These i and j correspond to the i-th row and the j-th column. For example, the coordinates of the vertexes of the four corners of the pixel specified by the index [0, 0] are (+ 0.5L, -0.5L), (+ 0.5L, + 0.5L), (-0.5L, + 0.5L). ) And (−0.5L, −0.5L), and the center coordinates of the pixel are (0, 0). In general, the coordinates of the vertices at the four corners of the pixel specified by the index [i, j] are ((i + 0.5) L, (j-0.5) L), ((i + 0.5) L, (j + 0.5) L), ((i-0.5) L, (j + 0.5) L) and ((i-0.5) L, (j-0.5) L), and the center coordinates of the pixel are (iL, jL).
[0048]
Further, x or y representing coordinate values and index values i, j or k, m for designating pixels are displayed using lower case letters as values relating to the light receiving plane of the first imaging device 10, that is, the oblique light receiving plane 10b. It is assumed that the value relating to the light receiving plane of the second imaging device 12, that is, the directly-facing light receiving plane 12b is displayed using capital letters. That is, the coordinate value of the light receiving plane 10b of the first imaging device 10 is (x, y), and the index for designating the pixels constituting the light receiving plane 10b of the first imaging device 10 is [i, j] or [k, m].
[0049]
On the other hand, the coordinate value of the light receiving plane 12b of the second imaging device 12 is (X, Y), and the index for designating the pixels constituting the light receiving plane 12b of the second imaging device 12 is [I, J]. Note that the image data obtained from the second imaging device 12 is referred to as directly-facing image data. Here, since the directly-facing position is a position directly above the imaging object, it is also referred to as directly-on image data. The image data obtained from the first imaging device 10 is referred to as oblique image data.
[0050]
With reference to FIG. 5, a method of acquiring image data of each pixel constituting the light receiving plane will be described. Figure 5 shows the indicator [i_p, j_q] (Where p = 1, 2, 3, 4, 5 and q = 1, 2). A case where a black belt-like image having a width is formed on this pixel will be described. The shaded area in FIG. 5 is the black image. It is assumed that light does not reach the black portion indicated by the oblique lines, and light reaches a uniform brightness in the other white portions. In addition, the light intensity received by each pixel is digitized into 256 gradations and received. That is, it is assumed that the pixel is set to output brightness information as digital data with 0 when no light is received and 255 when the brightest light is received. In this case, the indicator [i_p, j_q] (Where p = 1, 2, 3, 4, 5 and q = 1, 2), the digital values captured as one of the image data by the eight pixels given by
[i₁, j₂] = 0, [i₂, j₂] = 10, [i_Three, j₂] = 40, [i_Four, j₂] = 96, [i_Five, j₂] = 128,
[i₁, j₁] = 20, [i₂, j₁] = 0, [i_Three, j₁] = 0, [i_Four, j₁] = 0, [i_Five, j₁] = 0
Suppose that These values are proportional to the area occupied by the bright part in each pixel.
[0051]
Here, for the sake of simplicity, an example of an image composed of two areas of a white part and a black part has been described as an example, but the same applies to an image including a more complex intermediate brightness area. The digital value that the pixel captures as one of the image data is a value that is proportional to the product of the brightness of the image projected onto the pixel and the area that the brightness portion occupies on the pixel.
[0052]
Next, referring to FIG. 6, a pixel for acquiring the upper image data (hereinafter, simply referred to as “directly upper image pixel”) and a pixel for acquiring the oblique image data (hereinafter simply referred to as “oblique image pixel”). ”).) Is explained. FIG. 6 is a diagram in which pixels existing in the range from i to i + 2 with respect to the column positions of the diagonal image pixels and pixels existing in the range from j to j + 2 with respect to the row positions of the diagonal image pixels are separated by thin lines. Indicated. That is, the upper left pixel is a pixel given by the index [i, j], and the lower right pixel is a pixel given by the index [i + 2, j + 2]. In addition, the pixel directly above given by the index [I, J] is an image projected on the pixel from which the oblique image is acquired, and is indicated by a bold rectangle ABCD. In the following, pixels given by the index [I, J] or [i, j] are abbreviated as pixels [I, J] or pixels [i, j], respectively.
[0053]
The vertices A, B, C, and D of the pixel [I, J] projected onto the oblique image pixel are respectively A ((I-0.5) L, (J -0.5) L), B ((I + 0.5) L, (J-0.5) L), C ((I + 0.5) L, (J + 0.5) L), D ((I-0.5) L, ( J + 0.5) is a point existing in L). The coordinates of the four vertices of the pixel [I, J] ((I-0.5) L, (J-0.5) L), ((I + 0.5) L, (J-0.5) L), ((I + 0.5) L , (J + 0.5) L) and ((I-0.5) L, (J + 0.5) L) are at positions indicated by A, B, C, D on the light receiving plane 12a of the second imaging device 12, respectively. Projected.
[0054]
Next, the following equation (1) that gives resampling will be described.
Img2 (I, J) = Σ [W (k, m) x Img1 (k, m)] / ΣW (k, m) (1)
As shown in FIG. 6, consider a case where a pixel [I, J] is projected onto a pixel constituting the light receiving plane 10 b of the first imaging device 10. In this case, the range for obtaining the sum indicated by Σ is the range from i to (i + 2) for k, and the range from j to j + 2 for m.
[0055]
Here, the meaning of W (k, m) will be described. W (k, m) represents the area of the pixel [I, J] projected onto the pixel [k, m] constituting the light receiving plane 10b of the first imaging device 10. That is, a part of the pixel [I, J] is projected on the pixel [k, m], but W (k, m) is an image of the pixel [I, J] on the pixel [k, m]. Represents the area of the overlapping part. The area of the overlapping portion is also referred to as “partial projection area”. FIG. 7 illustrates the meaning of W (k, m) on the pixel [k, m] over the range from i to (i + 2) for k and from j to j + 2 for m. FIG. 6 is a diagram showing the portion of the image of pixel [I, J] that is projected and overlapped with hatched lines for easy viewing. Each square indicated by a thin line represents a pixel [k, m]. Polygon (part of the partial projection area where the image of pixel [I, J] is projected onto pixel [k, m] with a diagonal line that is bordered by a thick straight line and the area given by W (k, m) ).
[0056]
  formula (1)Is the range of pixels projected from the pixel [I, J] on the pixels constituting the light receiving plane 10b of the first imaging device 10 (range from i to (i + 2) for k, j for m To the range of j + 2), the weighted average value using W (k, m) as a weight is the value of the pixel [I, J] on the light receiving plane 12b of the second image pickup device 12 (digital for brightness). Value).
[0057]
  Img1 (k, m) is a digital value related to the brightness received by the pixel [k, m] (value of the pixel [k, m]), and W (k, m) is the pixel [k, m]. Since it is the image partial projection area of the pixel [I, J] above, W (k, m) × Img1 (k, m) is the value of the pixel [k, m] only W (k, m) The value takes into account the weight. Σ [W (k, m) × Img1 (k, m)] is the sum obtained over the range from i to (i + 2) for k and from j to j + 2 for m. Further, ΣW (k, m) represents the total projected area of the image of the pixel [I, J] projected onto the light receiving plane 10b of the first imaging device 10. Therefore,formula (1)The right side of the pixel is the pixel [I, J] received by the first imaging deviceflatW (k, m) for the range of pixels projected on the pixels that make up the surface (range from i to (i + 2) for k, range from j to j + 2 for m) Is used as a weighted average value.
[0058]
Next, the procedure for determining W (k, m) will be described. FIG. 8 is a diagram for explaining the procedure for obtaining W (k, m), and is an enlarged view of the peripheral portion of pixel [i, j]. The positions at which the four vertices of the pixel [I, J] are projected onto the light receiving plane 10b of the first imaging device 10 are denoted as A, B, C, and D. The four vertices of the pixel [i, j] on the light receiving plane 10b are A ′, B ′, C ′, and D ′. As already explained, these eight points are in the same (x, y) coordinate system.
[0059]
W (k, m) is obtained by the following procedure (process).
(A) It is examined whether or not each point of A, B, C, and D is included in the quadrilateral A'B'C'D '.
Since A is included, pick up A.
(B) It is examined whether or not each point of A ′, B ′, C ′, D ′ is included in the quadrilateral A B C D.
Since C 'is included, C' is picked up.
(C-1) The intersection of the line segment A′B ′ and the line segment A B, line segment B C, line segment CD, and line segment D A is examined.
There is no intersection.
(C-2) The intersection of the line segment B′C ′ and the line segment A B, the line segment B C, the line segment CD, and the line segment D A is examined.
Since a is an intersection (exists as an intersection of line segment B'C 'and line segment AB), a is picked up.
(C-3) The intersection of the line segment C′D ′ and the line segment A B, line segment B C, line segment CD, and line segment D A is examined.
Since b is an intersection (exists as an intersection of line segment C'D 'and line segment DA), b is picked up.
(C-4) The intersection of the line segment D′ A ′ and the line segment A B, line segment B C, line segment CD, and line segment D A is examined. There is no intersection.
(D) The upper left point is obtained for the picked up points (in this embodiment, four points are registered in the list in the order of A, C ′, a, and b). Specifically, as shown in FIG. 9, a straight line having a 45 degree inclination with respect to the X axis is drawn one by one so that each point passes through each point, and the intersection of the Y axis and this 45 degree straight line is drawn. Select the smallest Y coordinate.
The reason why the smallest Y coordinate is selected is that the coordinate set on the light receiving plane is the left-handed coordinate system. In this way, the first point is obtained. Here, it is point A.
(E-1) Using a straight line with a 45 degree slope passing through point A as a reference, rotate this line clockwise around point A to find the first contact point, which is the second point And In this case, point a is obtained as the second point.
(E-2) Next, the straight line Aa is rotated clockwise around the point a to obtain the first contact point, which is set as the third point. In this case, the C ′ point is obtained as the third point.
(E-3) Next, the straight line a C ′ is rotated clockwise around the C ′ point to obtain the first contact point, which is set as the fourth point. In this case, point b is obtained as the fourth point.
(E-4) The above steps are sequentially performed on the picked up points. By this process, the picked-up points are arranged clockwise from the top left.
(F) If the picked-up points are arranged clockwise from the top left, the coordinates of the four vertices of the quadrilateral have been found, so the quadrilateral A a C ′ The area of b, that is, W (k, m) can be obtained by a known method.
[0060]
Next, a method for examining whether or not the point of interest is included in the inside of the quadrilateral performed in steps (a) and (b) will be described with reference to FIG. FIG. 10A is a diagram showing a case where the O point is inside the quadrilateral EFGH, and FIG. 10B is a diagram showing a case where the O point is outside the quadrilateral EFGH. It is assumed that there is a point that moves from the point E of the quadrilateral EFGH so as to pass on the side in the order of point E, point F, point G, and point H. At this time, the angle θ stretched from point E to point F₁The direction from point E to point F is taken as the positive direction. Similarly, the angle stretched from point F to point G is θ2, the angle stretched from point G to point H is θ3, and the angle stretched from point H to point E is θ4. The direction toward is taken as the positive direction.
[0061]
When the point O shown in FIG. 10A is inside the quadrilateral EFGH, θ₁+ θ₂+ θ_Three+ θ_FourBecomes 2π. On the other hand, when the point O shown in FIG. 10B is outside the quadrilateral EFGH, θ₁+ θ₂+ θ_Three+ θ_FourBecomes 0. Although not shown, when the point O is on the side of the quadrilateral EFGH,₁+ θ₂+ θ_Three+ θ_FourBecomes π. As explained above, θ₁+ θ₂+ θ_Three+ θ_FourIf the value is 2π, 0, or π, it can be determined whether or not the point of interest O is included in the quadrilateral EFGH.
[0062]
Of the steps for obtaining W (k, m) described above, in step (a) and step (b), the point to be picked up is inside the quadrilateral even when it is on the side of the quadrilateral. Treated as.
[0063]
Next, referring to FIG. 11 and FIG. 12, the intersection of the line segment and the line segment performed in the steps (c-1), (c-2), (c-3) and (c-4) described above Explain the rules for requesting. FIG. 11 illustrates a case where the line segments do not overlap each other, and FIG. 12 illustrates a case where the line segments overlap each other. As shown in FIG. 11 (A), when an intersection exists on the extension of the line segment, it is treated as no intersection. It is determined that the intersection exists as shown in FIG. 11B when the intersection exists within the range where the line segment exists. As shown in FIG. 11C, it is determined that there is an intersection even when the end point of the line segment is an intersection. When the thin line segment 1 and the thick line segment 2 overlap as shown in FIG. 12, two points are picked up as intersections.
Further, the following processing is performed as a derivative process.
[0064]
(A) In the above-described step (a), when all points A, B, C, and D are not included in the quadrilateral A′B′C′D ′, W (k, m) = Set to 0 to finish.
(B) In the above-described step (b), when all points A ′, B ′, C ′, D ′ are not included in the quadrilateral ABCD, W (k, m) = 0 finish.
(C) When the above steps (c-1) to (c-4) are completed, the number of points to be picked up is 1 to 2, or 3 to 8.
[0065]
Therefore,
(C1) When the number of points to be picked up is 0 or 2, the process ends with W (k, m) = 0.
(C2) When the number of points to be picked up is 3 to 8, the process goes to step (f) to obtain W (k, m).
[0066]
In FIG. 6, the relationship between the pixel for the upper image and the pixel for the oblique image is illustrated. This indicates the relationship between the quadrilateral (the shape of the pixel for the upper image) and the square (the shape of the pixel for the oblique image). Is. In such a relationship, when the relationship between the quadrilateral and the square becomes the relationship shown in FIG. 13, the number of points picked up is 8, which is the maximum number. FIG. 13 shows the relationship between the image pixel directly above (a quadrilateral indicated by a thick line) and the diagonal image pixel (a square indicated by a thin line). For this reason, the maximum number of points picked up is eight.
[0067]
The procedure for obtaining W (k, m) described above is merely an example, and there may be a suitable procedure other than the method described here. However, it is obvious that whatever procedure is used to obtain W (k, m), the present invention can be applied to the image conversion method or the image conversion apparatus according to the present invention.
[0068]
The procedure for obtaining W (k, m) described above is summarized as follows with reference to the flowchart shown in FIG.
[0069]
The positions at which the four vertices of the pixels [I, J] constituting the second light receiving plane are projected onto the light receiving plane of the first imaging device are A, B, C, D, and the pixels [1] constituting the first light receiving plane [ The four vertices of i, j] are A ', B', C ', D'
(D) It is checked whether or not points A, B, C, and D are included in the quadrilateral A'B'C'D '(step S1). When neither is included (N), W (k, m) is set to W (k, m) = 0 (step S2), and this procedure is terminated. On the other hand, if any point is included (Y), the included point is picked up (step S3).
(E) It is checked whether or not the points A ′, B ′, C ′, and D ′ are included in the quadrilateral A B C D (step S4). If neither is included (N), W (k, m) is set to W (k, m) = 0 (step S5), and this procedure is terminated. On the other hand, if any point is included (Y), the included point is picked up (step S6).
(F) Next, the intersection of line segment A′B ′ and line segment A B, line segment B C, line segment C D, line segment D A,
Intersection of line segment B’C ′ and line segment A B, line segment B C, line segment C D, line segment D A,
The intersection of line segment C’D ′ with line segment A B, line segment B C, line segment C D, line segment D A, and
The intersections of the line segment D'A 'and the line segment A B, line segment B C, line segment CD and line segment D A are examined and picked up (step S7).
(G) W (k, m) is obtained from the coordinates of all the points picked up in the steps S3, S6 and S7 (step S8).
[0070]
<Description of Image Conversion Device>
Next, an apparatus for realizing the above-described image conversion method will be described with reference to FIGS. FIG. 15 is a diagram for explaining the configuration of the image conversion apparatus 32 of the present invention. The image conversion device 32 includes the imaging device 10 described with reference to FIG.
The image conversion device 32 includes a first image data acquisition unit 36, an image conversion processing unit 38, a storage device 40, and a second image data output unit 42. The image conversion processing unit 38 includes a CPU as will be described later. The first image data acquisition unit 36 receives an image captured by the first imaging device 10, and transmits this image to the image conversion processing unit 38 as first image data. The storage device 40 stores all parameters related to the coordinate conversion parameters and the equation (1) that gives resampling. Further, in the image conversion process, all parameters related to the expression (1) that gives resampling are called, transferred to the image conversion processing unit 38, and used when converted into second image data. Here, all parameters related to equation (1) giving resampling mean the range of k and m for pixel [I, J] and W (k, m) for each pixel [k, m]. The pixel [I, J] is the entire range of the second image data. The second image data converted by the image conversion processing unit 38 is output to the outside via the second image data output unit as necessary.
Since processing related to writing and calling information to the storage device 40 is a well-known matter in a control device using a computer, detailed description thereof is omitted.
[0071]
First, the test pattern 14 is placed on the transfer arm 26, and an operation for calibrating the position of the imaging device is performed on the test pattern 14 as described above. First, the test pattern 14 is imaged by the first imaging device 10 at the position A. This step corresponds to step ST1 shown in the flowchart of FIG. In order not to degrade the image quality of the directly-converted image data as much as possible as compared to the image quality of the oblique image data, a lattice point positioned at the center on the test pattern is located at the center point of the light receiving plane 10b of the first imaging device 10. The installation position of the test pattern is adjusted so that an image is taken. This step corresponds to step ST1-1 shown in the flowchart of FIG. Next, the scale of the second imaging device is set so that the image dimensions are equal in the central portion of the light receiving plane 10b of the first imaging device 10 and the central portion of the light receiving plane 12b of the second imaging device 12. Since the second imaging device is a virtual model, this setting can be easily performed. This step is step ST3-1 shown in the flowchart of FIG.
[0072]
  An (x, y) coordinate system is provided on the light receiving plane of the first imaging device 10, and the first image data of the test pattern is taken into the first image data acquisition unit 36 provided in the image conversion device 32. The first image data captured by the first image data acquisition unit 36 provided in the image conversion device 32 is sent to the image conversion processing unit 38 and used when calculating the coordinate conversion parameters. On the other hand, the second imaging device 12 is a virtual device that does not actually exist. Therefore, the coordinate system (X, Y) provided on the light receiving plane 12b of the second imaging device 12 exists only on the internal memory 54 of the image conversion processing unit 38 (see FIG. 16 described later). After the center position and scale of the second imaging device 12 are set, the coordinate value of the selected lattice point on the test pattern is input to the image conversion device 32. When the coordinate value of the selected grid point is input to the image conversion processing unit 38, for each grid point on the selected test pattern, the coordinate value on the light receiving plane of the second imaging device 12 is It is created on the internal memory 54 of the image conversion processing unit 38 (see FIG. 16 described later). In this way, data for obtaining coordinate conversion parameters is prepared. When these data are prepared, they are calculated in the data image conversion processing unit 38, and the coordinate conversion parameter bi (where i = 1, 2, 3, 4, 5, 6, 7, 8) Relational expressions used in sampling
  Img2 (I, J) = Σ [W (k, m) x Img1 (k, m)] / ΣW (k, m)(1)
Is determined. The coordinate transformation parameter and the resampling relational expression described above are sent to and stored in the storage device 40.
[0073]
Thus, the installation of the imaging device 10 that captures an actual imaging target and the setting of the parameters necessary for image conversion in the storage device of the image conversion device are completed. If the imaging object 10 is imaged by the imaging device 10, the imaging result is sent to the first image acquisition unit 36 and further sent to the image conversion processing unit 38. In the image conversion processing unit 38, the coordinate conversion parameters and the re-sampling relational expression described above are extracted from the storage device 40, and conversion to the directly-facing image data is performed using these parameters. It is created on the internal memory of the conversion processing unit 38 and image processing can be continued. Alternatively, in some cases, the directly-facing image data obtained as a result of the conversion is output to the outside from the second image data output unit 42, and further image processing or the like is performed outside.
[0074]
The configuration of the image conversion processing unit 38 provided in the image conversion device 32 will be described with reference to FIG. The image conversion processing unit 38 includes a control unit 46, a coordinate conversion unit 48, a resampling unit 50, and an internal memory 54. In the internal memory 54, W (k, m) and the directly-facing image data obtained by image conversion are temporarily stored in order to perform image conversion processing at high speed. The resampling unit 50 includes a W (k, m) calculating unit 51 and an Img2 (I, J) calculating unit 52.
[0075]
The control unit 46 controls the coordinate conversion unit 48 and the re-sampling unit 50, and controls data exchange and operation timing with the storage device 40 and the like. Further, the control unit 46 can control operations of the first image data acquisition unit 36 and the second image data output unit 42 described with reference to FIG. Note that the part of the control unit 46 that controls the first image data acquisition unit 36 and the second image data output unit 42 may be configured outside the image conversion processing unit 38 as a part of another CPU. How to configure the control unit 46 is merely a design matter.
[0076]
The coordinate conversion unit 48 reads the target coordinates of the first light receiving plane captured by the light receiving plane when a plurality of characteristic points in the test pattern are captured from an oblique direction. This step is step ST2 in the flowchart shown in FIG. Further, the center position and scale of the second light receiving plane 12b are set. This step is step ST3 in the flowchart shown in FIG. In addition, the coordinate values on the second light receiving plane 12b of the lattice points on the plurality of points having the selected features in the test pattern imaged from the oblique direction by the first imaging device are calculated. This step is step ST4 in the flowchart shown in FIG. Also, after this step, based on the coordinate values of the grid points on the corresponding characteristic points on the test pattern, the coordinate conversion parameter b_i(Where i = 1, 2, 3, 4, 5, 6, 7, 8). This step is step ST5 in the flowchart shown in FIG.
[0077]
  The re-sampling unit 50 performs a process of converting the first image data that is the oblique image data obtained by imaging the imaging object into the second image data that is the facing image data. The W (k, m) calculation unit 51 performs processing in a procedure corresponding to steps S1 to S8 shown in FIG. 14, and the I-th row and the J-th column of the second image data projected on the first light receiving plane. The pixel [I, J] calculates W (k, m) representing the area projected onto the pixel [k, m] on the first light receiving plane, and stores the result in the storage device 40. The Img2 (I, J) calculation unit 52 calculates the value of the light intensity Img1 (k, m) of the pixel [k, m] in the k-th row and m-th column of the first image data sent from the first image acquisition unit. With W (k, m) stored in the storage device 40(1)
  Img2 (I, J) = Σ [W (k, m) x Img1 (k, m)] / ΣW (k, m)(1)
Accordingly, the light intensity Img2 (I, J) of the pixel [I, J] of the second image data is calculated.
[0078]
According to the apparatus described above, if the element Img1 (k, m) of the oblique image data obtained by imaging the imaging object with the first imaging apparatus is given over the entire oblique light receiving plane, this is taken as the imaging object. Since the directly-facing image data Img2 (I, J), which should be obtained by imaging with the second imaging device, is calculated over all the directly-facing light receiving planes, desired image conversion is performed.
[0079]
<Example of conversion processing by image conversion method>
With reference to FIG. 17, an example of an image obtained by performing image conversion from diagonal image data obtained by imaging a grid, which is an imaging object, from an oblique direction to directly above image data will be described. FIG. 17A shows an image captured from obliquely above. In this image, it can be seen that the grid interval (interval of the intersections of the black lines) becomes narrower toward the right side. It can also be seen that the thickness of the black lines forming the grid image becomes thinner toward the right side.
[0080]
The image shown in FIG. 17B is obtained by converting the lattice image shown in FIG. 17A into an image that would be obtained if it was captured from directly above. In this image conversion, in order to determine the magnification of the image, the lattice spacing p on the representative black line in the central portion of the image forming the lattice image indicated by the arrow 16 in the figure also corresponds to the corresponding black in the converted image. The condition of being equal to the lattice spacing p ′ on the line was imposed. In the image-converted image shown in FIG. 17B, grid points (intersections of black lines) are arranged in a square shape, and the black lines have a uniform thickness. That is, it can be seen that this is equivalent to a lattice image captured from directly above.
[0081]
Note that black triangular portions indicated by arrows 17 and 18 in FIG. 17B are portions that are not shown in the image of FIG. That is, the black triangular portion is a portion where the corresponding image data does not exist in the oblique image data constituting the image shown in FIG.
[0082]
<Example of a semiconductor manufacturing apparatus equipped with an apparatus for realizing an image conversion method>
With reference to FIG. 18, an example of a semiconductor manufacturing apparatus having a mechanism for carrying out the image conversion method according to the present invention will be described. This semiconductor manufacturing apparatus has a function of taking out wafers 24 one by one by a transfer arm 26 from a wafer cassette 22 in which a plurality of wafers (single crystal thin plates) are stocked in a horizontal direction and transferring them to a stage 28. Is. The wafer 24 transferred to the stage 28 must be placed in a predetermined position and direction according to the orientation flat attached to the wafer 24.
[0083]
The orientation flat refers to a portion in which a portion of the periphery of the wafer 24 is cut into a straight line so that the crystal axis direction of a single crystal constituting the wafer 24 can be visually or mechanically read. For the same purpose of making the crystal axis direction readable, in addition to using an orientation flat, there is a case where a notch is called a notch in a part of the periphery of the wafer.
[0084]
At the time when the wafer 24 is set in the wafer cassette 22, the orientation flats are not aligned in a certain direction. For this reason, the orientation arm of the wafer 24 is detected by some method until it is taken out from the wafer cassette 22 by the transfer arm 26 and placed on the stage 28, and the wafer 24 is transferred so that the orientation flat faces in a predetermined direction. Must. Therefore, the wafer 24 taken out from the wafer cassette 22 by the orientation flat detection camera 20 is set as an imaging target. Here, the wafer 24 on the transfer arm 26 that is being transferred by the imaging device 20 is imaged by the orientation flat detection camera 20, and the orientation flat is detected based on this image to obtain the required rotation amount of the wafer. This information is given to the transfer arm 26, and the transfer arm 26 rotates the wafer 24 by a necessary amount based on this information and installs it on the stage 28.
[0085]
An imaging device provided in a semiconductor manufacturing apparatus is attached to capture an object such as a wafer and acquire images used for various purposes such as orientation flat and notch detection using a technique such as image search. Therefore, it is preferable to install the imaging apparatus so that the wafer can be imaged in a direction perpendicular to the upper surface of the wafer because there is no distortion of the image to be captured. This is because, in general, a program or the like for realizing a function such as an image search is made on the premise that there is no external image distortion such as that captured from an oblique direction in the captured image. It is.
[0086]
However, since the semiconductor manufacturing apparatus is installed in a clean room, it is required to make the size as small as possible. In order to reduce the size of the entire apparatus such as a semiconductor manufacturing apparatus, the imaging apparatus may not be attached so that the wafer can be imaged from a direction perpendicular to the wafer surface. Therefore, even if an image is captured by an imaging device installed at any location, the image is converted into an image that should be obtained by setting the imaging device to the original desired position and capturing the image. If possible, this problem will be solved.
[0087]
The image conversion method according to the present invention is an image conversion method suitable for achieving the above-described object. With reference to FIG. 14, FIG. 15 and FIG. 17, the configuration and handling of a semiconductor manufacturing apparatus having a mechanism for carrying out the image conversion method according to the present invention will be described. In the following description, process names shown in parentheses correspond to the process names shown in the flowcharts of FIGS.
[0088]
The test pattern on the transfer arm 26 is imaged from the diagonal direction by the imaging device 20 installed at the diagonal position (step ST1). A plurality of characteristic points set on the upper surface of the test pattern reads the coordinate value of the first light receiving plane imaged on the light receiving plane when imaged from an oblique direction (step ST2). This is performed by the first image acquisition unit 36 provided in the image conversion device 32 shown in FIG. These coordinate values (x, y) are sent to the image conversion processing unit 38. The center position and scale of the virtual second imaging device 12 are determined (step ST3). Further, the coordinate values (X, Y) on the second light receiving plane 12b of a plurality of characteristic points set on the upper surface of the test pattern imaged from the oblique direction by the first imaging device are calculated (step ST4). In addition, after this step, coordinate conversion is performed based on the first plane coordinate value (x, y) and the second plane coordinate value (X, Y) corresponding to a plurality of characteristic points set on the upper surface of the test pattern. Parameter b_i(Where i = 1, 2, 3, 4, 5, 6, 7, 8) is calculated by solving simultaneous equations or by the method of least squares (step ST5). The calculation of this parameter is performed in the coordinate conversion unit 48 in the image conversion processing unit 38 shown in FIGS.
[0089]
Subsequently, the light intensity Img2 (I, J) of the pixel [I, J] of the second image data constituting the light receiving plane of the second imaging device 12 and the light receiving plane of the first imaging device 10 are configured. The following equation (1) that gives the relationship with the light intensity Img1 (k, m) of the pixel [k, m] in the k-th row and m-th column of the first image data is obtained (step S1 to step S8).
[0090]
Img2 (I, J) = Σ [W (k, m) x Img1 (k, m)] / ΣW (k, m) (1)
This process is also performed in the W (k, m) calculation unit 51 and the Img2 (I, J) calculation unit 52 in the image conversion processing unit 38 shown in FIG.
[0091]
Parameter b for coordinate transformation described above_i(Where i = 1, 2, 3, 4, 5, 6, 7, 8) and the above formula (1) that gives the relationship between Img2 (I, J) and Img1 (k, m) All relevant parameters are stored in the storage device 40. Here, all parameters related to the above equation (1) that give the relationship between Img2 (I, J) and Img1 (k, m) are the ranges of k and m for the pixel [I, J], and This means W (k, m) for pixel [k, m], and pixel [I, J] is the entire range of the second image data.
[0092]
As described above, if the first imaging device 10 is set as a semiconductor manufacturing device, the wafer 24 on the transfer arm 26 is imaged by the first imaging device 10 every time it is taken out from the wafer cassette 22. One image data is converted into second image data and output to the outside. That is, if the first image acquisition unit 36 acquires oblique image data and sends it to the image conversion processing unit 38, all parameters related to the conditional expression (1) for resampling are called from the storage device 40, and the image The image data is transferred to the internal memory 45 of the conversion processing unit 38, and the first image data is converted into directly-facing image data that should be obtained by the second imaging device. The result is output from the second image data output unit 42 to the outside. That is, in this semiconductor manufacturing apparatus, in normal use, it is not necessary to install a second imaging device that is a directly-facing imaging position with respect to the wafer, and the first imaging device 20 is replaced with the second imaging device. In order to process the subsequent image processing at high speed, the second image data is held in the internal memory 54 of the image conversion processing unit 38, and the next image processing is performed.
[0093]
If the second image data is output from the second image data output unit 42 as described above, the orientation flat can be detected using a known image search program or the like, and the known orientation flat is directed in a predetermined direction from this result. A wafer can be set on the stage. Of course, not only the orientation flat, but also the notch or the like, it is obvious that the wafer can be set on the stage in a predetermined direction as described above according to the method of the present invention.
[0094]
In a large-scale manufacturing apparatus that needs to monitor imaging objects such as wafers at a plurality of locations, all images captured from the plurality of locations are captured in the imaging direction as well as the distance between the imaging device and the wafer. Are preferably imaged under the same condition. The large-scale manufacturing apparatus that needs to monitor imaging objects such as wafers at a plurality of locations described above may be a manufacturing line or a manufacturing plant. Even in this case, as in the example of the semiconductor manufacturing apparatus described above, there are many cases where the imaging apparatus cannot always be set to a position that satisfies the same condition with respect to the imaging target.
[0095]
Even in such a large-scale manufacturing apparatus that needs to monitor an object at a plurality of locations, the image conversion method and the image conversion apparatus of the present invention can produce the following advantages. If image data acquired by a plurality of imaging devices installed in a large-scale manufacturing apparatus is subjected to image conversion processing by the image conversion method of the present invention, all images taken at an equal distance from the vertical direction with respect to the object Each can be converted into data.
[0096]
In this way, if all of the objects are converted into image data that is equidistant and imaged from the vertical direction, based on the converted image, the position detection and shape defect of the imaging object at each imaging position are performed. It is possible to efficiently use an image processing program used for such determination. That is, in order to process an image including image distortion or the like based on a difference in imaging conditions or the like with the above-described image processing program, the parameters of the above-described image processing program are associated with each imaging device in consideration of the distortion. Etc. need to be changed. However, if the image conversion method of the present invention is implemented, the step of changing such parameters and the like can be omitted.
[0097]
【The invention's effect】
As described above, according to the image conversion method and the image conversion apparatus of the present invention, the oblique image data obtained by imaging the imaging object from the oblique direction is obtained by imaging from the direction facing the imaging object. Converted to the directly-facing image data. Since the facing image data is similar to the object to be imaged, various types such as identification of the object to be imaged, detection of a difference from the standard object of the object to be imaged, position measurement of the object to be imaged, shape measurement of the object to be imaged, etc. An existing program library that realizes an image processing function can be used effectively. In addition, since the screen search can be performed effectively, the image conversion method of the present invention is an effective technique in the detection of orientation flats in a semiconductor manufacturing apparatus and in monitoring and inspection in a manufacturing line composed of a plurality of manufacturing processes.
[Brief description of the drawings]
FIG. 1 is a diagram for explaining a state in which a lattice pattern is imaged from directly above and obliquely upward.
FIGS. 2A, 2B, and 2C are diagrams showing lattice pattern images captured from directly above and obliquely upward. FIG.
FIG. 3 is a flowchart for obtaining parameters for coordinate transformation.
FIG. 4 is a diagram for explaining an index for designating coordinates and pixels of a light receiving plane.
FIG. 5 is a diagram for explaining an image data acquisition method for pixels constituting a light receiving plane;
FIG. 6 is a diagram for explaining a relationship between an upper image pixel and an oblique image pixel;
FIG. 7 is a diagram for explaining W (k, m).
FIG. 8 is a diagram for explaining how to obtain W (k, m).
FIG. 9 is a diagram for explaining points picked up to obtain W (k, m).
FIGS. 10A and 10B are diagrams for explaining the relationship between a point O and a quadrilateral EFGH, respectively.
FIGS. 11A, 11B, and 11C are diagrams for explaining the determination of whether or not there is an intersection between a line segment and line segment, respectively.
FIG. 12 is a diagram for explaining a case where line segments overlap with each other.
FIG. 13 is a diagram for explaining a relationship between an upper image pixel and an oblique image pixel;
FIG. 14 is a flowchart for obtaining W (k, m).
FIG. 15 is a diagram for explaining a configuration of an image conversion apparatus;
FIG. 16 is a diagram for explaining the configuration of an image conversion processing unit;
FIGS. 17A and 17B are diagrams showing examples of lattice image conversion, respectively. FIGS.
FIG. 18 is a schematic view of a semiconductor manufacturing apparatus having a mechanism for detecting orientation flats.
[Explanation of symbols]
10: First imaging device
12: Second imaging device
14: Lattice pattern
20: Alignment camera
22: Wafer cassette
24: Wafer
26: Transfer arm
28: Stage
32: Image conversion device
50: Resampling unit

Claims (12)

First image data obtained on an oblique light receiving plane (hereinafter referred to as “first light receiving plane”) by imaging an imaging object from an oblique direction by an imaging device including an imaging optical system and a light receiving plane. In the image conversion method for converting to the second image data obtained in the directly-facing light receiving plane (hereinafter referred to as “second light receiving plane”) when the imaging object is temporarily imaged from the facing direction by the imaging device,
A coordinate conversion step for determining a relationship between a first plane coordinate for determining a pixel position on the oblique light receiving plane and a second plane coordinate for determining a pixel position on the directly-facing light receiving plane;
A re-sampling step of obtaining the light intensity Img2 (I, J) of the pixel [I, J] of the second image data according to the following equation (1):
Img2 (I, J) = Σ [W (k, m) x Img1 (k, m)] / ΣW (k, m) (1)
Where I, J, k, and m are integers.
Img1 (k, m) is the light intensity of the pixel [k, m] in the k-th row and the m-th column of the first image data.
W (k, m) is the I row and J columns of pixels [I, J] of the second image data to be projected onto the first light receiving plane, the pixels on the first light receiving flat surface [k, m] Represents the area projected above.
Moreover, the range of the sum on the right hand side of equation (1), the pixel [I, J] of the second image data to be projected onto the first light receiving plane, the pixels on the first light receiving flat surface [k, m] If even a part is projected on the top, it will cover all the projected pixels [k, m]. The image conversion method according to claim 1, wherein the coordinate conversion step includes:
(A) Multiple points selected to include at least 4 points selected so that 3 or more points do not line up on a straight line, or at least 2 points outside this line, including 3 points on a straight line (These points are hereinafter referred to as “a plurality of characteristic points”.) An imaging object having a mark thereon, and the coordinate value of the mark is known by a coordinate system provided on the plane where the mark exists. An imaging step of imaging the test pattern with the imaging device using the imaging object as a test pattern,
(B) A coordinate reading step for reading the coordinate values of the first plane coordinates for marks on a plurality of characteristic points in the test pattern that are imaged on the first light receiving plane when imaged from an oblique direction;
(C) An arbitrary position in the test pattern is defined as the origin, and based on the image captured on the first light receiving plane of the test pattern captured from an oblique direction, the test pattern has the origin at the second light reception. When the image is taken from the opposite direction projected on the central portion of the plane, the position of the central portion of the second light receiving plane on which the origin is projected and the test pattern image taken from the oblique direction are geometrically A calibration process for virtually setting a scale for similarity conversion to
(D) Second light reception when each mark in the test pattern is imaged from a virtually opposite direction with respect to marks on a plurality of characteristic points imaged from an oblique direction A position calculating step of projecting on a plane and calculating the position of the mark in the second plane coordinates;
(E) the first coordinate of each of the marks on the plurality of characteristic points in the test pattern projected onto the first light receiving plane when imaged from an oblique direction, and a virtually opposite direction set On the second light receiving plane when imaged from the above, based on the second coordinates of the marks on the plurality of characteristic points in the calculated test pattern, coordinate conversion parameters b _i (where i = 1, 2, 3, 4, 5, 6, 7, 8).
An image conversion method comprising: In the image conversion method according to claim 2, a calibration step of virtually setting a center position and a scale in the second plane coordinates when imaging from the directly facing direction,
(C ′) The scale is determined so that the image dimensions are equal at the position of the central portion of the second light receiving plane that is virtually set to the position of the central portion of the first light receiving plane, and An image conversion method comprising a calibration step of virtually setting a position of a central portion when imaging the imaging object from a direction facing the central portion so that the positions of the central portion coincide with each other. 4. The image conversion method according to claim 2, wherein the coordinate reading step, the step of calculating the mark position in the second plane coordinate, and the parameter calculation step are (b ′) a plurality of characteristic features in the test pattern. A coordinate reading step of selecting points and reading the first plane coordinate values (X, Y) for each selected point;
(D ′) a position calculating step for calculating a mark position (x, y) in the second plane coordinates for each selected point selected in (b ′);
(E ′) Eight coordinate conversion parameters defined by the following equations (2) and (3) from the first plane coordinate values (X, Y) and the second plane coordinate values (x, y) b _i (where i = 1, 2, 3, 4, 5, 6, 7, 8), including a parameter calculation step for obtaining simultaneous equations or a method of least squares Image conversion method.
X = (b ₁ x + b ₂ y + b ₃ ) / (b ₇ x + b ₈ y + 1) (2)
Y = (b ₄ x + b ₅ y + b ₆ ) / (b ₇ x + b ₈ y + 1) (3) 2. The image conversion method according to claim 1, wherein a pixel [I, J] of the second image data projected onto the first light receiving plane is projected onto a pixel [k, m] on the first light receiving surface. An image conversion method characterized in that the step of obtaining W (k, m) representing a specific area includes the following steps.
The positions at which the four vertices of the pixels [I, J] constituting the second light receiving plane are projected onto the light receiving plane of the first imaging device are A, B, C, D, and the pixels [1] constituting the first light receiving plane [ The four vertices of i, j] are A ', B', C ', D'
(D) A process of checking whether or not each point of A, B, C, D is included in the quadrilateral A'B'C'D '
(E) A process of checking and picking up whether each point of A ′, B ′, C ′, D ′ is included in the quadrilateral ABCD,
(F-1) The process of picking up the intersection of line segment A'B 'and line segment AB, line segment BC, line segment CD, line segment DA;
(F-2) The process of checking and picking up the intersection of line segment B'C 'and line segment AB, line segment BC, line segment CD, line segment DA,
(F-3) Checking and picking up the intersection of line segment C'D 'and line segment AB, line segment BC, line segment CD, line segment DA;
(F-4) A process of examining and picking up the intersection of line segment D'A 'and line segment AB, line segment BC, line segment CD, line segment DA,
(G) A step of obtaining W (k, m) from the coordinates of all points picked up in the steps (d) to (f-4). First image data obtained on an oblique light receiving plane (hereinafter referred to as “first light receiving plane”) by imaging an imaging object from an oblique direction by an imaging device including an imaging optical system and a light receiving plane. In the image conversion device for converting into second image data obtained in the directly-facing light receiving plane (hereinafter referred to as “second light receiving plane”) when the imaging object is temporarily imaged from the direction facing the imaging object,
Coordinate conversion means for determining a relationship between a first plane coordinate for determining a pixel position on the oblique light receiving plane and a second plane coordinate for determining a pixel position on the directly-facing light receiving plane;
An image conversion apparatus comprising: a resampling unit that obtains the light intensity Img2 (I, J) of the pixel [I, J] of the second image data according to the following equation (1).
Img2 (I, J) = Σ [W (k, m) x Img1 (k, m)] / ΣW (k, m) (1)
Where I, J, k, and m are integers.
Img1 (k, m) is the light intensity of the pixel [k, m] in the k-th row and the m-th column of the first image data.
W (k, m) is the I row and J columns of pixels [I, J] of the second image data to be projected onto the first light receiving plane, the pixels on the first light receiving flat surface [k, m] Represents the area projected above.
Moreover, the range of the sum on the right hand side of equation (1), the pixel [I, J] of the second image data to be projected onto the first light receiving plane, the pixels on the first light receiving flat surface [k, m] If even a part is projected on the top, it will cover all the projected pixels [k, m]. The image conversion apparatus according to claim 6, wherein the coordinate conversion unit includes:
Multiple points selected to include at least two points outside the straight line, including four or more points selected so that three or more points are not aligned on a straight line, or three points on a straight line This point is referred to as “a plurality of characteristic points.”) An imaging object having a mark on the image, and the coordinate value of the mark is known by a coordinate system provided on a plane where the mark exists Imaging means for imaging the test pattern by the imaging device using the object as a test pattern;
Coordinate reading means for reading the coordinate values of the first plane coordinates for marks on a plurality of characteristic points in the test pattern imaged on the first light receiving plane when imaged from an oblique direction;
An arbitrary position in the test pattern is defined as the origin, and based on the image captured on the first light receiving plane of the test pattern captured from an oblique direction, the test pattern is centered on the second light receiving plane. Geometric similarity conversion between the position of the central part of the second light receiving plane where the origin is projected and the test pattern image taken from an oblique direction when the image is taken from the opposite direction projected onto the part Calibration means for virtually setting the scale to be
With respect to marks on a plurality of characteristic points imaged from an oblique direction, each mark in the test pattern is placed on the second light receiving plane when imaged from a virtually opposite direction. Projecting and calculating a position of the mark in the second plane coordinates;
Images are taken from the first coordinate of each of the marks on the plurality of characteristic points in the test pattern projected onto the first light receiving plane when imaged from an oblique direction, and from a virtually set opposite direction. Coordinate conversion parameter b _i (where i = 1) based on the second coordinates of the marks on the plurality of characteristic points in the calculated test pattern on the second light receiving plane , 2, 3, 4, 5, 6, 7, and 8). The image conversion apparatus according to claim 7, wherein calibration means for virtually setting a center position and a scale in the second plane coordinates when imaging from the facing direction,
The scale is determined so that the image dimensions are equal at the position of the central portion of the second light receiving plane that is virtually set to the position of the central portion of the first light receiving plane, and the positions of the central portions of the images of each other An image conversion apparatus comprising calibration means for virtually setting the position of the central portion when the imaging object is imaged from the facing direction so as to match. In the image conversion apparatus according to claim 7 or 8, the coordinate reading means, the means for calculating the mark position in the second plane coordinates, and the parameter calculation means,
Coordinate reading means for selecting a plurality of characteristic points in the test pattern and reading the first plane coordinate value (X, Y) for each selected point;
Position calculating means for calculating the mark position (x, y) in the second plane coordinates for each selected point selected by the coordinate reading means;
From the first plane coordinate value (X, Y) and the second plane coordinate value (x, y), eight coordinate transformation parameters b _i defined by the following equations (2) and (3) (where , I = 1, 2, 3, 4, 5, 6, 7, and 8)), a parameter calculation means for solving the simultaneous equations or a least squares method. apparatus.
X = (b ₁ x + b ₂ y + b ₃ ) / (b ₇ x + b ₈ y + 1) (2)
Y = (b ₄ x + b ₅ y + b ₆ ) / (b ₇ x + b ₈ y + 1) (3) 7. The image conversion apparatus according to claim 6, wherein the pixel [I, J] of the second image data projected onto the first light receiving plane is projected onto the pixel [k, m] on the first light receiving surface. An image conversion apparatus characterized in that means for obtaining W (k, m) representing an area to be obtained includes the following means.
The positions at which the four vertices of the pixels [I, J] constituting the second light receiving plane are projected onto the light receiving plane of the first imaging device are A, B, C, D, and the pixels [1] constituting the first light receiving plane [ The four vertices of i, j] are A ', B', C ', D'
Means 1 for checking whether or not each point of A, B, C, D is included in quadrilateral A'B'C'D ';
Means 2 for checking whether or not each point of A ′, B ′, C ′, D ′ is included in the quadrilateral ABCD;
Means 3 for checking and picking up the intersection of line segment A'B 'and line segment AB, line segment BC, line segment CD, line segment DA;
Means 4 for examining and picking up the intersection of line segment B'C 'and line segment AB, line segment BC, line segment CD, line segment DA;
Means 5 for examining and picking up the intersection of line C'D 'and line AB, line BC, line CD, line DA;
Means 6 for examining and picking up the intersection of line segment D'A 'and line segment AB, line segment BC, line segment CD, line segment DA;
Means 7 for obtaining W (k, m) from the coordinates of all points picked up by means 1 to means 6; 11. A semiconductor manufacturing apparatus comprising: a wafer cassette that stocks a plurality of wafers; and a transfer means that takes out the wafers one by one from the wafer cassette and transfers the wafers to a stage. A semiconductor manufacturing apparatus comprising an image conversion device, wherein the object to be imaged is the wafer taken out from the wafer cassette. A conveyance line comprising a conveyance means connecting a plurality of steps and one or more inspection means using images, comprising the image conversion device according to any one of claims 6 to 10, wherein the imaging object is A transport line characterized by being an object transported on the transport line.

Images