JP7523840B1 - PROGRAM, COMPUTER, INSPECTION SYSTEM AND INSPECTION METHOD - Google Patents

Abstract

[Problem] To provide a program, a computer, an inspection system, and an inspection method that can detect abnormalities in bumps 103 more reliably. [Solution] A receiving unit 32 receives an image 100 including an image 100a of a gap g between a substrate 101 and an object 102. A pre-processing unit 34 specifies a first boundary line b1 in the image 100 using a first trained model 220 generated by machine learning using training data 210 including an image of a gap between the substrate and the object bonded by a bump, from which a first boundary line between the bump and the object is obtained, and specifies a second boundary line b2 from the first boundary line b1 so as to include the lower end of the bump 103 between the first boundary line b1 and the first boundary line b1 based on the size of the gap g. A determining unit 36 determines whether the bump 103 is normal or not from the image portion 100a using a second trained model 320 generated by machine learning using training data 310 including an image of a normal bump. [Selected Figure] FIG. 5

Description

The present invention relates to a program, a computer, an inspection system, and an inspection method.

Conventionally, when mounting a semiconductor chip on a substrate, the semiconductor chip is attached to the substrate by bumps, but whether the bumps are properly attached to the substrate or not (checking the melted state and remaining amount of bumps, judging the appearance such as gloss) is inspected by a person using a microscope. For example, Patent Document 1 discloses a technique for magnifying and observing the main surface side of a chip on which bumps are formed using a microscope.

JP 2008-270628 A

As such, conventionally, bumps used to attach objects such as semiconductor chips to substrates were inspected visually using a microscope to see if they were normal. However, since most bumps are normal to begin with and abnormalities rarely occur, there was a problem that abnormalities in the bumps could easily be overlooked when inspected visually by humans.

The present invention has been made in consideration of these points, and aims to provide a program, computer, inspection system, and inspection method that can more reliably detect bump abnormalities.

The program of the present disclosure is
A program for causing a computer to function as a reception unit, a preprocessing unit, and a determination unit,
The receiving means receives an image including an image of a gap between a substrate and an object to be bonded to the substrate by a bump, the image being captured by an imaging device;
the preprocessing means uses a first trained model generated by machine learning by using training data including an image of a gap between a substrate and an object bonded by a bump, the image being an image from which a first boundary line between the bump and the object can be obtained, to identify a first boundary line in the image received by the receiving means, and identifies a second boundary line from the first boundary line in the image received by the receiving means based on a preset size of the gap between the substrate and the object such that a lower end of the bump is included between the first boundary line and the second boundary line;
the determination means uses a second trained model generated by machine learning by using teacher data including an image of a normal bump to determine whether the bump is normal or not from an image portion of a region between the first boundary line and the second boundary line in the image accepted by the acceptance means;
The first trained model is characterized in that it has been trained to detect all bumps with bounding boxes of the same size and shape.

In the program of the present disclosure,
The bounding box may be rectangular in shape.

In the program of the present disclosure,
The bounding boxes do not have to overlap.

In the program of the present disclosure,
The first learned model may identify the first boundary line based on the edges of all of the bounding boxes on the object side.

In the program of the present disclosure,
The first learned model may identify the first boundary line for each specified area of the image accepted by the accepting means.

In the program of the present disclosure,
The second trained model is generated by machine learning using an image of a normal bump that includes a real image of the bump and a mirror image of the bump projected onto a substrate in the same image;
the determination means uses the second learned model to determine whether the bump is normal or not from an image portion of a region between the first boundary line and the second boundary line in an image including a real image of the bump and a mirror image of the bump projected onto the substrate, the image portion being received by the receiving means;
The second boundary line may be a boundary line between a mirror image of the bump and a mirror image of the object in an image including a real image of the bump received by the receiving means and a mirror image of the bump projected onto the substrate.

In the program of the present disclosure,
The second trained model may be generated by non-defective product training using only training data including images of normal bumps.

In the program of the present disclosure,
The object may be a semiconductor chip.

The computer of the present disclosure includes:
A computer that functions as a reception unit, a preprocessing unit, and a determination unit by executing a program,
The receiving means receives an image including an image of a gap between a substrate and an object to be bonded to the substrate by a bump, the image being captured by an imaging device;
the preprocessing means uses a first trained model generated by machine learning by using training data including an image of a gap between a substrate and an object bonded by a bump, the image being an image from which a first boundary line between the bump and the object can be obtained, to identify a first boundary line in the image received by the receiving means, and identifies a second boundary line from the first boundary line in the image received by the receiving means based on a preset size of the gap between the substrate and the object such that a lower end of the bump is included between the first boundary line and the second boundary line;
the determination means uses a second trained model generated by machine learning by using teacher data including an image of a normal bump to determine whether the bump is normal or not from an image portion of a region between the first boundary line and the second boundary line in the image accepted by the acceptance means;
The first trained model is characterized in that it has been trained to detect all bumps with bounding boxes of the same size and shape.

The inspection system of the present disclosure comprises:
An inspection system including an imaging device and a computer to which an image captured by the imaging device is sent,
the imaging device acquires an image by capturing an image including an image of a gap between a substrate and an object bonded to the substrate by a bump;
The computer functions as a reception unit, a preprocessing unit, and a determination unit by executing a program;
The receiving means receives an image captured by the imaging device,
the preprocessing means uses a first trained model generated by machine learning by using training data including an image of a gap between a substrate and an object bonded by a bump, the image being an image from which a first boundary line between the bump and the object can be obtained, to identify a first boundary line in the image received by the receiving means, and identifies a second boundary line from the first boundary line in the image received by the receiving means based on a preset size of the gap between the substrate and the object such that a lower end of the bump is included between the first boundary line and the second boundary line;
the determination means uses a second trained model generated by machine learning using teacher data including an image of a normal bump to determine whether the bump is normal or not from an image portion of a region between the first boundary line and the second boundary line in the image accepted by the acceptance means;
The first trained model is characterized in that it has been trained to detect all bumps with bounding boxes of the same size and shape.

The inspection method of the present disclosure includes:
An inspection method using an inspection system including an imaging device and a computer to which an image captured by the imaging device is sent, comprising:
acquiring an image by the imaging device capturing an image including an image of a gap between a substrate and an object bonded to the substrate by a bump;
a step of receiving an image captured by the imaging device by the computer;
a step of the computer identifying a first boundary line from the received image using a first trained model generated by machine learning by using training data including an image of a gap between a substrate and an object bonded by a bump, the image being an image from which a first boundary line between the bump and the object is obtained, and identifying a second boundary line from the first boundary line in the received image based on a preset size of the gap between the substrate and the object such that the second boundary line includes a lower end of the bump between the first boundary line and the second boundary line;
A step in which the computer determines whether or not the bump is normal from an image portion of a region between the first boundary line and the second boundary line in the received image by using a second trained model generated by machine learning using teacher data including an image of a normal bump;
Equipped with
The first trained model is characterized in that it has been trained to detect all bumps with bounding boxes of the same size and shape.

In the inspection method of the present disclosure,
The second trained model is generated by machine learning using an image of a normal bump that includes a real image of the bump and a mirror image of the bump projected onto a substrate in the same image;
The imaging device may capture an image obliquely downward when capturing an image to be sent to the computer, and may obtain an image including a real image of the bump and a mirror image of the bump projected onto the substrate.

The program, computer, inspection system, and inspection method disclosed herein can more reliably detect bump abnormalities.

FIG. 1 is a diagram illustrating a schematic configuration of an inspection system according to an embodiment of the present disclosure. FIG. 2 is a diagram illustrating an exemplary flow of a detection method and information processing in the detection method according to an embodiment of the present disclosure. FIG. 2 illustrates an exemplary flow of information processing in an inspection system and detection method according to an embodiment of the present disclosure. 1A-1C show exemplary arrangements of a substrate, objects, gaps, and bumps imaged in an inspection system and detection method according to an embodiment of the present disclosure. 1A-1C illustrate example images captured and determined in an inspection system and detection method according to an embodiment of the present disclosure. 1A and 1B are conceptual diagrams showing a method for identifying a first boundary line in an inspection system and a detection method according to a conventional example and an embodiment of the present disclosure. 1A to 1C are diagrams for explaining a cut-out process in an inspection system and a detection method according to an embodiment of the present disclosure.

The following describes embodiments of the present disclosure, but the invention according to the present disclosure is not limited thereto. FIG. 1 is a diagram showing a schematic configuration of an inspection system according to an embodiment of the present disclosure. FIG. 2 is a diagram showing an exemplary flow of a detection method and information processing in the detection method according to an embodiment of the present disclosure. FIG. 3 is a diagram showing an exemplary flow of information processing in an inspection system and detection method according to an embodiment of the present disclosure. FIG. 4 is a diagram showing an exemplary arrangement of a substrate, an object, a gap, and a bump imaged in an inspection system and detection method according to an embodiment of the present disclosure. FIG. 5 is a diagram showing an exemplary image imaged and determined in an inspection system and detection method according to an embodiment of the present disclosure.

[Inspection system 1]
FIG. 1 shows an inspection system 1 according to the present disclosure. The inspection system 1 detects whether or not bumps 103 (e.g., FIG. 4B) for bonding an object 102 such as a semiconductor chip to a substrate 101 are normal. As shown in FIG. 1, the inspection system 1 includes an imaging device 2 and a computer 3 to which an image 100 (e.g., FIGS. 5A and 5B) captured by the imaging device 2 is sent. The computer 3 and the imaging device 2 are connected to be able to communicate with each other. The computer 3 judges whether or not the bumps 103 on the image 100 captured by the imaging device 2 are normal (e.g., FIG. 5C). According to the inspection system 1, the process from imaging to judgment can be performed automatically more reliably than visual inspection.

Note that FIG. 4A is a top view of the substrate 101 and the object 102 to be imaged, and the object 102 and the obstacle 104 are placed on the substrate 101. Examples of the object 102 to be bonded to the substrate 101 include semiconductor chips and chip molds. Note that FIG. 4 shows an example of an FCCSP. As in this example, defective products can be excluded in a judgment inspection in a bonded state before underfill filling. Examples of the obstacle 104 placed on the substrate 101 include chip capacitors, but are not limited to these. FIG. 4B is a diagram showing an example of the positional relationship between the substrate 101, the object 102, and the bumps 103, and the bumps 103 are interposed between the substrate 101 and the object 102, and the object 102 is bonded to the substrate 101 by the bumps 103. The example of the substrate 101 is not particularly limited. Furthermore, a metal layer or the like may be laminated on the substrate 101, or the substrate 101 may be laminated with epoxy resin or the like. As shown by the arrows in Figure 4(A), the accuracy of the judgment can be improved by using images from up to four directions in the judgment process. The bumps 103 are the joints between the substrate 101 and the object 102, and are usually made of solder, but the bumps 103 are not limited to this.

The inspection system 1 can be incorporated into a manufacturing line and can determine the quality of the bumps 103 and thus the target object 102 in real time.

<Imaging device 2>
Next, the configuration of the imaging device 2 will be described. In the inspection system 1, the imaging device 2 captures an image 100 (imaging range including the gap g) including an image 100a of a gap g (see FIG. 4C ) between a substrate 101 and an object 102 bonded to the substrate 101 by a bump 103 (hereinafter, also simply referred to as "imaging the gap g") to obtain the image 100. As shown in FIG. 1, the imaging device 2 includes a camera 21 and a communication unit 22.

The camera 21 is not particularly limited as long as it can capture an image of the gap g between the substrate 101 and the object 102 as shown in FIG. 4(C). Generally, information from at least 3 to 4 pixels is required for defect detection using image processing. Therefore, for example, if it is desired to visualize a 10 μm defect, this objective can be achieved by attaching a telecentric lens with a magnification of 1x to the size of the mounted CMOS (object 102), 2.7 μm/pix, to the camera 21.

4(C) shows the direction (direction indicated by solid line arrow and dotted line arrow) in which the imaging device 2 (camera 21) images the gap g. Note that FIG. 4(C) shows an example in which the bumps 103 are not present on the outer periphery of the object 102, but the present invention is not limited to this, and the present invention is also applicable to the case in which the bumps 103 are present on the outer periphery of the object 102. In order to determine whether the bumps for bonding the object to the substrate are normal or not, it is possible to take an image of the bumps 103 in the horizontal (sideways) direction, i.e., from the front, as shown by the dotted line arrows in FIG. 4(B) and FIG. 4(C). However, if there is an obstacle 104 for imaging such as a chip capacitor, it may be difficult to take an image of the bumps 103 from the horizontal direction. Therefore, as shown by the solid line arrow in FIG. 4(C), when taking an image 100 to be sent to the computer 3, it is also possible to take an image diagonally downward. The imaging angle (camera angle) at this time, i.e., "diagonally downward" is determined by the distance between the object 102 and the obstacle 104, and is not limited. It is preferable to adjust the camera angle so that the light reflected by the obstacle 104 does not overlap with the bump 103, and so that the bump 103 is not too far hidden by the object 102.

The communication unit 22 includes a communication interface for transmitting and receiving signals to and from an external device wirelessly or via a wired connection. The image 100 captured by the camera 21 is transmitted to the computer 3 (reception means 32) by the communication unit 22.

<Computer 3>
The configuration of the computer 3 of the present disclosure will be described with reference to Fig. 1. The computer 3 of the present embodiment is composed of an industrial computer, a tablet terminal, or the like, and includes a control unit 30, a storage unit 40, a communication unit 42, a display unit 44, and an operation unit 46, as shown in Fig. 1.

The control unit 30 is composed of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an AI inference device, etc., and controls the operation of the computer 3. Specifically, the control unit 30 functions as a reception means 32, a preprocessing means 34, a determination means 36, and an output means 38 by executing a program stored in the storage unit 40 described below. Furthermore, the control unit 30 may function as a first model generation means 200 and a second model generation means 300 by executing a program stored in the storage unit 40. Each of these means will be described later.

The storage unit 40 is composed of, for example, a hard disk drive (HDD), a random access memory (RAM), a read only memory (ROM), a solid state drive (SSD), etc. Furthermore, the storage unit 40 is not limited to being built into the computer 3, and may be a storage medium (e.g., a USB memory) that can be detachably attached to the computer 3. In this embodiment, the storage unit 40 is configured to store the program executed by the control unit 30, the first trained model 220, the second trained model 320, etc.

The communication unit 42 includes a communication interface for wirelessly or wiredly transmitting and receiving signals to and from an external device. The communication unit 42 allows the control unit 30 to transmit and receive signals to and from the imaging device 2.

The display unit 44 is, for example, a monitor, and displays various screens by receiving a display command signal from the control unit 30. The operation unit 46 is, for example, a keyboard, and is capable of giving various commands to the control unit 30. Note that in this embodiment, a display operation unit such as a touch panel in which the display unit 44 and the operation unit 46 are integrated may be used.

(Details of the control unit 30)
(Reception means 32)
The accepting unit 32 accepts an image 100 including an image 100a of a gap g between a substrate 101 and an object 102 bonded to the substrate 101 by a bump 103, the image being captured by the imaging device 2. Fig. 5(A) shows an example of the image 100 including the image 100a of the gap g.

(Pre-processing means 34)
The pre-processing means 34 (and the pre-processing step in the present disclosure) aims to extract (mask) a judgment region by identifying the first boundary line b1 and the second boundary line b2. The pre-processing means 34 identifies the first boundary line b1 in the image 100 accepted by the accepting means 32 using the first trained model 220 generated by machine learning by using the teacher data 210 including an image of the gap g between the substrate 101 and the object 102 bonded by the bump 103, from which the first boundary line b1 between the bump 103 and the object 102 is obtained (identification process 51 of the first boundary line b1 in FIG. 2), and identifies the second boundary line b2 from the first boundary line b1 so that the lower end of the bump 103 is included between the first boundary line b1 in the image accepted by the accepting means 32 based on the size of the gap g between the substrate 101 and the object 102 that is set in advance (identification process 52 of the second boundary line b2 in FIG. 2). The "first boundary line b1" and the "second boundary line b2" define the upper and lower ends of the image portion 100a that is to be the judgment area.

((First trained model 220))
The first trained model 220 is trained to detect all bumps 103 (which may include mirror images, which will be described later) with bounding boxes (BBs) of the same size and shape. Here, the "detection" of the bumps 103 by the first trained model 220 refers to identifying (annotating) each bump 103 that appears in the image 100 accepted by the accepting means 32 with a BB, and includes surrounding each bump 103 with a BB, including each bump 103 together with a margin (background), and partially identifying each bump 103 (see FIG. 6B). The shape of these BBs is preferably rectangular. In addition, it is preferable that the BBs do not overlap each other. This annotation work includes surrounding a standard single bump 103 as the only reference for the BB in the image 100, and applying (detecting) this BB to all bumps 103 in the image 100 without changing the size and starting point Y position (top side). Here, there is no particular limitation on the "standard bump", but bumps that are obviously larger, smaller, or more irregular than other bumps are excluded.

((Identification of the first boundary line b1))
As a result of detecting (annotating) all the bumps 103 with the BB, the first learned model 220 can identify the first boundary line b1 based on the side (upper side) of all the BBs on the side of the object 102. For example, if the heights of the bumps 103 captured in the images are the same as in Figures 5(A) and 5(B), all the bumps 103 are uniformly surrounded by the BBs, and the upper sides of the BBs can be connected to form the first boundary line b1. Even if the heights of all the bumps 103 are not the same, if all the bumps 103 are detected with BBs of the same size and shape, the upper sides of the BBs will be roughly aligned (Figure 6(B)). In this case, the first trained model 220 can specify the first boundary line b1 so as to pass through the upper side of the BB with the highest Y position (upper side) in the image 100 (for example, the center of FIG. 6B) or the BB with the lowest Y position (upper side) (for example, a BB other than the center of FIG. 6B), or can specify the first boundary line b1 so as to pass through the average height (Y position) of the upper side of the BB. As long as the first boundary line b1 can be specified as described above, the machine learning for generating the first trained model 220 is not particularly limited, and various methods such as deep learning can be used. The first trained model 220 is an object detection AI in a stage preceding the second trained model 320 for determining whether or not the model is good, which will be described later, and is trained using various images (which may include mirror images) including the bump as described above as the teacher data 210. Note that the first boundary line b1 is not necessarily actually specified in the image in which the first boundary line b1 between the bump 103 and the object 102 is obtained as the teacher data 210. As described below, BB is shown in the image serving as the teacher data 210. In other words, "an image from which the first boundary line b1 can be obtained" means an image from which the first boundary line b1 will be obtained by the preprocessing means 34 of the present disclosure.

((Identification of second boundary line b2))
In addition, "the second boundary line b2 is specified based on the size of the gap g between the substrate 101 and the object 102 that is set in advance" means that if the first boundary line b1 (upper end), which is the boundary line between the bump 103 and the object 102, is known, the second boundary line b2 (lower end) can be derived from the size of the gap g that is set in advance. Here, the size of the gap g can be defined from the number of pixels (pix) of the image. In addition, the setting for deriving the second boundary line b2 (lower end) including the lower end of the bump 103 in the image accepted by the accepting means 32 can be arbitrarily changed according to the type, purpose, and operation method of the object 102 and the bump 103. For example, if it is desired to judge only rough defects, the amount of information may be reduced (the second boundary line b2 is specified upward), or a large amount of mirror images may be included in order to obtain information that is not visible in the real image as described later (the second boundary line b2 can also be set to be specified downward. See b2' in FIG. 5B). 5A, which is an exemplary side view (side view in one direction) of the substrate 101, the object 102, etc., captured from the direction indicated by the dotted arrow in FIG. 4C, shows exemplary first boundary line b1 and second boundary line b2 specified by the pre-processing means 34, and an image portion 100a of the area between them. In this image portion 100a, the upper and lower ends of all bumps 103 are in contact with the first boundary line b1 and the second boundary line b2. As in this example, in order to remove the maximum amount of unnecessary information of the object 102, the substrate 101, etc., it is particularly preferable to specify the first boundary line b1 and the second boundary line b2 so as to be in contact with the upper and lower ends of the bumps 103.

((The significance of identifying the first boundary line b1 and the second boundary line b2))
In the present invention, the reason for identifying the first boundary line b1 and the second boundary line b2 before the judgment process is as follows. In the image 100, the area (amount of information) of the object 102 and the substrate 101 is larger than the area (amount of information) of the bump 103, so if the judgment area remains the image 100, it is difficult for the AI to judge the bump 103 part. In particular, if the cut surface of the object 102 is not managed in the manufacturing process, the appearance of the object 102 in the image will vary, and the difference between normal and abnormal bumps 103 will be more noticeable, causing erroneous detection and erroneous judgment. Therefore, the first trained model 220 according to the present invention identifies the first boundary line b1 and the second boundary line b2 in order to perform masking of the bump 103 in the image 100.

In general, when creating a learning model for object detection, the bumps 103 on the screen are surrounded one by one with a mouse. The inventor tried to create a learning model using this conventional manual method, but the first boundary line b1 (multiple bumps 103) was not neatly aligned. In other words, the more precisely the human surrounded the bumps and the more samples were used, the more uneven the first boundary line b1 became (FIG. 6(A)). The inventor then discovered that there are two main problems with aligning the first boundary line b1 for mask processing (identifying the first boundary line b1). One is that the object 102 (chip) and the board 101 vary greatly from one to another, and in some cases may be warped (not completely flat). In addition, the object 102 may be tilted relative to the board 101 due to differences in the amount of solder on the left and right bumps 103. Therefore, the first boundary line b1 cannot be simply determined by drawing a straight line between the left and right ends of the chip. Another reason is that there are individual variations in the bumps 103 (even the same type of bumps can vary across all four sides, and even on one side), so naturally the glossy parts that are illuminated for imaging look different. In other words, there are parts behind the glossy parts (dark parts) that are not illuminated, and an example of this part is the top (rounded part) of the bump 103. Furthermore, there are cases where bumps 103 are imaged in different positions at the front and back. In this way, the joint between the bump 103 and the object 102 may not appear in the image, and the first boundary line b1 does not exist between all of the bumps 103 and the object 102.

The inventors then, after careful consideration, changed their approach and decided to circle only one bump 103 with the mouse (FIG. 6B). Then, using such a reference value bump 103 from various images as teacher data 210 to construct a first trained model 220, they found that the first boundary line b1 (multiple bumps 103) was aligned. In other words, they found that when using this first trained model 220, the first boundary line b1 begins to align as if the reference value slides from one end of the chip to the other. This is thought to be because encircling each of the bumps 103, most of which are non-defective, ultimately increases the influence of human error. Note that FIGS. 6A and 6B show an example in which a mirror image is also reflected as in FIG. 5B, but the presence or absence of a mirror image is not essential. In addition, in order to identify the first boundary line b1, it is preferable that the judgment area of the bump 103 is as large as possible and extends to the top (rounded portion).

((Other processing))
Furthermore, by identifying the first boundary line b1 with high accuracy, in addition to the masking process, the position of the inspection image to be input to the second trained model 320 may also be corrected. For example, the Y coordinates (vertical direction) of the first boundary line b1 and the second boundary line b2 in the image may be corrected. In this way, the anomaly detection AI of the second trained model 320 can be more accurately determined.

((Cutout processing))
However, if the left and right ends of the object 102 or the board 101 are not horizontal, it may be difficult to align the first boundary line b1 when using the captured (one field of view) image 100. Therefore, the first trained model 220 may specify the first boundary line b1 for each predetermined region (e.g., a region of Y pixels (pix) length in FIG. 7) of the image 100 accepted by the accepting means 32. That is, the difference between the left and right ends of the object 102 or the board 101 can be reduced by cutting out (trimming) the learning area or the judgment area into a small area.

This cutout process may be performed in the following procedure. First, a feature point search is performed by pattern matching. The "feature point" is not particularly limited, but refers to a unique shape found in the image 100. Depending on the type of product, a feature that is unique in the bump array (for example, the gap between adjacent bumps 103 is different, etc.) can be used as the feature point. Next, a cutout region is determined based on the searched feature points. Specifically, the left and right end positions of the cutout region (i.e., the relative distance from the feature point in the X coordinate) are determined at a position offset (relative) by a predetermined pixel length from the detected X coordinate of the feature point in the image 100 (for example, if the detected feature point is in the center of the image, the start point of the cutout region can be expressed as minus X and the end point as plus X). Next, the distance from the left end position to the right end position of the cutout region is set to a fixed size (Y pix length in FIG. 7), and the cutout region is cut out sequentially from the image 100 while overlapping. In FIG. 7, Z pix indicates the length of the overlapping region. The values of "Y" and "Z" can be set appropriately according to the size of the bump 103 and the image 100. Next, in each cut-out region of Y pix length, the bump 103 is detected, and the first boundary line b1 and the second boundary line b2 are identified and masked. After the masking process, each cut-out region may be further divided into two, left and right (with a predetermined region overlapping in the center), according to the defect size to be judged and the resolution of the camera 21, and used as the judgment region. This division process makes it possible to avoid a situation in which the defect becomes invisible due to reduction when there is a limit to the input size to the AI (first trained model 220).

(Determination means 36)
The determination means 36 uses the second trained model 320 generated by machine learning using the teacher data 310 including an image of the normal bump 103 to determine whether the bump 103 is normal or not from the image portion 100a of the area between the first boundary line b1 and the second boundary line b2 of the image 100 accepted by the accepting means 32 (determination process 53 in FIG. 2). This "determination" corresponds to the appearance inspection that was originally performed visually by humans, and specifically includes checking the melting state and remaining amount of the bump 103, and judging the appearance such as gloss. In addition, various types of machine learning such as deep learning can be used for generating the second trained model 320. Note that FIG. 5(C) shows examples of bumps 103 judged to be normal (OK) and abnormal (NG).

In this way, instead of using the image 100 itself accepted by the accepting means 32, the judgment range is limited by masking, and the judgment means 36 makes a judgment from the image portion 100a (FIG. 5(A)) of the area (gap g) between the first boundary line b1 and the second boundary line b2. In addition, the image 100a may be further divided (cut out) so that each divided image contains only one bump 103. This division method is not particularly limited, and the above-mentioned cutting process may be performed. In this way, it is possible to prevent the features of the board 101 and the object 102 captured in the image 100 from being detected. More specifically, the second trained model 320 is used for the judgment to detect the position of the bump 103 from the image portion 100a, and erroneous judgment can be prevented by AI object detection.

If the substrate 101 reflects a mirror image (virtual image) of the bump 103 by reflecting light during imaging, the second trained model 320 may be generated by machine learning by using an image of the normal bump 103 that includes both the real image of the bump 103 and the mirror image of the bump 103 projected onto the substrate 101 in the same image. As shown in the example of FIG. 4(C) with the solid arrow, when the gap g is imaged diagonally downward to inspect the bump 103, the mirror image of the bump 103 projected onto the substrate 101 is also imaged and can be used for judgment. In the example shown in FIG. 5(B), each bump 103 is reflected in the image 100 in a daruma shape. That is, as a result of imaging the gap g obliquely downward, the area above the dotted line between b1' and b2' shows the real image of the bump 103, and the area below the dotted line shows the mirror image of the bump 103 (as in FIG. 4C, as a result of imaging the gap g obliquely downward in a state where there is no bump 103 on the outer periphery of the object 102, the upper end of the bump 103 is shown by a dotted line to show that the upper end of the real image of the bump 103 is hidden by the object 102). Here, as in FIG. 5B, the upper end of each bump 103, i.e., a part of the area of the bump 103 in the real image, may not be visible (the amount of information is small). Even in this case, b1' between the hidden upper end bumps 103 and the object 102 in the image 100 may be specified as the first boundary line, and further b2' may be specified as the second boundary line so that the area of the mirror image is also included in the judgment area. By doing so, the amount of information showing the difference between good and bad bumps 103 increases, and the accuracy of the judgment is improved or remains high. Therefore, when a mirror image is also captured as in the example shown in FIG. 5B, the second boundary line b2 can be a second boundary line b2' indicating the boundary line between the mirror image of bump 103 and the mirror image of object 102 in order to improve the judgment accuracy by using the mirror image as well. That is, the image portion 100a' that is the judgment area in this case corresponds to the area between the first boundary line b1' based on the real image and the second boundary line b2' based on the mirror image. This second boundary line b2' can also be specified based on the size of the gap g between the substrate 101 and object 102 that is set in advance.

Alternatively, the second trained model 320 may be generated by good product training using only the teacher data 310 including an image of the normal bump 103 (as described above, the upper end of the bump 103 may be hidden and/or may include a mirror image of the bump 103). The determination means 36 may use the second trained model 320 described above to determine whether the bump 103 is normal or not from the image portion 100a of the area between the first boundary line b1 or b1' and the second boundary line b2 or b2' of the image 100 including the real image of the bump 103 accepted by the acceptance means 32 and the mirror image of the bump 103 projected onto the substrate 101.

If it is not possible to judge, the bump 103 is judged to be abnormal (NG).

(Output means 38)
The output means 38 outputs the determination result via the display unit 44 .

(First model generation means 200, second model generation means 300)
The first model generation means 200 and the second model generation means 300 generate the first trained model 320 and the second trained model 220 by machine learning using the teacher data 210 and 310, respectively ( FIG. 2 ). The first trained model 220 and/or the second trained model 320 may function by a device other than the computer 3 executing a predetermined program, or the control unit 30 may function by executing a program stored in the storage unit 40. In the former case, the first trained model 220 and/or the second trained model 320 generated by the other device is transmitted to the computer 3 and stored in the storage unit 40 of the computer 3.

[Detection method]
Next, a description will be given of a detection method using the computer 3 (the inspection system 1) as described above. Note that the following process is performed by the control unit 30 executing a program stored in the storage unit 40.

First, the imaging device 2 captures an image 100 by capturing an image of the gap g between the substrate 101 and the object 102 that is attached to the substrate 101 by the bumps 103 (FIG. 3, step S1). Here, as described above, when capturing an image to be sent to the computer 3, the imaging device 2 captures an image obliquely downward, and can also capture an image that includes a real image of the bumps 103 and a mirror image of the bumps 103 projected onto the substrate 101.

Next, the computer 3 (reception means 32) receives the image captured by the imaging device 2 (Figure 3, step S2).

Next, the computer 3 (pre-processing means 34) uses the first trained model 220 generated by machine learning using training data including an image of the gap g between the substrate 101 and the object 102 bonded by the bump 103, from which the first boundary line b1 between the bump 103 and the object 102 is obtained, to identify the first boundary line b1 from the received image 100 (FIG. 2, identification process 51 of the first boundary line b1), and identifies the second boundary line b2 from the first boundary line b1 so as to include the lower end of the bump 103 between the first boundary line b1 in the received image 100 based on the size of the gap g between the substrate 101 and the object 102 that is set in advance (FIG. 2, identification process 51 of the second boundary line b2; FIG. 3, step S3). As described above, the first trained model 220 has been trained to detect all bumps 103 with bounding boxes of the same size and shape. In this way, the image portion 100a that is the judgment area is extracted from the first boundary line b1 and the second boundary line b2. Furthermore, as pre-processing for the image 100, angle measurement and rotation correction using contour extraction may be performed so that the board 101 and the object 102 are horizontal, or the image 100 may be enlarged.

Next, the computer 3 (determination means 36) uses the second trained model 320 generated by machine learning using training data including an image of the normal bump 103 to determine whether the bump 103 is normal from the image portion 100a of the area between the first boundary line b1 and the second boundary line b2 of the received image (FIG. 3, step S4). As described above, the second trained model 320 may be generated by machine learning using an image of the normal bump 103 that includes both the real image of the bump 103 and the mirror image of the bump 103 projected onto the substrate 101 in the same image.

Next, the computer 3 (output means 38) outputs the determination result via the display unit 44. The computer 3 (output means 38) may also transmit the determination result to an external device via the communication unit 42.

According to the program, computer 3, inspection system 1, and inspection method of the present disclosure configured as described above, the inspection system 1 includes an imaging device 2 and a computer 3 to which an image captured by the imaging device 2 is sent. The imaging device 2 acquires an image 100 by capturing an image 100 including an image 100a of the gap g between the substrate 101 and an object 102 bonded to the substrate 101 by the bumps 103. The computer 3 (control unit 30) executes the program to function as a receiving means 32, a preprocessing means 34, and a determining means 36. The receiving means 32 receives the image 100 including an image 100a of the gap g between the substrate 101 and the object 102 bonded to the substrate 101 by the bumps 103, captured by the imaging device 2. The preprocessing means 34 uses a first trained model 220 generated by machine learning by using teacher data including an image of the gap g between the substrate 101 and the object 102 bonded by the bump 103, in which the first boundary line b1 between the bump 103 and the object 102 is obtained, to specify the first boundary line b1 in the image 100 accepted by the accepting means 32, and specifies a second boundary line b2 from the first boundary line b1 in the image 100 accepted by the accepting means 32 based on a preset size of the gap g between the substrate 101 and the object 102 so that the lower end of the bump 103 is included between the first boundary line b1 and the first boundary line b1 in the image 100 accepted by the accepting means 32. The determination means 36 uses a second trained model 320 generated by machine learning by using teacher data including an image of the normal bump 103, to determine whether the bump 103 is normal or not from the image portion 100a of the region between the first boundary line b1 and the second boundary line b2 in the image 100 accepted by the accepting means 32. The first trained model 220 is trained to detect all bumps 103 with bounding boxes of the same size and shape.

Conventionally, a person would visually inspect bumps for bonding objects such as semiconductor chips to a substrate using a microscope to see if they were normal. However, most bumps are normal and abnormalities are rare, so there was a problem that abnormalities in the bumps were easily overlooked. In contrast, the program, computer 3, inspection system 1, and inspection method disclosed herein can detect abnormalities in bumps from a received image more reliably than visual inspection. In particular, the present invention identifies the first boundary line b1 and the second boundary line b2 in the received image 100, and determines whether the bumps 103 are normal from the image portion 100a between the first boundary line b1 and the second boundary line b2, thereby achieving efficient inspection.

In addition, in the program, computer 3, inspection system 1, and inspection method disclosed herein, the shape of the bounding box can be rectangular. Bounding boxes of various rectangular shapes can be set to match the bumps 103.

In addition, in the program, computer 3, inspection system 1, and inspection method disclosed herein, it is preferable that the bounding boxes do not overlap each other as a means of preventing unnecessary bounding boxes from being detected when identifying the first boundary line b1. Non-maximum suppression (NMS) can be used as a method of selecting or clarifying an appropriate BB for identifying the first boundary line b1 from the many BBs generated.

Furthermore, in the program, computer 3, inspection system 1, and inspection method disclosed herein, the first trained model 220 may identify the first boundary line b1 based on the side (top side) of the object 102 of all bounding boxes. In this way, the first boundary line b1 can be easily identified based on the top side of each bounding box.

Furthermore, in the program, computer 3, inspection system 1, and inspection method disclosed herein, the first trained model 220 may identify the first boundary line b1 for each predetermined area of the image 100 accepted by the accepting means 32. By adjusting the image size through the above-described cropping, it becomes possible to ignore the warping or tilt of the object 102 such as a chip (because short sections are approximated by straight lines).

In addition, in the program, computer 3, inspection system 1, and inspection method disclosed herein, the second learned model 320 is generated by machine learning by using an image of a normal bump 103 that includes a real image of the bump 103 and a mirror image of the bump 103 projected onto the substrate 101 in the same image, and the determination means 36 may use this second learned model 320 to determine whether the bump 103 is normal or not from an image portion 100a' in the area between the first boundary line b1 (b1') and the second boundary line b2' in the image 100 that includes the real image of the bump 103 and the mirror image of the bump 103 projected onto the substrate 101 and that is accepted by the accepting means 32, where the second boundary line b2' is the boundary line between the mirror image of the bump 103 and the mirror image of the object 102 in the image 100 that includes the real image of the bump 103 and the mirror image of the bump 103 projected onto the substrate 101 and that is accepted by the accepting means 32 (Figure 5 (B)). It is possible to photograph the bump 103 from the side as shown in FIG. 4B, but if there is an obstacle 104 to the image, such as a chip capacitor, it may be difficult to photograph the bump 103 from the side. Therefore, as shown in FIG. 4C, even if the image 100 is photographed diagonally downward, it is possible to determine whether the bump 103 is normal or not by using the actual image of the bump 103 contained in the image 100 and the mirror image of the bump 103 projected onto the substrate 101. In other words, according to the program, computer 3, inspection system 1, and inspection method disclosed herein, even if there is an obstacle 104 that obstructs the imaging of the bump 103 or the gap g, a more reliable judgment can be made. Furthermore, by using the mirror image, the amount of information for judgment is increased, and the judgment accuracy is improved.

In addition, in the program, computer 3, inspection system 1, and inspection method disclosed herein, the second trained model 320 may be generated by good product learning using only training data including images of normal bumps 103. The program, computer 3, inspection system 1, and inspection method disclosed herein are also suitable when the number of actual defective products is small, making it difficult to define defective products, or when it is difficult to collect data on defective products.

In addition, in the program, computer 3, inspection system 1, and inspection method of the present disclosure, the object 102 may be a semiconductor chip. In this manner, the program, computer 3, inspection system 1, and inspection method of the present disclosure can be suitably applied when, in addition to the object 102, there are other objects (such as an obstacle 104) placed on the substrate 101 in association with the object 102.

In addition, in the inspection method disclosed herein, the second trained model 320 is generated by machine learning using an image of a normal bump 103 that includes both a real image of the bump 103 and a mirror image of the bump 103 projected onto the substrate 101 in the same image, and the imaging device 2 may capture an image diagonally downward when capturing an image to be sent to the computer 3, thereby acquiring an image that includes both a real image of the bump 103 and a mirror image of the bump 103 projected onto the substrate 101 (see FIG. 4(C)). In this way, even if there is an obstacle 104 as shown in FIG. 4 that impedes the imaging of the bump 103, it is possible to accurately determine whether the bump 103 is normal or not.

Note that the computer 3, program, and detection method disclosed herein are not limited to the aspects and combinations described above, and various modifications can be made.

For example, as described above, even when the mirror image is included in the image 100, the preprocessing means 34 may identify the first boundary line b1 (b1') and the second boundary line b2' in the image 100, but it is sufficient that the height of the first boundary line b1 (b1') can be specified so that it includes about half (not too much, not too little) of the bump 103 in the mirror image. Since there is variation in the gap g even within the same sample, the first boundary line b1 (b1') and the second boundary line b2' cannot necessarily be identified with high accuracy, and good judgment results can be obtained even in this case.

1 Inspection system 2 Imaging device 21 Camera 22 Communication unit 3 Computer 30 Control unit 32 Reception means 34 Preprocessing means 36 Determination means 38 Output means 22a Model generation unit 24a Reception unit 26a Estimation unit 28a Output unit 40 Memory unit 42 Communication unit 44 Display unit 46 Operation unit 100, 100a, 100a' Image 101 Substrate 102 Object 103 Bump 104 Obstacle 200 First model generation means 210 Teacher data 220 First trained model 300 Second model generation means 310 Teacher data 320 Second trained model b1, b1' First boundary line b2, b2' Second boundary line g Gap

Claims (12)

A program for causing a computer to function as a reception unit, a preprocessing unit, and a determination unit,
The receiving means receives an image including an image of a gap between a substrate and an object to be bonded to the substrate by a bump, the image being captured by an imaging device;
the preprocessing means uses a first trained model generated by machine learning by using training data including an image of a gap between a substrate and an object bonded by a bump, the image being an image from which a first boundary line between the bump and the object can be obtained, to identify a first boundary line in the image received by the receiving means, and identifies a second boundary line from the first boundary line in the image received by the receiving means based on a preset size of the gap between the substrate and the object such that a lower end of the bump is included between the first boundary line and the second boundary line;
the determination means uses a second trained model generated by machine learning by using teacher data including an image of a normal bump to determine whether the bump is normal or not from an image portion of a region between the first boundary line and the second boundary line in the image accepted by the acceptance means;
The first trained model is trained to detect all bumps with bounding boxes of the same size and shape. The program of claim 1, wherein the bounding box has a rectangular shape. The program of claim 1, wherein the bounding boxes do not overlap. The program according to claim 1, wherein the first trained model identifies the first boundary line based on the object-side edges of all the bounding boxes. The program according to claim 1, wherein the first trained model identifies the first boundary line for each predetermined area of the image accepted by the accepting means. The second trained model is generated by machine learning using an image of a normal bump that includes a real image of the bump and a mirror image of the bump projected onto a substrate in the same image;
the determination means uses the second learned model to determine whether the bump is normal or not from an image portion of a region between the first boundary line and the second boundary line in an image including a real image of the bump and a mirror image of the bump projected onto the substrate, the image portion being received by the receiving means;
The program according to claim 1 , wherein the second boundary line is a boundary line between the mirror image of the bump and the mirror image of the object in an image including the real image of the bump accepted by the accepting means and a mirror image of the bump projected onto the substrate. The program of claim 1, wherein the second trained model is generated by non-defective learning using only training data that includes images of normal bumps. The program of claim 1, wherein the object is a semiconductor chip. A computer that functions as a reception unit, a preprocessing unit, and a determination unit by executing a program,
The receiving means receives an image including an image of a gap between a substrate and an object to be bonded to the substrate by a bump, the image being captured by an imaging device;
the preprocessing means uses a first trained model generated by machine learning by using training data including an image of a gap between a substrate and an object bonded by a bump, the image being an image from which a first boundary line between the bump and the object can be obtained, to identify a first boundary line in the image received by the receiving means, and identifies a second boundary line from the first boundary line in the image received by the receiving means based on a preset size of the gap between the substrate and the object such that a lower end of the bump is included between the first boundary line and the second boundary line;
the determination means uses a second trained model generated by machine learning by using teacher data including an image of a normal bump to determine whether the bump is normal or not from an image portion of a region between the first boundary line and the second boundary line in the image accepted by the acceptance means;
The first trained model is trained to detect all bumps with bounding boxes of the same size and shape. An inspection system including an imaging device and a computer to which an image captured by the imaging device is sent,
the imaging device acquires an image by capturing an image including an image of a gap between a substrate and an object bonded to the substrate by a bump;
The computer functions as a reception unit, a preprocessing unit, and a determination unit by executing a program;
The receiving means receives an image captured by the imaging device,
the preprocessing means uses a first trained model generated by machine learning by using training data including an image of a gap between a substrate and an object bonded by a bump, the image being an image from which a first boundary line between the bump and the object can be obtained, to identify a first boundary line in the image received by the receiving means, and identifies a second boundary line from the first boundary line in the image received by the receiving means based on a preset size of the gap between the substrate and the object such that a lower end of the bump is included between the first boundary line and the second boundary line;
the determination means uses a second trained model generated by machine learning using teacher data including an image of a normal bump to determine whether the bump is normal or not from an image portion of a region between the first boundary line and the second boundary line in the image accepted by the acceptance means;
The first trained model is trained to detect all bumps with bounding boxes of the same size and shape. An inspection method using an inspection system including an imaging device and a computer to which an image captured by the imaging device is sent, comprising:
acquiring an image by the imaging device capturing an image including an image of a gap between a substrate and an object bonded to the substrate by bumps;
a step of receiving an image captured by the imaging device by the computer;
a step of the computer identifying a first boundary line from the received image using a first trained model generated by machine learning by using training data including an image of a gap between a substrate and an object bonded by a bump, the image being an image from which a first boundary line between the bump and the object is obtained, and identifying a second boundary line from the first boundary line in the received image based on a preset size of the gap between the substrate and the object such that the second boundary line includes a lower end of the bump between the first boundary line and the second boundary line;
A step in which the computer determines whether or not the bump is normal from an image portion of a region between the first boundary line and the second boundary line in the received image by using a second trained model generated by machine learning using teacher data including an image of a normal bump;
Equipped with
The inspection method, wherein the first trained model is trained to detect all bumps with bounding boxes of the same size and shape. The second trained model is generated by machine learning using an image of a normal bump that includes a real image of the bump and a mirror image of the bump projected onto a substrate in the same image;
The inspection method according to claim 11 , wherein the imaging device captures an image to be sent to the computer obliquely downward, and obtains an image including a real image of the bump and a mirror image of the bump projected onto the substrate.

Images