CN118647861A - Vision Inspection Systems for Liquid Pharmaceutical Containers - Google Patents

Abstract

An automated visual inspection (AVI) system may include: at least one profile view imager having an optical axis passing through an inspection object; a proximal polarizing film axially aligned with the optical axis; a liquid crystal device axially aligned with the optical axis; a distal polarizing film axially aligned with the optical axis; and at least one light source oriented to emit illumination toward the distal polarizing film. Alternatively or additionally, an AVI system may include: a profile view imager having an optical axis passing through a sidewall of a container into a container; and a ring light coaxially aligned with a central axis of the container, located below the container and oriented to emit light toward a bottom of the container. The AVI system may also include a bottom imager coaxially aligned with the central axis and oriented to view the bottom of the container.

Description

Vision Inspection Systems for Liquid Pharmaceutical Containers

Technical Field

The present application relates generally to visual inspection systems for inspecting liquid drug containers, and more particularly to techniques for imaging containers/vessels of liquid drug without intentionally agitating the liquid.

Background Art

In some cases (e.g., quality control procedures for manufactured drug products), it is necessary to examine a sample (e.g., a fluid sample) for the presence of various particles (e.g., protein aggregates or debris). Under applicable quality standards, the acceptability of a given sample may depend on criteria such as the number and/or size of undesirable particles contained within the sample. If a sample has unacceptable criteria, it may be rejected and discarded.

Similarly, it is necessary to inspect the associated containers (e.g., vials, cartridges, syringes, vessels, seals, etc.) for various defects (e.g., scratches on vial seals, cracks in containers, etc.). Many times, different inspection systems (e.g., manual or automated visual inspection systems, etc.) are utilized to detect different defects (e.g., presence of particles, presence of particles resting on the bottom of the container, presence of particles floating on the surface of the product within the container, container defects, product defects, etc.).

To handle the volumes typically associated with commercial pharmaceutical production, particle and container inspection tasks are becoming increasingly automated. However, automated inspection systems have struggled to overcome various obstacles to achieve good particle measurement and container fidelity without system complexity. For example, liquid pharmaceuticals are often dispensed in glass vials. Inspecting these glass vials for foreign particles and vial seal crimping defects is one of the most difficult challenges in the associated automated visual inspection (AVI) process. One reason for the difficulty with known AVI systems is that the liquid needs to be stirred in order to reliably detect particles. AVI systems that require stirring are particularly highly dependent on the fluid properties and fill level of the associated liquid.

One known method for particle detection in a vial filled with a liquid, for example, involves rapidly spinning the vial (e.g., 1000-3000 RPM) and capturing a series of images as the vial spins. Due to centrifugal forces, heavy particles may be flung toward the inner surface of the vial sidewall. While the vial is illuminated by a backlight, the dark outline of the particle can be detected from a series of images acquired from an imager. The entire circumference of the vial can be inspected based on a series of images acquired from at least one fixed imager as the vial spins.

As another example, another method for particle detection in a vial filled with a liquid involves rotating the vial and abruptly stopping the rotation of the vial (i.e., a "spin-stop" method). Multiple images of the vial are then captured while the liquid is still in motion. In the spin-stop method, for example, image data associated with subsequent images of the vial can be compared with corresponding image data associated with previous images of the vial to infer particle presence and optionally particle time series trajectories.

These known techniques for particle detection in vials filled with liquids may be good at detecting defects once the associated liquid is purposefully agitated. However, each method is highly dependent on several parameters, such as vial rotation speed, vial rotation deceleration rate, fluid viscosity of the liquid in the vial, product fill level in the vial, fluid surface tension of the liquid in the vial, etc. In addition, false rejections of associated vials may be caused by parameters such as rotation speed, deceleration rate, fluid viscosity, fill level, fluid surface tension, bubbles, surface defects on the glass, liquid droplets formed on the neck area of the vial, light reflected from other imager stations in the associated AVI system, etc.

While agitating the liquid in the vial can improve detection of certain particles, excessive agitation of the liquid can produce agitation events such as the formation of bubbles within the vial, the formation of fluid droplets that appear to be cracks on the neck of the vial, etc. Due at least in part to the time required to optimize the rotation and inspection parameters for new products, known techniques for particle detection in vials filled with liquids are not ideal for high mix-low volume (HMLV) production environments (e.g., clinical operations, small batch products, etc.).

Summary of the invention

Embodiments described herein relate to systems and methods for improving conventional visual inspection techniques for containers of liquid products (e.g., pharmaceutical vessels, vials, vessels, etc.). In particular, systems embodying the invention provide for imaging vessels containing liquids by capturing two-dimensional (2D) images using an automated visual inspection (AVI) system that does not purposefully rely on agitating the liquid within the vessel.

As described herein, the AVI system may include a profile view imager having an optical axis passing through an at least partially translucent inspection object (e.g., a container, vessel, vial, syringe, cartridge, etc.). The inspection object is positioned at a first distance from the profile view imager. The AVI system may also include a near-side polarizing film that is axially aligned with the optical axis, positioned at a second distance from the profile view imager, and oriented perpendicular to the optical axis. The second distance is less than the first distance. The AVI system may further include a liquid crystal device that is axially aligned with the optical axis, positioned at a third distance from the profile view imager, and oriented parallel to the near-side polarizing film. The third distance is greater than the second distance and less than the first distance. The AVI system may further include a far-side polarizing film that is axially aligned with the optical axis, positioned at a fourth distance from the profile view imager, and oriented parallel to the near-side polarizing film and the liquid crystal device. The fourth distance is greater than the first distance. The AVI system may also include a light source that is oriented to emit light toward the far-side polarizing film.

A computer-implemented method for imaging an inspection object may include: emitting illumination from a light source. The method may also include: polarizing the illumination emitted from the light source using a distal polarizing film. The method may further include: transmitting the polarized illumination toward the inspection object, through a liquid crystal device, and through a proximal polarizing film. The method may still further include: capturing an image of a sidewall of the inspection object using a profile view imager, the profile view imager having an optical axis intersecting the sidewall of the inspection object.

Alternatively or additionally, an automatic visual inspection (AVI) system may include a profile view imager having an optical axis that passes through the sidewall of the container and enters the container. The container may be at least partially translucent. The AVI system may also include a ring light that is coaxially aligned with the central axis of the container, is located below the container and is oriented to emit light toward the bottom of the container.

The AVI system may further include a holding device for supporting and/or fixing the container. As described herein, the AVI system may also include a bottom imager that is coaxially aligned with the central axis and oriented to view the bottom of the container. Alternatively or additionally, the AVI system may include an optical axis redirection mechanism that is used to redirect the optical axis of the imager relative to the central axis of the container and/or associated light source.

A computer-implemented method for imaging a container containing a liquid sample may include: illuminating the container with a ring light coaxially aligned with a central axis of the container, located below the container and oriented to emit light toward a bottom of the container. The method may also include: capturing a profile view image with a profile view imager having an optical axis entering the container and intersecting a side wall of the container, the container being at least partially translucent.

Novel methods are provided for inspecting containers (e.g., vials, syringes, cartridges, etc.) for foreign particles or fibers, and/or other defects (e.g., damaged crimps, scratched seals, etc.) for high-mix-low-volume or other manufacturing environments based on captured images.

BRIEF DESCRIPTION OF THE DRAWINGS

The skilled artisan will appreciate that the drawings described herein are included for purposes of illustration and not to limit the present disclosure. The drawings are not necessarily drawn to scale, but rather emphasis is placed on demonstrating the principles of the present disclosure. It will be appreciated that in some cases, various aspects of the described embodiments may be shown as exaggerated or enlarged to facilitate understanding of the described embodiments. In the drawings, similar reference numerals generally refer to components that are similar in function and/or structure throughout the various figures.

1A and 1B depict various views of an example automated visual inspection system having polarizing optical elements on opposite sides of an inspection object and between an imager and a light source.

FIG. 1C depicts different states of a typical liquid crystal device.

2 depicts another example automated vision inspection system having a ring light positioned coaxially with a central axis of a container and oriented to emit light toward a bottom of the container and an imager having an optical axis that enters the container through a sidewall of the container.

3 depicts a further example automated visual inspection system that combines the systems of FIGS. 1A , 1B and 2 together with a bottom imager having an optical axis coaxial with the central axis and oriented to view the bottom of the container.

4 depicts a still further example automated visual inspection system that combines multiple systems of FIGS. 1A and 1B along with the system of FIG. 3 .

5A-5C depict various example container types that may be inspected using a vision inspection system (eg, any of the vision inspection systems of FIGS. 1-4 ).

6 is a simplified block diagram of an example system that can implement the various techniques described herein related to the training and/or use of one or more neural networks for automated visual inspection (AVI).

FIG. 7 depicts an example method of providing an AVI system, which may be similar to the AVI system of FIGS. 1A and 1B or FIG. 2 .

FIG. 8 depicts an example method of providing an AVI system, which may be similar to the AVI system of FIG. 2 , FIG. 3 , or FIG. 4 .

9A and 9B depict bottom views of example containers that may be inspected using the system of FIG. 3 or FIG. 4 .

10A and 10B depict bottom views of another example container that may be inspected using the system of FIG. 3 or FIG. 4 .

11A-14B depict profile views of example containers that may be inspected using any of the systems of FIGS. 1-4 .

15 depicts an example automated visual inspection method for detecting defects in containers using the system of FIGS. 1-4 and 6 .

DETAILED DESCRIPTION

The various concepts introduced above and discussed in greater detail below may be implemented in any of numerous ways, and the described concepts are not limited to any particular implementation.Examples of implementations are provided for illustrative purposes.

The disclosed automated visual inspection (AVI) system reduces the complexity associated with inspecting containers (e.g., vials 505c of FIG. 5C , cartridges 505b of FIG. 5B , syringes 505a of FIG. 5A , etc.) that include liquid products inside the containers. For example, the disclosed AVI system can reduce (if not eliminate) variables such as: container rotation speed, container deceleration rate, fluid viscosity of the product inside the container, product fill level inside the container, fluid surface tension of the product inside the container, bubbles inside the container, surface defects of the container glass (or plastic, etc.), liquid droplets formed on the neck area of the container, light reflected from other imager stations within the associated AVI system, etc. Although embodiments are described herein primarily with reference to AVI systems, it should be understood that various aspects may also be applied in manual visual inspection systems.

Compared to known systems, the AVI system of the present disclosure can accommodate increased throughput speeds of associated inspection processes. Additionally or alternatively, the AVI system can reduce the time required to set up an automated inspection protocol for a new product, making the AVI system particularly useful for high-mix, low-volume production scenarios (e.g., clinical operations, small batches of products, etc.). As described herein for certain embodiments, capturing images of vials or other containers without intentionally agitating the liquid product within the container virtually eliminates the complexity of introducing different fluid properties when optimizing associated inspection protocols.

1A and 1B depict various views of an example automatic visual inspection (AVI) system 100 having polarizing optical elements 115, 125 on opposite sides of an inspection object 105 and between an imager 110 and a light source 130. An "imager" may be a separate camera (e.g., a CCD camera) or include one or more external optical components (e.g., a lens, a mirror, etc.). As used herein, reference to the "optical axis" of an imager refers to the axis of the optical path of the imager in the region where the optical axis passes through the inspected object (e.g., a container). Thus, for example, the use of a mirror may result in the "optical axis" of the imager being orthogonal to the central axis of the container, even if the imager itself faces a direction extending parallel to the central axis. Numerous mirrors may be arranged around the container to combine various views of the container within the resulting field of view of a single imager 110.

The AVI system 100 may include a profile view imager 110 having an optical axis 111 passing through an at least partially translucent inspection object 105. Although FIGS. 1A and 1B show that the inspection object 105 is a vial, the inspection object 105 may instead be a different type of translucent or partially translucent container (e.g., a syringe 505a, a cartridge 505b, etc.) or an object other than a container. The inspection object 105 is positioned at a first distance from the profile view imager 110. The AVI system 100a, b may also include a near-side polarizing film 115 that is axially aligned with the optical axis 111, positioned at a second distance from the profile view imager 110, and oriented perpendicular to the optical axis 111. The second distance is less than the first distance. The AVI system 100a, b may further include a liquid crystal device 120 that is axially aligned with the optical axis 111, positioned at a third distance from the profile view imager 110, and oriented parallel to the near-side polarizing film 115. The third distance is greater than the second distance and less than the first distance. The AVI system 100a, b can further include a far side polarizing film 120, which is axially aligned with the optical axis 111, positioned at a fourth distance from the profile view imager 110 and oriented parallel to the near side polarizing film 115 and the liquid crystal device 120. The fourth distance is greater than the first distance. The AVI system 100a, b can also include a light source 130, which is oriented to emit light toward the far side polarizing film 125. The light source 130 can include at least one backlight, an angled lamp, etc. As used herein, the relative terms "near side" and "far side" represent the spacing relative to the imager (e.g., the profile view imager 110).

As used herein, reference to an object that is "axially aligned" with a particular reference axis means that the object is positioned so that the reference axis intersects or passes through the object. Of particular relevance to AVI systems 100a, b, because the near-side polarizing film 115, the liquid crystal device 120, the inspection object 105, and the far-side polarizing film 125 are axially aligned with the optical axis 111 of the profile view imager 110, the light emitted from the light source 130 passes through the far-side polarizing film 125, the inspection object 105, the liquid crystal device 120, and the near-side polarizing film 115 before being received by the profile view imager 110. In some embodiments, the imager 110 is not a "profile view" imager. For example, elements 110, 115, and 120 can be positioned below a well containing a sample, and elements 125 and 130 can be positioned above the well (or vice versa).

However, when using the arrangement shown in FIGS. 1A and 1B , the AVI system 100a,b may be particularly useful for particle inspection in vials or other containers. Although the profile view imager 110 is shown as being oriented horizontally, the imager 110 may instead be tilted upward or downward so that the optical axis 111 of the profile view imager 110 is not perpendicular to the central axis 106 of the imaged container 105. For example, multiple imagers similar to the profile view imager 110 may be oriented at different "elevation angles" so that the associated optical axes 111 are pointed slightly upward or slightly downward relative to the optical axis 111 shown in FIGS. 1A and 1B . This may be particularly useful, for example, for generating a composite three-dimensional image of a container 105 and its contents from multiple two-dimensional images.

As shown in Figures 14A and 14B (relative to Figures 13A and 13B), detection of fibers 1409a, b can benefit from polarizing films 115 and 125. When light source 130 is powered and liquid crystal device 120 is not powered, imager 110 can acquire images 1400a, b of Figures 14A and 14B. When both light source 130 and liquid crystal device 120 are powered, imager 110 can acquire images 1300a, b of Figures 13A and 13B.

FIG1B shows a modification of the liquid crystal device 100c by removing the polarizing filter 125c on the incoming side and placing the distal polarizing film 125 in front of the light source 130 so that the vial 105 is located between the distal polarizing film 125 and the liquid crystal device 120, which allows the polarization effect to be switched on or off (by powering off or on the liquid crystal device 100c, respectively), thereby allowing both filtered and unfiltered images to be captured. Thus, the AVI system 100a,b can switch polarization on/off electronically without mechanical parts.

With the polarizing filter in place, other types of inspections (e.g., inspections for defects on curled edges or cracks in container glass) may be negatively impacted. Therefore, the liquid crystal device 120 can quickly switch the polarizing filter on or off so that the associated inspections can be performed at high speed using a minimum number of imagers (i.e., one image can be acquired with the liquid crystal device 120 powered on, and another image can be acquired with the liquid crystal device 120 powered off).

Fig. 1C shows a polarization device 100c constructed in a classical manner, and shows a functional diagram. The device 100c has two polarization films 115c, 125c on each side of the liquid crystal cell 120c, and the far-side polarization film 125c is 90° out of phase with the near-side polarization film 115c. The charge 156c causes the liquid crystal to be arranged and maintain the same polarization as the light entering the device 100c. When not powered, the crystal rotates the light from the far-side polarization film 125c and is in phase with the near-side polarization film 115c. In other words, when the liquid crystal device 100c is not powered, the device rotates the light by 90 degrees. However, when the liquid crystal device 100c is powered, the liquid crystal is arranged and does not rotate the light. The typical liquid crystal device 100c can be used as an "electronic shutter" because the device 100c includes polarization films 115c, 125c on both sides of the cell 120c. This allows light to pass when not powered and be blocked when powered. It is noteworthy that the AVI systems 100 1A and 1B may represent embodiments of the liquid crystal device 120c in which the filter 125c is removed and repositioned with the distal polarizing film 125 as illustrated in Figures 1A and 1B. Although the liquid crystal device 100c is illustrated as a twisted nematic device, the device 100c may include any suitable cell 120c (e.g., a smectic cell, a cholesteric cell, etc.).

The AVI system 100a,b is particularly useful for applications where polarized light improves detection of certain types of defects, such as fibers (e.g., fibers 1409a,b of FIGS. 14A and 14B). On the other hand, other types of defects (e.g., defects in a seal bead or cracks in a vial glass, etc.) may have less contrast relative to background noise (e.g., bubbles, droplets, etc.) when the liquid crystal device 120 is powered off. The liquid crystal device 120 may be used to switch the polarization angle between light emitted from the light source and light entering the imager 110 between zero and 90 degrees of polarization.

FIG. 2 depicts another example AVI system 200 in which the central axis 241 of the ring light 240 is coaxially aligned with the central axis 206 of the container 205 and is oriented to emit light toward the bottom of the container 205. Although FIG. 2 (and FIG. 3 and FIG. 4 ) shows that the container 205 is a vial, the container 205 may instead be a different type of translucent or partially translucent container (e.g., a syringe 505a, a cartridge 505b, a vial 505c, etc.). The profile view imager 210 has an optical axis 211 that passes through the sidewall 212 of the container 205 and enters the container 205. The AVI system 200 may further include a holding device (not shown in FIG. 2 ) for supporting and/or fixing the container 205. Possible holding devices are discussed in further detail below.

As used herein, reference to an object being "coaxially aligned" with a particular reference axis means that the object is positioned so that the axis of the object (e.g., its central axis 206) is substantially aligned with (substantially the same as) the reference axis. Of particular relevance in the AVI system 200, with the central axis 241 of the ring light 240 coaxially aligned with the central axis 206 of the container 205, the light emitted from the ring light 240 can be projected uniformly across the bottom of the container 205 and around the perimeter of the container.

The AVI system 200 is particularly well suited for detecting scratch defects on the crimp of a vial. In fact, the ring light 240 (typically set for the bottom imager (e.g., bottom imager 335 of FIG. 3 )) is particularly useful for inspecting the crimp when the image is acquired from the profile view imager 210 with the ring light 240 powered (e.g., images 1200a, b of FIG. 12A and FIG. 12B ). It is worth noting that the scratch defect 1109a, b is one of the most difficult defects to detect on the crimp 1208a, b when using a conventional AVI setting. For example, a conventional visual inspection system may cause a false rejection of a container due to a shadow difference in the container seal (i.e., the seal shadow difference may appear as a scratch defect to a conventional AVI system). The AVI system 200 can reduce such false rejections.

Some of the same advantages of hem detection (e.g., inspection speed, defect clarity, etc.) also apply to particle inspection. For example, an AVI system that does not purposefully stir the liquid in the container can simplify the AVI system settings for new container types and/or new products. Additionally, an AVI system that does not purposefully stir the liquid in the container does not rely on particle movement for particle detection. This is particularly advantageous when inspecting containers from multiple angles and through the bottom of the container (e.g., AVI system 300 of FIG. 3 ) on the side profile. Other benefits of AVI system 200 (e.g., smaller variances of light and shadows) can improve the identification of fixed particles.

3 depicts a further example AVI system 300 that combines the systems of FIG1A and FIG2 and further adds a bottom imager 335 having an optical axis 336 coaxially aligned with the central axis of the container 305 and oriented to view the bottom of the container 305. Without agitation, most particles tend to settle on the bottom of the vial and show up with good contrast in the image acquired from the bottom imager 335. The profile view imager 310 can be used to inspect for fibers and floating particles, as well as cracks, other defects, and curled areas in the glass.

AVI system 300 may be faster than inspection based on rotation, because it is not necessary to rotate vial 305 (that is, it is not necessary to ramp up, take a picture, then ramp down), which may be a bottleneck in the AVI process.AVI system 300 can alleviate bottleneck problems, and can allow closer to real-time AVI.AVI system 300 may also cause setting/programming faster, because it is not necessary to experiment to determine which agitation speeds are too fast for different types of fluids/containers.The accuracy of AVI system 300 can be comparable to the method comprising a rotation-based technology.For example, AVI system 300 can detect glass and metal particles and fibers in the vial containing liquid products.

The AVI system 300 may include a holding device 345 (e.g., a glass plate, a carousel, a star wheel, or a robotic arm that can slowly rotate the container, etc.) for supporting and/or fixing the container 305. The holding device 345 may also act as an optical axis redirection mechanism, which is described in more detail herein. Two imagers 310, 335 in conjunction with different lighting arrangements (e.g., backlight 330 and ring light 340) can perform most of the required inspections of the automatic visual inspection system 300. Fewer imagers and the elimination of the need for agitation and fluid movement help reduce the setup and characterization time for new products, which is usually a requirement for HMLV operations. Object detection using such an arrangement of the AVI system 300 was found to successfully detect all particles and curling defects. The results show that the detection rate is higher than that of manual inspection, with a detection rate of 94% for 300um metal particles, 1000% for 1000um metals, 85% for glass particles, and 92% for fibers, all of which have no false rejections (i.e., good samples are classified as defective). By comparison, conventional AVI equipment may need to juggle more than 10 different imagers to perform an inspection.

FIG. 4 depicts yet a further example AVI system 400, which is similar to AVI system 300, but uses additional profile view imagers. In AVI system 200 or AVI system 300, some inspection processes (e.g., inspection of vial curling scratches) do require the vial to be rotated slowly so that images of the periphery around the container can be acquired from all profile viewing angles. This can significantly slow down the inspection process. However, placing five profile view imagers 410 as in FIG. 4 allows inspection of the entire container 405 without any requirement for rotating the container 405. Therefore, a series of images can be acquired by the multiple profile view imagers 410, each of which has a different optical axis relative to the container.

FIG. 4 shows a specific embodiment in which the need to rotate the container 405 is alleviated by placing five imagers 410 around the perimeter of the container 405. Five images taken around the perimeter of the container 405, or every 72 degrees, are sufficient to fully inspect the container 405 for particles and cracks or nicks on the container sidewall 412. However, other embodiments may include more (e.g., six) or fewer (e.g., four) profile view imagers 410. A number of mirrors may be arranged around the container 405 to combine various views of the container 405 within the resulting field of view of a single imager 410.

As seen in FIG. 4 , the AVI system 400 may also include a bottom imager 435 that is coaxially aligned with the central axis of the container 405 and the ring light 440 and oriented to view the bottom of the container 405 .

The system 400 may include an optical axis redirection mechanism (a plurality of imagers 410, each having a uniquely oriented optical axis) to change the orientation of the optical axis relative to the sidewall 412. The optical axis redirection mechanism (a plurality of imagers 410, each having a uniquely oriented optical axis) may include a container rotator. Alternatively or additionally, the optical axis redirection mechanism may include a plurality of profile view imagers 410, each having a respective optical axis passing through the sidewall of the container around the perimeter of the container 405.

FIG. 5A to FIG. 5C depict various example container types that can be used as samples for imaging by the visual inspection systems 100a, b of FIG. 1A and FIG. 1B , the visual inspection system 200 of FIG. 2 , the visual inspection system 300 of FIG. 3 , or the visual inspection system 400 of FIG. 4 . Referring first to FIG. 5A , an example syringe 505a includes a hollow barrel 502, a flange 504, a plunger 506 providing a movable fluid seal inside the barrel 502, and a needle shield 508 for covering a syringe needle (not shown in FIG. 5A ). For example, the barrel 502 and the flange 504 can be formed of glass and/or plastic, and the plunger 506 can be formed of rubber and/or plastic. The needle shield 508 is separated from the shoulder 510 of the syringe 505a by a gap 512. The syringe 505a contains a liquid (e.g., a drug product) 514 within the barrel 502 and above the plunger 506. Typically, the top of the liquid 514 forms a meniscus 516 , above which is an air gap 518 .

Next, referring to FIG. 5B , an example cartridge 505b includes a hollow barrel 522, a flange 524, a piston 526 providing a movable fluid seal within the interior of the barrel 522, and a Luer lock 528. For example, the barrel 522, the flange 524, and/or the Luer lock 528 can be formed of glass and/or plastic, and the piston 526 can be formed of rubber and/or plastic. The cartridge 505b contains a liquid (e.g., a drug product) 530 within the barrel 522 and above the piston 526. Typically, the top of the liquid 530 forms a meniscus 532, above which is an air gap 534.

Referring next to FIG. 5C , an example vial 505c includes a hollow body 542 and a neck 544, the transition between which forms a shoulder 546. At the bottom of the vial 505c, the body 542 transitions to a heel 548. A crimp 550 includes a stopper (not visible in FIG. 5C ) that provides a fluid seal at the top of the vial 505c, and a flip cap 552 covers the crimp 550. For example, the body 542, the neck 544, the shoulder 546, and the heel 548 can be formed of glass and/or plastic, the crimp 550 can be formed of metal, and the flip cap 552 can be formed of plastic. The vial 505c can include a liquid (e.g., a drug product) 554 within the body 542. Typically, the top of the liquid 554 forms a meniscus 556 (e.g., a very slightly curved meniscus if the body 542 has a relatively large diameter), above which is an air gap 558. In other embodiments, liquid 554 is instead a solid material within vial 505c.

6 is a simplified block diagram of an example system 600 that can implement various techniques related to training (and possibly validating and/or certifying) and/or using one or more neural networks or other machine learning (ML) systems. System 600 can also be used to test/certify non-ML AVI systems. In addition to or as an alternative to ML systems, system 600 can include a "computer vision" algorithm that does not use ML but instead uses fixed rules (e.g., empty vial, low fill, high fill, etc.).

FIG6 illustrates an embodiment in which a system 600 implements one or more neural networks. Once trained and identified, the neural network(s) may be used in production to detect defects associated with containers and/or the contents of those containers (e.g., the defects shown in FIGS. 9A to 14B ). In a pharmaceutical context, for example, the neural network(s) may be used to detect defects associated with syringes, cartridges, vials, or other container types (e.g., abraded bead/seals of the container, cracks, scratches, stains, missing parts, etc.), and/or to detect defects associated with liquid or lyophilized drug products within the container (e.g., the presence of fibers, metal particles, and/or other foreign particles, color changes in the product, etc.). As used herein, "defect detection" may refer to classifying a container image as exhibiting or not exhibiting a defect (or a particular defect class), and/or may refer to detecting a particular object or feature (e.g., a particle or crack) associated with whether a container and/or its contents should be considered defective, depending on the embodiment.

System 600 includes a visual inspection system (VIS) 602 communicatively coupled to a computer system 604. VIS 602 includes hardware (e.g., a transport mechanism, (multiple) light sources, (multiple) imagers, etc.), and firmware and/or software configured to capture a digital image of a sample (e.g., a container holding a fluid or a lyophilized material). VIS 602 may include any of the AVI systems 100a, b, 200, 300, 400 described herein with reference to, for example, FIGS. 1 to 4 , respectively, or may be some other suitable system.

For ease of explanation, the system 600 is described herein as using container images from the VIS 602 to train and validate one or more AVI neural networks, and then using the trained/validated neural network(s) to perform AVI/defect detection. However, it should be understood that this is not necessarily the case. For example, instead of or in addition to the VIS 602, the system 600 may use container images generated by several different visual inspection systems to perform training and/or validation. Furthermore, the training/validation may be performed by another system, and the system 600 may then use the trained neural network(s) (e.g., during commercial production). In some embodiments, some or all of the container images used for training and/or validation are generated using one or more offline (e.g., lab-based) "simulation stations" that closely replicate important aspects of commercial production line equipment stations (e.g., optics, lighting, etc.), thereby expanding the training and/or validation library without incurring excessive downtime of the commercial production line equipment.

The VIS 602 can image each of a number of containers in turn. To this end, the VIS 602 can include or operate in conjunction with a holding device (e.g., a Cartesian robot, a carousel, a star wheel, and/or any other holding device) that can move each container in turn into an appropriate position for imaging and then remove the container once imaging of the container is complete. Although not shown in FIG. 6 , the VIS 602 can include a communication interface and a processor to enable communication with the computer system 604. In other embodiments (e.g., a laboratory-based setting), the VIS 602 includes a simpler holding device (e.g., a stage with a hole covered by a glass plate).

Computer system 604 can be generally configured to control/automate the operation of VIS 602, and receive and process images captured/generated by VIS 602, as further discussed below. Computer system 604 can be a general-purpose computer specially programmed to perform the operations discussed herein, or can be a special-purpose computing device. As seen in FIG. 6 , computer system 604 includes user interface 606, processing unit 610, and memory unit 614. However, in some embodiments, computer system 604 includes two or more computers that are co-located or remote from each other. In these distributed embodiments, the operations described herein related to processing unit 610 and memory unit 614 can be divided among multiple processing units and/or memory units, respectively.

The processing unit 610 includes one or more processors, each of which may be a programmable microprocessor that executes software instructions stored in the memory unit 614 to perform some or all of the functions of the computer system 604 as described herein. For example, the processing unit 610 may include one or more graphics processing units (GPUs) and/or one or more central processing units (CPUs). Alternatively or in addition, some of the processors in the processing unit 610 may be other types of processors (e.g., application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc.), and some of the functions of the computer system 604 as described herein may instead be implemented in hardware.

The memory unit 614 may include one or more volatile and/or non-volatile memories. The memory unit 614 may include any suitable one or more memory types, such as read-only memory (ROM), random access memory (RAM), flash memory, solid-state drive (SSD), hard disk drive (HDD), etc. In general, the memory unit 614 may store one or more software applications, data received/used by those applications, and data output/generated by those applications.

The memory unit 614 stores software instructions for various modules that, when executed by the processing unit 610, perform various functions to achieve the purpose of training, validating, and/or authenticating one or more AVI neural networks. Specifically, in the example embodiment of FIG. 6 , the memory unit 614 includes an AVI neural network module 616 and a visual inspection system (VIS) control module 620. In other embodiments, the memory unit 614 may omit one or more of the modules 616, 620, and/or include one or more additional modules. Additionally or alternatively, one, some, or all of the modules 616, 620 may be implemented by a different computer system (e.g., a remote server coupled to the computer system 604 via one or more wired and/or wireless communication networks). Furthermore, the functionality of any of the modules 616 and 620 may be divided among different software applications and/or computer systems. As just one example, in an embodiment where the computer system 604 accesses a network service to train and use one or more AVI neural networks, the software instructions for the AVI neural network module 616 may be stored at the remote server.

The AVI neural network module 616 includes software for training one or more AVI neural networks using images stored in the image library 640. The image library 640 may be stored in the memory unit 614, or in another local or remote memory (e.g., memory coupled to a remote library server, etc.). In addition to training, the module 616 may also implement/run the trained AVI neural network(s), such as by applying newly acquired images from the VIS 602 (or another visual inspection system) to the neural network(s), possibly after performing some pre-processing on the images as discussed below. In various embodiments, the AVI neural network(s) trained and/or run by the module 616 may classify the entire image (e.g., defect vs. no defect, or the presence or absence of a particular type of defect (e.g., hem scratches or hem defects in general), etc.), detect objects in the image (e.g., detect the location of foreign objects that are not bubbles within a container image), or some combination thereof (e.g., one neural network classifies the image and another neural network performs object detection). As used herein, unless the context clearly indicates a more specific use, "object detection" broadly refers to techniques for identifying specific locations of objects within an image (e.g., particles, fibers, etc.) and/or identifying specific locations of features of larger objects (e.g., a scratched bead or seal, cracks or notches on the barrel of a syringe or cartridge, etc.), and may include, for example, techniques for performing segmentation (e.g., pixel-by-pixel classification) on container images or image portions, or techniques for identifying objects and placing bounding boxes (or other bounding shapes) around those objects.

In embodiments where the AVI neural network(s) detect container defects, the defects may be associated with any suitable container feature(s). Referring to the example containers of FIGS. 5A-5C , for example, a particular AVI neural network implemented by the AVI neural network module 616 may detect whether the barrel 502, barrel 522, or bottle 542 has cracks or stains, whether the flange 504 or 524 is misshapen, whether the needle shield 508 is not properly positioned, whether the plunger 506 or piston 526 has any defects, whether the Luer lock 528 has any defects, whether the crimp 550 is properly positioned and/or has any defects (e.g., scratches), whether the flip cap 552 is properly positioned and/or has any defects, and the like.

Module 616 can run the trained AVI neural network(s) for the purpose of validation, certification, and/or inspection during commercial production. In one embodiment, for example, module 616 is used only to train and validate the AVI neural network(s), and then transport the trained neural network(s) to another computer system for certification and inspection during commercial production (e.g., using another module similar to module 616). In some embodiments where AVI neural network module 616 trains/runs multiple neural networks, module 616 includes separate software for each neural network.

As described above with respect to FIG. 3 and FIG. 4, the ring lamps 340 and 440 for the bottom imagers 345 and 435 are particularly useful for inspecting the vial seal curling defects (e.g., the vial seal curling defects 1209 of FIG. 12A and FIG. 12B). For example, a total of 100 images per container can be captured on 13 containers, resulting in a total of 1,300 images. After enhancing the associated training images by adjusting brightness, vertical mirroring, adding noise and tilting images, and tilting bounding boxes, AVI neural network training can be performed on images from, for example, six vials (i.e., the training set can be multiplied by five times). In general, deep learning can be used to detect defects in images. Using (multiple) previously trained AVI neural networks further reduces the time required to set up automatic inspection schemes for new products. The AVI neural network of the present disclosure can be implemented for high-mix, low-volume production scenarios (e.g., clinical operations or small batch products), and then modern deep learning techniques (e.g., AVI neural network module 616 of FIG. 6) are used.

In some embodiments, the VIS control module 620 controls/automates the operation of the VIS 602 so that a container image can be generated with little or no human interaction. The VIS control module 620 can cause a given imager to capture a container image by sending a command or other electronic signal (e.g., generating a pulse on a control line, etc.) to the imager. The VIS 602 can send the captured container image to the computer system 604, which can store the image in the memory unit 614 for local processing. In alternative embodiments, the VIS 602 can be locally controlled, in which case the VIS control module 620 can have fewer functions than described herein (e.g., only handle image retrieval from the VIS 602), or can be omitted entirely from the memory unit 614.

FIG. 7 is an example method 700 for operating an AVI system. The AVI system may be similar to, for example, the AVI systems 100a, b of FIGS. 1A and 1B . The method may include providing a profile view imager 110 having an optical axis 111 passing through an at least partially translucent inspection object (e.g., vial 105) that is positioned at a first distance from the profile view imager (block 702). The field of view of the profile view imager 110 may be configured to acquire a desired image of the inspection object (e.g., image 1100a, image 1200a, image 1300a, image 1400a, etc.), for example including the entire inspection object or only a portion thereof. The method 700 may also include providing a near-side polarizing film 115 that is axially aligned with the optical axis 111, positioned at a second distance from the profile view imager 110 and oriented perpendicular to the optical axis 111, the second distance being less than the first distance (block 704). The method 700 may further include providing a liquid crystal device 120 axially aligned with the optical axis 111, positioned at a third distance from the profile view imager 110, and oriented parallel to the near-side polarizing film 115, the third distance being greater than the second distance and less than the first distance (block 708). The method 700 may still further include providing a far-side polarizing film 125 axially aligned with the optical axis 111, positioned at a fourth distance from the profile view imager 110, and oriented parallel to the near-side polarizing film 115 and the liquid crystal device 120, the fourth distance being greater than the first distance (block 710). The method 700 may include providing a light source 130 oriented to emit illumination toward the far-side polarizing film (block 712).

FIG8 is an example method 800 of operating an AVI system 800. The AVI system may be similar to, for example, the AVI system 200 of FIG2. The method 800 may include providing a profile view imager 210 having an optical axis 211 that enters a container (e.g., vial 205) through a sidewall of the container 205, the container being at least partially translucent (block 802). The field of view of the profile view imager 210 may be configured to acquire a desired image (e.g., image 1100a, image 1200a, image 1300a, image 1400a, etc.) of an inspection object, for example, including the entire inspection object or only a portion thereof. The method 800 may also include providing a ring light 240 that is coaxially aligned with the central axis 206 of the container 205, is located below the container, and is oriented to emit light toward the bottom of the container (block 804). The method 800 may further include providing a holding device 245 for supporting and/or fixing the container (block 806), as described elsewhere herein.

9A and 9B depict images 900a, b (the latter being a magnified view) of a bottom view of an example container 906 that may be inspected using the system of FIG. 3 (or the system of FIG. 4 plus a bottom imager oriented in a manner similar to imagers 335, 445, etc.). Bottom images 900a, b depict 1000 μm metal particles 907 imaged through the bottom of the container (here, a vial).

10A and 10B depict images 1000a, b (the latter being a magnified view) of a bottom view of another example container 1006 that may be inspected using the system of FIG. 3 (or the system of FIG. 4 plus a bottom imager oriented in a manner similar to imager 335, etc.). Bottom images 1006a, b depict 300 μm metal particles 1007 imaged through the bottom of the container (vial).

11A and 11B depict profile view images 1100a, b (the latter being a magnified view) of an example container 1108 that can be inspected using any of the systems of FIGS. 1 to 4. Profile view images 1100a, b depict fibers 1109 imaged through the sidewall of the container (vial). By using a polarizing film (e.g., polarizing films 115, 125 arranged as in FIG. 1A), the contrast of the fibers is improved.

12A and 12B depict profile view images 1200a, b (the latter being a magnified view) of another example container 1208 that may be inspected using any of the systems of FIGS. 1 to 4. The profile view images 1108a, b depict a scratched curled edge 1209, wherein the box in FIG. 11B represents the output/result of object detection performed (e.g., by the AVI neural network module 516) on the images 1200a, b from the profile view imager 110, and the liquid crystal device 120 is cut off.

13A and 13B depict images 1300a, b (the latter being a magnified view) of a profile view of a further example container 1308b that may be inspected using any of the systems of FIGS. 1-4 without polarization effects (e.g., with the liquid crystal device 120 turned on). When an image includes a fluid droplet formed on the neck of a vial (e.g., the image of FIG. 13A ), the associated AVI system may mistake the edge of the fluid droplet for, for example, a crack in the associated container.

FIG14A and FIG14B depict images 1400a, b (the latter being a magnified view) of the same profile view of the same example container 1408a, b as in FIG13A and FIG13B, which can be inspected using any of the systems of FIG1 to FIG4. Using a polarizing filter, the contrast of fiber 1409 relative to its surroundings is higher than the contrast of fiber 1309 relative to its surroundings. The neural network or other image processing of the AVI system is more likely to detect fiber 1409 using images 1400a, b of FIG14A and FIG14B than to detect fiber 1309 using images 1300a, b of FIG13A and FIG13B.

FIG. 15 depicts an example automated visual inspection method 1500 for detecting defects (e.g., particles 907, particles 1007, scratched seals 1109, fibers 1209, fibers 1309, etc.) in a container (e.g., syringe 505a, cartridge 505b, vial 505c, etc.). At least portions of method 1500 may be implemented using, for example, any of the systems of FIGS. 1 to 4 and 6. Method 1500 may include illuminating the container with a ring light (e.g., element 240, 340, or 440) located below the container, coaxially aligned with a central axis of the container, and oriented to emit light toward the bottom of the container (block 1502). Method 1500 may also include capturing one or more profile view images of a profile of the container using a profile view imager (e.g., imager 110, 210, 310, or 410) having an optical axis that passes through a sidewall of the container and enters the container (block 1504). The field of view of the profile view imager may be configured to acquire a desired image of the inspection object (eg, image 1100a, image 1200a, image 1300a, image 1400a, etc.).

The method 1500 may further include capturing one or more bottom images of the bottom of the container using a bottom imager (e.g., imager 335 or 435) coaxially aligned with the central axis of the container (block 1508). The field of view of the bottom imager may be configured to acquire a desired image of the container (e.g., image 900a, image 1000a, etc.).

The method 1500 may also include: using one or more processors (e.g., the processing unit 610 of FIG. 6 when executing the AVI neural network module 616) to analyze one or more profile view images (e.g., image 1100a, image 1200a, image 1300a, image 1400a, etc.) and/or one or more bottom images (e.g., image 900a, image 1000a, etc.) of the container to detect defects (block 1510). The processor(s) may implement one or more machine learning models (e.g., classification and/or object detection models) to detect defects. For example, the processor(s) may implement a classification model to classify the container as "acceptable" or "rejected". Additionally or alternatively, the processor(s) may implement multiple machine learning models to classify specific types of defects (e.g., a first machine learning model to classify the defect as a fiber within the container, a second machine learning model to classify the defect as a non-fiber particle within the container, a third machine learning model to classify the defect as a curling scratch, etc.).

Although the systems, methods, devices, and components thereof have been described in terms of exemplary embodiments, they are not limited thereto. The detailed description is to be construed as exemplary only and does not describe every possible embodiment of the invention, as it would be impractical, if possible, to describe every possible embodiment. Many alternative embodiments may be implemented using current technology or technology developed after the filing date of this patent application, and these embodiments still fall within the scope of the claims defining the invention.

Those skilled in the art will appreciate that various modifications, changes and combinations may be made to the embodiments described above without departing from the scope of the present invention, and such modifications, changes and combinations will be deemed to be within the scope of the inventive concept.

Claims (31)

1. An automated vision inspection system, comprising:

A contour view imager having an optical axis passing through an at least partially translucent inspection object positioned at a first distance from the contour view imager;

A proximal polarizing film axially aligned with the optical axis, positioned at a second distance from the profile view imager and oriented perpendicular to the optical axis, the second distance being less than the first distance;

A liquid crystal device axially aligned with the optical axis, positioned at a third distance from the profile view imager and oriented parallel to the proximal polarizing film, the third distance being greater than the second distance and less than the first distance;

a distal polarizing film axially aligned with the optical axis, positioned a fourth distance from the profile view imager, the fourth distance being greater than the first distance, and oriented parallel to the proximal polarizing film and the liquid crystal device; and

A light source oriented to emit illumination toward the distal polarizing film.

2. The system of claim 1, wherein the inspection object is a container.

3. The system of claim 2, wherein the container is selected from the group consisting of: vials, syringes or cartridges.

4. A system as claimed in any one of claims 1 to 3, further comprising:

An annular lamp coaxially aligned with the central axis of the vessel, below the vessel and oriented to emit light toward the bottom of the vessel.

5. The system of any one of claims 1 to 4, further comprising:

a bottom imager coaxially aligned with the central axis and oriented to view the bottom of the container.

6. The system of any one of claims 1 to 5, further comprising:

At least one of the following: the container rotation mechanism or one or more additional contour view imagers oriented parallel to the respective optical axis to view at least a portion of the respective contour of the inspection object.

7. The system of claim 6, wherein the one or more additional contour view imagers consist of four contour view imagers.

8. A method for imaging an at least partially translucent inspection object, the method comprising:

emitting illumination from a light source;

polarizing illumination emitted from the light source using a distal polarizing film;

passing the polarized illumination through at least a portion of the inspection object, then through a liquid crystal device, and then through a proximal polarizing film; and

One or more images of the inspection object are captured with a contour view imager having an optical axis intersecting a sidewall of the inspection object.

9. The method of claim 8, wherein the inspection object is a container.

10. The method of claim 9, wherein the container is selected from the group consisting of: vials, syringes or cartridges.

11. The method of any of claims 7 to 10, further comprising:

the one or more images of the container are analyzed by one or more processors to detect at least one defect associated with the container and/or the contents of the container.

12. The method of claim 11, wherein the at least one defect comprises particles or fibers within the container.

13. An automated vision inspection system, comprising:

A contour view imager having an optical axis passing through a sidewall of the container into the container, the container being at least partially translucent;

An annular lamp coaxially aligned with the central axis of the container, below the container and oriented to emit light toward the bottom of the container; and

Retaining means for supporting and/or securing the container.

14. The system of claim 13, further comprising:

At least one of the following: a container rotator or one or more additional profile view imagers oriented parallel to the respective optical axis to view at least a portion of the respective profile of the container.

15. The system of claim 14, wherein the one or more additional contour view imagers consist of four contour view imagers.

16. The system of any of claims 13 to 15, further comprising:

a bottom imager coaxially aligned with the central axis and oriented to view the bottom of the container.

17. The system of any of claims 13 to 16, further comprising:

A proximal polarizing film axially aligned with the optical axis, positioned at a second distance from the profile view imager and oriented perpendicular to the optical axis, the second distance being less than the first distance;

A liquid crystal device axially aligned with the optical axis, positioned at a third distance from the profile view imager and oriented parallel to the proximal polarizing film, the third distance being greater than the second distance and less than the first distance;

a distal polarizing film axially aligned with the optical axis, positioned a fourth distance from the profile view imager, the fourth distance being greater than the first distance, and oriented parallel to the proximal polarizing film and the liquid crystal device; and

A light source oriented to emit illumination toward the distal polarizing film.

18. The system of any of claims 13 to 17, further comprising:

A container rotator.

19. A method for imaging a container that is at least partially translucent and holds a liquid sample, the method comprising:

Illuminating the container with an annular lamp coaxially aligned with a central axis of the container, positioned below the container and oriented to emit light toward a bottom of the container;

capturing one or more contour view images with a contour view imager having an optical axis passing through a sidewall of the container into the container; and

One or more bottom images are captured with a bottom imager coaxially aligned with the central axis and oriented to view the bottom of the container.

20. The method of claim 19, further comprising:

The one or more profile view images of the container are analyzed by one or more processors to detect at least one defect associated with the container and/or the contents of the container.

21. The method of claim 20, wherein the at least one defect comprises particles or fibers within the container.

22. The method of claim 20, wherein the at least one defect comprises an scratched container seal.

23. The method of any of claims 19 to 22, further comprising:

The one or more bottom images of the container are analyzed by one or more processors to detect at least one defect associated with the container and/or the contents of the container.

24. The method of claim 23, wherein the at least one defect comprises particles or fibers within the container.

25. The method of any of claims 19 to 24, further comprising:

The one or more contour view images of the container are analyzed by one or more processors to classify at least one defect associated with the container and/or the contents of the container.

26. The method of claim 25, wherein the at least one defect associated with the container and/or the contents of the container is classified as one of: particles in the container, fibers in the container, or scratched container seals.

27. The method of any one of claims 19 to 26, further comprising:

The one or more bottom images of the container are analyzed by one or more processors to classify at least one defect associated with the container and/or the contents of the container.

28. The method of claim 27, wherein the at least one defect associated with the container and/or the contents of the container is classified as a particle within the container or a fiber within the container.

29. The method of any one of claims 19 to 28, further comprising:

the one or more contour view images of the container are analyzed by one or more processors to classify the container as either acceptable or rejected.

30. The method of any of claims 19 to 29, further comprising:

The one or more bottom images of the container are analyzed by one or more processors to classify the container as either acceptable or rejected.

31. The method of any one of claims 19 to 30, wherein the container is a vial.