CN110288528B - An image stitching system and method for micro-nano visual observation - Google Patents

Abstract

The invention discloses an image stitching system and method for micro-nano-level visual observation. The system comprises: a micro-nano motion platform, which is used for fixing an observation object, and is arranged under a microscope objective lens; the micro-nano motion platform is further provided with There are one or more grating sensors; an image acquisition device, which is arranged on the microscope eyepiece, is used to capture the image of the observation object; a computing device is connected in communication with the micro-nano motion platform and the image acquisition device, and controls the movement of the micro-nano motion platform, and each time After moving, the image of the observation object is collected by the image acquisition device, and the images of the observation object obtained in two adjacent times are partially overlapped, and the position information of the micro-nano motion platform transmitted by the grating sensor is obtained at the same time; The location information of each frame is registered and stitched. The invention uses the micro-nano motion platform as the carrier of the observation object, records the motion information of the image through the grating sensor, and can meet the precision requirements of the micro-nano image registration.

Description

An image stitching system and method for micro-nano visual observation

technical field

The invention belongs to the technical field of image processing, and in particular relates to an image stitching system and method for micro-nano visual observation.

Background technique

The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

The core technology of image stitching technology is image registration technology. As early as 1992, Lisa Gottesfeld Brown of Cambridge University has summarized the basic theory of image registration technology in various fields and their main methods. These fields include medical image analysis, remote sensing Data processing as well as computer vision, pattern recognition, etc.

In 1996, Richard Szeliski (Microsoft Corporation) proposed a motion-based panorama image stitching model, using the Levenberg-Marquardt iterative nonlinear minimization method (L-M algorithm for short), by finding the geometric exchange relationship between images for image registration. , because this method has good effect, fast convergence speed, and can process images to be stitched with translation, rotation, affine and other transformations, it has become a classic algorithm in the field of image stitching, and Richard Szeliski has since become the field of image stitching. The founder of his theory has become a classic theoretical system, and many people are still studying his splicing theory today. In 2000, Shmuel Peleg (memberIEEE), Benny Rousso, Alex Rav-Acha and Assaf Zomet made further improvements on the basis of Richard Szeliski and proposed an adaptive image collage model, which is based on the different movements of the camera. Adapt to the selection stitching model, and complete the image stitching by dividing the image into narrow strips for multiple projections. This research result has undoubtedly promoted the further development of image stitching technology, and the self-adaptive problem has since become a new hot spot in the field of image stitching. At the same time, the development of other registration algorithms has also been applied to image stitching techniques. In addition to the classical algorithms mentioned above, there are two main methods. One is the phase correlation method, and the other is the image registration method based on geometric features.

The phase correlation method was first proposed by Kuglin and Hines in 1975, which is scene independent and can accurately align purely two-dimensional translation images. Later, De Castro and Morandi discovered that Fourier exchange is used to determine rotational alignment, just as translation alignment is determined; in 1996, Reddy and Chaueqi improved TDe Castro's algorithm, greatly reducing the number of required transformations. The translation vector of the two images can be directly calculated from the }fl bits of their Cross Power Spectrum. The application of Fourier transform for image registration is another research achievement in the field of image stitching. With the proposal of fast Fourier transform algorithm and the mature application of Fourier transform in the field of signal processing, image stitching technology has also been developed accordingly.

Image registration method based on geometric features is another research hotspot of image stitching technology. In 1994, Blaszka obtained some low-level feature models, such as edge models, corner models and vertex models, through two-dimensional Gaussian blur filtering. Based on this, more and more people began to study the method of image stitching based on the low-level features in these images. In 1997, Zoghlami I, Faugeras O. Dedche R. proposed an image alignment algorithm based on geometric corner model, because the corner model provides more information than coordinate points; then in 1999, Bao P, Xu D. proposed the use of wavelet transform Extract the edge-preserving visual model for image alignment; while Nielsen F. proposed a matching method based on geometric point feature optimization. 2000. Kang E, Cohen I, Medioni G. Propose a method for image stitching based on high-level features of images, which utilizes feature-image relationship graphs for image alignment. By using the low-level features of the image and then using its high-level features, people's analysis and understanding of the image are getting deeper and deeper, and the research on image stitching technology has gradually matured.

At present, the research on image stitching technology at home and abroad is mainly aimed at the realization of universal algorithms, and there is no research on the stitching method of images observed by micro-nano vision. At present, the image registration of image stitching technology is mainly based on the information provided by the image, and this information is generally obtained from the correlation and the geometric characteristics of the image. The error is generally at the pixel level, but in the micro-nano vision The error is relatively large, so these two methods are not suitable for image stitching in micro-nano vision.

SUMMARY OF THE INVENTION

In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides an image stitching system and method for micro-nano visual observation, which utilizes the measurement information of the grating sensor inside the micro-nano motion platform as the information source for image registration, thereby achieving high accuracy. The precise image registration process finally enables the image stitching method for micro-nano visual observation to have high stitching accuracy.

To achieve the above object, one or more embodiments of the present invention provide the following technical solutions:

An image stitching system for micro-nano visual observation, comprising:

The micro-nano motion platform is used to fix the observation object, and is arranged under the microscope objective lens, and makes the objective lens align the observation object; The micro-nano motion platform is also provided with one or more grating sensors;

an image acquisition device, which is arranged on the microscope eyepiece and is used to capture the image of the observation object;

The computing device is connected in communication with the micro-nano motion platform and the image acquisition device, controls the movement of the micro-nano motion platform, and collects the image of the observed object through the image acquisition device after each movement, and the images of the observed object obtained in two adjacent times are partially overlapped At the same time, the position information of the micro-nano motion platform transmitted by the grating sensor is obtained; according to the position information of the micro-nano motion platform when each frame of images is captured, each frame of images is registered and spliced.

An image stitching method for micro-nano visual observation. When performing micro-nano visual observation, an observation object is fixed based on a micro-nano motion platform, images are collected based on an image acquisition device on an eyepiece, and collected by moving the micro-nano motion platform. The complete observation object image data, and the observation object images obtained twice adjacently have partial overlap, and at the same time, the position information of the micro-nano motion platform is obtained for each movement;

The method includes: registering and splicing each frame of images according to the position information of the micro-nano motion platform when each frame of images is taken.

One or more of the above technical solutions have the following beneficial effects:

The invention uses the micro-nano motion platform as the carrier of the observed object of the microscope, adopts a camera placed at the eyepiece, and drives the micro-nano motion platform to move to obtain a plurality of images of the observed object, and records the images through the grating sensor of the micro-nano motion platform at the same time. The position information of the micro-nano motion platform when each image is taken is used as the information source of image registration to achieve high-precision image registration, and finally the image stitching method for micro-nano visual observation can have high stitching accuracy. .

Description of drawings

The accompanying drawings that form a part of the present invention are used to provide further understanding of the present invention, and the exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention.

Fig. 1 is a working principle diagram of an image stitching system for micro-nano visual observation in one or more embodiments of the present invention;

2 is a diagram of the steady-state motion accuracy of a micro-nano motion platform in one or more embodiments of the present invention;

3 is a marker used in a calibration experiment in one or more embodiments of the present invention;

Fig. 4 is the relationship diagram of the calibration experiment coordinate system in one or more embodiments of the present invention;

5 is a schematic diagram of an image stitching algorithm in one or more embodiments of the present invention;

6 is a schematic diagram of a hat function in one or more embodiments of the present invention;

7 is a set of pictures collected by a camera in one or more embodiments of the present invention;

FIG. 8 is a mosaic image obtained experimentally in one or more embodiments of the present invention.

Detailed ways

It should be noted that the following detailed description is exemplary and intended to provide further explanation of the invention. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

It should be noted that the terminology used herein is for the purpose of describing specific embodiments only, and is not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural as well, furthermore, it is to be understood that when the terms "comprising" and/or "including" are used in this specification, it indicates that There are features, steps, operations, devices, components and/or combinations thereof.

Embodiments of the invention and features of the embodiments may be combined with each other without conflict.

Example 1

The present embodiment provides an image stitching system for micro-nano visual observation, as shown in Figure 1, including:

a camera, which is installed on the eyepiece of the microscope and is used to capture an image of the observed object, and the captured image is transmitted to the host computer;

The micro-nano motion platform, the objective lens of the microscope is located directly above the micro-nano motion platform; the observation object is fixed on the upper surface of the micro-nano motion platform, the objective lens of the microscope is aligned with the observation object, and the micro-nano motion platform is driven to press the set Move a fixed path to take a series of images of the observation object;

The control system of the micro-nano motion platform is a Simulink xPC system, which can collect the information of two grating sensors built in the micro-nano motion platform in real time, thereby obtaining the position of the observed object and providing conditions for the closed-loop control of the micro-nano motion platform. The Simulink xPC lower computer system is connected with the upper computer through the network cable, and communicates with the upper computer.

The built-in two grating sensors of the micro-nano motion platform are respectively installed in parallel with the X direction and the Y direction, and the two grating sensors are perpendicular to each other, and are used to sense the position of the micro-nano motion platform during the movement in the X and Y directions. Information, the measurement accuracy of the grating is 2 nm, and the position control error of the final experiment is within the range of -5 nm to +5 nm, as shown in Fig. 2.

A gold-plated film is provided between the upper surface of the micro-nano platform and the observation object, and the gold-plated film is used as the background of the observation object, so that the cleanliness of the background where the observation object is located can be guaranteed, and the image processing will not be disturbed.

The microscope and the micro-nano platform are enclosed in a glass cover, thereby reducing the contamination of the reference object caused by the tiny particles in the air.

The image acquisition process ensures that the camera does not shake, the light input and light intensity of the camera are continuously stable, and the acquisition environment must have high cleanliness; the lens height and horizontal plane of the microscope can be finely adjusted, so that markers can be found more quickly, and the camera can be guaranteed. There is good focus quality; the microscope is a 50x lens. The camera model is MER-531-20GM/C-P.

The host computer obtains the motion information fed back by the micro-nano motion platform control system and the image sequence of the observed object, and executes the following registration methods:

The image stitching method for micro- and nano-level visual observation performs the stitching process of super-large images according to the acquired multiple original images and the position information between the images, as shown in Figure 5. The specific steps are as follows:

Step (1): obtain the image sequence of the observed object, and select the first frame image in the image sequence as the reference reference image;

Step (2): utilize Gaussian function to carry out noise reduction processing and image equalization processing to the input image;

Step (3): According to the relationship between the image coordinate system and the micro-nano motion platform coordinate system obtained in the calibration experiment, the position information recorded by the grating sensor is transformed into the image coordinate system.

Step (4): Establish the translation transformation model as shown in the figure, as shown below:

和

为由当前图像变换到参考基准图像坐标系的像素点。where θ is the rotation angle of the image, m ₂ and m ₅ are the translation amounts, x and y are the horizontal and vertical coordinates of the pixels in the current image, respectively,

and

is the pixel point transformed from the current image to the reference reference image coordinate system.

Step (5): the image is registered, and the position information of each frame image recorded by the grating relative to the reference reference image is used as the input information of the translation amounts m ₂ and m ₅ , and then according to the established image translation transformation model, Transform the pixels in each frame image to the coordinate system where the reference reference image is located.

Step (6): Use the weighted average fusion method to fuse the reference reference images that have been registered. This method makes the pixel values of the overlapping area not simply superimposed, but firstly weighted and then superimposed and averaged. The specific process is as follows. First, let f represent the fused image, and f ₁ and f ₂ represent the two images to be spliced, respectively, as follows:

Here w ₁ and w ₂ are the weights of the pixels corresponding to the overlapping areas in the first image and the second image, respectively, and satisfy w ₁ +w ₂ =1, 0<w ₁ <1, and 0<w ₂ <1. Selecting appropriate weights can achieve smooth transitions in overlapping areas and eliminate splicing marks. The present invention selects the hat function weighted average method to select suitable weights. This method assigns higher weights to the pixels in the central area of the image, and the pixels in the edge area of the image have lower weights, and the weight function adopts a hat function (hatfunction). :

where width _i and height _i represent the width and height of the i-th image, and the hat function w(x) is shown in Figure 6.

The specific steps of the image sequence acquisition process of the observed object in the step (1) are as follows:

Step (1.1): Obtain the first frame of image captured by the camera at the initial position of the micro-nano motion platform as a reference frame.

Step (1.2): Drive the micro-nano motion platform to move to the next position according to the set motion trajectory, acquire the image captured by the camera (at this time, the image and the previous frame image must overlap), and record the value of the grating sensor at this time .

Step (1.3): determine whether the image reaches the specified boundary position, if it reaches the boundary position, complete the image acquisition, otherwise repeat step (1.2).

Since the coordinate system of the motion information recorded by the grating sensor information is different from the image coordinate system, it is necessary to calibrate the two coordinate systems. In the step (3), the image coordinate system and the micro-nano motion platform coordinate system are obtained through the calibration process. Relationships include:

Step (3.1): place the marker (as shown in Figure 3) on the micro-nano motion platform, drive the platform to move slowly along the X-axis, while the camera records the sequence image Ix of the marker at a speed of 100 frames per second.

Step (3.2): Drive the platform to move slowly along the Y axis, while the camera records the sequence image Iy of the marker at a speed of 100 frames per second.

Step (3.3): Binarize the sequence images Ix and Iy, then extract the coordinates of the center of mass of the marker, and draw these coordinates into a rectangular coordinate system to obtain two orthogonal straight lines (as shown in Figure 4) , the angle α between the orthogonal coordinate system formed by these two straight lines and the X-O-Y rectangular coordinate system is the angle between the image coordinate system and the micro-nano motion platform coordinate system. The marker used in this embodiment is in the shape of a cross, and its center of mass is marked as the cross center of the cross.

Step (3.4): At present, it is known that the pixel physical size of the camera is 4.8 microns, the magnification of the microscope objective lens used is 50 times, and the distance corresponding to the image translation by one pixel corresponds to the actual moving distance of the platform is the pixel physical size/microscope magnification, that is, in In this experiment, one pixel represents that the platform has moved 4.8/50=0.096 microns, so the motion information recorded by the grating and the unit conversion relationship in the pixel coordinate system can be obtained.

Experimental results:

The nano-motion platform is driven to move according to the trajectory shown in Table 1, where the control accuracy of the nano-motion platform is ±5 nm (as shown in Figure 2), and the 9 original images finally collected from the camera are shown in Figure 7. The present invention uses the algorithm described above to perform image stitching on it, and finally generates the result shown in FIG. 8 . It can be found that the stitching effect in Figure 8 is good, there is no obvious edge trace, and the transition is smooth.

Table 1

group order 1 2 3 4 5 6 7 8 9 X(nm) 0 24576 49152 49152 24576 0 0 24576 49152 Y(nm) 0 0 0 24576 24576 24576 49152 49152 49152

Embodiment 2

The purpose of this embodiment is to provide an image stitching method for micro-nano visual observation.

In order to achieve the above purpose, this embodiment discloses an image stitching method for micro-nano visual observation. When performing micro-nano visual observation, the observation object is fixed based on the micro-nano motion platform, and the image is collected based on the image acquisition device on the eyepiece. , collect the complete observation object image data by moving the micro-nano motion platform, and the observation object images obtained twice adjacently have partial overlap, and at the same time, each movement obtains the position information of the micro-nano motion platform;

The method includes: registering and splicing each frame of images according to the position information of the micro-nano motion platform when each frame of images is taken.

The steps involved in the second embodiment above correspond to the first embodiment, and the specific implementation can refer to the relevant description part of the first embodiment. The term "computer-readable storage medium" should be understood to include a single medium or multiple media including one or more sets of instructions; it should also be understood to include any medium capable of storing, encoding or carrying for use by a processor The executed instruction set causes the processor to perform any of the methods of the present invention.

The above one or more embodiments have the following technical effects:

The invention uses the micro-nano motion platform as the carrier of the observed object of the microscope, adopts a camera placed at the eyepiece, and drives the micro-nano motion platform to move to obtain a plurality of images of the observed object, and records the images through the grating sensor of the micro-nano motion platform at the same time. The position information of the micro-nano motion platform when each image is taken is used as the information source of image registration to achieve high-precision image registration, and finally the image stitching method for micro-nano visual observation can have high stitching accuracy. .

Those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by the computing device, so that they can be stored in a storage device. The device is executed by a computing device, or they are separately fabricated into individual integrated circuit modules, or multiple modules or steps in them are fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Although the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, they do not limit the scope of protection of the present invention. Those skilled in the art should understand that on the basis of the technical solutions of the present invention, those skilled in the art do not need to pay creative efforts. Various modifications or deformations that can be made are still within the protection scope of the present invention.

Claims (4)

1. an image splicing system for micro-nano-level visual observation, is characterized in that, comprises: A micro-nano motion platform, used for fixing the observation object, is arranged under the microscope objective lens, and makes the objective lens align with the observation object; the micro-nano motion platform is further provided with one or more grating sensors; an image acquisition device, which is arranged on the microscope eyepiece and is used to capture the image of the observation object; The computing device is connected in communication with the micro-nano motion platform and the image acquisition device, controls the movement of the micro-nano motion platform, and collects the image of the observed object through the image acquisition device after each movement, and the images of the observed object obtained in two adjacent times are partially overlapped At the same time, the position information of the micro-nano motion platform transmitted by the grating sensor is obtained; according to the position information of the micro-nano motion platform when each frame of image is taken, the images of each frame are registered and spliced; The process of collecting the image of the observation object includes: judging whether the image reaches the specified boundary position after each acquisition of the image; image; if achieved, image acquisition is complete; The process of registering each frame of images includes: Transform the position information of the micro-nano motion platform into the image coordinate system to obtain the position information of each frame of image; Use one frame in the image sequence as the reference image, and for each other frame: Calculate the translation transformation model of the image to the reference image; Calculate the position information of the image relative to the reference image according to the position information of each frame image, and use the position information as the input of the translation transformation model, and transform the image to the coordinate system where the reference image is located; The method for image splicing is: after each frame of image registration is completed, image splicing is also performed, and weighted fusion is performed on the overlapping area between two adjacent frames of images after image splicing; The process of transforming the position information of the micro-nano motion platform to the image coordinate system is based on the relationship between the micro-nano motion platform coordinate system and the image coordinate system, wherein the method for obtaining the relationship between the micro-nano motion platform coordinate system and the image coordinate system for: Place the marker on the micro-nano motion platform, drive the platform to move along the X-axis and the Y-axis respectively, acquire the marker image during the movement, and obtain the sequence images along the X-axis and the Y-axis; For each image in the two sets of sequence images, the centroid coordinates of the markers are extracted, and these centroid coordinates are drawn into a rectangular coordinate system, and two orthogonal straight lines are obtained by connecting the lines. The included angle between the formed orthogonal coordinate system and the rectangular coordinate system is the included angle between the image coordinate system and the micro-nano motion platform coordinate system; According to the pixel physical size of the image acquisition device and the magnification of the microscope objective lens, the conversion relationship between the position information of the micro-nano motion platform and the image coordinate system is obtained. 2 . The image stitching system for micro-nano-level visual observation according to claim 1 , wherein after acquiring the image sequence of the observed object, noise reduction and image equalization processing are also performed on each frame of image. 3 . 3 . The image stitching system for micro-nano visual observation as claimed in claim 1 , wherein the weight of each pixel in the overlapping area is obtained based on a hat function. 4 . 4. An image stitching method for micro-nano visual observation, characterized in that, when performing micro-nano visual observation, the observation object is fixed based on the micro-nano motion platform, the image is collected based on the image acquisition device on the eyepiece, and the image is collected by moving the object. The micro-nano motion platform collects complete image data of the observed object, and the images of the observed object obtained twice adjacently overlap, and at the same time, the position information of the micro-nano motion platform is acquired for each movement; the method includes: according to The position information of the micro-nano motion platform when each frame of image is taken, and the registration and splicing of each frame of image are performed; The process of collecting images includes: judging whether the image reaches the specified boundary position after each acquisition of the image, if not, driving the micro-nano motion platform to move to the next position according to the set motion trajectory, and continue to acquire the image of the observed object; If so, the image acquisition is completed; The process of registering each frame of images includes: Transform the position information of the micro-nano motion platform into the image coordinate system to obtain the position information of each frame of image; Use one frame in the image sequence as the reference image, and for each other frame: Calculate the translation transformation model of the image to the reference image; Calculate the position information of the image relative to the reference image according to the position information of each frame image, and use the position information as the input of the translation transformation model, and transform the image to the coordinate system where the reference image is located; The method for image splicing is: after each frame of image registration is completed, image splicing is also performed, and weighted fusion is performed on the overlapping area between two adjacent frames of images after image splicing; The process of transforming the position information of the micro-nano motion platform to the image coordinate system is based on the relationship between the micro-nano motion platform coordinate system and the image coordinate system, wherein the method for obtaining the relationship between the micro-nano motion platform coordinate system and the image coordinate system for: Place the marker on the micro-nano motion platform, drive the platform to move along the X-axis and the Y-axis respectively, acquire the marker image during the movement, and obtain the sequence images along the X-axis and the Y-axis; For each image in the two sets of sequence images, the centroid coordinates of the markers are extracted, and these centroid coordinates are drawn into a rectangular coordinate system, and two orthogonal straight lines are obtained by connecting the lines. The included angle between the formed orthogonal coordinate system and the rectangular coordinate system is the included angle between the image coordinate system and the micro-nano motion platform coordinate system; According to the pixel physical size of the image acquisition device and the magnification of the microscope objective lens, the conversion relationship between the position information of the micro-nano motion platform and the image coordinate system is obtained.

Images