CN115526850A - Training method and device for real-space decoder of refrigeration electron microscope and electronic equipment - Google Patents

Abstract

The invention provides a training method, device and electronic equipment for a cryo-electron microscope real-space decoder. Among them, the method includes: obtaining the input projection image and label information of the cryo-electron microscope; inputting the projection parameters of the input projection image into the decoder, and outputting the predicted projection image; using the mask to reconstruct the machine learning representation and prediction of the three-dimensional reconstruction target structure The projection image and the input projection image are masked; the loss value is calculated based on the masked input projection image and the predicted projection image, and the parameters of the decoder are adjusted based on the loss value; until the preset training conditions are met, the trained decoding is obtained. device. This method is suitable for machine learning scenarios, so that the 3D reconstruction of cryo-EM based on machine learning can better meet the actual needs. It can focus on the key local information of the 3D reconstruction target structure, and effectively separate the foreground and background, thereby improving the quality of reconstruction. Better results can also be achieved with fewer images.

Description

Training method, device and electronic equipment for cryo-electron microscope real-space decoder

technical field

The invention relates to the technical field of cryo-electron microscopy, in particular to a training method, device and electronic equipment for a real-space decoder of a cryo-electron microscope.

Background technique

Single particle cryo-electron microscopy is an important experimental technique in the field of structural biology. This technology uses two-dimensional images of biological macromolecules taken by electron microscopes to analyze the three-dimensional structure of the macromolecules, thereby helping biologists understand the functional mechanism of biological macromolecules and assisting in the design of new drugs. Three-dimensional reconstruction is a core step in single-particle cryo-EM technology, which realizes the conversion from two-dimensional electron microscope images to three-dimensional molecular density maps.

Traditional cryo-EM 3D reconstruction algorithms can be divided into two categories. The first is an algebraic reconstruction algorithm, and the second is a Fourier transform algorithm. The algorithm of three-dimensional reconstruction in frequency space based on Fourier transform algorithm is currently commonly used algorithm, but this algorithm has the following problems: this type of algorithm needs to perform interpolation calculation in frequency space, and this interpolation algorithm is difficult to achieve fast And accurate. Commonly used fast interpolation algorithms lead to artifacts due to loss of accuracy and loss of resolution of the reconstructed 3D density map. This defect is particularly prominent on small data sets, because small data sets have more missing information in the frequency space. The representation of the frequency domain space only retains the description of the global structure, and is not sensitive to the irregularity of the local structure of the real space, so it is difficult to optimize a specific spatial region.

Although traditional algebraic algorithms can avoid these two problems, these algorithms themselves have low computational efficiency and cannot be directly applied to the framework of machine learning, so their computational efficiency is far inferior to the 3D reconstruction algorithm based on machine learning acceleration, and It is difficult to combine with other machine learning algorithms, such as three-dimensional classification of electron microscope images based on machine learning. Therefore, this type of method hinders the development of efficient real-space 3D reconstruction algorithms based on machine learning.

Contents of the invention

In view of this, the object of the present invention is to provide a training method, device and electronic equipment for a cryo-electron microscope real-space decoder, so that the advantages of cryo-electron microscope real-space three-dimensional reconstruction can be fully utilized under the framework of machine learning.

In the first aspect, an embodiment of the present invention provides a training method for a cryo-electron microscope real-space decoder. The method includes: obtaining the input projection image of the cryo-electron microscope and the label information of the input projection image from the sample set; wherein, the label information of the input projection image The information includes: the projection parameters and mask of the input projection image, the projection parameters include projection angle and image translation; the projection parameters of the input projection image are input into the decoder, and the machine learning representation of the 3D reconstruction target structure inside the decoder is determined. The projected field of view is spatially transformed, and the predicted projection image is output; among them, the machine learning representation includes explicit representation and implicit representation; the machine learning representation of the 3D reconstruction target structure, the predicted projection image and the input projection image through the mask Perform mask processing; calculate the loss value based on the input projection image and predicted projection image after mask processing, adjust the parameters of the decoder based on the loss value; perform multiple executions to obtain the input projection image of the cryo-electron microscope and the label of the input projection image from the sample set The step of information until the preset training conditions are met, and the trained decoder is obtained.

In an optional embodiment of the present application, the above mask includes a 3D local mask, a 2D local mask and/or a planar foreground and background mask.

In an optional embodiment of the present application, the above-mentioned 3D local mask includes an inscribed spherical mask or a weighted mask, and the 2D local mask is determined based on the 3D local mask.

In an optional embodiment of the present application, spatial transformation and projection calculation are performed on the 3D local mask based on projection parameters to obtain a 2D local mask.

In an optional embodiment of the present application, the above-mentioned planar foreground and background masks are used to distinguish the foreground area and the background area of the projected image.

In an optional embodiment of the present application, the explicit representation of the three-dimensional reconstruction target structure includes: encoding the Coulomb potential energy value of the three-dimensional structure through the grid points of the three-dimensional lattice matrix; the implicit representation of the three-dimensional reconstruction target structure includes: Functions that map spatial coordinates to Coulomb potential energy values of three-dimensional structures are represented by neural networks.

In an optional embodiment of the present application, after the above-mentioned step of inputting the projection parameters of the input projection image into the decoder, the method further includes: spatially performing the projection field of view of the machine learning representation of the 3D reconstructed target structure inside the decoder. Transformation and subsequent processing to output predicted projected images.

In an optional embodiment of the present application, the step of calculating the loss value based on the masked input projection image and the predicted projection image includes: masking the input projection image based on the 2D local mask and the planar foreground and background mask Mask processing to obtain the input projection image after mask processing; perform mask processing on the predicted projection image based on the plane foreground and background mask to obtain the predicted projection image after mask processing; based on the mask processed input projection image and predicted projection image determines the loss value for the specified region of the input projected image.

In an optional embodiment of the present application, the above masked input projection image includes the designated area and other areas; after the step of obtaining the masked input projection image, the method further includes: suppressing other area noise.

In an optional embodiment of the present application, the above method further includes: inputting the first electron microscope image of the cryo-electron microscope into the projection parameter prediction encoder, and outputting the space transformation parameters of the first electron microscope image; wherein, the space transformation parameters represent the first The projection parameters of the electron microscope image; input the space transformation parameters into the cryo-electron microscope real space decoder, and perform training iterations; where the loss value generated by the training iterations is passed back to the encoder and decoder through the machine learning calculation graph, and based on the training iterations The resulting loss values update the parameters of the encoder and decoder.

In the second aspect, an embodiment of the present invention also provides a training device for a cryo-electron microscope real-space decoder, the device comprising: an input projection image acquisition module, configured to acquire an input projection image of a cryo-electron microscope and label information of the input projection image from a sample set ; Wherein, the tag information of the input projection image includes: the projection parameter and the mask of the input projection image, and the projection parameter includes projection angle and image translation; the predicted projection image output module is used to input the projection parameters of the input projection image into the decoder, Perform spatial transformation on the projection field of view of the machine learning representation of the three-dimensional reconstruction target structure inside the decoder, and output the predicted projection image; wherein, the machine learning representation includes explicit representation and implicit representation; the mask processing module uses It is used to perform mask processing on the machine learning representation of the three-dimensional reconstruction target structure, the predicted projection image and the input projection image through the mask; the loss value calculation module is used to calculate the loss based on the masked input projection image and the predicted projection image value, adjust the parameters of the decoder based on the loss value; the decoder training completion module is used to perform the steps of obtaining the input projection image of the cryo-electron microscope and the label information of the input projection image from the sample set for multiple times until the preset training conditions are met, Get the trained decoder.

In the third aspect, the embodiment of the present invention also provides an electronic device, which is characterized in that it includes a processor and a memory, the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to achieve the above-mentioned A training method for a real-space decoder for cryo-EM.

In the fourth aspect, the embodiment of the present invention also provides a computer-readable storage medium, which is characterized in that the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable The instruction prompts the processor to implement the above-mentioned training method of the cryo-EM real-space decoder.

Embodiments of the present invention bring the following beneficial effects:

The embodiment of the present invention provides a training method, device, and electronic equipment for a cryo-electron microscope real-space decoder, which are suitable for machine learning scenarios, so that the three-dimensional reconstruction based on machine learning in electron microscope real space can better meet actual needs, and can focus on The key local information of the 3D reconstruction target structure can effectively separate the foreground and background, thereby improving the reconstruction quality, avoiding the disadvantages of the frequency domain 3D reconstruction interpolation algorithm, and achieving relatively good results when there are few images.

Other features and advantages of the present disclosure will be set forth in the following description, or some of the features and advantages can be inferred or unambiguously determined from the description, or can be known by implementing the above-mentioned techniques of the present disclosure.

In order to make the above-mentioned objects, features and advantages of the present disclosure more comprehensible, preferred embodiments will be described in detail below together with the accompanying drawings.

Description of drawings

In order to more clearly illustrate the specific implementation of the present invention or the technical solutions in the prior art, the following will briefly introduce the accompanying drawings that need to be used in the specific implementation or description of the prior art. Obviously, the accompanying drawings in the following description The drawings show some implementations of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative work.

Fig. 1 is the flowchart of the training method of a kind of cryo-electron microscope real space decoder provided by the embodiment of the present invention;

Fig. 2 is a flowchart of another training method for a cryo-electron microscope real-space decoder provided by an embodiment of the present invention;

FIG. 3 is a schematic diagram of a decoder algorithm provided by an embodiment of the present invention;

Fig. 4 is a kind of schematic diagram with encoder and decoder algorithm provided by the embodiment of the present invention;

5 is a schematic structural diagram of a training device for a cryo-electron microscope real-space decoder provided by an embodiment of the present invention;

FIG. 6 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention.

detailed description

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. the embodiment. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Single particle cryo-electron microscopy is an important experimental technique in the field of structural biology. This technology uses two-dimensional images of biological macromolecules taken by electron microscopes to analyze the three-dimensional structure of the macromolecules, thereby helping biologists understand the functional mechanism of biological macromolecules and assisting in the design of new drugs.

Three-dimensional reconstruction is a core step in single-particle cryo-EM technology, which realizes the conversion from two-dimensional electron microscope images to three-dimensional molecular density maps. The focus of this embodiment is to realize the reconstruction from the two-dimensional projection image to the three-dimensional object under the given key parameters (spatial angle/image translation) of the three-dimensional reconstruction of the electron microscope.

At present, the traditional 3D reconstruction algorithms for cryo-EM can be divided into two categories. The first is the algebraic reconstruction algorithm (such as SIRT, ART). This type of algorithm directly uses the physical imaging principle of the electron microscope image to estimate the three-dimensional volume corresponding to a set of electron microscope projection images. The second type is the Fourier transform algorithm (such as the method used in Relion and cryoSPARC electron microscope software). This type of algorithm is based on the Fourier slice theorem, and converts the projection problem into the interpolation problem of frequency space.

The design of 3D reconstruction methods based on machine learning (such as cryoDRGN) is generally based on the Fourier slice theorem, using a neural network to fit a function that describes the 3D representation of the target object in frequency space, thereby achieving 3D reconstruction.

However, the algorithm for three-dimensional reconstruction in frequency space based on Fourier transform algorithm is currently commonly used algorithm, but this algorithm has the following problems: this type of algorithm needs to perform interpolation calculation in frequency space, and this interpolation algorithm is difficult to do Fast and accurate. Commonly used fast interpolation algorithms lead to artifacts due to loss of accuracy and loss of resolution of the reconstructed 3D density map. This defect is particularly prominent on small data sets, because small data sets have more missing information in the frequency space. The representation of the frequency domain space only retains the description of the global structure, and is not sensitive to the irregularity of the local structure of the real space, so it is difficult to optimize a specific spatial region.

Although traditional algebraic algorithms can avoid these two problems, these algorithms themselves have low computational efficiency and cannot be directly applied to the framework of machine learning, so their computational efficiency is far inferior to the 3D reconstruction algorithm based on machine learning acceleration, and It is difficult to combine with other machine learning algorithms, such as three-dimensional classification of electron microscope images based on machine learning. Therefore, this type of method hinders the development of efficient real-space 3D reconstruction algorithms based on machine learning.

Based on this, the embodiments of the present invention provide a training method, device and electronic equipment for a cryo-electron microscope real space decoder, specifically relating to a three-dimensional reconstruction of a cryo-electron microscope real space with a mask that is suitable for machine learning decoder.

Most of the existing 3D reconstructions are done in the frequency space. Limited by the accuracy of the frequency space signal interpolation algorithm, the 3D reconstruction results in the frequency space are easily distorted in the reconstruction of high frequency real space signals, resulting in inaccurate reconstruction results. Accurate, especially on small datasets. In addition, frequency-space reconstruction is not sensitive to irregularities in local spatial structure changes, making it difficult to achieve local optimization of the target volume. Traditional real-space 3D reconstruction algorithms cannot be directly applied to the machine learning framework without modification. The method provided in this embodiment can avoid the defect of frequency-space three-dimensional reconstruction, and is also applicable to the scene of machine learning.

In order to facilitate the understanding of this embodiment, a training method for a cryo-electron microscope real-space decoder disclosed in the embodiment of the present invention is firstly introduced in detail.

Embodiment one:

An embodiment of the present invention provides a training method for a cryo-electron microscope real-space decoder. Referring to the flowchart of a training method for a cryo-electron microscope real-space decoder shown in FIG. 1 , the training method for the cryo-electron microscope real-space decoder includes the following step:

Step S102, acquiring the input projection image of the cryo-electron microscope and the label information of the input projection image from the sample set; wherein, the label information of the input projection image includes: projection parameters and a mask of the input projection image, and the projection parameters include projection angle and image translation.

The decoder in this embodiment is a network capable of predicting the projection image from the projection angle and projection translation of the electron microscope. When training the decoder, it is necessary to use the input projection image of the cryo-electron microscope and the label information of the input projection image as samples for training. Among them, the label information includes: the projection angle of the input projected image, image translation and mask. The mask in this embodiment is not necessarily a single mask, and may include multiple masks, for example: 3D local mask, 2D local mask, planar foreground/background mask, and the like.

Step S104, input the projection parameters of the input projection image into the decoder, perform spatial transformation on the projection field of view of the machine learning representation of the 3D reconstruction target structure inside the decoder, and output the predicted projection image.

Among them, machine learning representations include explicit representations and implicit representations, and the decoder can spatially perform explicit representations, implicit representations, or projection views of the three-dimensional reconstruction target structure (3DR) according to the input projection parameters. transform, the decoder can output a predicted projected image.

For example, the spatial transformation can be performed on the projection view of the explicit representation and the implicit representation, and the spatial transformation and subsequent processing can be directly performed on the explicit representation to output the predicted projection image.

Step S106, performing mask processing on the machine learning representation of the three-dimensionally reconstructed target structure, the predicted projection image, and the input projection image through a mask.

In this embodiment, mask processing can be performed on the machine learning representation of the 3D reconstruction target structure, the predicted projection image, and the input projection image. Through mask processing, it can be ensured that the same area is used for the predicted projection image and the input projection image. Subsequent calculations.

Step S108, calculating a loss value based on the masked input projection image and the predicted projection image, and adjusting the parameters of the decoder based on the loss value.

Based on the preset loss function, the loss value of the input projection image and the predicted projection image after mask processing can be calculated, and the parameters of the decoder can be adjusted according to the loss value, thereby completing one iteration of the decoder.

In step S110, the step of obtaining the input projection image of the cryo-electron microscope and the label information of the input projection image from the sample set is performed multiple times until the preset training conditions are met, and a trained decoder is obtained.

In this embodiment, the total number of iterations (for example, 3000) can be preset as the training condition. When the number of iterations reaches the total number, it can be considered that the preset training condition is met, and a trained decoder can be obtained.

The embodiment of the present invention provides a training method for a cryo-electron microscope real space decoder, which is suitable for machine learning scenarios, so that the three-dimensional reconstruction based on machine learning in electron microscope real space can better meet the actual needs, and can focus on three-dimensional reconstruction of the target structure The key local information can effectively separate the foreground and background, thereby improving the reconstruction quality, avoiding the disadvantages of the three-dimensional reconstruction interpolation algorithm in the frequency domain, and achieving relatively good results when there are few images.

Embodiment two:

This embodiment provides another training method for a cryo-electron microscope real-space decoder, which is implemented on the basis of the above-mentioned embodiments, as shown in Figure 2 is a flowchart of another training method for a cryo-electron microscope real-space decoder , the training method of a cryo-electron microscope real space decoder in the present embodiment comprises the following steps:

Step S202, acquiring the input projection image of the cryo-electron microscope and the label information of the input projection image from the sample set; wherein, the label information of the input projection image includes: projection parameters and a mask of the input projection image, and the projection parameters include projection angle and image translation.

For ease of understanding, refer to a schematic diagram of a decoder algorithm shown in FIG. 3 , the above decoder algorithm may include a training process and an inference process. During the training process, the electron microscope images and their label information are input into the network for training, and the training is terminated when the training loss no longer decreases or reaches a certain number of training iterations.

The input projection image and its label information used in the training process. The input projection image includes the cryo-electron microscope projection image obtained in the experiment. The label information includes: the projection angle, image translation and mask of the input projection image.

Step S204, inputting the projection parameters of the input projection image into the decoder, performing spatial transformation on the projection field of view of the machine learning representation of the 3D reconstruction target structure inside the decoder, and outputting the predicted projection image.

Machine learning representations include explicit representations and implicit representations. In this embodiment, spatial transformation and subsequent processing can be performed on the projection field of view of the machine learning representation of the 3D reconstruction target structure inside the decoder, and the predicted projection image can be output. .

Among them, the explicit representation of the 3D reconstruction target structure includes: encoding the Coulomb potential energy value of the 3D structure through the grid points of the 3D lattice matrix; the implicit representation of the 3D reconstruction target structure includes: mapping the spatial coordinates with the neural network representation to the function of the Coulomb potential energy value of the three-dimensional structure.

One implementation form of explicit 3DR is a three-dimensional lattice matrix, and each lattice point can encode the Coulomb potential energy value of the three-dimensional structure with a numerical value or a function with parameters (such as a spherical harmonic function). One implementation of implicit 3DR is to use a neural network to represent a function that can calculate the Coulomb potential energy value of a three-dimensional structure corresponding to each coordinate value in the space.

As shown in Figure 3, during the training process, the network needs to input a set of projection parameters, project the 3DR according to the projection angle, output the projection map at this angle, and calculate the loss function with the input experimental electron microscope image.

For the two-dimensional planar projection calculation of explicit 3DR, the three-dimensional lattice matrix can be transformed to a new spatial orientation and position according to the input space transformation parameters (including rotation and translation information), and then the voxel information of the original three-dimensional lattice matrix can be used Do an interpolation algorithm to calculate the voxel information of the new 3D grid matrix. Then integrate or calculate the mean in the projected direction.

In addition, the above-mentioned explicit 3DR can also be replaced by implicit representations based on neural networks, such as: neural networks based on position encoding, networks based on sine/cosine activation functions such as SIREN, and generation technologies based on GAN/VAE/diffusion model network etc.

When obtaining the projection, this embodiment can use the method of rotating the three-dimensional lattice matrix to obtain the projection, and can also use other methods to obtain the projection, for example: do not rotate the three-dimensional lattice point, and directly perform projection on the corresponding rotation angle and translation position; through The neural network directly predicts the Coulomb potential energy value of the 3D reconstructed target object within the projection field of view corresponding to the projection parameters and performs projection; using the above method, the 3DR is processed by foreground/background mask, local mask, etc. before projection.

Electron microscope projection parameters can be predicted by other professional software; predicted by numerical search algorithm combined with three-dimensional reconstruction method; predicted by neural network suitable for image processing; predicted by neural network in the training process, and simultaneously optimize the prediction network and three-dimensional reconstruction Network structure; add correction items to predict by any of the above methods.

Step S206, performing mask processing on the machine learning representation of the three-dimensionally reconstructed target structure, the predicted projection image, and the input projection image through a mask.

Specifically, the mask in this embodiment includes a 3D local mask, a 2D local mask and/or a planar foreground and background mask. Among them, the 3D local mask includes an inscribed spherical mask or a mask with weights, and the 2D local mask is determined based on the 3D local mask, for example: spatial transformation and projection of the 3D local mask based on projection parameters Calculate to get a 2D local mask.

There are many design schemes for 3D local masks, such as inscribed spherical masks, which set the mask value at the corner of 3DR to zero, and set the mask value inside the 3DR inscribed sphere to 1. 3D local masks can also be Set as a weighted mask, that is, set different mask weight values (between 0 and 1) for different areas of 3DR.

The design of the 2D local mask is based on the 3D local mask. A possible implementation solution is to perform projection calculation on the 3D local mask according to the input projection parameters after spatial transformation.

As shown in Figure 3, the explicit 3DR can be formed by a three-dimensional lattice matrix, and the implicit 3DR can be formed by a neural network. In addition, 3DR needs to undergo a three-dimensional mask preprocessing before the rotation/translation transformation to avoid false signals at the corners of the eight cubes during the three-dimensional reconstruction process. The three-dimensional mask (that is, the 3D local mask) used in this embodiment may be an inscribed sphere of the three-dimensional lattice matrix, that is, the voxel signals inside the inscribed sphere are reserved, and the voxel signals outside the inscribed sphere are set to zero.

The decoder provided in this embodiment operates entirely in real space, and the three-dimensional mask here can be designed as a weighted three-dimensional foreground mask according to user needs. For example, if the 3D reconstructed target object contains three spatial regions A, B, and C, the user is most interested in region A, slightly interested in region B, and not interested in region C. Then the three-dimensional mask here can set the highest weight for the A region, set a lower weight for the B region, and set a very low weight for the C region or directly reset the weight to zero.

A three-dimensional mask can be converted into a two-dimensional mask (that is, a 2D local mask). For example, a two-dimensional mask can be obtained by performing the same spatial transformation and projection on the three-dimensional mask as 3DR. This two-dimensional mask can preprocess the input electron microscope image during subsequent loss function calculations to ensure that the predicted image and the input image use the same area for correlation calculations.

In step S208, a loss value is calculated based on the masked input projection image and the predicted projection image, and parameters of the decoder are adjusted based on the loss value.

Specifically, in this embodiment, the input projection image may be masked based on the 2D local mask and the planar foreground and background mask to obtain the masked input projection image; the predicted projection image may be masked based on the planar foreground and background mask. The mask processing is to obtain a mask-processed predicted projection image; based on the mask-processed input projection image and the predicted projection image, the loss value of a designated area of the input projection image is determined.

In addition, this embodiment can also suppress noise in other regions through regularization.

As shown in Figure 3, the planar foreground/background mask is designed to separate the foreground from the background. The foreground here refers to the area corresponding to the 3D reconstruction target structure in the projection image, and the background refers to other areas in the projection image.

Two-dimensional foreground/background masks can be used during training to improve reconstruction quality. The specific method is: in the step of calculating the loss function, two-dimensional foreground/background mask processing can be performed on the input and predicted projection images, the model only calculates the loss function of the global or local foreground, and uses a regularization method for the background part to suppress noise .

The foreground/background mask can be manually designed, calculated by other professional software, or directly predicted by projection results. The foreground mask method can reduce the noise while preserving the signal, thereby improving the performance of the model.

For the global foreground mask, this embodiment designs an automatic two-dimensional foreground/background mask generation scheme. In this scheme, the input image can be smoothed (such as using a Gaussian filter). If the pixel value of the processed input projection image exceeds a certain threshold, these areas are set as the foreground, and other areas are set as the background, and then Make binary or weighted foreground/background masks from these regions. The transition area between the foreground and the background can be processed gradually with a smooth function (such as a sigmoid function).

If you want to design a foreground mask with weights, the weights can be calculated based on the smoothed pixel values (such as normalization or inputting these pixel values into a weight generation function to obtain mask weights). The local foreground mask can be generated by the aforementioned three-dimensional to two-dimensional area mask. It is worth emphasizing that the same projection image can be processed with multiple layers of masks.

When calculating the loss function, L2loss can be used for the comparison between the predicted projection image and the input projection image, and L1 loss can be used for the reconstructed three-dimensional volume. The final loss function contains these two parts. In addition, the prefactor beta can be added to the L1 loss of the 3D volume to optimize and reduce the negative impact of the noise in the input image on the 3D reconstruction. Preferably, setting beta to 0.05 has the best effect on 3D reconstruction.

In addition, the loss function used in this embodiment can be replaced by other similar functions, including: smooth L1loss, Huber loss, Hinge loss, cross-entropy loss, etc.; the regularization term can be replaced by other regularized loss functions, including: L1 loss, L2 loss, L1+L2 loss, TV loss, etc.

In step S210, the step of obtaining the input projection image of the cryo-electron microscope and the label information of the input projection image from the sample set is performed multiple times until the preset training conditions are met, and a trained decoder is obtained.

In this embodiment, AdamW optimizer can be used for machine learning iteration. Preferably, the reconstruction effect of setting the weight decay parameter of AdamWoptimizer to 1.0E-6 is the best.

Step S212, generating a three-dimensional reconstructed structure matching the input projection image data set.

During the inference process, the network of the encoder can directly output the 3D lattice matrix representing the Coulomb potential energy of the 3D reconstructed target structure as the reconstruction result. The 3D reconstruction algorithm of this embodiment can also run on a low-end graphics card and obtain relatively good reconstruction results. Preferably, setting the batch size to 8 can complete relatively good 3D reconstruction. Setting the learningrate to 0.1 works best.

In addition, in this embodiment, the first electron microscope image of the cryo-electron microscope can also be input into the projection parameter prediction encoder, and the spatial transformation parameters of the first electron microscope image can be output; wherein, the spatial transformation parameters represent the projection parameters of the first electron microscope image shown; The transformation parameters are input to the cryo-electron microscope real-time decoder, and the training iteration is performed; the loss value generated by the training iteration is passed back to the encoder through the machine learning calculation graph, and based on the loss value generated by the training iteration, the encoder and decoder are compared. The parameters are updated.

Among them, the space transformation parameters include various representation forms of space transformation, such as space transformation matrix, quaternion, Rodrigues space rotation vector, etc., as shown in FIG. 4 for a schematic diagram with encoder and decoder algorithms.

Specifically, the electron microscope sample image of the cryo-electron microscope can be input into the initial encoder, and the spatial transformation parameters are output; the initial encoder is trained using the spatial transformation parameters of the electron microscope sample image as the projection angle of the three-dimensional reconstruction target structure until the preset Training conditions, get the trained projection parameter prediction encoder.

The projection parameters can be predicted by the neural network. For example, the first electron microscope image is input to the angle prediction network, and the output is a spatial rotation and translation transformation parameter or an equivalent representation. The transformation parameter is added to the training process as a 3DR spatial transformation parameter. Implement joint training of 3D reconstruction network and projection parameter prediction network.

To sum up, the above method provided by this embodiment can be represented in real space during the reconstruction process; a real space projection algorithm is used in the reconstruction process; and a real space mask can be applied in the reconstruction process. In this method, the differentiable three-dimensional reconstruction of electron microscopy in real space under the framework of machine learning is realized, which avoids various disadvantages of three-dimensional reconstruction in frequency space; a real-space mask suitable for machine learning is designed to improve three-dimensional Focused optimization of reconstructed overall quality and local features.

Embodiment three:

Corresponding to the above-mentioned method embodiment, the embodiment of the present invention provides a training device for a cryo-electron microscope real-space decoder. Referring to the schematic structural diagram of a training device for a cryo-electron microscope real-space decoder shown in FIG. The training setup for the spatial decoder consists of:

The input projection image acquisition module 51 is used to obtain the input projection image of the cryo-electron microscope and the label information of the input projection image from the sample set; wherein, the label information of the input projection image includes: the projection parameters and the mask of the input projection image, and the projection parameters include projection angle and image translation;

The predicted projection image output module 52 is used to input the projection parameters of the input projection image into the decoder, perform spatial transformation on the projection field of view of the machine learning representation of the three-dimensional reconstruction target structure inside the decoder, and output the predicted projection image; wherein, Machine learning representations include explicit representations and implicit representations;

The mask processing module 53 is used to perform mask processing on the machine learning representation of the three-dimensional reconstruction target structure, the predicted projection image and the input projection image through a mask;

The loss value calculation module 54 is used to calculate the loss value based on the input projection image and the predicted projection image processed by the mask, and adjust the parameters of the decoder based on the loss value;

The decoder training completion module 55 is used to perform the step of obtaining the input projection image of the cryo-electron microscope and the label information of the input projection image from the sample set for multiple times until the preset training conditions are met to obtain a trained decoder.

The embodiment of the present invention provides a training device for a cryo-electron microscope real space decoder, which is suitable for machine learning scenarios, so that the three-dimensional reconstruction based on machine learning in electron microscope real space can better meet the actual needs, and can focus on three-dimensional reconstruction of the target structure The key local information can effectively separate the foreground and background, thereby improving the reconstruction quality, avoiding the disadvantages of the three-dimensional reconstruction interpolation algorithm in the frequency domain, and achieving relatively good results when there are few images.

The aforementioned masks include 3D local masks, 2D local masks and/or planar foreground and background masks.

The above-mentioned 3D local mask includes an inscribed spherical mask or a weighted mask, and the 2D local mask is determined based on the 3D local mask. Based on the projection parameters, space transformation and projection calculation are performed on the 3D local mask to obtain a 2D local mask.

The above planar foreground and background masks are used to distinguish the foreground area and the background area of the projected image.

The explicit representation of the above three-dimensional reconstruction target structure includes: encoding the Coulomb potential energy value of the three-dimensional structure through the grid points of the three-dimensional lattice matrix; the implicit representation of the three-dimensional reconstruction target structure includes: using neural network representation to map the spatial coordinates to A function of the Coulomb potential energy value of the three-dimensional structure.

The above-mentioned predictive projection image output module is also used to perform spatial transformation and subsequent processing on the projection field of view of the machine learning representation of the three-dimensional reconstruction target structure inside the decoder, and output the predictive projection image.

The above loss value calculation module is used to perform mask processing on the input projection image based on the 2D local mask and the planar foreground background mask to obtain the input projection image after mask processing; based on the planar foreground and background mask, the predicted projection image is Mask processing, obtaining a predicted projection image after mask processing; determining a loss value of a designated area of the input projection image based on the input projection image after mask processing and the predicted projection image.

The input projected image processed by the above mask includes the specified area and other areas; the above device further includes: a noise suppression module, which is used to suppress noise in other areas by means of regularization.

The above-mentioned device also includes: a projection parameter prediction encoder processing module, which is used to input the first electron microscope image of the cryo-electron microscope into the projection parameter prediction encoder, and output the space transformation parameters of the first electron microscope image; wherein, the space transformation parameters represent the first The projection parameters of an electron microscope image; input the space transformation parameters into the cryo-electron microscope real space decoder, and perform training iterations; wherein, the loss value generated by the training iterations is passed back to the encoder and decoder through the machine learning calculation graph, and based on the training The loss values generated by the iterations update the parameters of the encoder and decoder.

Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the training device of the cryo-electron microscope real-space decoder described above can refer to the implementation of the aforementioned training method of the cryo-electron microscope real-space decoder The corresponding process in the example will not be repeated here.

Embodiment four:

The embodiment of the present invention also provides an electronic device for running the training method of the cryo-electron microscope real space decoder; refer to the schematic structural diagram of an electronic device shown in FIG. 6 , the electronic device includes a memory 100 and a processor 101 , wherein the memory 100 is used to store one or more computer instructions, and one or more computer instructions are executed by the processor 101 to implement the above-mentioned training method of the cryo-electron microscope real-space decoder.

Further, the electronic device shown in FIG. 6 further includes a bus 102 and a communication interface 103 , and the processor 101 , the communication interface 103 and the memory 100 are connected through the bus 102 .

Wherein, the memory 100 may include a high-speed random access memory (RAM, Random Access Memory), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the Internet, wide area network, local network, metropolitan area network, etc. can be used. The bus 102 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus can be divided into address bus, data bus, control bus and so on. For ease of representation, only one double-headed arrow is used in FIG. 6 , but it does not mean that there is only one bus or one type of bus.

The processor 101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 101 or instructions in the form of software. The above-mentioned processor 101 can be a general-purpose processor, including a central processing unit (Central Processing Unit, referred to as CPU), a network processor (Network Processor, referred to as NP), etc.; it can also be a digital signal processor (Digital Signal Processor, referred to as DSP) , Application Specific Integrated Circuit (ASIC for short), Field Programmable Gate Array (Field-Programmable Gate Array, FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Various methods, steps and logic block diagrams disclosed in the embodiments of the present invention may be implemented or executed. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like. The steps of the methods disclosed in the embodiments of the present invention may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register. The storage medium is located in the memory 100, and the processor 101 reads the information in the memory 100, and completes the steps of the methods in the foregoing embodiments in combination with its hardware.

An embodiment of the present invention also provides a computer-readable storage medium, the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are invoked and executed by a processor, the computer-executable instructions cause the processor to implement For the training method of the cryo-electron microscope real-space decoder, see the method embodiment for specific implementation, and details are not repeated here.

The computer program product of the training method, device and electronic equipment of the cryo-electron microscope real-space decoder provided by the embodiments of the present invention includes a computer-readable storage medium storing program codes, and the instructions included in the program codes can be used to execute the foregoing method embodiments For the specific implementation of the method, please refer to the method embodiment, which will not be repeated here.

Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the system and/or device described above can refer to the corresponding process in the foregoing method embodiment, and details are not repeated here.

In addition, in the description of the embodiments of the present invention, unless otherwise specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection or a detachable connection , or integrally connected; it may be mechanically connected or electrically connected; it may be directly connected or indirectly connected through an intermediary, and it may be the internal communication of two components. Those of ordinary skill in the art can understand the specific meanings of the above terms in the present invention in specific situations.

If the functions described above are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes. .

In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" etc. The indicated orientation or positional relationship is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the referred device or element must have a specific orientation, or in a specific orientation. construction and operation, therefore, should not be construed as limiting the invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and should not be construed as indicating or implying relative importance.

Finally, it should be noted that: the above-described embodiments are only specific implementations of the present invention, used to illustrate the technical solutions of the present invention, rather than limiting them, and the scope of protection of the present invention is not limited thereto, although referring to the foregoing The embodiment has described the present invention in detail, and those skilled in the art should understand that any person familiar with the technical field can still modify the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention Changes can be easily thought of, or equivalent replacements are made to some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of the present invention within the scope of protection. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (13)

1. A training method of a real-space decoder of a cryoelectron microscope is characterized by comprising the following steps:

acquiring an input projection image of a cryoelectron microscope and label information of the input projection image from a sample set; wherein the inputting of the label information of the projection image includes: inputting projection parameters and a mask of a projection image, wherein the projection parameters comprise a projection angle and image translation;

inputting the projection parameters of the input projection image into a decoder, performing spatial transformation on a projection visual field of a three-dimensional reconstruction target structure in the decoder in a machine learning representation form, and outputting a prediction projection image; wherein the machine learning representation comprises an explicit representation and an implicit representation;

masking the machine learning representation of the three-dimensional reconstructed target structure, the predicted projection image and the input projection image through the mask;

calculating a loss value based on the input projection image and the predicted projection image after the mask processing, and adjusting a parameter of the decoder based on the loss value;

and repeatedly acquiring the input projection image of the cryoelectron microscope and the label information of the input projection image from the sample set until a preset training condition is met, so as to obtain the trained decoder.

2. The method of claim 1, wherein the mask comprises a 3D local mask, a 2D local mask, and/or a planar foreground background mask.

3. The method of claim 2, wherein the 3D local mask comprises an inscribed sphere mask or a weighted mask, and wherein the 2D local mask is determined based on the 3D local mask.

4. The method of claim 2, wherein the 2D local mask is obtained by performing spatial transformation and projection calculation on the 3D local mask based on projection parameters.

5. The method of claim 2, wherein the planar foreground background mask is used to distinguish between foreground and background regions of the projected image.

6. The method of claim 1, wherein the explicit representation of the three-dimensional reconstructed target structure comprises: encoding a coulombic potential value of a three-dimensional structure through lattice points of a three-dimensional lattice point matrix;

the implicit representation of the three-dimensional reconstructed target structure comprises: a function mapping the spatial coordinates to the coulomb potential value of the three-dimensional structure is characterized by a neural network.

7. The method of claim 1, wherein after the step of inputting projection parameters of the input projection image into a decoder, the method further comprises:

and carrying out spatial transformation and subsequent processing on the projection visual field of the three-dimensional reconstruction target structure in the decoder in a machine learning representation form, and outputting the prediction projection image.

8. The method of claim 2, wherein the step of calculating a loss value based on the masked input projection image and the predicted projection image comprises:

masking the input projection image based on the 2D local mask and a plane foreground background mask to obtain the masked input projection image;

masking the predicted projection image based on the plane foreground background mask to obtain the masked predicted projection image;

determining a loss value of a designated region of the input projection image based on the masked input projection image and the predicted projection image.

9. The method according to claim 8, wherein the input projection image after mask processing includes a designated area and other areas; after the step of obtaining the masked input projection image, the method further includes:

and suppressing the noise of the other areas in a regularization mode.

10. The method of claim 1, further comprising:

inputting a first electron microscope image of a cryoelectron microscope into a projection parameter prediction encoder, and outputting a spatial transformation parameter of the first electron microscope image; the space transformation parameters represent projection parameters of the first electron microscope image;

inputting the space transformation parameters into the real space decoder of the refrigeration electron microscope, and performing training iteration;

and transmitting the loss values generated by the training iteration to the encoder and the decoder through a machine learning calculation graph, and updating the parameters of the encoder and the decoder based on the loss values generated by the training iteration.

11. A training device for a real-space decoder of a cryoelectron microscope, the device comprising:

the system comprises an input projection image acquisition module, a data acquisition module and a data processing module, wherein the input projection image acquisition module is used for acquiring an input projection image of a cryoelectron microscope and label information of the input projection image from a sample set; wherein the label information of the input projection image includes: inputting projection parameters and a mask of a projection image, wherein the projection parameters comprise a projection angle and image translation;

the predicted projection image output module is used for inputting the projection parameters of the input projection image into a decoder, performing spatial transformation on a projection visual field of a three-dimensional reconstruction target structure in the decoder in a machine learning representation form, and outputting a predicted projection image; wherein the machine-learned representation comprises an explicit representation and an implicit representation;

the mask processing module is used for performing mask processing on a machine learning representation form of a three-dimensional reconstruction target structure, the prediction projection image and the input projection image through the mask;

a loss value calculation module for calculating a loss value based on the input projection image and the predicted projection image after the mask processing, and adjusting a parameter of the decoder based on the loss value;

and the decoder training completion module is used for repeatedly executing the step of obtaining the input projection image of the cryoelectron microscope and the label information of the input projection image from the sample set until a preset training condition is met, so as to obtain a trained decoder.

12. An electronic device comprising a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the training method of a cryoelectron microscopy real-space decoder as defined in any one of claims 1 to 10.

13. A computer-readable storage medium storing computer-executable instructions that, when invoked and executed by a processor, cause the processor to carry out the training method of a cryoelectron microscopy real-space decoder according to any one of claims 1 to 10.

Images