JP7711764B2 - Image processing method and image processing device - Google Patents

Description

Embodiments of the present disclosure relate generally to the field of image processing, and more particularly to methods, apparatus, and computer-readable media for image processing.

In real clinical practice, accurate diagnosis requires considering all aspects of evidential data. For example, in transnasal endoscopy diagnosis, physicians often manually switch between different light sources at the same camera position to examine suspicious lesions or tumors. Lesions or tumors under different light sources may exhibit different features, which can provide a large amount of information useful for physicians to make more accurate and reliable decisions. Therefore, it is reasonable to assume that these features also contribute to improving the performance of image processing tasks (e.g., semantic segmentation, instance segmentation, and/or object identification) on transnasal endoscopy images.

Overall, exemplary embodiments of the present disclosure provide methods, apparatus, and computer-readable media for image processing.

In a first aspect, a method of image processing is provided. The method includes obtaining a plurality of images of an object captured under different illuminants, the plurality of images including a target image and at least one related image, and generating a segmentation label for the target image based on segmentation results of the plurality of images.

In a second aspect, a method of image processing is provided, the method including obtaining a plurality of images of an object captured under different illuminants, registering the plurality of images, and identifying the object based on the registered plurality of images.

In a third aspect, an apparatus for image processing is provided. The apparatus includes at least one processor configured to obtain a plurality of images of an object captured under different illuminants, the plurality of images including a target image and at least one related image, and generate a segmentation label for the target image based on segmentation results of the plurality of images.

In a fourth aspect, an apparatus for image processing is provided. The apparatus includes at least one processor configured to obtain a plurality of images of an object captured under different illuminants, register the plurality of images, and identify the object based on the registered plurality of images.

In a fifth aspect, a computer-readable storage medium is provided having instructions stored thereon that, when executed on at least one processor, cause the at least one processor to perform a method according to the first or second aspect of the present disclosure.

In a sixth aspect, a computer program product is provided that includes machine executable instructions that, when executed on at least one processor, cause the at least one processor to perform a method according to the first or second aspect of the present disclosure.

It should be understood that this Summary is not intended to identify key or essential features of the embodiments of the present invention, nor to limit the scope of the present invention. Other features of the embodiments will be readily apparent from the following description.

The above and other objects, features and advantages of the present disclosure will become more apparent from a more detailed description of several embodiments of the present disclosure in the drawings, in which like reference numerals generally refer to like components within the embodiments of the present disclosure.
FIG. 1 illustrates an exemplary image processing system in which embodiments of the present invention may be implemented. FIG. 2 is a schematic diagram of image processing according to some embodiments of the present disclosure. FIG. 2 is a schematic diagram of image processing according to some embodiments of the present disclosure. FIG. 2 is a schematic diagram of image processing according to some embodiments of the present disclosure. FIG. 2 is a schematic diagram of image processing according to some embodiments of the present disclosure. FIG. 1 is a schematic diagram of image registration in accordance with some embodiments of the present disclosure. FIG. 2 illustrates an exemplary method of image processing according to some embodiments of the present disclosure. FIG. 2 illustrates an exemplary method of image processing according to some embodiments of the present disclosure. FIG. 1 is a schematic block diagram of an apparatus suitable for implementing embodiments of the present disclosure.

The principles of the present disclosure will now be described with reference to some exemplary embodiments. It should be understood that these embodiments are provided for illustrative purposes only, to aid those skilled in the art in understanding and implementing the present disclosure, and do not imply any limitations on the scope of the present disclosure. The disclosure described herein can be implemented in a variety of ways different from those described below.

In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

As used herein, the singular forms "a," "an," and "said" include the plural unless the context clearly indicates otherwise. The term "comprises" and variations thereof should be understood as open-ended terms meaning "including, but not limited to." The term "based on" should be understood as "based at least in part on." The terms "one embodiment" and "embodiment" should be understood as "at least one embodiment." The term "another embodiment" should be understood as "at least one other embodiment." The terms "first," "second," etc. may refer to different or the same object. The following may include other explicit and implicit definitions.

In some instances, values, procedures, or devices are referred to as "best," "lowest," "highest," "minimum," "maximum," etc. It will be understood that such descriptions are intended to indicate that selections may be made from among many functional alternatives used, and that such selections are not necessarily better, smaller, higher, or otherwise more preferred than other selections.

As used herein, a "neural network" or "network" can process inputs and provide corresponding outputs, and typically includes an input layer, an output layer, and one or more hidden layers between the input layer and the output layer. A neural network typically includes multiple layers connected in sequence, where the output of the previous layer is provided as the input of the next layer, the input layer accepts the input of the neural network, and the output of the output layer serves as the final output of the neural network. Each layer of a neural network includes one or more nodes (also referred to as processing nodes or neurons), and each node processes the input from the previous layer. Hereinafter, the terms "neural network", "model", "network" and "neural network model" can be used interchangeably.

As mentioned above, images of the same object captured under different light sources can exhibit different features and provide a large amount of useful information to improve the performance of image processing. Conventional image processing solutions can perform image segmentation (e.g., semantic segmentation, instance segmentation) or object identification on a single image to obtain processing results. However, conventional solutions cannot utilize these features or useful information to improve the performance of image processing. Furthermore, since images of the same object can be captured at different times under different light sources, the objects in the images may have slight distortions. In this case, performing image segmentation or object identification directly on the images may result in poor segmentation/identification accuracy.

Embodiments of the present disclosure provide a solution for image processing to solve the above problems and one or more other potential problems. In some embodiments, multiple images of an object captured under different light sources may be obtained, where the multiple images include a target image and at least one related image. A segmentation label may be generated for the target image based on the segmentation results of the multiple images. In this way, the segmentation accuracy of the target image may be improved by combining the segmentation results of the images of the object captured under different light sources to obtain a final segmentation result for the target image. In some other embodiments, multiple images of the object captured under different light sources may be obtained and registered. The object may be identified based on the registered multiple images. In this way, the accuracy of object identification may be improved by removing the effect of slight deformation of the object between different images by image registration.

Hereinafter, some exemplary embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be readily understood by those skilled in the art that the detailed description given in this specification with respect to these drawings is provided for illustrative purposes only and does not imply any limitation on the scope of the present disclosure.

1 is a diagram illustrating an exemplary image processing system 100 in which embodiments of the present invention may be implemented. As shown in FIG. 1, the system 100 may include an image acquisition device 110 and an image processing device 120. In some embodiments, the devices 110 and 120 may be implemented in different physical devices. Alternatively, the devices 110 and 120 may be implemented in the same physical device. It should be understood that the configuration of the system 100 is shown for illustrative purposes only and does not imply any limitation on the scope of the present disclosure. The embodiments of the present disclosure may also be applied to other systems having different configurations.

The image acquisition device 110 may collect images 101 to be processed by the image processing device 120. In some embodiments, the image acquisition device 110 may collect multiple images of an object captured under different light sources at the same camera position. For example, the different light sources may be associated with different wavelengths or different combinations of wavelengths. In some embodiments, the image acquisition device 110 may be a medical support device or an endoscopy support device. The images 101 may be medical images, such as endoscopy images. For example, as described above, in a transnasal endoscopy diagnosis, a physician may manually switch between different light sources at the same camera position to examine a suspected lesion and capture multiple images of a nasal lesion. The images 101 collected by the image acquisition device 110 may be provided to the image processing device 120. The image processing device 120 may process the images 101 to generate an image processing result 102.

In some embodiments, the image processing device 120 may perform an image segmentation task. For example, the multiple images of the same object may include a target image and at least one related image. The image processing device 120 may generate a segmentation label for the target image based on the segmentation results of the multiple images. The image processing result 102 may indicate the segmentation label for the target image. As used herein, the "segmentation result" of an image may indicate the respective probability that each pixel in the image belongs to different predefined categories. For example, the segmentation result may be represented as a heat map in which the brightness of a pixel is used to indicate the probability that the pixel belongs to a category. The "segmentation label" of an image may indicate that each pixel in the image belongs to one of the predefined categories. For example, the segmentation label may be represented as a vector or array indicating the corresponding category of each pixel, or may be represented as a visual image in which pixels of different categories are identified with different colors.

In semantic segmentation, the segmentation result of an image may indicate the respective probability that each pixel in the image belongs to different predefined semantic categories. The segmentation label of the image may indicate that each pixel in the image belongs to one of the predefined semantic categories. Examples of semantic categories may include, but are not limited to, background, person, animal, vehicle, etc. In instance segmentation, the segmentation result of an image may indicate the respective probability that each pixel in the image belongs to different predefined instance categories. The segmentation label of the image may indicate that each pixel in the image belongs to one of the predefined instance categories. For example, a semantic segmentation network may classify pixels corresponding to different people in an image into the same semantic category, e.g., person. However, the instance segmentation network may classify these pixels into different instance categories corresponding to different people. In the following, some embodiments are described with reference to semantic segmentation. However, it should be understood that these embodiments are also applicable to instance segmentation.

2A-2C are schematic diagrams of image processing according to some embodiments of the present disclosure. As shown in FIGS. 2A-2C, the image processing device 120 may include a registration module 121 and a segmentation module 122. The segmentation module 122 may be a semantic segmentation module or an instance segmentation module. In the example shown in FIGS. 2A-2C, for example, the segmentation module 122 is a semantic segmentation module. The images processed by the image processing device 120 may include images 201, 202, and 203 of nasal lesions captured under different light sources.

In FIG. 2, for example, image 201 may be selected as a target image on which semantic segmentation is performed. Images 202 and 203 are related images. Registration module 121 may register related images 202 and 203 to target image 201 to obtain registration images 204 and 205. In some embodiments, for each of related images 202 and 203, registration module 121 may generate a transformed image by registering the related image to target image 201 based on a first transformation, and then generate a registered image by registering the transformed image to target image 201 based on a second transformation. In some embodiments, the first transformation may be an affine transformation or a rigid transformation. Examples of the first transformation may include, but are not limited to, translation, rotation, scaling, and the like. In some embodiments, the second transformation may be a deformable transformation or a non-rigid transformation. For example, the second transformation may refer to a transformation of pixels or content of the image. In some embodiments, the registration module 121 may use a trained image registration network to register the related images 202 and 203 to the target image 201, as described below with reference to FIG. 4.

As shown in FIG. 2A, the target image 201 and the registration images 204 and 205 may be input to the segmentation module 122. The segmentation module 122 may perform semantic segmentation on the images 201, 204, and 205 to generate semantic segmentation results thereof. The semantic segmentation may be performed by using a trained semantic segmentation network or by using any other suitable algorithm now known or developed in the future. The segmentation module 122 may generate the final segmentation result 231 by combining the semantic segmentation results of the images 201, 204, and 205 based on the weights of the images 201, 204, and 205. In some embodiments, the respective weights associated with the images 201, 204, and 205 may be predetermined. For example, the weight associated with the target image 201 may be the highest because no transformation is performed on the target image 201. The weights associated with the registered image 204 and the weights associated with the registered image 205 may be the same or different. The segmentation module 122 may determine a weighted sum of the semantic segmentation results of the images 201, 204, and 205 as a final segmentation result 211. The final segmentation result 211 may be input to an argmax function 123 to generate a semantic segmentation label 212 for the target image 201. For example, as described above, the semantic segmentation label 212 may indicate a semantic category of each of the pixels in the target image 201. In some embodiments, the semantic segmentation label 212 may be provided to a doctor or an automated diagnostic system for disease diagnosis.

In some embodiments, for a group of images of the same object captured under different light sources, each image can be selected as a target image, so that a group of segmentation labels can be generated for the group of images. In some embodiments, the group of segmentation labels can be provided directly to a doctor or an automated diagnostic system for disease diagnosis. Alternatively, the group of segmentation labels can be combined in a suitable manner and then provided to a doctor or an automated diagnostic system for disease diagnosis.

2B, for example, image 203 may be selected as the target image, and images 202 and 201 may be selected as related images. Registration module 121 may register related images 202 and 201 to target image 203 to obtain registration images 206 and 207. Segmentation module 122 may perform semantic segmentation on images 203, 206, and 207 and combine the semantic segmentation results into a final segmentation result 221. The final segmentation result 221 may be input to argmax function 123 to generate a semantic segmentation label 222 for target image 203. As shown in FIG. 2C, for example, image 202 may be selected as the target image, and images 203 and 201 may be selected as related images. The registration module 121 may register the related images 203 and 201 to the target image 202 to obtain registered images 208 and 209. The segmentation module 122 may perform semantic segmentation on the images 202, 208, and 209 and combine the semantic segmentation results into a final segmentation result 231. The final segmentation result 231 may be input to an argmax function 123 to generate a semantic segmentation label 232 for the target image 203.

For example, the semantic segmentation labels 212, 222 and 232 may be provided directly to a doctor or an automated diagnostic system for disease diagnosis. Alternatively, the semantic segmentation labels 212, 222 and 232 may be combined in a suitable manner and then provided to a doctor or an automated diagnostic system for disease diagnosis.

In some embodiments, the image processing device 120 shown in FIG. 1 may perform an object identification task. For example, in response to obtaining multiple images of an object captured under different light sources, the image processing device 120 may register the multiple images and identify the object based on the registered multiple images.

Figure 3 is a schematic diagram of image processing according to some embodiments of the present disclosure. As shown in Figure 3, the image processing device 120 may include the same registration module 121 as in Figures 2A-2C and the object identification module 124. The images processed by the image processing device 120 may include images 301, 302, and 303 of nasal lesions captured under different light sources.

3, for example, image 301 may be selected as a target image. Images 302 and 303 are related images. Registration module 121 may register related images 302 and 303 to target image 301 to obtain registration images 304 and 305. Images 301, 304, and 305 may be input to object identification module 124. In some embodiments, object identification module 124 may identify objects based on images 301, 304, and 305. For example, object identification module 124 may perform object identification on each of images 301, 304, and 305 by using a trained object identification network or by using any suitable algorithm now known or developed in the future to obtain their respective object identification results. Then, object identification module 124 may combine the object identification results into a final object identification result 306 based on their respective weights. Alternatively, in some embodiments, the object identification module 124 may perform semantic segmentation or instance segmentation on each of the images 301, 304, and 305 to obtain their respective segmentation results, and combine the segmentation results into a final segmentation result based on their respective weights. Then, the object identification module 124 may identify objects based on the final segmentation results to obtain a final object identification result 306. For example, the final object identification result 306 may be provided to a doctor or an automatic diagnostic system for disease diagnosis.

In some embodiments, for a group of images of the same object captured under different light sources, each image can be selected as a target image, so that a group of object identification results can be generated for the group of images. In some embodiments, the group of object identification results can be provided directly to a doctor or an automated diagnostic system for disease diagnosis. Alternatively, the group of object identification results can be combined in an appropriate manner and then provided to a doctor or an automated diagnostic system for disease diagnosis.

4 is a schematic diagram of image registration according to some embodiments of the present disclosure. FIG. 4 illustrates an image registration network 400 that can be used in the registration network 121 illustrated in FIGS. 2A-2C and 3. The image registration network 400 includes sub-networks 410 and 420. The sub-network 410 may be trained to register images based on a first transformation. As described above, the first transformation may be an affine transformation or a rigid transformation. Examples of the first transformation may include, but are not limited to, translation, rotation, scaling, and the like. The sub-network 410 may be trained to register images based on a second transformation. As described above, the second transformation may be a deformable transformation or a non-rigid transformation. For example, the second transformation may refer to a transformation of pixels or content of an image. The image registration network 400 may be trained based on a group of training image pairs, where each training image pair includes a fixed image and a translation image relative to the fixed image. The fixed and moving images in the group of training image pairs may have the same number of pixels.

As shown in FIG. 4, during the training phase, each training image pair including a fixed image 402 (hereinafter also denoted as “F”) and a moving image 401 (hereinafter also denoted as “M”) is input to a sub-network 410 to generate a first transformation function
For example, the first transformation function
is an affine transformation function or a rigid transformation function.
A first transformed image 403 can be generated by registering the moved image 401 to the fixed image 402 based on
The first transformed image 403 and the fixed image 402 may be input to a sub-network 420 to extract a second transformation function. For example, the second transformation function
is a deformable or non-rigid transformation function.
A second transformed image 404 can be generated by registering the first transformed image 403 to the fixed image 402 based on
It may be expressed as:

In some embodiments, a target loss for training the image registration network 400 may be determined based on the fixed image 402, the first transformed image 403, and the second transformed image 404. Network parameters of the image registration network 400 may be iteratively updated such that the target loss is minimized. In some embodiments, the target loss for training the image registration network 400 may be determined as a weighted sum of a group of losses.

As shown in FIG. 4, in some embodiments, the first similarity loss 441 is calculated by multiplying a fixed image 402 (i.e., F) and a first transformed image 403 (i.e.,
For example, the first similarity loss 441 may be expressed as:
Here, N represents the number of pixels contained in a single image (i.e., a fixed image or a moving image).

As shown in FIG. 4, in some embodiments, the second similarity loss 442 is calculated by multiplying the fixed image 402 (i.e., F) and the second transformed image 404 (i.e.,
For example, the second similarity loss 442 may be expressed as:

As shown in FIG. 4, in some embodiments, the third similarity loss 443 is calculated based on the first transformed image 403 (i.e.,
) and a second transformed image 404 (i.e.
For example, the third similarity loss 443 may be expressed as:
where
The third similarity loss
is to improve the registration accuracy.
and
and the backward similarity loss on

As shown in FIG. 4, in some embodiments, the spatial smoothness loss 444 is applied to the first transformed image 403 (i.e.,
) and a second transformation 432 (i.e.,
For example, the spatial smoothness loss 444 may be expressed as:
Here, the spatial smoothness loss is used to enforce spatially smooth deformations.
In some embodiments, the target loss L can be determined as a weighted sum of all the losses above, i.e.,
Where:
It is.

In view of the above, it can be seen that embodiments of the present disclosure provide a solution for image processing. According to some embodiments of the present disclosure, the accuracy of image segmentation (e.g., semantic segmentation or instance segmentation) for a target image can be improved by combining segmentation results of images for an object captured under different light sources to obtain a final segmentation result for the target image. Additionally, the accuracy of image segmentation and/or object identification can be improved by removing the effects of slight deformations of the object between different images through image registration.

FIG. 5 illustrates an exemplary method 500 of image processing according to some embodiments of the present disclosure. Method 500 may be implemented in image processing device 120 such as that shown in FIG. 1. It should be understood that method 500 may include additional blocks not shown and/or omit some of the blocks shown, and that the scope of the present disclosure is not limited in this respect.

In block 510, the image processing device 120 may obtain multiple images of the same object captured under different light sources, where the multiple images include a target image and at least one related image.

In block 520, the image processing device 120 may generate a segmentation label for the target image based on the segmentation results of the multiple images.

In some embodiments, the method 500 may further include generating at least one registered image by registering the at least one related image to the target image, and generating the segmentation result by performing semantic segmentation or instance segmentation on the target image and the at least one registered image.

In some embodiments, generating the at least one registration image includes, for each related image of the at least one related image, generating a transformed image by registering the at least one related image to the target image based on a first transformation, and generating a registration image by registering the transformed image to the target image based on a second transformation.

In some embodiments, the first transformation is an affine transformation and the second transformation is a deformable transformation.

In some embodiments, registering the at least one related image to the target image includes registering the at least one related image to the target image by using a trained image registration network.

In some embodiments, the method 500 may further include training the image registration network based on a group of training image pairs, where each training image pair includes a fixed image and a translation image relative to the fixed image.

In some embodiments, training the image registration network includes generating a first transformed image by registering the moving image to the fixed image based on an affine transformation, generating a second transformed image by registering the first transformed image to the fixed image based on a deformable transformation, determining a target loss for training the image registration network based on the fixed image, the first transformed image, and the second transformed image, and training the image registration network such that the target loss is minimized.

In some embodiments, determining the target loss includes determining a first similarity loss based on the fixed image and the first transformed image, determining a second similarity loss based on the fixed image and the second transformed image, determining a spatial smoothness loss based on the first transformed image and a function corresponding to the deformable transformation, determining a third similarity loss based on the first transformed image and the second transformed image, and determining the target loss based on a weighted sum of the first similarity loss, the second similarity loss, the spatial smoothness loss, and the third similarity loss.

In some embodiments, performing semantic segmentation or instance segmentation on the target image and the at least one registration image includes performing the semantic segmentation on the target image and the at least one registration image by using a trained semantic segmentation network, or performing the instance segmentation on the target image and the at least one registration image by using a trained instance segmentation network.

In some embodiments, generating the semantic segmentation label for the target image includes determining a final segmentation result for the target image based on a weighted sum of the segmentation results, and generating the segmentation label for the target image based on the final segmentation result.

In some embodiments, different light sources are associated with different wavelengths or different combinations of wavelengths.

FIG. 6 illustrates an exemplary method 600 of image processing according to some embodiments of the present disclosure. Method 600 may be implemented in an image processing device 120 such as that shown in FIG. 1. It should be understood that method 500 may include additional blocks not shown and/or omit some of the blocks shown, and the scope of the present disclosure is not limited in this respect.

In block 610, the image processing device 120 may acquire multiple images of the same object captured under different light sources.

In block 620, the image processing device 120 may register the multiple images.

In block 630, the image processing device 120 may identify the object based on the registered images.

In some embodiments, the plurality of images includes a target image and at least one related image, and registering the plurality of images includes generating at least one registered image by registering the at least one related image to the target image, where the registered plurality of images includes the at least one registered image and the target image.

In some embodiments, identifying the object based on the registered images includes performing semantic or instance segmentation on the target image and the at least one registered image to generate a segmentation result, and identifying the object based on the segmentation result.

In some embodiments, generating the at least one registration image includes, for each related image of the at least one related image, generating a transformed image by registering the at least one related image to the target image based on a first transformation, and generating a registration image by registering the transformed image to the target image based on a second transformation.

In some embodiments, the first transformation is an affine transformation and the second transformation is a deformable transformation.

In some embodiments, registering the at least one related image to the target image includes registering the at least one related image to the target image by using a trained image registration network.

In some embodiments, the method 600 may further include training the image registration network based on a group of training image pairs, where each training image pair includes a fixed image and a translation image relative to the fixed image.

In some embodiments, training the image registration network includes generating a first transformed image by registering the moving image to the fixed image based on an affine transformation, generating a second transformed image by registering the first transformed image to the fixed image based on a deformable transformation, determining a target loss for training the image registration network based on the fixed image, the first transformed image, and the second transformed image, and training the image registration network such that the target loss is minimized.

In some embodiments, determining the target loss includes determining a first similarity loss based on the fixed image and the first transformed image, determining a second similarity loss based on the fixed image and the second transformed image, determining a spatial smoothness loss based on the first transformed image and a function corresponding to the deformable transformation, determining a third similarity loss based on the first transformed image and the second transformed image, and determining the target loss based on a weighted sum of the first similarity loss, the second similarity loss, the spatial smoothness loss, and the third similarity loss.

In some embodiments, performing semantic segmentation or instance segmentation on the target image and the at least one registration image includes performing the semantic segmentation on the target image and the at least one registration image by using a trained semantic segmentation network, or performing the instance segmentation on the target image and the at least one registration image by using a trained instance segmentation network.

In some embodiments, generating the semantic segmentation label for the target image includes determining a final segmentation result for the target image based on a weighted sum of the segmentation results, and generating the segmentation label for the target image based on the final segmentation result.

In some embodiments, different light sources are associated with different wavelengths or different combinations of wavelengths.

7 is a schematic block diagram of an apparatus 700 that can be used to implement an embodiment of the present disclosure. For example, the image acquisition device 110 and/or the image processing device 120 can be implemented by the apparatus 700. For example, the apparatus 700 can be used to implement a medical support device or an endoscopy support device that can capture images of a suspicious lesion or tumor under different light sources. As shown in FIG. 7, the apparatus 700 includes a central processing unit (CPU) 701 that can perform various suitable operations and processes based on computer program instructions stored in a read-only memory (ROM) 702 or uploaded from a storage unit 708 to a random access memory (RAM) 703. The RAM 703 further stores various programs and data required for the operation of the apparatus 700. The CPU 701, the ROM 702, and the RAM 703 are interconnected via a bus 704. An input/output (I/O) interface 705 is also connected to the bus 704.

The I/O interface 705 is connected to components including an input section 706 such as a keyboard and a mouse, an output section 707 such as various displays and speakers, a storage section 708 such as a magnetic disk and an optical disk, and a communication section 709 such as a network card, a modem, a wireless communication transceiver, etc. The communication section 709 enables the device 700 to exchange data/information with other devices via a computer network such as the Internet and/or a telecommunications network.

The above-mentioned methods or processes, e.g., methods 500 and/or 600, may be executed by the processing unit 701. For example, in some implementations, the method 500 may be implemented as a computer software program tangibly contained in a machine-readable medium, such as the storage unit 708. In some implementations, the computer program may be partially or fully loaded and/or mounted on the device 700 by the ROM 702 and/or the communication unit 709. When the computer program is uploaded to the RAM 703 and executed by the CPU 701, it may perform one or more steps of the above-mentioned methods 500 and/or 600.

In some embodiments, the image processing device comprises a circuit configured to obtain a plurality of images of an object captured under different illuminants, the plurality of images including a target image and at least one related image, and generate a segmentation label for the target image based on segmentation results of the plurality of images.

In some embodiments, the circuitry is further configured to generate at least one registered image by registering the at least one related image to the target image, and to generate the segmentation result by performing semantic segmentation or instance segmentation on the target image and the at least one registered image.

In some embodiments, the image processing device includes circuitry configured to obtain a plurality of images of an object captured under different illuminants, register the plurality of images, and identify the object based on the registered plurality of images.

In some embodiments, the plurality of images includes a target image and at least one related image, and the circuitry is further configured to generate at least one registered image by registering the at least one related image to the target image, where the registered plurality of images includes the at least one registered image and the target image.

In some embodiments, the circuitry is further configured to perform semantic or instance segmentation on the target image and the at least one registered image to generate a segmentation result and identify the object based on the segmentation result.

In some embodiments, the circuitry is further configured to, for each related image of the at least one related image, generate a transformed image by registering the at least one related image to the target image based on a first transformation, and generate a registered image by registering the transformed image to the target image based on a second transformation.

In some embodiments, the first transformation is an affine transformation and the second transformation is a deformable transformation.

In some embodiments, the circuitry is further configured to register the at least one related image to the target image by using a trained image registration network.

In some embodiments, the circuitry is further configured to train the image registration network based on a group of training image pairs, where each training image pair includes a fixed image and a translation image relative to the fixed image.

In some embodiments, the circuitry is further configured to generate a first transformed image by registering the moving image to the fixed image based on an affine transformation, generate a second transformed image by registering the first transformed image to the fixed image based on a deformable transformation, determine a target loss for training the image registration network based on the fixed image, the first transformed image, and the second transformed image, and train the image registration network such that the target loss is minimized.

In some embodiments, the circuitry is further configured to determine a first similarity loss based on the fixed image and the first transformed image, determine a second similarity loss based on the fixed image and the second transformed image, determine a spatial smoothness loss based on the first transformed image and a function corresponding to the deformable transformation, determine a third similarity loss based on the first transformed image and the second transformed image, and determine the target loss based on a weighted sum of the first similarity loss, the second similarity loss, the spatial smoothness loss, and the third similarity loss.

In some embodiments, the circuitry is further configured to perform the semantic segmentation of the target image and the at least one registered image by using a trained semantic segmentation network, or to perform the instance segmentation of the target image and the at least one registered image by using a trained instance segmentation network.

In some embodiments, the circuitry is further configured to determine a final segmentation result for the target image based on the weighted sum of the segmentation results, and generate the segmentation label for the target image based on the final segmentation result.

In some embodiments, different light sources are associated with different wavelengths or different combinations of wavelengths.

The present disclosure may be realized as a system, a method, and/or a computer program product. When the present disclosure is implemented as a system, the components described herein may be realized in the form of a cloud computing architecture in addition to being realized on a single device. In a cloud computing environment, these components may be located remotely and work together to realize the functions described in the present disclosure. Cloud computing may provide computing, software, data access, and storage services. The physical location and configuration of the systems and hardware that provide these services need not be known to the end user. Cloud computing may provide services over a wide area network (e.g., the Internet) using an appropriate protocol. For example, a cloud computing provider may provide applications over a wide area network, which may be accessed through a browser or any other computing component. Cloud computing components and corresponding data may be stored on a remote server. Computing resources in a cloud computing environment may be centralized in a remote data center, or these computing resources may be distributed. A cloud computing infrastructure may provide services through these shared data centers, even if the shared data centers appear to users as a single access point. Thus, a cloud computing architecture may be used to provide various functions described herein from a remote service provider. Alternatively, these functions may be provided from a conventional server or may be installed directly or otherwise on a client device. Additionally, the present disclosure may be embodied as a computer program product. The computer program product may include a computer-readable storage medium loaded with computer-readable program instructions for carrying out various aspects of the present disclosure.

A computer-readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures in grooves in which instructions are recorded, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transitory signal, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse traveling through an optical cable), or an electrical signal transmitted over a wire.

The computer readable program instructions described herein may be downloaded from a computer readable storage medium to a corresponding computing/processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in the respective computing/processing device.

The computer-readable program instructions for carrying out the operations of the present disclosure may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, and traditional procedural programming languages such as the "C" programming language. The computer-readable program instructions may run entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry to perform aspects of the present disclosure.

Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, may be implemented by computer readable program instructions.

These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that the instructions, which execute through the processor of the computer or other programmable data processing device, generate means for implementing the functions/operations specified in one or more blocks of the flowcharts and/or block diagrams to produce a machine. These computer-readable program instructions may be stored on a computer-readable storage medium capable of directing a computer, programmable data processing device, and/or other device to function in a particular manner, such that the computer-readable storage medium storing the instructions includes a product including instructions implementing aspects of the functions/operations defined in one or more blocks of the flowcharts and/or block diagrams.

The computer readable program instructions may be loaded into a computer, other programmable data processing device, or other device to generate a computer implemented process such that a series of operational steps are executed on the computer, other programmable data processing device, or other device, thereby causing the instructions executing on the computer, other programmable data processing device, or other device to implement the functions/acts specified in the flowchart and/or block diagram blocks.

The flowcharts and block diagrams illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, snippet, or portion of code that includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions recorded in the blocks may occur in a different order than the order recorded in the diagram. For example, depending on the functionality involved, two blocks shown in succession may actually be executed substantially simultaneously, or these blocks may sometimes be executed in the reverse order. It should also be noted that each block in the block diagrams and/or flowchart diagrams, as well as combinations of blocks in the block diagrams and/or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs a particular function or operation, or a combination of dedicated hardware and computer instructions.

The description of various embodiments of the present disclosure has been presented for illustrative purposes, but is not intended to be an exhaustive or limiting description of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification have been selected to best explain the principles of the embodiments, practical applications or technical improvements of the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims (16)

1. A method of image processing, comprising the steps of:
obtaining a plurality of images of an object captured under different light sources, the plurality of images including a target image and at least one associated image;
generating a segmentation label for the target image based on the segmentation results of the plurality of images;
generating at least one registered image by registering the at least one related image to the target image;
performing semantic or instance segmentation on the target image and the at least one registered image to generate the segmentation result;
Including,
Registering the at least one related image to the target image includes:
registering the at least one related image to the target image by using a trained image registration network;
The method comprises:
training the image registration network based on a group of training image pairs;
Each training image pair includes a fixed image and a translation image relative to the fixed image;
Training the image registration network includes:
generating a first transformed image by registering the moved image to the fixed image based on an affine transformation;
generating a second transformed image by registering the first transformed image to the fixed image based on a deformable transformation;
determining a target loss for training the image registration network based on the fixed image, the first transformed image, and the second transformed image;
training the image registration network such that the target loss is minimized; and
Including,
Determining the target loss comprises:
determining a first similarity loss based on the fixed image and the first transformed image;
determining a second similarity loss based on the fixed image and the second transformed image;
determining a spatial smoothness loss based on the first transformed image and a function corresponding to the deformable transformation;
determining a third similarity loss based on the first transformed image and the second transformed image;
determining the target loss based on a weighted sum of the first similarity loss, the second similarity loss, the spatial smoothness loss, and the third similarity loss;
A method comprising : 1. A method of image processing, comprising the steps of:
Obtaining a plurality of images of an object captured under different light sources;
registering the plurality of images; and
identifying the object based on the registered images;
Including,
The plurality of images includes a target image and at least one related image, and registering the plurality of images includes:
generating at least one registered image by registering the at least one related image to the target image;
the registered plurality of images includes the at least one registered image and the target image;
Registering the at least one related image to the target image includes:
registering the at least one related image to the target image by using a trained image registration network;
The method comprises:
training the image registration network based on a group of training image pairs;
Each training image pair includes a fixed image and a translation image relative to the fixed image;
Training the image registration network includes:
generating a first transformed image by registering the moved image to the fixed image based on an affine transformation;
generating a second transformed image by registering the first transformed image to the fixed image based on a deformable transformation;
determining a target loss for training the image registration network based on the fixed image, the first transformed image, and the second transformed image;
training the image registration network such that the target loss is minimized; and
Including,
Determining the target loss comprises:
determining a first similarity loss based on the fixed image and the first transformed image;
determining a second similarity loss based on the fixed image and the second transformed image;
determining a spatial smoothness loss based on the first transformed image and a function corresponding to the deformable transformation;
determining a third similarity loss based on the first transformed image and the second transformed image;
determining the target loss based on a weighted sum of the first similarity loss, the second similarity loss, the spatial smoothness loss, and the third similarity loss;
A method comprising : Identifying the object based on the registered plurality of images includes:
performing semantic or instance segmentation on the target image and the at least one registered image to generate a segmentation result;
identifying the object based on the segmentation result;
3. The method of claim 2 comprising: Producing the at least one registration image includes:
For each related image of the at least one related image,
generating a transformed image by registering the at least one related image to the target image based on a first transformation;
generating a registered image by registering the transformed image to the target image based on a second transformation; and
The method according to claim 1 or 2 , comprising: The method of claim 4 , wherein the first transformation is an affine transformation and the second transformation is a deformation transformation. Performing semantic or instance segmentation on the target image and the at least one registered image includes:
performing the semantic segmentation on the target image and the at least one registered image by using a trained semantic segmentation network; or performing the instance segmentation on the target image and the at least one registered image by using a trained instance segmentation network;
The method according to claim 1 or 3 , comprising: Generating the segmentation labels for the target image includes:
determining a final segmentation result for the target image based on a weighted sum of the segmentation results;
generating the segmentation label for the target image based on the final segmentation result;
2. The method of claim 1, comprising: The method according to any one of the preceding claims, wherein the different light sources are associated with different wavelengths or different combinations of wavelengths. An image processing apparatus comprising at least one processor,
The at least one processor
acquiring a plurality of images of an object captured under different light sources, the plurality of images including a target image and at least one related image;
generating a segmentation label for the target image based on the segmentation results of the plurality of images;
generating at least one registered image by registering the at least one related image to the target image;
performing a semantic or instance segmentation on the target image and the at least one registered image to generate the segmentation result;
registering the at least one related image to the target image by using a trained image registration network;
configured to train the image registration network based on a group of training image pairs;
Each training image pair includes a fixed image and a translation image relative to the fixed image;
The at least one processor further comprises:
generating a first transformed image by registering the moved image to the fixed image based on an affine transformation;
generating a second transformed image by registering the first transformed image to the fixed image based on a deformable transformation;
determining a target loss for training the image registration network based on the fixed image, the first transformed image, and the second transformed image;
training the image registration network such that the target loss is minimized;
determining a first similarity loss based on the fixed image and the first transformed image;
determining a second similarity loss based on the fixed image and the second transformed image;
determining a spatial smoothness loss based on the first transformed image and a function corresponding to the deformable transformation;
determining a third similarity loss based on the first transformed image and the second transformed image;
determining the target loss based on a weighted sum of the first similarity loss, the second similarity loss, the spatial smoothness loss, and the third similarity loss;
4. An image processing device configured as follows: An image processing apparatus comprising at least one processor,
The at least one processor
Obtaining a plurality of images of an object captured under different light sources;
registering the plurality of images;
configured to identify the object based on the registered plurality of images;
the plurality of images includes a target image and at least one related image;
The at least one processor further comprises:
configured to generate at least one registered image by registering the at least one related image to the target image;
the registered plurality of images includes the at least one registered image and the target image;
The at least one processor further comprises:
registering the at least one related image to the target image by using a trained image registration network;
configured to train the image registration network based on a group of training image pairs;
Each training image pair includes a fixed image and a translation image relative to the fixed image;
The at least one processor further comprises:
generating a first transformed image by registering the moved image to the fixed image based on an affine transformation;
generating a second transformed image by registering the first transformed image to the fixed image based on a deformable transformation;
determining a target loss for training the image registration network based on the fixed image, the first transformed image, and the second transformed image;
training the image registration network such that the target loss is minimized;
determining a first similarity loss based on the fixed image and the first transformed image;
determining a second similarity loss based on the fixed image and the second transformed image;
determining a spatial smoothness loss based on the first transformed image and a function corresponding to the deformable transformation;
determining a third similarity loss based on the first transformed image and the second transformed image;
determining the target loss based on a weighted sum of the first similarity loss, the second similarity loss, the spatial smoothness loss, and the third similarity loss;
4. An image processing device configured as follows: The at least one processor further comprises:
performing a semantic or instance segmentation on the target image and the at least one registered image to generate a segmentation result;
The apparatus of claim 10 , configured to identify the object based on the segmentation result. The at least one processor further comprises:
For each related image of the at least one related image,
generating a transformed image by registering the at least one related image to the target image based on a first transformation;
11. Apparatus according to claim 9 or 10 , configured to generate a registered image by registering the transformed image to the target image based on a second transformation. The apparatus of claim 12 , wherein the first transformation is an affine transformation and the second transformation is a deformation transformation. The at least one processor further comprises:
12. The apparatus of claim 9 or 11, configured to: perform the semantic segmentation on the target image and the at least one registered image by using a trained semantic segmentation network; or perform the instance segmentation on the target image and the at least one registered image by using a trained instance segmentation network. The at least one processor further comprises:
determining a final segmentation result for the target image based on a weighted sum of the segmentation results;
The apparatus of claim 9 , configured to generate the segmentation label for the target image based on the final segmentation result. 16. Apparatus according to any one of claims 9 to 15 , wherein the different light sources are associated with different wavelengths or different combinations of wavelengths.