ES2997308T3 - Automated determination of a canonical pose of a 3d objects and superimposition of 3d objects using deep learning - Google Patents

Abstract

A method is described for automatically determining a canonical position of a 3D object represented by a 3D data set, the method comprising: providing one or more voxel blocks of a voxel representation of the 3D object associated with a first coordinate system as input to a first 3D deep neural network, the first 3D neural network being trained to generate canonical position information associated with a canonical coordinate system defined relative to a position of a portion of the 3D tooth structure; receiving canonical position information from the output of the first 3D deep neural network, the canonical position information for each voxel of the one or more blocks comprising a prediction of a position of the voxel in the canonical coordinate system, the position being defined by canonical coordinates; using the canonical coordinates to determine an orientation and scale of the axes of the canonical coordinate system and a position of the origin of the canonical coordinate system relative to the axis and origin of the first 3D coordinate system, and using the orientation and position to determine transformation parameters for transforming coordinates of the first coordinate system into canonical coordinates; and determining a canonical representation of the 3D dental structure, including determining and applying the transformation parameters to the coordinates of the voxels of the voxel representation or of the 3D data set used to determine the voxel representation. (Automatic translation with Google Translate, no legal value)

Description

description

Automated determination of a canonical pose of a 3D object and 3D object superposition using deep learning

Field of the invention

The invention relates to automated determination of a canonical pose of a 3D object, such as 3D dental structure, and 3D object superposition using deep learning; and in particular, though not exclusively, to methods and systems for automated determination of a canonical pose of a 3D object and to methods and systems for automated 3D object superposition, and to a computer program product that enables a computer system to perform such methods.

Background of the invention

Accurate 3D models of a patient's dentition and jawbones (maxilla and mandible) are essential for 3D computer-aided dental applications (orthodontic treatment planning, dental implant planning, orthognathic surgery (maxillary surgery) planning, etc.). The formation of such 3D models is based on 3D image data of the patient, typically 3D computed tomography (CT) data of a 3D object representing, for example, a dentomaxillofacial complex or other body part. A CT scan typically results in a voxel representation representing (part of) the 3D object, where each voxel is associated with an intensity value, typically a radiodensity of the scanned volume. In medical applications, such as dental applications, CT scans are commonly acquired using cone beam CT (CBCT), due to the lower radiation dose to the patient, the lower acquisition price of the equipment, and the greater ease of use compared to fan beam CT.

However, CBCT techniques are sensitive to artifacts (especially in the presence of metal), and there is no industry-wide standard for converting the CBCT scanner sensor output into radiation values representing the radiodensity in Hounsfield units (HU) in the scanned volume. Furthermore, the use of low doses results in relatively poor contrast, making it difficult to distinguish between structures in a 3D object, e.g., a dentomaxillofacial complex, that have similar densities. These problems can result in discrepancies in 3D models derived from such voxel representations using, e.g., a thresholding technique. Therefore, 3D models derived from a voxel representation of CBCT data are not suitable, or at least less suitable, for designing precisely fitting tooth-supported patterns as used, for example, in orthodontics (invisible aligner therapy), jaw surgery (orthognathic surgery), implant surgery (implantology), cosmetic dentistry (crowns, bridges), etc.

In order to address this problem, a voxel representation of a CBCT dataset or a 3D model derived from such a voxel representation can be complemented, augmented, and/or (partially) replaced by optical scanning data. Optical scanning data, such as intraoral scanning (IOS) data, are generated by a surface scan (e.g., structured light or laser) of tooth surfaces (usually the tooth crowns and the surrounding gingival surface) derived from a plaster model (or impression) of a patient's dentition, or by generating intraoral scanning (IOS) data of the patient's dentition. Compared to CBCT/CT data, the advantages are the absence of radiation during data acquisition and the higher spatial resolution. Typical accuracies of optical (extraoral) and IOS examinations are estimated to be within the range of 5–10 micrometers and 25–75 micrometers, respectively. However, the examination results do not distinguish between teeth (crowns) and gingival regions. Furthermore, no information beyond visible surfaces can be captured, in particular no information about tooth roots, jawbone, nerves, etc. Intraoral examinations can be supplemented by generalized model data, for example derived from a database of crown shapes with corresponding root shapes, to estimate the underlying structures. However, as such, a generalization does not take into account information that considers the true 3D shape of the desired volumes. Therefore, such model-based estimates are inherently inaccurate.

More generally, when processing 3D image data, e.g. to generate accurate 3D models, to repair missing data in a 3D dataset, but also to analyze and evaluate, e.g., (possible) treatment effect(s)/outcomes, or for the purpose of disease progression analysis, it is advantageous or even necessary to combine 3D image datasets from different sources. This may mean aligning one or more voxelized 3D datasets, such as CBCT datasets, and/or one or more 3D surface mesh or point cloud datasets, e.g., IOS datasets, of the same 3D object, e.g., the same tooth structure or bone structure, and merging the aligned sets to give a dataset that can be used to determine an accurate 3D model or to perform an analysis of the tooth structure. The procedure of aligning different image datasets is called image overlay or image registration. The superposition-registration problem therefore concerns finding a one-to-one mapping between one or more coordinate systems such that corresponding features of a model, for example 3D tooth structures, in the different coordinate systems map to each other. Combining the aligned data sets to give a data set representing the tooth structure is often referred to as fusion.

In CT and CBCT imaging, popular 3D superimposition techniques include point-based/landmark-based superimposition, surface-based/contour-based superimposition, and voxel-based superimposition. Examples of such techniques are described in GKANTIDIS, et al., Evaluation of 3-dimensional superimposition techniques on various skeletal structures of the head using surface models, PLoSOne 2015, Vol. I.10, No. 2; and JODA, et al., Systematic literature review of digital 3D superimposition techniques to create virtual dental patients, Int J Oral Maxillofacial Implants, 2015, March–April, Vol. I.

30, no. 2). Typically, these techniques require human intervention, e.g., human input.

The accuracy of point-based and surface-based superposition techniques depends on the accuracy of identifying landmarks and 3D surface models, respectively. This can be especially problematic in the presence of artifacts and low-contrast regions. When matching disparate data sets, it will be difficult to identify corresponding landmarks with sufficient accuracy. Point-based matching algorithms, such as iterative closest point (ICP), generally require user interaction to provide an initial state that is already relatively closely aligned. Voxel-based superposition can overcome some of the limitations of landmark-based and surface-based superposition techniques. This technique employs 3D volume information stored as voxel representations. Similarities between the superimposed 3D data (to be superimposed) can be inferred from the voxel-level intensities in corresponding reference structures. This technique is particularly challenging when combining low-contrast, unnormalized CBCT data from different sources, or when combining data from different imaging modalities (e.g., CT and MRI, or CBCT and binary 3D image data such as those derived from surface meshes enclosing or not enclosing a volume). Additional difficulties can arise when the data sets only partially overlap. State-of-the-art voxel-based superposition methods are often computationally expensive.

The large size of 3D datasets and the fact that clinical implementation requires a very strict standard on accuracy make it difficult to use conventional image overlay methods on high-dimensional medical images. With recent developments in deep learning, there have been some efforts to apply deep learning to the field of image registration. In one approach, deep learning is used to estimate a similarity metric, which is then used to drive an iterative optimization scheme. This is reported, for example, by Simonovsky et al., A Deep Metric for Multimodal Registration (MICCAI 2016) (Springer, Cham) pp. 10–18, where the problem is presented as a classification task, where a CNN is trained to distinguish between alignment and misalignment of two overlapping image fragments. In another approach, a deep regression (neural) network is used to predict transformation parameters between images. For example, EP3121789 describes a method that uses a deep neural network to directly predict transformation parameters between a 3D CT image and a 2D X-ray image. P2742857 discloses a method that registers features from shape data that represent one or more crowns with corresponding voxels from volumetric imaging of a patient's tooth. Furthermore, there is the article by Li et al., “Non-rigid image registration using fully convolutional networks with deep self-supervision,” September 3, 2017, in which a trained neural network is given two images and computes, for each pixel, a dx,dy,dx deformation that is used to register one image with respect to the other. This method requires two input images that already have a certain similarity and is therefore unable to handle 3D datasets from different modalities. Therefore, the problem of registering 3D data sets of different modalities (in terms of pose, data type, coordinate system, etc.) of a particular 3D object is not addressed in the prior art.

Large variations in these 3D datasets (in terms of data formats/modalities, coordinate systems, 3D object position and orientation, image data quality, different amounts of overlap between the structures present, etc.) that the superposition system must be able to cope with make the problem of accurate automated superposition of 3D objects (e.g., superimposing 3D dental structures without any human intervention) a significant task. Known superposition systems cannot address these problems in a reliable and robust manner. More generally, large variations in 3D datasets from different modalities pose a problem for accurate processing by a deep neural network system. This is not only a problem for accurate registration, but also for accurate segmentation and/or classification by a deep neural network.

Therefore, there is a need in the art for a method that can fully automatically, rapidly, and robustly superimpose 3D datasets of 3D objects, such as 3D dentomaxillofacial structures. More specifically, there is a need in the art for a solution where, for example, a dental expert can obtain superimposition results, required for any of a variety of purposes, without requiring any additional interaction or knowledge from said expert, with known accuracy and timeliness of results.

Summary of the invention

As will be appreciated by one of skill in the art, aspects of the present invention are implemented as a system, a method, or a computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be generally referred to herein as a “circuit,” “module,” or “system.” The functions described in this disclosure may be implemented as an algorithm executed by a microprocessor of a computer. Furthermore, aspects of the present invention may take the form of a computer program product implemented on one or more computer-readable media having computer-readable program code implemented, e.g., stored, therein.

Any combination of one or more computer-readable media may be used. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection having one or more cables, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPRO or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that can comprise, or store, a program for use by, or in connection with, a system, apparatus, or instruction execution device.

A computer-readable signal medium may include a propagated data signal with computer-readable program code implemented therein, for example, in baseband or as part of a carrier wave. Such propagated signal may take a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and that can communicate, propagate, or transport a program for use by, or in connection with, a system, apparatus, or instruction execution device.

Program code implemented on a computer-readable medium may be transmitted using any appropriate media, including, but not limited to, wireless, wired, fiber optic, cable, RF, etc., or any suitable combination thereof. Computer program code for performing operations for aspects of the present invention may be written in any combination of one or more programming languages, including a functional programming language, an object-oriented programming language such as Java(TM), Scala, C++, Python, or the like, and conventional procedural-oriented programming languages, such as the “C” programming language, or similar programming languages. The program code may execute entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer, server, or virtualized server. In the latter situation, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, via the Internet using an Internet service provider).

Aspects of the present invention are described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, may be implemented by computer program instructions. These computer program instructions may be provided to a processor, in particular a microprocessor, central processing unit (CPU), or graphics processing unit (GPU), of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus for producing a machine, such that the instructions, being executed by the processor of the computer, other programmable data processing apparatus, or other devices, create means for implementing the functions/actions specified in the block blocks of the flowchart and/or block diagram.

These computer program instructions may also be stored on a computer-readable medium that can direct a computer, or other programmable data processing apparatus, or other device, to operate in a particular manner, such that the instructions stored on the computer-readable medium produce an article of manufacture that includes instructions that implement the function/action specified in the block blocks of the flowchart and/or block diagram.

Computer program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause a series of operating steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented procedure such that the instructions executed on the computer or other programmable apparatus provide procedures for implementing the functions/actions specified in the block blocks of the flowchart and/or block diagram.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code, comprising one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions indicated in the blocks may occur out of the order indicated in the figures. For example, two blocks shown in succession may actually execute substantially simultaneously, or the blocks may sometimes execute in reverse order, depending on the functionality involved. It will also be noted that each block in the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, may be implemented by systems based on special-purpose hardware that perform the specified functions or actions, or combinations of computer instructions and special-purpose hardware.

In this application, “image” may refer to a data set comprising information in two, three, or more spatial dimensions. “3D image data” may refer to any kind of three-dimensional data set, for example voxel intensity values, surface mesh definitions, etc. Adjusting the coordinate system of one or more images to match that of another reference image comprising (portions of) the same structures is variously known as data or image registration, matching, superposition, or alignment. Except where the context requires otherwise, these terms (and other words derived from these terms) are used interchangeably. “Transformation (parameter) set” is, in this context, a generic term for information on how to rotate, translate, and/or scale a data set in order to superimpose it on another data set or to represent the data set in an alternative coordinate system; it may be represented by a single matrix, but also, for example, by a collection of matrices, vectors, and/or scalars.

In one aspect, the invention relates to a computer-implemented method as defined in claim 1.

In one embodiment, the data points in the 3D data set may represent voxels. In another embodiment, the data points in the 3D data set may define points in a point cloud or points and the normal of a 3D surface mesh.

In one embodiment, the first 3D deep neural network may be configured as a convolutional deep neural network that is configured to process voxelized 3D data.

In another embodiment, the first 3D deep neural network may be implemented as a deep multi-layer perceptron (MLP)-based network, which may process points from a 3D point cloud or a 3D surface mesh.

In one embodiment, the transformation parameters may include rotation, translation, and/or scaling parameters.

In one embodiment, the canonical representation of the 3D object may be a canonical voxel representation or a canonical 3D mesh representation of the 3D object.

In one embodiment, the canonical pose information may include one or more voxel maps for associating (linking) a voxel of the voxel representation with (a) a prediction of a position of the voxel in the canonical coordinate system.

In one embodiment, the one or more voxel maps may include a first 3D voxel map associating (linking) a voxel with (a) a prediction of a first canonical coordinate x' of the canonical coordinate system, a second 3D voxel map associating (linking) a voxel with (a) a prediction of a second canonical coordinate y' of the canonical coordinate system, and a third 3D voxel map associating (linking) a voxel with (a) a prediction of a third canonical coordinate z' of the canonical coordinate system.

In one embodiment, determining an orientation of an axis of the canonical coordinate system may further comprise: determining for a voxel of the voxel representation a local gradient in a canonical coordinate of one of the one or more 3D voxel maps, the local gradient representing a vector in the space defined by the first coordinate system, wherein the orientation of the vector represents a prediction of the orientation of a canonical axis and/or wherein the length of the vector defines a scaling factor associated with the canonical axis.

Thus, the method enables the automated determination of a canonical representation of the 3D object, such as a 3D dental structure. The method may be used to transform different 3D data modalities of a 3D object into the canonical pose of the 3D object, which may be used in the process of superimposing different 3D data sets. Alternatively and/or additionally, the method may be used as a preprocessing step before providing the 3D data set as a 3D input to one or more 3D deep neural networks that are configured to segment the 3D object, e.g., a 3D dental structure, and/or to determine a taxonomy of segmented 3D objects, e.g., teeth. Such a pre-processing stage substantially increases the accuracy of 3D object segmentation and classification since the accuracy of such a trained neural network may be affected if the pose of the 3D object represented by the 3D dataset being fed into the system deviates too much (especially with respect to orientation) from a normalized pose.

In a further aspect, the invention relates to a computer-implemented method as defined in claim 4.

Thus, two or more different 3D data sets of 3D dental structures (representing dentomaxillofacial complexes), typically from the same patient, can be superimposed by transforming the coordinates associated with the 3D data sets into coordinates of a canonical coordinate system. In a typical example, different 3D data sets may be different scans of (part of) a patient's dentition. Typically, the different 3D data sets are at least partially overlapping, i.e. both data sets have at least part of an object element (e.g., a tooth or tooth crown in the case of a 3D dental structure) in common. A 3D deep neural network, e.g., a 3D convolutional neural network, determines the canonical pose at least for parts of a 3D object. The 3D deep neural network processes the coordinates block by block, in order to comply with computational constraints. A computer then applies further processing to infer the relative position of the canonical origin and canonical axes (directions and scale) for each voxel provided to the 3D convolutional network. Transformation parameters are then inferred to superimpose or align the two image data sets, and the 3D image data sets are aligned using those transformation parameters. The first 3D image data set may be transformed to align with the second 3D image data set, or the second 3D image data set may be transformed to align with the first 3D image data set, or both 3D image data sets may be transformed such that they align in a third orientation different from any received orientation.

The 3D deep neural network can be trained to be very robust to variations in 3D datasets because the 3D deep neural network is trained based on a large number of typical 3D dentomaxillofacial structures, where the structures show large spatial variation (translation, rotation, and/or scaling). Problems related to the (limited) memory size of the 3D deep neural network can be solved by training the deep neural network based on subsamples (blocks) of voxel representations. To do this, a voxel representation can be divided into voxel blocks of a predetermined size before being provided as input to the 3D deep neural network. An additional advantage of using blocks is that the network can determine the canonical pose even from a limited amount of data, for example, a few teeth instead of an entire dentition. Due to the fact that the canonical coordinate system is defined with respect to a known (predetermined) norm for the objects (e.g. dentomaxillofacial structures), the obtained first and second canonical 3D data sets are aligned, where the accuracy may depend on the training time, training sample variation and/or available blocks per received data set.

In this disclosure, a canonical pose defines a pose, comprising a position, an orientation, and a scale, and is a pose that is defined by assigning predetermined positions and/or orientations to a (preferably) reliably and unambiguously identifiable portion of a 3D object, e.g., the dental arch in the case of a 3D dental structure. In a similar manner, a canonical coordinate system may be defined by assigning the origin to a reliably and unambiguously identifiable position relative to an identifiable portion of the 3D object, and to define the coordinate axes in a systematic manner, e.g., the x-axis along the tangent of the point of maximum curvature of the dental arch. Such a canonical coordinate system can define a predetermined, normalized, unambiguous coordinate system that is systematic across 3D object data of a certain type (e.g., all dentomaxillofacial image data can be defined with respect to the typical position, orientation, and scale of reliably identifiable dentomaxillofacial structures). Its function is to ensure that a 3D object in different image data sets is at the same relative position, orientation, and scale. Such a function can be used in a multitude of applications where 3D data, such as voxel representations, point clouds, or 3D arrays, are processed by a trained neural network. Embodiments in this disclosure use the knowledge that if two or more 3D objects are transformed into the canonical coordinate system, the 3D objects are also aligned with each other. Furthermore, it uses the knowledge that a canonical pose of a dentomaxillofacial structure can be automatically determined using a 3D deep neural network, preferably a 3D convolutional neural network, avoiding the need for human interaction.

In one embodiment, the first and second canonical representations of the first and second 3D objects, preferably the first and second 3D surface meshes, may be 3D surface meshes, the determining an overlay further including: segmenting the first canonical representation of the first 3D object into at least one 3D surface mesh of a 3D object element, for example a first 3D dental object element, of the first 3D object and segmenting the second canonical representation of the second 3D object into at least one surface mesh of a second 3D object element, for example a second 3D dental element, of the second 3D object; selecting at least three first and second non-collinear keypoints of the first and second 3D surface meshes, a keypoint (a point of interest on the surface of a 3D surface mesh); and aligning the first and second 3D dental elements based on the first and second non-collinear key points. In one embodiment, a key point may define a local and/or global maximum and minimum in the surface curvature of the first surface mesh.

In one embodiment, the first and second canonical representations of the first and second 3D objects may be voxel representations. In one embodiment, determining an overlap may further include: providing at least part of the first canonical voxel representation of the first 3D object and at least part of the second canonical voxel representation of the second 3D object as input to a second 3D deep neural network, the second 3D deep neural network being trained to determine transformation parameters, preferably rotation, translation, and/or scaling parameters, to align the first and second canonical voxel representations; aligning the first and second canonical representations of the first and second 3D objects based on the transformation parameters provided by the output of the second 3D deep neural network.

In one embodiment, determining an overlap may further include: the processor determining a volume of overlap between the canonical representation of the first 3D object and the canonical representation of the second 3D object.

In one embodiment, determining an overlap may further include: the processor determining a first volume of interest comprising first voxels of the first canonical representation in the overlapping volume; and, determining a second volume of interest comprising second voxels of the second canonical representation in the overlapping volume.

In one embodiment, the method may further comprise: the processor providing first voxels contained in the first volume of interest (VOI) as input to a third 3D deep neural network, the third 3D deep neural network being trained to classify and segment voxels; and, the processor receiving activation values for each of the first voxels in the first volume of interest and/or for each of the second voxels in the second volume of interest from the output of the third 3D deep neural network, wherein an activation value of a voxel represents the probability that the voxel belongs to a predetermined 3D object element, for example a 3D dental element (such as a tooth) of a 3D dental structure; and, the processor using the activation values to determine first and second voxel representations of first and second 3D object elements in the first and second VOIs respectively.

In one embodiment, the processor may use the first and second voxel representations of the first and second 3D object elements to determine first and second 3D surface meshes of the first and second 3D object elements.

In one embodiment, the method may further comprise: the processor selecting at least three first and second non-collinear key points of the first and second 3D surface meshes, preferably a key point defining a local and/or global maximum and minimum in the surface curvature of the first surface mesh; and, the processor aligning the first and second 3D object elements based on the first and second non-collinear key points, preferably using an iterative closest point algorithm.

In one embodiment, the method may further comprise: providing the processor the first voxel representation of the first 3D dental element and the second voxel representation of the second 3D dental element to a fourth 3D deep neural network, the fourth 3D deep neural network being trained to generate an activation value for each of a plurality of candidate structure labels, an activation value associated with a candidate label representing the probability that a voxel representation received by the input of the fourth 3D deep neural network represents a type of structure as indicated by the candidate structure label; the processor receiving, from the output of the fourth 3D deep neural network, a plurality of first and second activation values, selecting a first structure label with the highest activation value from the first plurality of activation values and selecting a second structure label with the highest activation value from the second plurality of activation values and assigning the first and second structure labels to the first and second 3D surface meshes respectively.

In one embodiment, the method may further comprise: the processor selecting at least three non-collinear first and second key points of the first and second 3D surface meshes, preferably a key point defining a local and/or global maximum minimum in the surface curvature of the first surface mesh; the processor labeling the first and second key points based on the first structure label assigned to the first 3D surface mesh and the second structure label assigned to the second 3D surface mesh respectively; and, the processor aligning the first and second 3D dental elements based on the first and second key points and the first and second structure labels of the first and second 3D surface meshes respectively, preferably using an iterative closest point algorithm.

In a further aspect, the invention relates to a computer-implemented method for training a 3D deep neural network as defined in claim 12.

In another aspect, the invention relates to a computer system as defined in claim 13.

In yet another aspect, the invention relates to a computing system as defined in claim 14. In one embodiment, at least one of the first and second voxel representations may comprise CBCT/CT data wherein the voxel values represent radiodensities.

In one embodiment, at least one of the first and second voxel representations may comprise voxelized surface data or volume data obtained from a surface, preferably laser-structured light surface scanning data, more preferably intraoral scanning (IOS) data.

In a further aspect, the invention also relates to a computer program product as defined in claim 15.

The invention will be further illustrated with reference to the accompanying drawings, which schematically show embodiments according to the invention. It will be understood that the invention is in no way limited to these specific embodiments. The scope of the invention and its embodiments are defined by the appended claims.

Brief description of the drawings

Figure 1 represents a schematic overview of a computing system for superimposing 3D dentomaxillofacial image data using deep learning according to an embodiment of the invention;

Figure 2 represents a schematic of a system for determining a canonical pose of a 3D dental structure according to an embodiment of the invention;

Figures 3A-3D represent schemes illustrating a method of determining a canonical pose of a 3D dental structure according to an embodiment of the invention;

Figures 4A-C illustrate training and prediction data employed by system components, in accordance with an embodiment of the invention;

Figure 5 depicts an example of a 3D deep neural network architecture for generating canonical coordinates according to an embodiment of the invention;

Figure 6 represents a schematic overview of a system component for segmenting 3D dentomaxillofacial image data according to an embodiment of the invention;

Figures 7A and 7B depict examples of a 3D deep neural network architecture for segmentation of 3D dentomaxillofacial image data according to an embodiment of the invention;

Figure 8 represents a schematic summary of a system component for 3D dentomaxillofacial image data taxonomy according to an embodiment of the invention;

Figure 9 depicts an example of a 3D deep neural network architecture for 3D dentomaxillofacial image data taxonomy according to an embodiment of the invention;

Figures 10A and 10B illustrate examples of generated keypoints;

Figure 11 represents a schematic overview of a system component for the direct determination of transformation parameters for superimposing voxel representations according to an embodiment of the invention;

Figures 12A and 12B represent received and transformed data as used in, and resulting from, a system component for the direct generation of transformation parameters according to an embodiment of the invention;

Figure 13 represents an example of a 3D deep neural network architecture for a system component for direct derivation of transformation parameters according to an embodiment of the invention;

Figure 14 represents a flow diagram of the system logic for the selection/determination of the transformation parameters to be applied, according to an embodiment of the invention;

Figures 15A and 15B depict transformation results on two exemplary 3D dentomaxillofacial image data sets from individual methods according to various embodiments of the invention; and

Figure 16 is a block diagram illustrating an exemplary data processing system that can be used to execute methods and software products described in this disclosure.

Detailed description

In this disclosure, embodiments of computer systems and computer-implemented methods are described that use 3D deep neural networks for fully automated, fast, accurate, and robust superposition of different 3D data sets representing a 3D object, such as 3D dentomaxillofacial structures originating from dentomaxillofacial complexes. The methods and systems enable superposition of at least two 3D data sets using a 3D deep neural network that is trained to determine a canonical pose for each of the two 3D data sets. The output of the trained neural network is used to determine transformation parameters that are used to determine superimposed canonical 3D data sets, wherein one canonical 3D data set represents a canonical representation of a 3D object, such as a dentomaxillofacial structure. Additional 3D deep learning networks and/or overlay schemes can be used to further improve the overlay accuracy. The systems and methods will be described in more detail later in this paper.

Figure 1 depicts a high-level schematic of a computing system for automated superposition of image data representing a 3D object, in this example a 3D dentomaxillofacial complex, using deep learning according to one embodiment of the invention. The computing system 102 may comprise at least two inputs for receiving at least two 3D data sets, for example a first data set 106 comprising a first 3D object, such as a first 3D dental structure, associated with a first coordinate system and a second data set 108 comprising a second 3D object, such as a second 3D dental structure, associated with a second coordinate system. The 3D data sets may represent a first 3D dental structure and a second 3D dental structure originating from a 3D dentomaxillofacial complex 104, preferably from the same patient. The first and second 3D objects may have at least one part in common, for example, a common tooth part in the case of a 3D dental structure. The 3D data sets may be generated by different scanning devices, for example, different CBCT/CT scanning devices and/or different optical scanning devices. Such scanning devices may include a cone beam CT scanning device, a fan beam CT scanning device, an optical scanning device such as an intraoral scanning device, etc.

In the case of a CBCT scanning device, the 3D data set may include a voxel representation of X-ray data generated by a CBCT scanning device. The voxel representation may have a predetermined format, for example, the DICo M format or a derivative thereof. The voxel representation defines a 3D voxel space of a predetermined size, for example, a 400x400x400 voxel space, where each voxel is associated with a particular volume, and the position of the voxels in the voxel space may be defined based on a predetermined coordinate system.

Alternatively, in the case of an optical scanning device, the 3D data set may comprise surface mesh data, for example a set of points or vertices in a 3D space connected by edges defining a set of faces, the set of faces in turn defining a surface in 3D space. The 3D data set may also comprise point cloud data representing points in a 3D space defined by a 3D coordinate system. In one embodiment, a 3D data set representing a surface mesh may be generated using an intraoral scanning device, wherein the 3D data set may have a predetermined format, for example the STLoun format derived therefrom. Again, the 3D surface mesh representation defines a 3D space of a predetermined size in which the position of the points and/or vertices are based on a predetermined coordinate system (which differs from the coordinate system used for the other 3D data set).

In some embodiments, a 3D surface mesh of a 3D tooth structure may be segmented into segmented (i.e., separate) individual 3D tooth elements, e.g., tooth crowns and surfaces belonging to the gum. Segmenting 3D surface meshes into individual 3D surface meshes is a technique well known in the art, as described, e.g., by W.U. Ket al., Toothsegmentation on dental meshes using morphologic skeleton. Comput Graph, February 2014, vol. I.38, 199-211.

The 3D data sets may be generated (approximately) at the same time or at different points in time (pre-operative and post-operative scans using the same or different scanning systems), where the representation of the 3D dentomaxillofacial complex may be defined based on a 3D coordinate system as defined by image processing software such that the orientation and/or scale of the 3D dentomaxillofacial structures in the 3D dentomaxillofacial complex of the different 3D sets may vary substantially. The 3D dentomaxillofacial complex may include 3D dentomaxillofacial structures, in short 3D dental structures such as jaw, teeth, gums, etc.

Large variations in the 3D data sets (in terms of data formats/modalities, coordinate systems, position and orientation of the 3D structure, quality of image data, different amounts of overlap between the structures present, etc.) that can be offered as input to the computer system make the problem of accurate automated superimposition of 3D dental structures (i.e. superimposition of 3D dental structures without any human intervention) a not insignificant task. Known superimposition systems cannot address these problems in a reliable and robust manner.

In order to address this problem, the system of Figure 1 may include a first trained 3D deep neural network 112 that is configured to receive a voxel representation of different 3D data sets originating from a 3D dentomaxillofacial complex, preferably from a patient. The 3D deep neural network is trained to determine a canonical pose in a canonical coordinate system of a 3D dental structure in the 3D dentomaxillofacial complex, wherein the canonical coordinate system defines a coordinate system with respect to a position in a common dentomaxillofacial structure, for example a position in the dental arch. The 3D deep neural network is configured to determine first transformation parameters 114 (regarding translation, rotation, and/or scaling) for a voxel representation of a 3D data set as encoded in the memory of the 3D deep neural network. The first transformation parameters are determined based on translation, orientation, and/or scaling information of typical dentomaxillofacial features as encoded in the 3D deep neural network and are used to transform coordinates based on the first coordinate system of the first 3D data set and coordinates of the second coordinate system based on the second 3D data set to give coordinates based on a canonical coordinate system. The first and second 3D data sets thus obtained represent first and second 3D dental structures superimposed on the canonical coordinate system.

In the case where the first and/or second 3D data set is optical scanning data, such data may be preprocessed before being provided as input to the first 3D deep neural network. In this case, the preprocessing may include transforming the 3D scanning data, for example a 3D mesh, into a voxel representation such that it can be processed by the 3D deep neural network. For example, a 3D surface mesh may be voxelized, for example, such that a 3D voxel space represents at least the same volume in the real world as that comprised within the 3D surface mesh data. For example, such a voxelized 3D surface mesh may have a binary voxel representation having default voxel values of a first value (e.g., “0”) at which no surface in the mesh data matches the representative voxel, and voxel values of a second value (e.g., “1”) at which the mesh data matches. Where a received 3D surface mesh defines an “open” 3D surface structure, the structure may be “closed” with additional surfaces. Voxelization may be performed as above, where voxels located within an enclosed volume may also have a second value (e.g., “1”). In this manner, a voxel representation of a volume is formed. The resolution (size of a voxel) may be appropriately selected in order to provide accurate results throughout the system, while still respecting requirements considering, for example, available memory and processing.

In one embodiment, a 3D deep neural network can be used that can determine the canonical pose of optical scanning data (a 3D point cloud) based directly on the point cloud data. An example of such a network is a multilayer perceptron (MLP)-based deep neural network. MLP deep neural network architectures include PointNet (Qi, C.R., et al.: Pointnet: Deep learning on point sets for 3d classication and segmentation. Proc. Computer Vision and Pattern Recognition (CVPR), IEEE 1(2), 4 (2017)) or PointCNN (Li et al. “PointCNN: convolution on %-transformed points”, arXiv:1801.07791v 5, November 5, 2018, to be published in Neural Information Processing Systems (NIPS) 2018). These MLP deep neural networks can directly process points from a point cloud. Such neural networks can be trained to determine canonical pose information based on the optical scan data as described in this application. In effect, this will lead to being able to skip such a voxelization step as a preprocessing step, lead to faster processing and the ability to deliver higher accuracy results depending on the granularity of the point cloud data.

An additional preprocessing step may include dividing the first and second 3D data sets into blocks of a predetermined size. The block size may depend on the size of the 3D input space of the first 3D deep neural network and the memory space of the 3D deep neural network.

In one embodiment, the computer may determine first and second overlapping canonical data sets by determining first transformation parameters for the first 3D data set and first transformation parameters for the second 3D data set and applying the thus determined transformation parameters to the first and second 3D data sets. The 3D deep neural network may be trained to be very robust to large variations in the 3D data sets because the 3D deep neural network is trained based on a large number of typical 3D dentomaxillofacial structures where the structures exhibit large spatial variation (translation, rotation, and/or scaling). Issues related to the (limited) memory size of the 3D deep neural network are addressed by training the deep neural network based on subsamples (blocks) of voxel representations. To achieve this, a voxel representation is first divided into blocks of predetermined sizes before being offered as input to the 3D deep neural network. Because the canonical coordinate system is defined relative to a known (predetermined) norm for dentomaxillofacial structures, the first and second canonical 3D datasets obtained are aligned, where accuracy may depend on training time, training sample variation, and/or available blocks per dataset. Additionally, as described in more detail below, a specific network architecture can be used to encode large amounts of 3D image information while taking spatial variation into account.

In some cases, it may be advantageous to further refine the accuracy of the superposition of canonical 3D data sets. Thus, in some embodiments, further refinement of the superposition may be obtained by using (partially overlapping) canonical voxel representations 118 of the first and second 3D data sets and evaluating the overlap of the canonical voxel representations using a second additional 3D deep learning network. In these embodiments, the computer may determine the overlap between the volumes defined by the 3D dental structures represented by the overlapping first and second canonical data sets. In this case, the overlap may be defined as the volume within the space defined by the canonical coordinate system that is common to the 3D dental structures of the first and second data sets. The overlap may be used to select a volume of interest (VOI) in the canonical voxel representation of the first and second 3D data sets. In this manner, a first VOI from the canonical voxel representation of the first 3D data set and a second VOI from the canonical voxel representation of the second 3D data set may be selected for input to the second 3D deep neural network 120, which is configured to determine second transformation parameters 122. This 3D deep neural network may be referred to as a forward transformation deep neural network since the neural network generates transformation parameters in response to providing a canonical voxel representation as input to the neural network. Applying the second transformation parameters to each of the first and second canonical 3D data sets (as obtained based on the first transformation parameters) may further improve the accuracy of the superposition 116.

Alternatively and/or additionally, in some embodiments, further refinement of the superposition may be obtained by using canonical voxel representations of the first and second 3D data sets and evaluating the superposition of the canonical voxel representations based on an analytical superposition algorithm. In particular, in this embodiment, canonical voxel representations 124 of the first and second 3D data may be determined. Also in this case, the overlap between the volumes defined by the 3D dental structures represented by the first and second overlapping canonical data sets may be used to determine one or more first VOIs of the canonical voxel representation of the first 3D data set and one or more second VOIs of the canonical voxel representation of the second 3D data set, which may be provided as input to a third 3D deep neural network 126. This deep neural network is configured to classify voxels of a VOI of a voxel representation of a 3D dental structure and form voxel representations of different segmented 3D dental elements, e.g., tooth, jawbone, gum, etc. Additionally, in some embodiments, a post-processing step may be applied in which a segmented 3D model of a segmented 3D dental element is generated based on the classified voxels of a segmented 3D dental structure. Additionally, in some embodiments, a fourth additional 3D deep neural network may be used to label voxel representations of segmented 3D dental elements according to a known taxonomy scheme, e.g., that uniquely and systematically identifies individual teeth.

Segmentation and taxonomy procedures can benefit from the information derived from the first 3D deep neural network. In particular, determining and applying a first set of initial transformation parameters by the first 3D deep neural network can result in a canonical voxel representation of a 3D dataset, allowing for more accurate segmentation and/or taxonomy results since the accuracy of 3D deep neural networks used for segmentation and/or taxonomy is relatively sensitive to large rotational variations of the 3D input data.

Furthermore, as explained above, the amount of overlap may be used by the third 3D deep neural network to determine in which volume of the space defined by the canonical coordinate system overlapping structures (e.g., 3D dental elements) from the first and second 3D data sets that are identical are present. The identification of volumes (VOl) comprising overlapping structures in the first and second 3D data sets may be used to determine so-called keypoints. The keypoints are used to mark the same (overlapping) structures within the two different data sets. Thus, a set of keypoints identifies the exact 3D position of multiple points in the first 3D data set, which are linked to an associated set of keypoints in the second 3D data set. A distance minimization algorithm may use the keypoints to calculate appropriate third transformation parameters 130 for accurate superposition of the first and second 3D data sets.

In one embodiment, the computer may use the overlapping first and second canonical 3D data sets (as determined based on the first transformation parameters and optionally the second and/or third transformation parameters) to create a single focused 3D data set 132 in a predetermined data format. Fusing 3D data sets is known in the art, reference is made, for example, to the article by JUNG Wet et al., Combining volumetric dental CT and optical sonic data for teeth modeling, Computer Aided Design, October 2015, vol. I, 67-68, 24-37.

Figure 2 depicts a schematic of a system for determining a canonical pose of a 3D dental structure in a canonical coordinate system according to an embodiment of the invention. The system 200 comprises at least one 3D deep neural network 222 having an input and an output. The system may include a training module 201 for training the 3D deep neural network based on a training set 212. Additionally, the system may include an inference module 203 that is configured to receive a 3D data set representing a 3D object in a certain coordinate system and to determine transformation parameters for transforming coordinates of the voxels of the 3D data set into canonical coordinates of a canonical coordinate system as encoded into the 3D neural network during training.

The network may be trained based on a training set 212 that includes 3D image samples and associated canonical coordinates for the 3D image samples. The training data may comprise 3D data sets (e.g., voxel intensity values, e.g., radiodensities in the case of CBCT/CT data, or binary values, e.g., in the case of voxelized surface scan data). Canonical coordinate data, which may be represented as an (x,y,z) vector for each input voxel, may be used as target data.

A canonical coordinate system may be selected that is appropriate for a class of 3D objects, e.g., 3D dental structures. In one embodiment, for a 3D dental structure, a canonical coordinate system may be determined to have an origin (0,0,0) at a systematic point (between patients and within a patient). Hereinafter, when referring to “real-world coordinates,” they are considered to have axis directions relative to the patient's viewpoint, with a patient standing upright, “inferior-superior” meaning “top-down” from the patient's viewpoint, “front-back” meaning “front-back” from the patient's viewpoint, and “left-right” meaning “left-right” from the patient's viewpoint. “Real-world” is intended to refer to the situation from which information, such as 3D data sets, is obtained. Such a systematic point can be, for example, the lowest point (in real-world coordinates) at which both anteriorly positioned teeth (FDI system index 11 and 21) are still in contact, or would be in contact (if, for example, either of those teeth were missing). Given the axis directions, the real-world directions (seen as the patient) down-up, left-right, and front-back can be defined and encoded respectively as x , y , and z values ranging from low to high. In order to scale to real-world dimensions, various methods can be employed as long as this is done consistently across all the training data, since the same scaling will be the output of the 3D deep learning network. For example, a value of 1 coordinate unit can be employed for every real-world translation of 1 mm.

In order to obtain a 3D deep neural network that is robust against variations in data and/or data modalities, a wide variety of training samples 212 may be generated based on an initial training set 202 that includes 3D data sets, for example voxel representations of 3D dental structures and associated canonical coordinate data. To this end, the training module may include one or more modules for preprocessing the training data. In one embodiment, in order to respect the processing and memory requirements of the 3D deep neural network 222, a downscaling module 204 may be used to downscale a 3D data set to give a downscaled 3D data set and associated canonical coordinates of a predetermined resolution. Such a downscaling operation results in a smaller 3D image data set, for example by downscaling the voxel resolution in each direction to 1 mm. In another embodiment, a transformation module 206 may be used to generate different variations of a 3D data set by applying random rotations to the (downscaled) 3D data and associated canonical coordinates. Note that this may be done for any available patient, effectively providing a combination of data from which to draw potential training samples, having a multitude of patient data sets and a multitude of rotations per data set.

In a further embodiment, the training module may include a splitting module 208 for splitting the (downscaled) 3D data sets and associated canonical coordinates into blocks (3D image samples), each block having a predetermined size and being a subset of the total volume of the 3D data set. For example, a 3D data set provided as input to the training module may include a volume of 400x400x400 voxels where each voxel has a dimension of 0.2 mm in each orthogonal direction. This 3D data set may be downscaled to give a downscaled 3D data set having a volume of, for example, 80x80x80 voxels of 1 mm in each direction. The splitting module may then split the downscaled 3D dataset into 3D data blocks of a predetermined size (e.g., 24x24x24 voxels of 1 mm in each direction). These blocks may be used to train the 3D deep neural network using the canonical coordinates as a target. In one embodiment, the splitting module may include a random picker to randomly select blocks that form the training set 212 for the 3D deep neural network 222.

Note that such a 3D deep learning network will inherently be trained with both variable rotations (from 206) and translations (from random selections 208). Optionally, in another embodiment, samples may be presented at a multitude of scales as may be generated from 204.

With an appropriately trained 3D deep learning network 222, new 3D image data 214 (having an arbitrary position and orientation) can be presented as input to the system and appropriately pre-processed analogously to the training 3D image data, more specifically by employing a predetermined scaling 216, dividing the downscaled dataset into image blocks of a predetermined size 218, and presenting the 3D image blocks 220 as required by the 3D deep neural network. By presenting image blocks that cover the entire space of the received 3D image data at least once, canonical coordinates can be predicted by the 3D deep neural network for each (downscaled) voxel in the 3D image dataset.

Such prediction data may be further processed 224 to generate a set of general transformation parameters, which define how the received data may be transformed to align as closely as possible with the canonical pose. This processing will be described and illustrated in more detail later in this document. Note that, with sufficient training samples from a relatively large real-world 3D space, a canonical pose may be determined for data received from a smaller volume (provided it is representatively comprised within the training data). Note that in reality, the resolution of the input data may be approximately 1.25 mm. Predictions may be produced by the 3D deep neural network 222 in floating point values.

Figures 3A-3D depict schematics illustrating a method of determining a canonical pose of a 3D object such as a 3D dental structure according to an embodiment of the invention. Figure 3A schematically depicts a voxel representation 300 of a 3D object, for example a dental object such as a tooth. A voxel may be associated with an intensity value, for example a radiodensity obtained from a CBCT/CT scan. Alternatively, a voxel may be associated with a binary value. In that case, a voxel representation may be a binary voxel representation of a voxelized surface or a voxelized surface derived volume obtained from a structured light scan or laser surface scan. The 3D object may have specific characteristics that identify a top (e.g., a crown), a bottom (e.g., a root), a front, a back, and a left and right. The voxel representation is associated with a first (orthogonal) coordinate system (x,y,z) 302, for example a coordinate system that is used by the scanning software to represent the scanned data in 3D space. These coordinates may be provided, for example, as (meta)data in a DICOM image file. The 3D object may have a certain orientation, position and size in the 3D space defined by the first coordinate system. Note, however, that such a coordinate system may not yet correspond to a system as can be defined with respect to the object, illustrated in this case as “left”, “right”, “front”, “behind”, “bottom” and “top”. Using a trained 3D deep neural network, the 3D object is (spatially) “normalized” (i.e., reoriented, repositioned, and scaled) 308 and defined based on a canonical (orthogonal) coordinate system. In the canonical (x',y',z') coordinate system 306, the normalized 3D object 305 has a canonical pose, where specific features of the 3D object are aligned with the axis of the canonical coordinate system. The system therefore receives a voxel representation of a 3D dental structure having a given orientation, position and size in a 3D space defined by a coordinate system defined by the scanning system and determines a canonical voxel representation of the 3D object, wherein the 3D object is defined in a canonical coordinate system in which the size of the object is scaled and in which specific features of the 3D dental structure are aligned with axes of the canonical coordinate system.

Figure 3B depicts a 3D deep neural network 318 that may be trained to receive voxels from a voxel representation 310 of a 3D object, wherein the voxels may have a certain position defined by a coordinate system 302 (x,y,z). The 3D deep neural network is configured to generate so-called canonical pose information 303 associated with the voxel representation. The canonical pose information comprises, for each voxel 304 (x,y,z) of the voxel representation, a prediction of a coordinate (x',y',z') in a space defined by the canonical coordinate system. The canonical coordinate system may be defined with respect to a typical position, orientation, and scale of reliably identifiable dentomaxillofacial structures, for example features of the dental arch. The information required to derive such a canonical coordinate system can be encoded into the 3D deep neural network during the network training phase. In this way, the canonical pose information can be used to place different varieties and/or modalities of 3D data representing the same dentomaxillofacial structure in the same relative position, orientation, and scale.

Thus, for each input voxel 304, three corresponding output values 314, 324, 334 are generated by the 3D deep neural network, comprising predictions for the values, respectively, of the x ', and y 'coordinates of the input voxel in the canonical coordinate system. In one embodiment, the canonical pose information may include three 3D voxel maps 312, 322, 332 wherein each 3D voxel map links a voxel of a voxel representation in the input of the 3D neural network to a canonical coordinate.

Before providing the voxel representation to the input of the 3D deep neural network, the voxel representation may be partitioned into a set of voxel blocks (illustrated here by 316, hereinafter abbreviated as “blocks”), where the dimensions of a voxel block match the dimensions of the input space of the 3D deep neural network. The block size may depend on the data storage capabilities of the 3D deep neural network. Thus, the 3D deep neural network can process the voxels in each of the blocks of the voxel representation and produce canonical pose information for voxels in each block, i.e., coordinate predictions (x',y',z') of a canonical coordinate system for each voxel in a block. In one embodiment, the 3D deep neural network may generate three voxel maps 312, 322, 332, a first voxel map 312 comprising, for each voxel in a block offered to the input of the 3D deep neural network, a corresponding x' coordinate; a second voxel map 322 comprising, for each voxel in a block, a y' coordinate; and, a third voxel map 332 comprising, for each voxel in a block, a z' coordinate.

Figure 3C schematically shows a voxel representation of a 3D object 300 that is provided as input to the 3D deep neural network, and is defined based on a first coordinate system (x,y,z) 302, for example a coordinate system used by the image processing software of the scanning device that was used to produce the 3D images. These coordinates or information for determining these coordinates may be included in the data file, for example a DICOM file, as metadata. Based on canonical pose information generated by the 3D deep neural network, a prediction of the canonical pose of the 3D object in a canonical coordinate system is generated. Thus, the canonical pose information 350 may link a position (x,y,z) of each voxel in the first coordinate system to a position (x',y',z') in the canonical coordinate system. This information may be used to determine a transformation 360 that allows the system to transform the 3D object defined in the first coordinate system into its canonical posture 362 defined in the canonical coordinate system.

The posture information is used to determine an orientation and a scaling factor associated with the axis of the canonical coordinate system (the canonical axes). In this case, the orientation is an orientation of the canonical axes in the space defined by the first coordinate system. The posture information is also used to determine the position of the origin of the canonical coordinate system.

An orientation of a canonical axis may be determined based on a (local) gradient at one or more voxels in a 3D voxel map as determined by the 3D deep neural network. For example, for each, or at least several, voxels of the first 3D voxel map associated with the x 'component of a canonical coordinate, a local gradient may be determined. The local gradient may be represented as a 3D vector in the x ,y ,z space defined by the first coordinate system. The direction of the vector represents a prediction of the orientation of the canonical x 'axis at the voxel position. Furthermore, the length of the vector represents a prediction of a scaling factor associated with the canonical x 'axis. In one embodiment, a prediction for the orientation and scaling factor associated with the canonical x 'axis may be determined based on x 'values of the first 3D voxel map. For example, a statistically representative measure of the predictions for voxels in the first 3D voxel map may be determined, e.g., the median or average of the gradient. In one embodiment, the x values in the first 3D voxel map may be preprocessed, e.g., smoothed and/or filtered. For example, in one embodiment, a median filter may be used to remove outliers (local). Similarly, a prediction of an orientation and a scaling factor for the canonical y axis may be determined based on the y values in the second 3D voxel map, and a prediction of an orientation and a scaling factor for the canonical z axis may be determined based on the z values in the third 3D voxel map. The predicted orientations of the canonical x', y', z' axes can be post-processed to ensure that the axes are orthogonal or even orthonormal. Various schemes, for example the Gram-Schmidt procedure, can be used to achieve this. Rotation and scaling parameters can be obtained by comparing the received coordinate system 302 and the coordinate system as derived from predictions.

The position of the origin of the canonical coordinate system (as a translation vector in the space of the first coordinate system) may be obtained by determining a prediction of the canonical coordinates of the center of a voxel representation that is provided as input to the 3D deep learning network. These coordinates may be determined based on, for example, the average or median value of predicted x 'values from the first 3D voxel map, y 'values from the second 3D voxel map, and z 'values from the third 3D voxel map. A translation vector may be determined based on the predicted canonical coordinates (xo,yo',zo') of the center of the block and the coordinates of the center of the blocks based on the first coordinate system, for example by using simple subtraction. Alternatively, the origin of the canonical coordinate system can be determined by aggregating multiple predictions of such blocks, the latter effectively processing canonical coordinates as determined for the same-sized space as the received voxel representation. The procedure described above can be repeated for each, or at least a large portion, of the blocks in a 3D data set. The information determined for each block (orientation, scale, and origin of the canonical coordinate system) can be used to obtain average values that provide an accurate prediction.

Thus, the systems and methods depicted in Figures 2 and 3 provide an efficient way of determining a canonical pose of a 3D dental structure. As shown in Figure 3D, these methods include a first step 380 of providing, to a processor of a computer, a voxel representation of a 3D dental structure associated with a first coordinate system as an input to a 3D deep neural network, the neural network being configured to generate canonical pose information associated with a second canonical coordinate system. Thereafter, in step 382, the processor may receive, from the output of the 3D deep neural network, canonical pose information, wherein, for each voxel of the voxel representation, the canonical pose information comprises a prediction of a canonical coordinate of the voxel. The processor may subsequently execute a processing step 384 wherein the canonical pose information is used to determine an orientation (and, if applicable, a scaling) of the axes of the canonical coordinate system (e.g., by determining a vector representing the local gradient of a voxel position) and a position of the origin of the canonical coordinate system (e.g., by determining a vector representing average (x',y',z') values, hence an average 3D distance to the canonical origin), and wherein the orientation and position (and, if applicable, the scaling) are subsequently used to determine transformation parameters for transforming coordinates of the first 3D coordinate system into coordinates of the second canonical coordinate system. Thereafter, in step 386, the processor determines a canonical pose of the 3D tooth structure in the space represented by the second canonical coordinate system by applying the transformation parameters to the received 3D data set. If the 3D dataset is a voxel representation, the parameters can be applied to the voxels. Alternatively, if the 3D dataset is a mesh representation, the parameters can be applied to the mesh coordinates.

In this way, a canonical representation of a 3D object such as a 3D dental structure can be realized. The method can be used to transform different modalities of 3D data associated with a 3D object into the canonical pose of the 3D object that can be used in the process of superimposing the different 3D data sets. Alternatively and/or additionally, the method can be used as a preprocessing step before offering a 3D data set to an input of one or more 3D deep neural networks that are configured to segment the 3D object and (optionally) to determine a taxonomy of the segmented parts of the 3D object. Such a pre-processing step substantially increases the accuracy (and/or reduces the training time or memory requirements for the same accuracy for such 3D deep neural networks) of 3D object segmentation and classification since the accuracy of such a trained neural network may be affected if the pose of the 3D object represented by the 3D data set being input into the system deviates too much (especially with respect to orientation) from a normalized pose.

Figures 4A-C show illustrations of training objectives and results as may be used by a method as described with reference to Figures 3A-3D. Figure 4A depicts three slices 400i -3 of a 3D data set, in this example a CBCT scan of a 3D tooth structure and associated slices of 3D voxel maps for the coordinates x', y' and z' as may be used to train a 3D deep neural network. These 3D voxel maps comprise the desired predictions of the canonical x' coordinate 402i, the canonical y' coordinate 4022 and the canonical z' coordinate 4023. The grayscale values visualize the gradients of (encoded) values for coordinates according to the canonical coordinate system. The (x, y, z) coordinates indicate the position of a voxel of the 3D tooth structure based on a coordinate system associated with the CBCT scan. The axes as displayed including their directions are designated at the top left for each image. It is also worth noting that the grayscale values of the displayed gradients have been appropriately scaled to have the same grayscale value for the same value throughout Figures 4A-C. This allows for a better visual comparison of what are actually translations toward the canonical coordinate system as encoded (for training) or predicted. Finally, note that all visualizations are 2D representations of a single central “slice” (actually 2D image data pixels), as sliced from the actually employed 3D data set and the associated voxel maps, as designated by the slice number visible at the top left for each illustration.

In order to train the system, a 3D dataset representing a 3D dental structure can be attributed with a canonical coordinate system, as illustrated by Figure 4B. In the case of these illustrations, for the illustrations showing gradients, the value for black is -40.0 mm and the value for white is 40 mm, actually having the center of this patient scan as the origin (0,0,0). This data (both the 3D image data and the canonical system representation) has already been downscaled in an appropriate manner as will result from processor 204. This data may then be rotated (e.g., by employing a linear interpolation method or another) to provide 3D data as shown in the illustrations at 406. Depending on the exact method of performing this rotation, the size of the image space may also be expanded to include all of the voxels of the received 3D image data set, which is not the case in these illustrations.

Figure 4B illustrates training data 408 as may result from random selection (as may be performed by processor 208) of an appropriately sized block 412, in this case a subsample having dimension 24x24x24 voxels, from the randomly rotated input voxel representation 406. Note that when the three encoded directions of the canonical coordinate system are displayed in the same y-z view (the central y-z slice of the 3D cube of voxels) as performed at 408, a gradient direction may be observed, which effectively encodes (in the case of this 2D display) the direction of the 2D component (in the y-z plane) of the 3D direction vectors encoding the directions of the axes of the canonical coordinate system. Similarly, the value for each voxel effectively encodes the voxel’s x’, y’, and z’ coordinates according to the canonical coordinate system. Note that when processing, for example, an entire 3D predicted dataset, the 3D vectors for each axis can be determined to follow the canonical coordinate system. Subsampling for training can be done such that the smallest sample size selected only includes voxels that are part of the received 3D image dataset (i.e., not including the “empty” chunks of voxels along the edges, as seen in the illustrations, resulting from the applied rotation).

Figure 4C shows a new entry 416 after resizing (as may result from processor 216). For the purposes of this illustration, this entry has been arbitrarily rotated. For the purposes of illustrations 418 and 420, only x-y views (slices) have been displayed. Array 418 shows slices of the predicted canonical x', y', and z' coordinates. It can be seen that the received image data has been divided into blocks (or subsamples), coordinate prediction performed on the blocks, and these prediction blocks placed back into the overall received 3D image array space (which can be thought of as a square-like structure observed in the images, indicating the dimensions of the blocks). Note that this effectively illustrates that predictions for rotation, translation, and optionally scaling parameters for the transformation into the canonical coordinate system can be made using 3D image data of size 30x30x30mm. The figure also illustrates that the trained network is relatively robust to “empty” data fragments as they result from the rotation employed for this visualization (i.e., “empty” data are systematically given a value of 0 for the network trained for the purposes of these illustrations).

Illustrated at 420 are what the encoded coordinate values would have been if 416 had been training data, or the desired “target” values as they should result from the 3D deep neural network. It can be seen that the overall values (indicating distance per voxel from the origin) and the overall directions of the gradients are very similar. In the case of real-world data being rotated, the “empty” chunks as seen in the illustration will not be present. Within processor 224, the system may perform, for example, 3D averaging filtering on the predicted coordinate data 418, outlier removal, and/or other methods to smooth the resulting values. A representative measure, such as the average or median, of the predicted coordinate values may be used to determine the position of the center of the voxel representation 416 relative to the canonical coordinate system. A translation can be determined based on the difference between the position of the center of the voxel representation 416 with respect to the received coordinate system and with respect to the canonical coordinate system.

A relatively computationally inexpensive 3D gradient derivation algorithm can provide three additional value cubes for each axis cube of values, effectively providing for each axis cube all three components of the vector describing the axis direction. This can result in one directional 3D vector for all x , y , and z axes for each voxel. A representative measure, such as the average or median, can be determined for these vectors for the desired coordinate axes. If applicable, these axis-wise vectors can be converted into their unit vector equivalent. Additionally, the system can guarantee that these three vectors become their nearest perfectly orthonormal set of three by minimizing the sum of the angular distances between the first predicted axis-wise vectors and the resulting orthogonal set.

From these three (effectively) predicted directions of the canonical axes, appropriate transformation parameters can be calculated taking into account rotations of the received 3D image dataset towards the canonical orientation, as part of the canonical pose. The system can then determine what the average distance to the canonical origin will be, per axis, for the received 3D image dataset. From this, transformation parameters can be calculated for the translation of the received 3D image dataset, effectively determining where the canonical origin should lie within (or with respect to) the received 3D image dataset, or conversely, where the canonical position should lie in the coordinate system.

In another embodiment, the 3D deep neural network can be trained with varying scales, and the gradient magnitude of the resulting prediction values can be used to determine the scale of the received 3D image dataset. This can be used to calculate transformation parameters toward a desired scaling of the received data.

Figure 5 depicts an example of a 3D deep neural network architecture for determining canonical coordinates according to an embodiment of the invention. The 3D deep neural network may have an architecture similar to a 3D U-shaped network, which is actually a 3D implementation of the 2D U-shaped network as is well known in the art.

The network can be implemented using a variety of 3D neural network layers such as (dilated) convolutional layers (3D CNNs), 3D max-pooling layers, 3D deconvolution layers (3D de-CNNs), and densely connected layers. The layers can use a variety of activation functions such as linear, tanh, ReLU, PreLU, sigmoid, etc. The 3D CNN and de-CNN layers can vary in the number of filters, filter sizes, and subsampling parameters. The 3D CNN and de-CNN layers, as well as the densely connected layers, can vary in their parameter initialization methods. Dropout and/or batch normalization layers can be employed throughout the architecture.

Following a 3D U-net architecture, during training the various filters within the 3D CNN and 3D de-CNN layers learn to encode meaningful features as this will aid in the prediction accuracy effort. During training, 502 matched 3D image datasets and 540 encoded matched canonical coordinates are used to optimize towards predicting the latter from the former. A loss function can be employed as a measure to be minimized. This optimization effort can be aided by using optimizers such as SGD, Adam, etc.

Such an architecture may employ a variety of resolution scales, effectively performing downscaling 506, 510, 514 as outputs from a prior set of 3D CNN layers 504, 508, 512 via convolutional (dilated and/or downsampling) or max-pooling layers. The term “meaningful features” refers to (successive) derivations of information relevant to determining target output values, however they are also encoded by the 3D de-CNN layers, which effectively perform upscaling while employing filters. By combining 520, 526, 532 data resulting from such 3D de-CNN layers 518, 524, 534 with data from the “last” 3D CNN layers operating at the same resolution (512 to 520, 508 to 526, and 504 to 532), highly accurate predictions can be achieved. Along the upscaling path, additional 3D CNN layers 522, 528, 534 may be used. Additional logic may be encoded within the network parameters by making use of densely connected layers that distill, for example, voxel-wise logic based on the results of the incoming 3D CNN layer 534 filters.

When used for inference, having been trained to have internal parameters encoded in such a way that validation provides sufficiently accurate results, an input sample can be presented and the 3D deep learning network can provide predicted canonical coordinates for each 542 voxel.

Figure 6 represents a schematic overview of a system component for segmenting 3D dento-maxillofacial image data according to an embodiment of the invention. Methods and systems for automatic segmentation based on deep learning are described in European patent application No. 017179185.8 entitled "Classification and 3D modeling of 3D dento-maxillofacial structures using deep learning methods".

In particular, the computer system 602 may be configured to receive a 3D image data stack 604 of a dentomaxillofacial structure. The structure may include, for example, jawbone structures, teeth, and nerves. The 3D image data may comprise voxels, i.e., 3D spatial elements associated with a voxel value, for example, a grayscale value or a color value, representing a radiation intensity or density value. Preferably, the 3D image data stack may include CBCT image data in a predetermined format, for example, the DICOM format or a derived format thereof.

Particularly in CBCT scans, radiodensity (measured in Hounsfield units (HU)) is imprecise because different areas on the scan appear with different grayscale values depending on their relative positions in the organ being scanned. HU measured from the same anatomical area with both CBCT and medical-grade CT scanners are not identical and are therefore unreliable for determining radiographically determined site-specific bone density.

Furthermore, dental CBCT systems do not employ a standardized system for scaling the gray levels representing the reconstructed density values. These values are therefore arbitrary and do not allow for an assessment of bone quality. In the absence of such normalization, it is difficult to interpret the gray levels or impossible to compare the resulting values from different machines.

The structures of teeth and jawbone have a similar density, so it is difficult for a computer to distinguish between voxels belonging to teeth and voxels belonging to a jawbone. Additionally, CBCT systems are very sensitive to what is known as beam hardening, which produces dark lines between two high-attenuation objects (such as metallobone), with surrounding light lines.

Due to the reasons mentioned above, and as will be described in more detail later in this document, it is particularly beneficial for the overlay system to use the system component as described herein with reference to Figure 6.

The system component may comprise a segmentation preprocessor 606 for preprocessing the 3D image data before being fed as input to a first 3D deep neural network 612, which is trained to produce a 3D set of classified voxels as output 614. Such preprocessing may comprise, for example, normalizing voxel values to a range that is most beneficial to a neural network. As will be described in more detail hereinafter, the 3D deep neural network may be trained according to a predetermined training scheme such that the trained neural network can accurately classify voxels in the 3D image data stack into voxels of different classes (e.g., voxels associated with tooth, jawbone, and/or nervous tissue). The 3D deep neural network may comprise a plurality of connected 3D convolutional neural network (3D CNN) layers.

The computer system may further comprise a segmentation post-processor 616 for accurately reconstructing 3D models of different parts of the dentomaxillofacial structure (e.g., tooth, jaw, and nerve) using the voxels classified by the 3D deep neural network. The classified voxels 614 may comprise sets of voxels representing, e.g., all voxels classified as belonging to a tooth, maxilla, or nerve structure. It may be beneficial to create 3D data of these types of structures such that individual teeth and/or jaws (e.g., upper, lower) are represented by separate 3D models. This may be achieved by a volume reconstruction 620. For the case of separating sets of voxels belonging to individual teeth, this may be achieved by (combinations of) 3D binary erosion, 3D marker creation, and 3D splitting transform. For the combination separation into upper and lower jaw parts, a distance from the origin along the up-down axis (real-world coordinate system) can be found at which the sum of the voxels in the plane perpendicular to this direction is at a minimum compared to other intersecting planes along the same axis. The division into upper and lower jaw parts can be performed using this distance. In another embodiment, the jaws can be automatically divided by the deep network by classifying the corresponding voxels as separate jaw classes. Other portions of the classified voxels, for example voxels that were classified by the 3D deep neural network as belonging to nerves, may be post-processed using an interpolation function 618 and stored as 3D nerve data 622. Following segmentation, which performs post-processing on the 3D data of the various portions of the dentomaxillofacial structure, the nerve, jaw, and teeth data 622-626 may be combined and formatted into separate 3D models 628 that accurately represent the dentomaxillofacial structures in the 3D image data that were fed into the input of the computer system. Note that both the classified voxels 614 as well as the 3D models 628 are defined in the same coordinate system as the input data 604.

In order to make the 3D deep neural network robust against the variability present in, for example, current CBCT scan data, the 3D deep neural network may be trained using a module 638 to use 3D models of portions of the dentomaxillofacial structure as represented by the 3D image data. The 3D training data 630 may be properly aligned to a CBCT image presented at 604 for which the associated target output is known (e.g., 3D CT image data of a dentomaxillofacial structure and an associated 3D segmented representation of the dentomaxillofacial structure). Conventional 3D training data may be obtained by manually segmenting the input data, which may be a significant amount of work. Additionally, manual segmentation results in low reproducibility and consistency of input data to be used.

In order to counteract this problem, in one embodiment, optically produced training data 630, i.e., accurate 3D models of (parts of) dentomaxillofacial structure(s), may be used instead of or in addition to manually segmented training data. The dentomaxillofacial structures that are used to produce the training data may be scanned using a 3D optical scanning device. Such 3D optical scanning devices are known in the art and may be used to produce high quality 3D jaw and tooth surface data. The 3D surface data may include 3D surface meshes 632 that may be populated (by determining which specific voxels are part of the volume encompassed by the mesh) and used by a voxel classifier 634. In this manner, the voxel classifier may generate highly accurate classified voxels 636 for training. Additionally, as mentioned above, training voxels manually classified by the training module can also be used to train the network. The training module can use the classified training voxels as target CT training data and associated data as input.

Figures 7A and 7B depict examples of 3D deep neural network architectures for segmentation of 3D dentomaxillofacial image data in accordance with various embodiments of the invention. As shown in Figure 7A, the network may be implemented using 3D convolutional neural networks (3D CNNs). The convolutional layers may employ an activation function associated with the neurons in the layers such as a sigmoid function, tanh function, relu function, softmax function, etc. A plurality of 3D convolutional layers may be used wherein minor variations in the number of layers and their defining parameters, for example different activation functions, numbers and sizes of kernels, and additional functional layers such as dropout and/or batch normalization layers may be used in the implementation without losing the essence of the 3D deep neural network design.

The network may include a plurality of convolutional paths, in this example three convolutional paths, a first convolutional path associated with a first set of 3D convolutional layers 704, a second convolutional path associated with a second set of 3D convolutional layers 706, and a third set of 3D convolutional layers 708. A computer performing data processing may provide a 3D data set 702, for example CT image data, as inputs to the convolutional paths. The 3D data set may be a voxel representation of a 3D dental structure.

The function of the different trajectories is illustrated in more detail in Figure 7B. As shown in this figure, voxels from the voxel representation may be provided as input to the 3D deep neural network. The voxels from the voxel representation may define a predetermined volume, which may be referred to as an image volume 7014. The computer may divide the image volume into first blocks of voxels and provide a first block as input to the first trajectory. The 3D convolutional layers of the first trajectory 703i may perform a 3D convolution operation on the first block of voxels 701i. During processing, the output of a 3D convolution layer in the first trajectory is the input to a subsequent 3D convolution layer in the first trajectory. In this way, each 3D convolutional layer can generate a 3D feature map representing portions of the first block of pixels provided as input to the first trajectory. Therefore, a 3D convolutional layer configured to generate such feature maps can be referred to as a 3D CNN feature layer.

As shown in Figure 7B, the convolutional layers of the second path 7032 may be configured to process second voxel blocks 7012 of the voxel representation, wherein a second voxel block represents a downscaled version of an associated first voxel block, and wherein the first and second voxel blocks have the same centered origin. The rendered volume of the second block is larger than the volume of the first block. Furthermore, the second voxel block represents a downscaled version of an associated first voxel block. The downscaling factor may be any appropriate value. In one embodiment, the downscaling factor may be selected between 20 and 2, preferably between 5 and 3.

The first path 703i may define a first set of 3D CNN feature layers (e.g., 5-20 layers), which are configured to process input data (e.g., first voxel blocks at predetermined positions in the image volume) at the target voxel resolution (i.e., voxels in the image volume being classified). The second path may define a second set of 3D CNN feature layers (5-20 layers), which are configured to process second voxel blocks, wherein each block of the second voxel blocks 7012 has the same center point as its associated block of the first voxel block 701i. Furthermore, the voxels in the second blocks are processed at a resolution that is lower than the resolution of 701i. Thus, the second voxel blocks represent a larger volume in real-world dimensions than the first blocks. In this way, the second feature layers of a 3D CNN process voxels to generate 3D feature maps that include information about the direct neighbors of associated voxels that are processed by the first feature layers of a 3D CNN. In this way, the second trajectory allows the 3D deep neural network to determine contextual information, i.e., information about the context (e.g., their surroundings) of voxels in the 3D image data that are presented as input to the 3D deep neural network.

In a similar manner, a third path 7033 may be used to determine additional contextual information of the first voxel blocks 7 OI3. Thus, the third path may comprise a third set of 3D CNN feature layers (5-20 layers) configured to process third voxel blocks, wherein each block of the third voxel blocks 7 OI3 has the same center point as its associated block of the first voxel block 701i and the second voxel block 7 OI3. Furthermore, the voxels of the third blocks are processed at a resolution that is lower than the resolution of the first and second voxel blocks. This downscaling factor may again be set to an appropriate value. In one embodiment, the downscaling factor may be selected between 20 and 3, preferably between 16 and 9.

Using three trajectories or more trajectories, both the 3D image data (the input data) and contextual information about voxels in the 3D image data can be processed in parallel. Contextual information is important for classifying dentomaxillofacial structures, which typically include closely packed dental structures that are difficult to distinguish.

The output of the 3D CNN feature layer sets are then fused and fed into the input of a set of 3D fully connected CNN layers 410, which are trained to derive the intended classification of voxels 412 provided as input to the neural network and processed by the 3D CNN feature layers.

The 3D CNN feature layer sets can be trained (via their learnable parameters) to derive and pass the optimally useful information that can be determined from their specific input, the fully connected layers encode parameters that will determine how the information from the previous three trajectories should be combined to provide optimal classified voxels 712. In this case, the output (the last layer) of the fully connected layers may provide a plurality of activations for each voxel. Such a voxel activation may represent a probability measure (a prediction) that defines the probability of a voxel belonging to one of a plurality of classes, for example classes of dental structures, for example a tooth, jaw and/or nerve structure. For each voxel, a threshold of voxel activations associated with different dental structures may be determined in order to obtain a classified voxel. After that, classified voxels belonging to different classes of dental structures can be presented in the image space 714. Therefore, the output of the 3D deep neural network are classified voxels in an image space that corresponds to the image space of the input voxels.

Note that although a segmentation 3D deep neural network as described with respect to Figure 6 and Figure 7 may be inherently translation invariant across the 3D image data space, it may be beneficial to employ information from processor 114 to apply an initial pre-alignment step 124 to adjust at least rotations (albeit relatively roughly) to obtain a canonical pose. By having real-world orthogonal directions (e.g., up-down, left-right, and in front of-behind the patient) present in the 3D image data used in a predefined canonical direction (e.g., up-down in the z direction, left-right in the x direction, and in front of-behind in the y direction, respectively, of the internal representation (3D dataset)), the memory bandwidth required for the segmentation 3D deep neural network may be reduced, training time may be reduced, and segmentation accuracy may be improved. This can be done by specific training and performing inference (prediction on non-training samples) on data having said 3D data sets previously aligned into account rotations for said canonical rotations.

Figure 8 represents a schematic overview of a system component for taxonomy of 3D dentomaxillofacial image data in accordance with an embodiment of the invention. Methods and systems for automatic taxonomy based on deep learning are described in European patent application No. 017194460.6 entitled “Automated classification and taxonomy of 3D teeth data using deep learning methods”. The system 800 may include two different processors, a first training module 802 for executing a method for training the 3D deep neural network 826 and a second classification module 814 for executing a classification method based on new input data 816.

As shown in Figure 8, the training module may comprise one or more repositories or databases 806, 812 of data sources intended for training. Such a repository may be obtained by an input 804 that is configured to receive input data, for example 3D image data including dentitions, which may be stored in various formats together with respective desired labels. More specifically, at least a first repository or database 806 may be used to store 3D image data of dentitions and associated labels of the teeth within this dentition, which may be used by a computing system 808 that is configured to segment and extract volumes of interest 810 representing individual teeth that may be used for training. Such a system may be implemented as described with respect to Figure 6 and Figure 7 in the case of voxel data (e.g., CT/TAC), or it may be, for example, 3D surface meshes of individual tooth crowns as may be segmented from a 3D surface mesh comprising teeth and gums (e.g., IOS data). Similarly, a second repository or database 812 may be used to store other formats of 3D data, for example, 3D surface meshes generated by optical scanning and individual tooth labels that may be employed during training of the network.

The 3D training data may be pre-processed 826 to yield a (voxelized) 3D voxel representation that is optimized for the 3D deep neural network 828. The training process may terminate at this stage since the 3D deep neural network processor 826 may only require training on individual tooth samples. In one embodiment, 3D tooth data, such as a 3D surface mesh, may also be determined based on the segmented 3D image data originating from appropriately labeled whole dentition scans (808 to 812).

When the taxonomy module 800 is used to taxonomy a new (part of a) dentition 816, again multiple data formats may be employed when translating the physical dentition into a 3D representation that is optimized for the 3D deep neural network 828. As described above, the taxonomy system may use, for example, 3D image data of the dentition 106, 108 and use a computing system 820 (which is 602) that is configured to segment and extract volumes of interest comprising individual teeth 822 (which are 626) in a manner analogous to a training processor 808. Alternatively, another representation such as per-tooth surface meshes 824 resulting from optical scans may be used. Note again that whole dentition data can be used to extract other 3D representations of volumes of interest (820 to 824).

Pre-processing 826 may be implemented to format as required for the 3D deep neural network 828. Note that within the context of the overall overlay system, in the case of a received 3D image dataset which is e.g. CBCT/CAT data, the taxonomy of data segmented by the network 828 may be performed using (subsets of) data generated in the volume reconstruction 620 directly. In the case of a received 3D image dataset which is e.g. IOS data, the taxonomy by 828 may be performed directly with (subsets of) data from 3D surface mesh segmentation and crown voxelization.

The outputs of the 3D deep neural network may be fed into a taxonomy post-processing stage 830 designed to leverage dentition-specific knowledge (e.g., the fact that each individual tooth index can only appear once in a single dentition) to ensure taxonomy accuracy across the set of labels applied to the teeth in the dentition. This may result in the system outputting tooth labels for each individual tooth object identified. In one embodiment, correct labels may be fed back into the training data for the purpose of increasing future accuracy upon further training of the 3D deep neural network.

Figure 9 depicts an example of a 3D deep neural network architecture for 3D dentomaxillofacial image data taxonomy according to an embodiment of the invention. The network may be implemented using 3D convolutional layers (3D CNNs). The convolutions may use an activation function. A plurality of 3D convolutional layers, 904-908, may be used, wherein minor variations in the number of layers and their defining parameters, e.g., different activation functions, numbers of kernels, use of subsampling and sizes, and additional functional layers such as dropout and/or batch normalization layers, may be used in the implementation without losing the essence of the 3D deep neural network design.

In part to reduce the dimensionality of the internal representation of the data within the 3D deep neural network, a 3D max pooling layer 910 may be employed. At this point in the network, the internal representation may be passed to a densely connected layer 912 intended to be an intermediate layer for translating the representation in 3D space to activations of possible labels, particularly tooth type labels.

The final output layer 914 may have the same dimensionality as the desired number of encoded labels and may be used to determine an activation value (analogous to a prediction) for each possible label 918.

The network may be trained using a dataset with, as input to the 3D CNN layers, a pre-processed dataset of 3D data 902, i.e. a 3D voxel representation of a tooth. For each sample (which is a 3D representation of a single tooth) a matched representation of the correct label 916 may be used to determine a loss between the desired and actual output 914. This loss may be used during training as a measure to tune parameters within the 3D deep neural network layers. Optimizing functions may be used during training to aid in the efficiency of the training effort. The network may be trained for several iterations until the internal parameters lead to a desired accuracy of the results. When properly trained, an unlabeled sample may be presented as input and the 3D deep neural network may be used to derive a prediction for each possible label.

Thus, as the 3D deep neural network is trained to taxonomy a 3D data sample of a tooth into one of a plurality of tooth types, for example 32 tooth types in the case of a healthy adult dentition, the output of the neural network will be activation values and associated possible tooth type labels. The possible tooth type label with the highest activation value may indicate to the taxonomy system that the 3D data sample of a tooth most likely represents a tooth of the type indicated by the label. The possible tooth type label with the lowest or relatively low activation value may indicate to the taxonomy system that the 3D data set of a tooth is least likely to represent a tooth of the type indicated by such label.

Note that it may be required to train separate specific network models (having the same architectures but different final parameters after specific training) based on the type of input volume, for example, the input voxel representation being a full tooth volume, or the input voxel representation being just a tooth crown.

Note also that although a taxonomy 3D deep neural network as described with respect to Figure 8 and Figure 9 (as was the case with the segmentation 3D deep neural network) may be inherently translation invariant throughout the 3D image data space, it may be beneficial to use information from the processor 114 to apply an initial pre-alignment step 124 to adjust at least rotations (albeit relatively roughly) to obtain a canonical pose. By having real-world orthogonal directions (e.g., up-down, left-right, and in front of-behind the patient) present in the 3D image data used in a predefined canonical direction (e.g., up-down of z-direction, left-right of x-direction, and in front of-behind of y-direction of internal representation respectively (3D dataset)), the memory bandwidth required for the taxonomy 3D deep neural network can be reduced, the training time can be reduced, and the taxonomy accuracy can be improved. This can be done by targeted training and performing inference on data having said 3D datasets pre-aligned into account rotations to said canonical coordinate system.

Figure 10A and Figure 10B show examples of keypoints generated on two exemplary 3D dentomaxillofacial datasets, respectively, with and without the inclusion of taxonomy information. From at least the 3D image data (surface volumes) defining structures representing independent teeth or tooth crowns, and from, for example, processing as described with respect to Figure 6 and Figure 7 in the case of CBCT/tAC data, or by employing a more typical determination of surface meshes of individual tooth crowns in the case, for example, of IOS data, keypoints characterizing the surfaces can be determined. In effect, this can be considered as a reduction step to reduce all available points within a surface mesh to a more relevant (more prominent) set of points. This reduction is beneficial since it reduces the processing time and memory requirements. Furthermore, methods for determining such points can be selected that are expected to provide approximately the same set of points even though the inputs for generation are slightly divergent (sets of) 3D surface meshes (still representing the same structures). Well-known methods in the art for determining keypoints from surface meshes typically include the determination of local or global surface descriptors (or features) that can be done by hand (hand-designed) and/or automatically learned and optimized for repeatability across (slightly varying) input surface meshes, and can be optimized for performance (speed of determining keypoints or salient points), for example as taught by TONIONIa, et al. Learning to detect good 3D keypoints. Int J Comput Vis. 2018, vol I.126, pp. 1-20. Examples of such features are local and global minima or maxima in the surface curvature.

Computer representations of two received 3D image data sets are shown in Figure 10A and Figure 10B, including the edges and vertices defining the surface face meshes and thus showing the points defining the surfaces. The top four objects are individually processed and segmented tooth crowns derived from an intraoral scan. The bottom four objects are individual teeth derived from a CBCT scan according to the method referring to Figure 6 and Figure 7. These two sets of four teeth are obtained from the same patient at approximately the same point in time. They have been roughly pre-aligned by a processor as described in more detail above with reference to Figure 3, Figure 4 and Figure 5, the processor 114 determining the canonical pose as described above. From information according to 114, overlapping volumes were determined, and 3D structures were segmented to give independent surface meshes representing individual teeth. In the case of Figure 10B, additionally, a taxonomy of 3D image data of individual teeth was performed according to methods as described with reference to Figure 8 and Figure 9.

In particular, in Figure 10A, labeled points have been displayed according to the format P[received dataset no. 0H point no. 0]; the number of points has been reduced for display purposes. As can be seen, each 3D image dataset received after keypoint generation has its own set of keypoints from prominent volume features, where the same points along the surfaces will be marked with a keypoint (albeit arbitrarily numbered). Note that it would be possible to subgroup such points for each individual tooth within the source 3D dataset, but this would not yield additional benefits since the (same) individual tooth cannot be identified across the different 3D datasets.

In Figure 10B, using information from the additional taxonomy stage, the format of the labels has been visualized as P[received dataset no.]-[identified tooth index]-[point no.]. This identified tooth index is the same index across both received datasets for the same real-world teeth. It should be noted that, within each subgroup of individual teeth, the keypoint numbering is still arbitrary.

It is worth noting that 3D surface mesh data (and point cloud data or a collection of keypoints) are generally stored in an orthogonal x , y , and z coordinate format using floating-point numbers. This opens up the possibility of highly accurate determination of keypoint locations and, therefore, highly accurate alignment results, with transformation parameters determined based on, for example, methods that minimize a computed distance between such keypoint clouds, as may be the case when employing, for example, an iterative nearest point method.

The added information by considering which keypoint belongs to which tooth (and matching the representation of the same tooth in the other received 3D image data set), as shown in Figure 10B, can be specifically used to achieve more accurate determination of alignment transformation parameters. For example, in cases where no prior alignment has been performed, the average coordinate for each tooth can be used to determine a prior alignment of one received 3D image data set with respect to the other (in effect, approximately orienting the matched teeth as closely as possible to each other). In other cases, it may be beneficial to first determine a set of transformation parameters (one for each matched tooth between the two received 3D image data sets) and determine the final transformation parameters based on an average of such a set. This can be especially beneficial in cases where the overlapping volume between the two received 3D image data sets has not been determined (adequately enough).

Note that, for the determination of alignment transformation parameters, at least three non-collinear points need to be determined.

Figure 11 represents a schematic overview of a system component for the direct determination of transformation parameters for superimposing voxel representations according to an embodiment of the invention. The system 1100 may be employed to directly predict transformation parameters, for example, applicable 3D rotations, 3D translations, and 3D scaling that define how one received 3D image data set may be aligned relative to another. The training data 1102 and inference data 1116 may consist of 3D image data, for example, voxel intensity values, for example, radiodensities in the case of CBCT/CT data, or binary values, for example, in the case of voxelized surface scan data. The intensity values may be binarized by thresholding, for example, in the case of CBCT/CT data, by setting all voxel values greater than a value of, for example, 500 HU to 1, and the remaining voxel to 0. In particular, for the purpose of generating training data, this threshold may be randomly chosen over samples ranging from 0 to 100. to be generated, for example, in an interval of 400 to 800u h.

The system can be used to predict parameters from 3D image data that have different modalities between the two received 3D image data sets. The same network can be trained with different sources comprising information taking into account different structures. For example, in the case of matching CBCT/CT information with IOS information, tooth crown surfaces can be distinguished with both received data sets, while, for example, gums are expected to be distinguishable only within the IOS data, and, for example, tooth roots can only be distinguished in the dec b c t /t a c data.

During the training effort, the internal parameters of the 3D deep neural network 1114 may be optimized so that the network provides results with sufficiently high accuracy. This may be achieved by employing a collection of 3D image data sets 1102 that may be of varying modalities but comprise at least one volume partially overlapping real-world structures. In order to train such a network, it is desirable that these two input sets be aligned or overlapping with each other 1104. If this is not already the case in the data 1102, it may be done either manually or automatically based on, for example, information following a method as described with respect to Figures 6-10. The accuracy of the overlap of the training data may affect the accuracy of the output data.

It is beneficial (taking into account accuracy, memory bandwidth requirements, and possible processing speed) for the data presented for training to the network 1114 to comprise the same real-world structures and to be scaled to the same real-world resolution in the voxel representations that will be provided to the 3D deep neural network 1114. If sufficient overlap is not already present within the received datasets, this may be done either manually or automatically based on a determined overlap region in the canonical coordinate system, for example by following a method as described with respect to Figure 3. If the input datasets have different resolutions, as may be known from metadata within the received data or, for example, derived by a method as described with respect to Figure 3, further scaling of the high-resolution data to the resolution of the low-resolution data 1108 may be beneficial.

Note that, in order to generate a multitude of training samples from the same set of received 3D image data sets, overlap region selection 1106 can be used such that not only the maximally overlapping volumes of interest (VOl) are selected, but also smaller volumes within such maximally overlapping volumes, effectively performing a 3D “augmentation” on a subset of the matched structure data.

In order to generate a multitude of training samples, random translation, rotation, and/or scaling transformations 1110 may be applied, effectively de-aligning the alignment present in the processed data upon reaching the processor 1110. This introduced misalignment may be passed, in the form of applicable transformation parameters, to the 3D deep neural network 1114 to serve as a training target for the predicted transformations. Rotations and/or translations of the voxel representations of any pre-processed dataset sample, or optionally both samples, may be performed, for example, by rotation methods employing linear (or other) interpolation as is known in the art.

A multitude of samples resulting from pre-processing of various sets of 3D image datasets comprising similar structures may be stored in a database 1112, and thus training of the network 1114 may be performed with a multitude of samples.

In another embodiment, independent 3D deep neural networks having a similar architecture can be trained for specific conditions, e.g., matching specific imaging modalities, comprising real-world structures, and/or specific size scaling of voxel representations. This can provide higher possible accuracy of results for specific cases, while still respecting hardware requirements such as available system memory, processing speed, etc.

Having been sufficiently trained 1114, “new data” 1116 can be presented for prediction or inference. This new data can be of the same type as described above, taking into account that it is a voxel representation of possibly different image modalities, etc. The canonical pose of dental structures in the input dataset may be determined by a first 3D deep neural network 1118, followed by the selection of data subsets representing VO1s 1120 that overlap in the canonical coordinate system, for example by a method as described with reference to Figure 3. If the input datasets have different resolutions, as may be known from metadata within the received data or, for example, derived by a method as described with respect to Figure 3, a rescaling of the high-resolution data to the resolution of the low-resolution data 1122 may be performed. This results in both datasets being pre-processed for reception by the 3D deep learning network 1114. Note that the pre-alignment and selection of overlapping VO1s is expected to be less accurate than the method being described here and, with respect to this, it may be considered that this method is a more highly precise refinement of a method as described, for example, with respect to Figure 3.

The trained 3D deep neural network may post-process the pre-processed data 1114 and output as a prediction the transformation parameters 1126 for the superposition of sample 1 and sample 2. Such a parameter set may comprise, for example, a 6-valued vector, encoding the first 3 applicable rotations to be executed in order along the three orthogonal axes of the received coordinate system for the data sample to be transformed (e.g., sample 2), and the last three values being the applicable translations, positive and/or negative, in order to align or superimpose, for example, sample 2 to sample 1.

In another embodiment, training can be performed with these parameters in the form of, for example, a rotation and/or translation matrix and/or a transformation matrix achieving the same desired result of alignment or superposition.

Note that, in the case where 1118, 1120 and/or 1122 were employed, the transformation parameters as predicted for the received samples may not yet provide the parameters for alignment or superposition of the originally received 3D image data sets. In such a case, processor 1128 may be used taking into account information considering any previously processed transformations from these three previous processors, i.e., “stacking” any previous transformations together with the transformation predicted for the samples, with system 1100 providing as output 1128 transformation parameters as may be applicable to the received 3D image data sets.

Note that the inference usage of this system can be considered relatively computationally unintensive, hence relatively fast. The accuracy of this system can be significantly higher when using pre-alignment and selection stages 1118 and 1120. This system can be highly robust to different imaging modalities and can operate at multiple resolutions (voxel sizes in the received voxel representations), employing voxel resolutions of 0.5–1 mm depending on the amounts of overlap between structures. However, it may not be accurate enough in the case of insufficient overlap. The elements constituting the various transformation parameter sets can be in the form of floating-point values.

Figures 12A and 12B show illustrations of received and transformed data as employed within, and resulting from, a system component for direct derivation of transformation parameters according to an embodiment of the invention. More specifically, they are visualizations of two received 3D image data sets (1202 and 1204). The visualizations are computer representations of the 3D image data sets ensuring voxel representation.

In these specific visualizations, the voxel sizes as used can be seen, which are 1 mm in any orthogonal direction. Although 1202 as received by the system component originated from CBCT data, for the purposes of this visualization it is shown as the 3D volume resulting from thresholding the CBCT data above 500 Hounsfield units. 1204 is the voxelized representation of an IOS from the same patient, and both received 3D image data were obtained at approximately the same moment in time.

From Figure 12B it can be seen that with the application of the transformation parameters from the system component, 1204 has been aligned or superimposed by means of 3D rotation and 3D translation. The received 3D image data sets were already scaled the same in the case of this example.

Figure 13 depicts an example of a 3D deep neural network architecture for a system component for direct derivation of transformation parameters according to an embodiment of the invention. The received (pre-processed) 3D image data, two voxel representations 1302, 1304 matching the voxel space of the 3D deep neural network input, may be passed through and processed by various layers in the network 1306-1320. The first layers of the network may include a plurality of 3D convolutional layers 1306-1314.

Once the data has passed through the convolutional layers, the internal representation can be passed to a series of densely connected layers 1316-1318 that infer the rotational and translational distance between the 3D data.

Variations in the number of layers and their definition parameters, for example different numbers of activation function kernels, use of subsampling and size, and additional functional layers such as dropout and/or batch normalization layers, can be used in the implementation without losing the essence of the 3D deep neural network design.

The final output layer 1320 may represent a prediction of the translation along the three axes and rotations along three axes that need to be applied to the data to obtain a correct overlay of the received 3D image data sets.

The training data may comprise sets of two voxel representations as input 1302, 1304 whose translations and rotations are known. For each data set of voxel representations to be processed, a randomized translation and rotation may be applied to either of them, and the difference in total translation and rotation may be used to determine a loss between the desired output 1322 and the actual output 1320. This loss may be used during training as a measure to adjust parameters within the layers of the 3D deep neural network. Such a loss may be calculated such that optimally accurate predictions are obtained from the 3D deep learning network. Optimizing functions may be used during training to aid in the efficiency of the training effort. The network may be trained for several iterations until the internal parameters lead to a desired accuracy of the results. When trained appropriately, two different voxel representations, for example, of maxillofacial structures can be presented as input and the 3D deep neural network can be used to derive a prediction of the translation and rotation needed to accurately superimpose the input set.

Layers can use a variety of activation functions such as linear, tanh, ReLU, PreLU, sigmoid, etc. 3D CNN layers can vary in the number of filters, filter sizes, and subsampling parameters, etc. Like densely connected layers, they can vary in their parameter initialization methods.

Figure 14 represents a flowchart of the system logic for selecting/determining transformation parameters to be applied in accordance with an embodiment of the invention. Note that this is an exemplary configuration of the system logic in accordance with various embodiments of the invention as described above. For the purposes of the flowchart, it is mentioned that the two input data sets have already been appropriately voxelized. The two input data sets may be received at step 1402, at which time a first set of transformation parameters for a canonical pose may be determined. In an exemplary embodiment, this step may be robust against large variations in transformation parameters to be applied for alignment or superposition purposes. The accuracy may be lower, and the resolution of the voxel representations of the received image data may be approximately 1 mm in any orthogonal direction.

From information at 1402, pre-alignment 1404 and a determination of sufficient overlap 1406 may be performed. Note that in one embodiment, this step may perform two determinations of sufficient overlap, one for each optional subsequent method to be performed (starting at 1410 and 1416 respectively). If the amount of overlap is insufficient, such as according to a threshold as may be determined experimentally and subsequently verified programmatically, the system may choose not to perform either or both of the methods starting at 1410, 1416. That is, this may be considered as the system determining that the resulting transformation parameters starting at 1426 will not improve due to unviable results from either or both of these additional methods.

In the case of sufficient overlap, the direct inference method may be performed starting at step 1410, which is expected, especially in the case of having performed prior alignment 1404 and VOI selection 1408, to provide more accurate results while being robust against different image modalities within the received 3D image data sets. Note that applicable information from prior transformations, such as may possibly follow from 1404 and 1408, may be relayed for use in determining transformation parameters 1412 following the direct inference method. The preprocessing 1410 employed within this method may be expected to result in voxel representations at a voxel resolution of 0.5-1.0 mm.

A sanity check 1414 of the viability of the results can still be performed. This can be done by comparing the resulting parameters of 1402 with those of 1414 and/or 1424. If the deviation is too great, the system may choose, for example, not to forward the parameters to 1426, or it may assign a weight of 0 to the resulting transformation parameters.

After determining applicable overlap, the system may employ the segmentation-based method starting at step 1416. Segmentation 1416 may be performed automatically on both sets of received 3D image data, either by employing deep neural network-based methods as described above, or by other methods known in the art such as may be the case with IOS data. Note that in the latter case, such tooth crown segmentations may be performed on the received 3D image data in the form of surface mesh data.

Taxonomy 1418 can be performed on the (segmented) structure data and the resulting information can be relayed to the keypoint generation stage 1420. The ability to include the identification of the same teeth in the different received data sets is expected to provide more robustness against possible variations in the amount of overlap and data quality of the received data sets.

The generated clouds of selected (sparse, closely matching) key points may be used in step 1422 to determine the applicable transformation parameters for alignment or superposition. Note that, again, any prior transformations possibly from 1404, 1408 may be taken into account by 1422 to determine transformation parameters for each set.

A sanity check 1424 may again be performed for this method, for example by checking deviations against resulting parameters from 1414 and/or 1402. In case of large discrepancies, the system may choose not to retransmit parameters to 1426. Alternatively, 1426 may assign a weight of 0 to the resulting transformation parameter set. Unviable results may be the results of inaccurate data received, such as artifacts present in CBCT data, incorrect surface representation from IoS data, etc.

Point data for surface meshes are saved with floating-point precision, providing potentially highly accurate results. This method can be considered the most accurate within this system, while at the same time being the least robust. However, it can be considered significantly more robust than current methods in the art due to the inclusion of pre-alignment determination, overlap, and segmentation and taxonomy of individual structures.

Transformation parameters can be represented internally in a variety of ways, for example as three 3-valued vectors respectively describing rotations in order, three translations about an origin, and/or three values determining an applicable scaling, all having positive and/or negative magnitudes of values pertaining to a specific axis in an orthogonal 3D coordinate system. Alternatively, any combination of matrices as known in linear algebra can be employed, more specifically any rotation, transformation, scaling, and/or combination as can be determined in an (affine) transformation matrix.

Prior knowledge may be employed taking into account accuracies, robustness, etc., for example, to determine an importance weighting of any/all of the transformation parameters received by 1426. Thus, step 1426 may programmatically combine the parameters received from the various methods to provide the most accurate desired transformation parameters for alignment or overlay.

Note that the transformation parameters, depending on the desired results from such a system, may be parameters that match set 2 to set 1, set 1 to set 2, and/or superimpose both on an alternative (desired) coordinate system.

Figure 15A and Figure 15B depict transformation results on two exemplary data sets received from various embodiments of the invention. More specifically, Figure 15A and Figure 15B show computer representations of two 3D image data sets 1502 and 1504. These 3D image data sets were respectively obtained from a CBCT scanner and an intraoral scanner. In accordance with a system configuration as described with respect to Figure 14, sufficient overlap was determined by the system and all three methods for transformation parameter generation were performed, employing pre-alignment according to the canonical pose method.

For the purposes of this visualization, the 3D CBCT image data are reproduced by means of the surface meshes generated for each tooth structure resulting from the segmentation method. In Figure 15A, the image data are shown oriented as received, and it can be seen that the scaling between both 3D image data sets was the same (e.g., 1 mm in real-world dimensions represents a unit value on each orthogonal axis for both received data sets). It can also be seen that 1502 and 1504 are misaligned considering both rotations and translations, to a significant degree.

The most accurate set of transformation parameters was determined to result from the segmentation and taxonomy method, matching and minimizing the distance between keypoints generated for both segmented (and labeled) teeth (crowns in the case of IOS data) as identified, therefore, in the case of this example, none of the applied transformation was a direct result of the other two methods. However, transformation parameters from the canonical pose method were employed while preprocessing was performed for the segmentation and taxonomy-based method.

Figure 15B shows the transformation parameters as determined by the system applied to 1504, the IOS data, with the system configured to determine and apply transformation parameters that align one received 3D image data set to the other. Note that although the overlap is only present for image volumes defining tooth indices 41, 31, 32, 33, 34, 35, and 36 (as can be determined using FDI notation), the final alignment or overlay step as applied based on the determined transformation parameters was performed automatically to a significantly high accuracy.

The aligned or overlapping data as shown can be further merged or combined, for example for teeth that have overlapping surface data (referenced to 132 in Figure 1). In the case of the data as displayed, specifically showing the results of the segmentation step providing complete teeth including accurate roots from the CBCT data and the combination with more accurate information considering tooth crowns from the IOS data, a surface fusion providing IOS tooth crowns fused to CBCT tooth roots will be very beneficial, for example in the fields of implantology or orthodontics as described above. Such fusion methods are known in the field and can greatly benefit from the accurate alignment that can be produced as a result of the system as described.

The methods described above can provide the most accurate results for the available overlay, while being robust to a large variability of conditions given the input data. This variability accounts for a variable but potentially large magnitude of misalignment between received 3D image datasets, different imaging modalities, and robustness to possibly low data quality (e.g., misinterpreted surfaces, artifacts in CBCT data, etc.). The system can operate fully automatically and can deliver the most accurate alignment or overlay results quickly. It should be noted that for any implementation of a 3D deep learning network, results and robustness are expected to increase with longer training periods/using more (varied) training data.

Although the examples in the figures are described with reference to 3D dental structures, it is clear that the embodiments in this application can generally be used to automatically determine (thus, without any human intervention) a canonical pose for the 3D object in 3D data sets from different modalities. Furthermore, the embodiments in this application can be used for the automatic superposition of a first 3D object with a second 3D object, wherein the first and second 3D objects can be represented by 3D data sets from different modalities.

Figure 16 is a block diagram illustrating an exemplary data processing system that may be used as described in this disclosure. The data processing system 1600 may include at least a processor 1602 coupled to memory elements 1604 via a system bus 1606. As such, the data processing system may store program code within the memory elements 1604. In addition, the processor 1602 may execute program code accessed from the memory elements 1604 via the system bus 1606. In one aspect, the data processing system may be implemented as a computer that is suitable for storing and/or executing program code. However, it should be appreciated that the data processing system 1600 may be implemented as any system that includes a processor and memory that can perform the functions described within this specification.

The memory elements 1604 may include one or more physical memory devices such as, for example, local memory 1608 and one or more large capacity storage devices 1610. Local memory may refer to random access memory or other non-persistent memory device(s) generally used during actual execution of the program code. A large capacity storage device may be implemented as a hard drive or other persistent data storage device. The processing system 1600 may also include one or more caches (not shown) that provide temporary storage of at least some program code in order to reduce the number of times that program code must be retrieved from the large capacity storage device 1610 during execution.

Optionally, input/output devices (1/0), represented as input device 1612 and output device 1614, may be coupled to the data processing system. Examples of input devices may include, but are not limited to, for example, a keyboard, a pointing device such as a mouse, or the like. Examples of output devices may include, but are not limited to, for example, a monitor or display, speakers, or the like. The input device and/or output device may be coupled to the data processing system either directly or through intermediate 1/0 drivers. A network adapter 1616 may also be coupled to the data processing system to allow it to be coupled to other systems, computer systems, remote network devices, and/or remote storage devices through intermediate private or public networks. The network adapter may comprise a data receiver for receiving data transmitted by said systems, devices, and/or networks, and a data transmitter for transmitting data to said systems, devices, and/or networks. Modems, cable modems, and Ethernet cards are examples of different types of network adapters that may be used with the data processing system 1600.

As shown in Figure 16, memory elements 1604 may store an application 1618. It should be appreciated that data processing system 1600 may further execute an operating system (not shown) that may facilitate execution of the application. The application, implemented as executable program code, may be executed by data processing system 1600, for example, by processor 1602. In response to executing the application, the data processing system may be configured to perform one or more operations that will be described in more detail herein.

In one aspect, for example, the data processing system 1600 may represent a client data processing system. In that case, the application 1618 may represent a client application that, when executed, configures the data processing system 1600 to perform the various functions described herein with reference to a “client.” Examples of a client may include, but are not limited to, a personal computer, a laptop computer, a mobile phone, or the like.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It is further understood that the terms “comprises” and/or “comprising,” when used herein, specify the presence of the mentioned features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

The structures, materials, actions, and corresponding equivalents of all step means or elements plus function in the following claims are intended to perform any structure, material, or action to perform the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention as disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art within the scope of the appended claims. The embodiment was chosen and described so as to best explain the principles of the invention and practical application, and to enable others of ordinary skill in the art to understand the invention in various embodiments with various modifications as are suitable for the particular use contemplated.

Claims (1)

v i n d i c a t i o n s A computer-implemented method for automatically determining a canonical pose of a 3D dental structure represented by data points from a 3D data set, the method comprising: providing one or more blocks of data points from the 3D data set associated with a first coordinate system as input to a first 3D deep neural network, the first 3D deep neural network being trained to generate canonical pose information associated with a canonical coordinate system defined with respect to a position of part of a 3D dental structure and with a canonical representation of the 3D dental structure, wherein the size of the 3D dental structure is scaled and wherein one or more predetermined features of the canonical representation of the 3D dental structure are aligned with axes of the canonical coordinate system; receiving canonical pose information from the output of the first 3D deep neural network, the canonical pose information comprising, for each data point of the one or more blocks, a prediction of a position of a data point in the canonical coordinate system, the position of the data point being defined by canonical coordinates; using the canonical coordinates to determine an orientation and scaling of the axes of the canonical coordinate system and a position of the origin of the canonical coordinate system relative to the axis and origin of the first coordinate system, and using the orientation and position to determine transformation parameters, preferably rotation, translation, and/or scaling parameters, to transform coordinates of the first coordinate system into canonical coordinates; and determining a canonical representation, preferably a canonical voxel representation or a canonical 3D mesh representation, of the 3D tooth structure, including determining and applying the transformation parameters to coordinates of the data points of the 3D data set. 1. The method of claim 1 , wherein the canonical pose information includes one or more voxel maps for linking a voxel of a voxel representation to a prediction of a position of the voxel in the canonical coordinate system, the one or more voxel maps preferably including a first 3D voxel map linking a voxel to a prediction of a first x-coordinate of the canonical coordinate system, a second 3D voxel map linking a voxel to a prediction of a second y-coordinate of the canonical coordinate system, and a third 3D voxel map linking a voxel to a prediction of a third z-coordinate of the canonical coordinate system. Method according to claim 2, wherein determining an orientation of an axis of the canonical coordinate system further comprises: determining for a voxel of the voxel representation a local gradient at a canonical coordinate of one of the one or more 3D voxel maps, the local gradient representing a vector in the space defined by the first coordinate system, wherein the orientation of the vector represents a prediction of the orientation of a canonical axis and/or wherein the length of the vector defines a scaling factor associated with the canonical axis. A computer-implemented method for the automated superposition of a first 3D dental structure, represented by a first 3D data set, and a second 3D dental structure, represented by a second 3D data set, preferably the first and second 3D dental structures being of the same person, the method comprising: providing one or more first voxel blocks of a first voxel representation of the first 3D tooth structure associated with a first coordinate system and one or more second voxel blocks of a second voxel representation of the second 3D tooth structure associated with a second coordinate system to the input of a first 3D deep neural network, the first 3D deep neural network being trained to generate canonical pose information associated with a canonical coordinate system defined with respect to a position of part of a first and second 3D tooth structures and with a canonical representation of the 3D tooth structure, wherein the size of the 3D tooth structure is scaled and wherein one or more predetermined features of the canonical representation of the 3D tooth structure are aligned with axes of the canonical coordinate system; receiving first and second canonical pose information from the output of the 3D deep neural network, the first canonical pose information comprising, for each voxel of the one or more first blocks, a prediction of a first position of the voxel in the canonical coordinate system; and, the second canonical pose information comprising, for each voxel of the one or more second blocks, a prediction of a second position of the voxel in the canonical coordinate system, the first and second positions being defined by first and second canonical coordinates respectively; using the first canonical pose information to determine a first orientation and scale of the axes and a first position of the origin of the axes in the first coordinate system and using the second canonical pose information to determine a second orientation and scale of the axes and a second position of the origin of the axes of the canonical coordinate system in the second coordinate system; using the first orientation, scale, and the first position to determine first transformation parameters, preferably first rotation, translation, and/or scaling parameters, to transform coordinates of the first coordinate system into coordinates of the canonical coordinate system; and, using the second orientation, scale, and the second position to determine second transformation parameters, preferably second rotation, translation, and/or scaling parameters, to transform coordinates of the second coordinate system into canonical coordinates; and, determining a superposition of the first 3D tooth structure and the second 3D tooth structure, including determining using the first and second transformation parameters to form first and second canonical representations of the first and second 3D tooth structures respectively. Method according to claim 4 wherein the first and second canonical representations of the first and second 3D dental structures are 3D surface meshes, including determining an overlay further: segmenting the first canonical representation of the first 3D tooth structure into at least one 3D surface mesh of at least one 3D tooth element of the first 3D tooth structure and segmenting the second canonical representation of the second 3D tooth structure into at least one 3D surface mesh of at least one second 3D tooth element of the second 3D tooth structure; selecting at least three non-collinear first and second key points of the first and second 3D surface meshes, preferably defining a key point as a local and/or global maximum minimum in the surface curvature of the first 3D surface mesh; and, align the first and second tooth structures in 3D based on the first and second non-collinear key points. Method according to claim 4, wherein the first and second canonical representations of the first and second 3D dental structures are voxel representations, including determining an overlay further: providing at least part of the first canonical voxel representation of the first 3D tooth structure and at least part of the second canonical voxel representation of the second 3D tooth structure as input to a second 3D deep neural network, the second 3D deep neural network being trained to determine transformation parameters, preferably rotation, translation and/or scaling parameters, to align the first and second canonical voxel representations; aligning the first and second canonical representations of the first and second 3D dental structures based on the transformation parameters provided by the output of the second 3D deep neural network. Method according to claim 4, wherein determining an overlap further includes: determining an overlap volume between the canonical representation of the first 3D tooth structure and the canonical representation of the second 3D tooth structure; and, determining a first volume of interest comprising first voxels of the first canonical representation in the overlapping volume; and, determining a second volume of interest comprising second voxels of the second canonical representation in the overlapping volume. Method according to claim 7, further comprising: providing first voxels contained in the first volume of interest, VOI, as input to a third 3D deep neural network, the third 3D deep neural network being trained to classify and segment voxels; and, receiving activation values for each of the first voxels in the first volume of interest and/or for each of the second voxels in the second volume of interest from the output of the third 3D deep neural network, wherein an activation value of a voxel represents the probability that the voxel belongs to a predetermined 3D object class, for example a tooth of a 3D dental structure; using the activation values to determine first and second voxel representations of a first and second 3D dental element in the first and second VOI, respectively; and; optionally, using the first and second voxel representations of the first and second 3D dental elements to determine first and second 3D surface meshes of the first and second 3D dental elements. Method according to claim 8, further comprising: selecting at least three non-collinear first and second key points of the first and second 3D surface meshes, preferably defining a key point as a local and/or global maximum minimum in the surface curvature of the first surface mesh; and, align the first and second dental elements in 3D based on the first and second non-collinear key points, preferably using an iterative closest point algorithm. Method according to claim 8, further comprising: providing a first voxel representation of a first 3D dental element and a second voxel representation of a second 3D dental element to a fourth 3D deep neural network, the fourth 3D deep neural network being trained to generate an activation value for each of a plurality of candidate structure labels, an activation value associated with a candidate label representing the probability that a voxel representation received by the input of the fourth 3D deep neural network represents a type of structure as indicated by the candidate structure label; receiving, from the output of the fourth 3D deep neural network, a plurality of first and second activation values, selecting a first structure label with the highest activation value from the first plurality of activation values and selecting a second structure label with the highest activation value from the second plurality of activation values, and assigning the first and second structure labels to the first and second 3D surface meshes respectively. Method according to claim 10, further comprising: selecting at least three non-collinear first and second key points of the first and second 3D surface meshes, preferably defining a key point as a local and/or global maximum minimum in the surface curvature of the first surface mesh; label the first and second key points based on the first structure label assigned to the first 3D surface mesh and the second structure label assigned to the second 3D surface mesh respectively; aligning the first and second 3D dental elements based on the first and second key points and the first and second structure labels of the first and second 3D surface meshes respectively, preferably using an iterative closest point algorithm. Computer-implemented method for training a 3D deep neural network to automatically determine a canonical pose of a 3D dental structure represented by a 3D dataset, comprising: receiving training data and associated target data, the training data including a voxel representation of a 3D tooth structure, and the target data including canonical coordinate values of a canonical coordinate system for each voxel of the voxel representation, wherein the canonical coordinate system is a predetermined coordinate system defined with respect to a position of a portion of the 3D tooth structure, the canonical coordinate system being associated with a canonical representation of the 3D tooth structure, wherein the size of the 3D tooth structure is scaled, and wherein one or more predetermined features of the canonical representation of the 3D tooth structure are aligned with axes of the canonical coordinate system; selecting one or more voxel blocks, the one or more blocks representing one or more subsamples of the voxel representation of a predetermined size, and applying a random 3D rotation to the one or more subsamples; apply the same rotation to the target data; providing the one or more blocks as input to the 3D deep neural network and predicting the 3D deep neural network, for each voxel in the one or more blocks, canonical coordinates of the canonical coordinate system; and Optimize network parameter values of the 3D deep neural network by minimizing a loss function that represents a deviation between the coordinate values predicted by the 3D deep neural network and the canonical coordinates associated with the target data. A computer system adapted to automatically determine a canonical posture of a 3D dental structure represented by a 3D data set, comprising: a computer-readable storage medium having computer-readable program code implemented therein, the program code including at least one trained 3D deep neural network, and at least one processor, preferably a microprocessor, coupled to the computer-readable storage medium, wherein, in response to execution of the computer-readable program code, the at least one processor is configured to perform executable operations comprising: providing one or more voxel blocks of a voxel representation of the 3D tooth structure associated with a first coordinate system as input to a first 3D deep neural network, the first 3D deep neural network being trained to generate canonical pose information associated with a canonical coordinate system defined with respect to a position of part of the 3D tooth structure and with a canonical representation of the 3D tooth structure, wherein the size of the 3D tooth structure is scaled and wherein one or more predetermined features of the canonical representation of the 3D tooth structure are aligned with axes of the canonical coordinate system; receiving canonical pose information from the output of the first 3D deep neural network, the canonical pose information comprising, for each voxel of the one or more blocks, a prediction of a position of the voxel in the canonical coordinate system, the position being defined by canonical coordinates; using the canonical coordinates to determine an orientation and scale of the axes of the canonical coordinate system and a position of the origin of the canonical coordinate system with respect to the axis and origin of the first coordinate system and using the orientation, scale and position to determine transformation parameters, preferably rotation, translation and/or scaling parameters, to transform coordinates of the first coordinate system into canonical coordinates; and, determining a canonical representation, preferably a canonical voxel representation or a canonical 3D mesh representation, of the 3D tooth structure, including determining and applying the transformation parameters to coordinates of the voxels of the voxel representation or the 3D data set used to determine the voxel representation. A computer system adapted for the automated superposition of a first 3D dental structure represented by a first 3D data set and a second 3D dental structure represented by a second 3D data set, comprising: a computer-readable storage medium having computer-readable program code implemented therein, the program code including at least one trained 3D deep neural network, and at least one processor, preferably a microprocessor, coupled to the computer-readable storage medium, wherein, in response to execution of the computer-readable program code, the at least one processor is configured to perform executable operations comprising: providing one or more first voxel blocks of a first voxel representation of the first 3D tooth structure associated with a first coordinate system and one or more second voxel blocks of a second voxel representation of the second 3D tooth structure associated with a second coordinate system as input to a 3D deep neural network; the 3D deep neural network being trained to generate canonical pose information associated with a canonical coordinate system defined with respect to a position of part of the first and second 3D tooth structures and with a canonical representation of the 3D tooth structure, wherein the size of the 3D tooth structure is scaled and wherein one or more predetermined features of the canonical representation of the 3D tooth structure are aligned with axes of the canonical coordinate system; receiving first and second canonical pose information from the output of the 3D deep neural network, the first canonical pose information comprising, for each voxel of the one or more first blocks, a prediction of a first position of the voxel in the canonical coordinate system; and, the second canonical pose information comprising, for each voxel of the one or more second blocks, a prediction of a second position of the voxel in the canonical coordinate system, the first and second positions being defined by first and second canonical coordinates respectively; using the first canonical pose information to determine a first orientation and scale of the axes and a first position of the origin of the axes in the first coordinate system and using the second canonical pose information to determine a second orientation and scale of the axes and a second position of the origin of the axes of the canonical coordinate system in the second coordinate system; using the first orientation and the first position to determine first transformation parameters, preferably first rotation, translation and/or scaling parameters, to transform coordinates of the first coordinate system into coordinates of the canonical coordinate system; and, using the second orientation and the second position to determine second transformation parameters, preferably second rotation, translation and/or scaling parameters, to transform coordinates of the second coordinate system into canonical coordinates; and, determining a superposition of the first 3D tooth structure and the second 3D tooth structure, including determining using the first and second transformation parameters to form first and second canonical representations of the first and second 3D tooth structures respectively. Computer program product comprising portions of software code configured to, when executed in the memory of a computer, execute the method steps according to any of claims 1-11.