Computational pathology: A survey review and the way forward^☆

2.1. Clinical pathology workflow

This subsection provides a general overview of the clinical workflow in pathology covering the collection of a tissue sample, its subsequent processing into a slide, inspection by a pathologist, and compilation of the analysis and diagnosis into a pathology. Fig. 2 summarizes these steps at a high level and provides suggestions for corresponding CPath applications. The steps are organized under the conventional pathology phases for samples: pre-analytical, analytical, and post-analytical. These phases were developed to categorize quality control measures, as each phase has its own set of potential sources of errors,⁶⁸ and thus potential sources of corrections during which CPath and healthcare artificial intelligence tools could prove useful. For details about each step of the workflow, please refer to the Appendix A.1.

Pre Analytical Phase The first step of the pre-analytical phase is a biopsy performed to collect a tissue sample, where the biopsy method is dependent on the type of sample required and the tissue characteristics. Sample collection is followed by accessioning of the sample which involves entering of the patient and specimen information into a Laboratory Information System (LIS) and linking to the Electronic Medical Records (EMR) and potentially a Slide Tracking System (STS). After accessioning, smaller specimens that have not already been preserved by fixation in formalin are fixated. Once the basic specimen preparation has occurred, the tissue is analyzed by the pathology team without the use of a microscope; a step called grossing. Grossing involves cross-referencing the clinical findings and the EMR reports, with the operator localizing the disease, locating the pathological landmarks, describing these landmarks, and measuring disease extent. Specific sampling of these landmarks is performed, and these samples are then put into cassettes for the final fixation. Subsequently, the samples are then sliced using a microtome, stained using the relevant stains for diagnosis, and covered with a glass slide.

Analytical Phase After a slide is processed and prepared, a pathologist views the slide to analyze and interpret the sample. The approach to interpretation varies depending on the specimen type. Interpretation of smaller specimens is focused on diagnosis of any disease. Analysis is performed in a decision-tree style approach to add diagnosis-specific parameters, e.g. esophagus biopsy $\to$ type of sampled mucosa $\to$ presence of folveolar-type mucosa $\to$ identify Barrett’s metaplasia $\to$ identify degree of dysplasia. Once the main diagnosis has been identified and characterized, the pathologist sweeps the remaining tissue for secondary diagnoses which can also be characterized depending on their nature. Larger specimens are more complex and usually focus on characterizing the tissue and identifying unexpected diagnoses beyond the prior diagnosis from a small specimen biopsy. Microscopic interpretation of large specimens is highly dependent on the quality of the grossing and the appropriate detection and sampling of landmarks. Each landmark (e.g., tumor surface, tumor at deepest point, surgical margins, lymph node in mesenteric fat) is characterized either according to guidelines, if available, or according to the pathologist’s judgment. After the initial microscopic interpretation additional deeper cuts (“levels”), special stains, immunohistochemistry (IHC), and/or molecular testing may be performed to hone the diagnosis by generating new material or slides from the original tissue block.

Post-Analytical Phase The pathologist synthesizes a diagnosis by aggregating their findings from grossing and microscopic examination in combination with the patient’s clinical information, all of which are included in a final pathology report. The classic sections of a pathology report are patient information, a list of specimens included, clinical findings, grossing report, microscopic description, final diagnosis, and comment. The length and degree of complexity of the report again depends on the specimen type. Small specimen reports are often succinct, clearly and unambiguously listing relevant findings which guide treatment and follow-up. Large specimen reports depend on the disease, for example, in cancer resection specimens the grossing landmarks are specifically targeted at elements that will guide subsequent treatment.

In the past, pathology reports had no standardized format, usually taking a narrative-free text form. Free text reports can omit necessary data, include irrelevant information, and contain inconsistent descriptions.⁶⁹ To combat this, synoptic reporting was introduced to provide a structured and standardized reporting format specific to each organ and cancer of interest.^69,70 Over the last 15 years, synoptic reporting has enabled pathologists to communicate information to surgeons, oncologists, patients, and researchers in a consistent manner across institutions and even countries. The College of American Pathologists (CAP) and the International Collaboration on Cancer Reporting (ICCR) are the two major institutions publishing synoptic reporting protocols. The parameters included in these protocols are determined and updated by CAP and ICCR respectively to remain up-to-date and relevant for diagnosis of each cancer type. For the field of computational pathology, synoptic reporting provides a significant advantage in dataset and model creation, as a pre-normalized set of labels exist across a variety of cases and slides in the form of the synoptic parameters filled out in each report. Additionally, suggestion or prediction of synoptic report values are a possible CPath application area.

2.2. Diagnostic tasks

Computational pathology systems that focus on diagnostic tasks can broadly be categorized as: (1) disease detection, (2) tissue subtype classification, (3) disease diagnosis, and (4) segmentation. These tasks are visually depicted in Fig. 3. Note how the detection tasks all involve visual analysis of the tissue in WSI format. Thus computer vision approach is primarily adopted towards tackling diagnostic tasks in computer aided diagnosis (CAD). For additional detail on some previous works on these diagnostic tasks, we refer the reader to Appendix A.2

Detection We define the detection task as a binary classification problem where inputs are labeled as positive or negative, indicating the presence or absence of a certain feature. There may be variations in the level of annotation required, e.g. slide-level, patch-level, pixel-level detection depending on the feature in question. Although detection tasks may not provide an immediate disease diagnosis, it is a highly relevant task in many pathology workflows as pathologists incorporate the presence or absence of various histological features into synoptic reports that lead to diagnosis. Broadly, detection tasks fall into two main categories: (1) screening the presence of cancers and (2) detecting histopathological features specific to certain diagnoses.

Cancer detection algorithms can assist the pathologists by filtering obviously normal WSIs and directing pathologist’s focus to metastatic regions.⁷¹ Although pathologists have to review all the slides to check for multiple conditions regardless of the clinical diagnosis, an accurate cancer detection CAD would expedite the workflow by pinpointing the ROIs and summarizing results into synoptic reports, ultimately leading to a reduces time per slide. Due to this potential impact, cancer detection tasks have been explored in a broad set of organs. Additionally, the simple labeling in binary detection tasks allows for deep learning methods to generalize across different organs where similar cancers form.72, 73, 74

Tissue Subtype Classification Treatment and patient prognosis can vary widely depending on the stage of cancer, and finely classifying specific tissue structures associated with a specific disease type provides essential diagnostic and prognostic information.⁷⁵ Accordingly, accurately classifying tissue subtypes is a crucial component of the disease diagnosis process. As an example, discriminating between two forms of glioma (a type of brain cancer), glioblastoma multiforme and lower grade glioma, is critical as they differ by over $45\%$ in patient survival rates.⁷⁶ Additionally, accurate classification is key in colorectal cancer (CRC) diagnosis, as high morphological variation in tumor cells⁷⁷ makes certain forms of CRC difficult to diagnose by pathologists.⁷⁸ We define this classification of histological features as tissue subtype classification.

Disease Diagnosis The most frequently explored design of deep learning in digital pathology involves emulating pathologist diagnosis. We define this multi-class diagnosis problem as a disease diagnosis task. Note the similarity with detection–disease diagnosis can be considered a fine-grained classification problem which subdivides the general positive disease class into finer disease-specific labels based on the organ and patient context.

Segmentation The segmentation task moves one step beyond classification by adding an element of spatial localization to the predicted label(s). In semantic segmentation, objects of interest are delineated in an image by assigning class labels to every pixel. These class labels can be discrete or non-discrete, the latter being a more difficult task.⁷⁹ Another variant of the segmentation task is instance segmentation, which aims to achieve both pixel-level segmentation accuracy as well as clearly defined object (instance) boundaries. Segmentation approaches can accurately capture many morphological statistics⁸⁰ and textural features,⁸¹ both of which are relevant for cancer diagnosis and prognosis. Most frequently, segmentation is used to capture characteristics of individual glands, nuclei, and tumor regions in WSIs. For instance, glandular structure is a critical indicator of the severity of colorectal carcinoma,⁸² thus accurate segmentation could highlight particularly abnormal glands to the pathologist as demonstrated in.82, 83, 84 Overall, segmentation provides localization and classification of cancer-specific tumors and of specific histological features that can be meaningful for the pathologist’s clinical interpretation.

2.3. Prognosis

Prognosis involves predicting the likely development of a disease based on given patient features. For accurate survival prediction, models must learn to both identify and infer the effects of histological features on patient risk. Prognosis represents a merging of the diagnosis classification task and the disease-survivability regression task.

Training a model for prognosis requires a comprehensive set of both histopathology slides and patient survival data (i.e. a variant of multi-modal representation learning). Despite the complexity of the input data, ML models are still capable of extracting novel histological patterns for disease-specific survivability.85, 86, 87 Furthermore, strong models can discover novel prognostically-relevant histological features from WSI analysis.^88,89 As the quality and comprehensiveness of data improves, additional clinical factors could be incorporated into deep learning analysis to improve prognosis.

2.4. Prediction of treatment response

With the recent advances in targeted therapy for cancer treatment, clinicians are able to use treatment options that precisely identify and attack certain types of cancer cells. While the number of options for targeted therapy are constantly increasing, it becomes increasingly important to identify patients who are potential therapy responders to a specific therapy option and avoid treating non-responding patients who may experience severe side effects. Deep learning can be used to detect structures and transformations in tumour tissue that could be used as predictive markers of a positive treatment response. Training such deep learning models usually requires large cohorts of patient data for whom the specific type of treatment option and the corresponding response is known.

2.5. Organs and diseases

This section presents an overview of the various anatomical application areas for computational pathology grouped by the targeted organ. Each organ section gives a brief overview of the types of cancers typically found and the content of the pathology report as noted from the corresponding CAP synoptic reporting outline (discussed at 2.1). Fig. 4 highlights the intersection between the major diagnostic tasks and the anatomical focuses in state-of-the-art research. The majority of papers are dedicated to the four most common cancer sites: breast, colon, prostate, and lung.⁹⁰ Additionally, a significant amount of research is also done on cancer types with highest mortality, brain and liver.⁹⁰ Note that details of some additional works that may be of interest for each organ type can be found in Appendix A.7 (see Fig. 5).

Breast Breast cancers can start from different parts of the breast and majorly consist of 1) Lobular cancers that start from lobular glands, 2) Ductal cancers, 3) Paget cancer which involves the nipple, 4) Phyllodes tumor that stems from fat and connective tissue surrounding the ducts and lobules, and 5) Angiosarcoma which starts in the lining of the blood and lymph vessels. In addition, based on whether the cancer has spread or not, breast cancers can be categorized into in situ or invasive/infiltrating forms. DCIS is a precancerous state and is still confined to the ducts. Once the cancerous cells grow out of the ducts, the carcinoma is now considered invasive or infiltrative and can metastasize.⁹¹

Synoptic reports for breast cancer diagnosis are divided based on the type of cancers mentioned above. For DCIS and invasive breast cancers, synoptic reports focus on the histologic type and grade, along with the nuclear grade, evidence of necrosis, margin, involvement of regional lymph nodes, and biomarker status. Notably, architectural patterns are no longer a valuable predictive tool compared to nuclear grade and necrosis to determine a relative ordering of diagnostic importance for DCIS.⁹² In contrast to DCIS and invasive cancers, Phyllodes tumours vary due to their differing origin in the fat and connective tissue, focusing on analyzing the stroma characteristics, existence of heterologous elements, mitotic rate, along with the involvement of lymph nodes. Finally, to determine therapy response and treatments, biomarker tests for Estrogen, Progesterone⁹³ and HER-2⁹⁴ receptors are recommended, along with occasional tests for Ki67 antigens.^95,96

Most breast cancer-focused works in CPath propose various solutions for carcinoma detection and metastasis detection, an important step for assessing cancer stage and morbidity. Metastasis detection using deep learning methods was shown to outperform pathologists’ exhaustive diagnosis by $9\%$ free-response receiver operating characteristic (FROC) in.⁹⁷

Prostate Prostate cancer is the second most prevalent cancer among the total population and the most common cancer among men (both excluding non-melanoma skin cancers). However, most prostate cancers are not lethal. Prostate cancer can occur in any of the three prostate zones: Central (CZ), Peripheral (PZ), and Transition (TZ), in increasing order of aggressiveness. Prostate cancers are almost always adenocarcinomas, which develop from the gland cells that make prostate fluid. The other types of prostate cancers are small cell carcinomas, neuroendocrine tumors, transitional cell carcinomas, isolated intraductal carcinoma, and sarcomas (which are very rare). Other than cancers, there are multiple conditions that are important to identify or diagnose as precursors to cancer or not. Prostatic intraepithelial neoplasia (PIN) is diagnosed as either low-grade PIN or high-grade PIN. Men with high-grade PIN need closely monitored follow-up sessions to screen for prostate cancer. Similarly, atypical small acinar proliferation (ASAP) is another precancerous condition requiring follow-up biopsies.⁹⁸

To grade and score tumours, pathologists use a Tumor, Nodes, Metastasis (TNM) framework. In the synoptic report, pathologists identify and report the histologic type and grades, and involvement of regional lymph nodes to help grade and provide prognosis for any tumours. Specifically for prostate analysis, tumour size and volume are both important factors in prognosis according to multiple studies.99, 100, 101, 102 Similarly, location is important to note for both prognosis and therapy response.¹⁰³ Invasion to nearby (except perineural invasion) tissues is noted and can correlate to TMN classification.¹⁰⁴ Additionally, margin analysis is especially important in prostate cancers as the presence of a positive margin increases the risk of cancer recurrence and metastasis.¹⁰⁵ Finally, intraductal carcinoma (IDC) must be identified and distinguished from PIN and PIA; as it is strongly associated with a high Gleason score, a high-volume tumor, and metastatic disease.106, 107, 108, 109, 110

After a prostate cancer diagnosis is established, pathologists assign a Gleason Score to determine the cancer’s grade: a grade from 1 to 5 is assigned to the two most common areas and those two grades are summed to make a final Gleason Score.¹¹¹ For Gleason scores of 7, where survival and clinical outcomes demonstrate large variance, the identification of Cribriform glands is key in helping to narrow possible outcomes.^112,113

Ovary Ovarian cancer is the deadliest gynecologic malignancy and accounts for more than $\mathrm{14,000}$ deaths each year.¹¹⁴ Ovarian cancer manifests in three types: 1) epithelial cell tumors that start from the epithelial cells covering the outer surface of the ovary, 2) germ cell tumors which start from the cells that produce eggs, and 3) stromal tumors which start from cells that hold the ovary together and produce the hormones estrogen and progesterone. Each of these cancer types can be classified into benign, intermediate and malignant categories. Overall, epithelial cell tumors are the most common ovarian cancer and have the worst prognosis.¹¹⁵

When compiling a synoptic report for ovarian cancer diagnosis, pathologists focus on histologic type and grade, extra-organ involvement, regional lymph nodes, T53 gene mutations, and serous tubal intraeptithelial carconma (STIC). Varying histologic tissue types are vital to determine the pathology characteristics and determining eventual prognosis. For example, generally endometrioid, mucinous, and clear cell carcinomas have better outcomes than serous carcinomas.¹¹⁶ Additionally, lymph node involvement and metastasis in both regional and distant nodes has a direct correlation to patient survival, grading, and treatment. Determining the presence of STICs correlates directly to the presence of ovarian cancer, as $60\%$ of ovarian cancer patients will also have an associated STIC.¹¹⁴ Finally, T53 gene mutations are the most common in epithelial ovarian cancer; which has the worst prognosis among ovarian cancers, so determining their presence is critical to patient cancer risk and therapy response.^117,118 There are not a large number of works dedicated to the ovary specifically, but most works on ovary focus on classification of its five most common cancer subtypes: high-grade serous (HGSC), low-grade serous (LGSC), endometriod (ENC), clear cell (CCC), and mucinous (MUC).^119,120

Lung Lung cancer is the third most common cancer, next to breast and prostate cancer.¹²¹ Lung cancers mostly start in the bronchi, bronchioles, or alveoli and are divided into two major types, non-small cell lung carcinomas (NSCLC) ($80-85\%$) and small cell lung carcinomas (SCLC) ($10-15\%$). Although NSCLS cancers are different in terms of origin, they are grouped because they have similar outcomes and treatment plans. Common NSCLS cancers are 1) adenocarcinoma, 2) squamous cell carcinoma 3) large cell carcinoma, and some other uncommon subtypes.¹²²

For reporting, histologic type helps determine NSCLC vs SCLC and the subtype of NSCLC. Although NSCLC generally has favourable survival rates and prognosis as compared to SCLC, certain subtypes of NSCLC can have lower survival rates due to co-factors.¹²³ Histologic patterns are applicable in adenocarcinomas, consisting of favourable types: lepidic, intermediate: acinar and papillary, and unfavourable: micropapillary and solid.¹²⁴ Grading each histologic type aids in categorization but is differentiated based on each type, and thus is out of scope for this paper. Importantly for lung cancers, tumour size is an independent prognostic factor for early cancer stages, lymph node positivity, and locally invasive disease. Additionally, the size of the invasive portion is an important factor for prognosis of nonmucinous adenocarcinoma with lepidic pattern.^123,125, 126, 127, 128, 129 Other important lung specific features are visceral pleural invasion, which is associated with worse prognosis in early-stage lung cancer even with tumors <3cm,¹³⁰ and lymphatic invasion which indicates an unfavourable prognostic finding.^125,131

Colon and Rectum Colorectal cancers are two of the five most common cancer types.⁹⁰ Cancer cells usually start to develop in the innermost layer of the colon and rectum walls, known as the mucosa, and continue their way up to other layers. In other layers, there are lymph and blood vessels that can be used by cancer cells to travel to nearby lymph nodes or other organs.¹³² Colorectal cancers usually start with the creation of different types of polyps, each possessing a unique risk of developing into cancer. Most colorectal cancers are adenocarcinomas, which are split into three well-studied subtypes: classic adenocarcinoma (AC), signet ring cell carcinoma (SRCC), and mucinous adenocarcinoma (MAC). In most cases, AC has a better prognosis than MAC or SRCC. Other types, albeit uncommon, of colorectal cancers are: carcinoid tumors, gastrointestinal stromal tumors (GISTs), lymphomas, and sarcomas.¹³³

As in other cancers, histologic grade is the most important factor in cancer prognosis along with regional lymph node status and metastasis. The tumor site is also important in determining survival rates and prognosis.¹³⁴ Vascular invasion of both small and large vessels are important factors in adverse outcomes and metastasis,135, 136, 137 and perineural invasion has been shown in multiple studies to be an indicator of poor prognosis.137, 138, 139 Additionally, microsatellite instability (MSI) is shown to be a good indicator of prognosis and is divided into three stages in decreasing adversity of Stable (MSI-S), Low (MSI-L), and High (MSI-H).¹⁴⁰ Finally, some studies have indicated the usefulness of biomarkers in colorectal cancer treatment, with biomarkers such as BRAF mutations, KRAS mutations, MSI, APC, Micro-RNA, and PIK3CA.¹⁴¹

Works are relatively well-distributed among various tasks including disease diagnosis, segmentation, and detection. Expanding on colorectal cancer detection, work from¹⁴² used feature analysis for colorectal and mucinous adenocarcinomas using heatmap visualizations. They discovered that adenocarcinoma is often detected by ill-shaped epithelial cells and that misclassification can occur due to lumen regions that resemble the malformed epithelial cells. Similarly for mucinous carcinoma, the model again recognizes the dense epithelium, but this time ignores the primary characteristic of the carcinoma (abundance of extracellular mucin). These findings suggest that a thorough analysis of class activation maps can be helpful for improving the classifier’s accuracy and intuitiveness.

Bladder There are several layers within the bladder wall with most cancers starting in the internal layer, called the urothelium or transitional epithelium. Cancers remaining in the inner layer are non-invasive or carcinoma in situ (CIS) or stage 0. If they grow into other layers such as the muscle or fatty layer, the cancer is now invasive. Nearly all bladder cancers are urothelial carcinomas or transitional cell carcinomas (TCC). However, there are other types of cancer such as squamous cell carcinomas, adenocarcinomas, small cell carcinomas, and sarcomas which all are very rare. In the early stages, all types of bladder cancers are treated similarly but as their stage progresses, and chemotherapy is needed, different drugs might be used based on the type of the cancer.¹⁴³ As with other organs, histologic type and grade also play a role in prognosis and treatment,¹⁴⁴ and lymphovascular invasion is independently associated with poor prognosis and recurrence.¹⁴⁵

Works focusing on the bladder display promising results that could lead to rapid clinical application. For example, a prediction method for four molecular subtypes (basal, luminal, luminal p53, and double negative) of muscle-invasive bladder cancer was proposed in,¹⁴⁶ outperforming pathologists by $30\%$ in classification accuracy when restricted to a tissue morphology-based assessment. Further improvements in accuracy could help expedite diagnosis by complementing traditional molecular testing methods.

Kidney Each kidney is made up of thousands of glomeruli which feed into the renal tubules. Kidney cancer can occur in the cells that line the tubules (renal cell carcinoma (RCC)), blood vessels and connective tissue (sarcomas), or urothelial cells (Urothelial carcinoma). RCC accounts for about $90\%$ of kidney cancers and comes in two types: 1) clear cell renal carcinoma, which are most common and 2) non-clear cell renal carcinoma consisting of papillary, chromophobe and some very rare subtypes.¹⁴⁷ The CAP’s cancer protocol template for the kidney is solely focused on RCCs,¹⁴⁸ likely due to their high probability. Tumour size is directly associated with malignancy rates, with $1$cm size increase resulting in $16\%$ increase in malignancy chance.¹⁴⁹ Additionally, the RCC histologic type is correlated with metastasis, with clear cell, capillary, collecting ducts (Bellini), and medullary being the most aggressive ones.¹⁵⁰

Many works are focused on glomeruli segmentation, as the number of glomeruli and glomerulosclerosis constitute standard components of a renal pathology report.¹⁵¹ In addition to glomeruli detection, some works have also detected other relevant features such as tubules, Bowman’s capsules, and arteries.¹⁵² The results display strong performance on PAS-stained nephrectomy samples and tissue transplant biopsies, and there seems to be a strong correlation between the visual elements identified by the network and those identified by renal pathologists.

Brain There are two main types of brain tumors: malignant and non-malignant. Malignant tumors can be classified as primary tumors (originating from the brain) or secondary (metastatic).^153,154 The most common type of brain cancers is gliomas, occurring $50.4\%$ of the time, and are classified into four grades.¹⁵⁵ In the synoptic reporting, tumour location is noted as it has some impact on the prognosis, with parietal tumours showing better prognosis compared to other locations.¹⁵³ Additionally, focality of glioblastomas (a subtype of gliomas) is important to determine as multifocal glioblastoma is far more aggressive and resistant to chemotherapy as compared to unifocal.¹⁵⁴ A recent summary of the World Health Organization’s (WHO) classification of tumors of the central nervous system has indicated that biomarkers as both ancillary and diagnostic predictive tools.¹⁵⁶ Additionally, in a recent WHO edition of classification of tumours of the central nervous system, molecular information is now integrated with histologic information into tumor diagnosis for cases such as diffuse gliomas and embryonal tumors.¹⁵⁷

Accordingly, most works focus on gliomas and more specifically glioblastoma, the most aggressive and invasive form of glioma. Due to glioblastoma’s extremely low survival rate of $5\%$ after 5 years, compared to a low grade glioma’s survival rate of over $50\%$ after 5 years,^76,158 it is critical to distinguish the two forms for improved patient care and prognosis.

Liver Liver cancer is one of the most common causes of cancer death.¹⁵⁹ In particular, hepatocellular carcinoma (HCC) is the most common type of primary liver cancer and has various subtypes, but they generally have little impact on treatment.¹⁶⁰ Histogolical grade is divided into nuclear features and differentiation, which directly correlate to tumour size, presentation, and metastatic rate.^161,162 Notably, high-grade dysplastic nodules are included in synoptic reports for HCC but are difficult to assess and have high inter-observer disagreement,¹⁶³ and thus is an area where CAD systems could be leveraged to normalize assessments. Current grading of this cancer suffers from an unsatisfactory level of standardization,¹⁶⁴ likely due to the diversity and complexity of the tissue. This could explain why relatively low number of works are dedicated to liver disease diagnosis and prognosis. Instead, most works focus on the segmentation of cancerous tissues.

Lymph Nodes There are hundreds of lymph nodes in the human body that contain immune cells capable of fighting infections. Cancer manifests in lymph nodes in two ways: 1) cancer that originates in the lymph node itself known as lymphoma and 2) cancer cells from different origins that invade lymph nodes.¹⁶⁵ As mentioned in the prior organ sections, lymphocytic infiltration is correlated with cancer recurrence on multiple organs and lymph nodes are the most common site for metastasis. The generalizable impact to multiple organs and importance of detecting lymphocytic infiltration is why many works focused on lymph nodes address metastasis detection.¹⁶⁶

Organ Agnostic The remaining papers focus on segmentation, diagnosis, and prognosis tasks that attempt to generalize to multiple organs, or target organ agnostic applications. An interesting approach to increase the generalization capability of deep learning in histopathology is proposed in.¹⁶⁷ Currently, publicly available datasets with thorough histological tissue type annotations are organ specific or disease specific and thus constrain the generalizability of CPath research. To fill this gap, a novel dataset called Atlas of Digital Pathology (ADP) is proposed.¹⁶⁷ This dataset contains multi-label patch-level annotations of Histological Tissue Types (HTTs) arranged in a hierarchical taxonomy. Through supervised training on ADP, high performance on multiple tasks is achieved even on unseen tissue types.

3.1. Tissue slide preparation

For the application development stages in CPath, the creation of a new WSI dataset must begin with selection of relevant glass slides. High quality WSIs are required for effective analysis, however, considerations must be made for potential slide artifacts and variations inherently present. As described in Section 2.1, pathology samples are categorized as either biopsy or resection samples, with most samples being prepared as permanent samples and some intra-operative resection samples being prepared as frozen samples.

Variations and Irregularities Throughout the slide sectioning process, artifacts and irregularities can occur which reduce the slide quality, including: uncovered portions, air bubbles in between the glass seal, tissue chatter artifacts, tissue folding and tears, ink markings present on the slide, and dirt, debris, microorganisms, or cross-contamination of slides by unrelated tissue from other organs.171, 172, 173 Frozen sections can present unique irregularities and variations, such as freezing artifacts, cracks in the tissue specimen block, or delay of fixation causing drying artifacts.^174,175 Beyond these irregularities, glass slides may vary in stain colouring, occurring due to differences in slide thickness, tissue thickness, fixation, tissue processing schedule, patient variation, stain variation, and lab variation.^174,176, 177, 178, 179, 180

All such defects and variations are important to keep in mind when selecting glass slides for the development and application process in CPath, as they can both reduce the quality of the WSI as well as impact the performance of developed CAD tools trained with these WSIs.^171,172,177 A more detailed discussion on the surveyed works in CPath which seek to identify and correct issues in slide artifacts and colour variation in WSIs is found in Section 3.2. However, prior to digitization, artifacts, and irregularities can be kept at a minimum by following good pathology practices. While an in-depth discussion of this topic is outside the scope of this paper, some research provides an extensive list of recommendations for reducing such errors in slide sectioning.¹⁷³

3.2. Whole slide imaging (WSI)

WSI Scan Once a glass slide is prepared, it must be digitized into a WSI. The digitization and processing workflow for WSIs can be summarized as a four-step process¹⁸¹: (1) Image acquisition via scanning; (2) Image storage; (3) Image editing and annotation; (4) Image display.¹⁸² As the first two steps of the digitization workflow are the most relevant for WSI collection and with regards to the CPath workflow, they are discussed to a greater extent below.

Slide scanning is carried out through a dedicated slide scanner device. A plethora of such devices currently exist or are in development; see Appendix Table 1 for a collection of commercially available WSI scanners. Additionally, some research has investigated and compared the capabilities and performances of various WSI scanners.183, 184, 185, 186

In order to produce a WSI that is in focus, which is especially important for CPath works, appropriate focal points must be chosen across the slide either using a depth map or by selecting arbitrarily spaced tiles in a subset.¹⁸⁷ Once focal points are chosen, the image is scanned by capturing tiles or linear scans of the image, which are stitched together to form the full image.^180,187 Slides can be scanned at various magnification levels depending on the downstream task and analysis required, with the vast majority being scanned at $20\times$ ($\sim 0.5\mathit{\mu m}$/pixel) or $40\times$ ($\sim 0.25\mathit{\mu m}$/pixel) magnification.¹⁸⁰

WSI Storage and Standards WSIs are in giga-pixel dimension format.^30,188 For instance a tissue in $1\mathit{cm}\times 1\mathit{cm}$ size scanned $@0.25\mathit{\mu m}$/pixel resolution can produce a $4.8$GB image (uncompressed) with $\mathrm{50,000}\times \mathrm{50,000}$ pixels. Due to this large size, hardware constraints may not support viewing entire WSIs at full resolution, thus WSIs are most often stored in a tiled format so only the viewed portion of the image (tile) is loaded into memory and rendered.¹⁸⁹ When building CAD tools for CPath, this large WSI dimensionality must be taken into account in determining how much compute is required to analyze a WSI. Alongside the WSI, metadata regarding patient, tissue specimen, scanner, and WSI information is stored for reference.^30,188,190 Due to their clinical use, it is important to develop effective storage solutions for WSI images and metadata, allowing for robust data management, querying of WSIs, and efficient data retrieval.^191,192 Further details on WSI image formats and storage methods are discussed in Appendix A.6.

To develop CPath CAD tools in a widespread and general manner, a standardized format for WSIs and their corresponding metadata is essential.¹⁸⁸ However, there is a general lack of standardization for WSI formats outputted by various scanners, as shown in Table 1, especially regarding metadata storage. The Digital Imaging and Communications in Medicine (DICOM) standard provides a format for CPath image formatting and data management through Supplement 145,^190,193 and has been shown in research to allow for efficient access and interoperability of data between varying medical centers and devices.¹⁸⁸ However, few scanners are DICOM-compliant and thus there are challenges to using different models of scanners, thus different image formats and metadata structures, in the context of dataset aggregation and processing.

Apart from storage format, a general framework for storing and distributing WSIs is also an important pillar for CPath. In other medical imaging fields such as radiology, images are often stored in a picture archiving and communications systems (PACS) in a standardized DICOM format, with DICOM storage and retrieval protocols to interface with other systems.¹⁸⁹ The need for standardization persists in pathology for WSI storage solutions; few works have proposed solutions to incorporate DICOM-based WSIs in a PACS, although some research has successfully implemented a WSI PACS consistent using the DICOM standard using a web-based service for viewing and image querying.¹⁸⁹

WSI Defects and Variations Certain aspects of the slide scanning process can introduce unfavorable irregularities and variations.¹⁹⁴ A major source of defects is out-of-focus regions in a generated WSI; often caused by glass slide artifacts, such as air bubbles and tissue folds, which interfere with selection of focus points for a slide.^171,195 Out-of-focus regions degrade WSI quality and are detrimental to the performance of CAD tools developed with these WSIs, presenting concerns for performance with studies showing high false-positive errors.^196,172 Additionally, as WSIs are scanned in strips or tiles, any misalignment between sections can introduce striping/stitching errors in the final image.¹⁹⁷ Another source of error may appear during tissue-background segmentation where the scanner may misidentify some tissue regions as background, potentially missing crucial tissue areas on the glass slide from being digitized.¹⁹⁸

Variations in staining refers to differences in colour and contrast of the tissue structures in the final WSI occurring due to differences in the staining process, staining chemicals, and tissue state. Variations in colour can lead to difficulty in generalizing CAD tools to WSIs from different labs, institutions, and settings.^199,200 Even identical staining techniques can yield different WSIs due to scanner differences in sensor design, light source and calibration,^180,201 creating challenges for cross-laboratory dataset generation. These additional sources of variation add layers of complexity to the WSI processing workflow, and must be kept in mind during slide selection and dataset curation for CAD tool development and deployment.

Addressing Irregularities and Variations Much work has gone into identifying areas of irregularities within WSIs, most notably blur and tissue fold detection.^195,196 Some research has explored automated deep learning tools to identify these irregularities at a more efficient pace than manual inspection.^195,196 Developing techniques for addressing staining variation has also been a significant research area^177,202, 203, 204, 205, 206, 207 as the use of techniques addressing stain variation is important for all future works. We list some computational approaches proposed to address these issues: An example method proposed in²⁰² uses a stain normalization technique, attempting to map the original WSI onto a target color profile. In this technique, a color deconvolution matrix is estimated to convert each image to a target hematoxylin and eosin (H$\&$E) color space and each image is normalized to a target image colour profile through spline interpolation.²⁰² A second approach applies color normalization using the H channel with a threshold on the H channel on a Lymphocyte Detection dataset.²⁰⁵ Recent studies have shown promise in having deep neural networks accomplish the stain normalization in contrast to the previous classical approaches,^203,208, 209, 210 commonly applying generative models such as generative adversarial networks (GANs) to stain normalization. Furthermore, Histogram Equalization (HE) technique for contrast enhancement is used in,²¹¹ where novel preprocessing technique is proposed to select and enhance a portion of the images instead of the whole dataset, resulting in improved performance and computational efficiency.

An alternative approach to address the impact of stain variation on training CAD tools is data augmentation. Such methods augment the data with duplicates of the original data, containing adjustments to the color channels of the image, creating images of varying stain coloration, and training train models that are accustomed to stain variations.²⁰⁰ This method has been frequently used as a pre-processing step in the development of training datasets for deep learning.212, 213, 214 A form of medically-irrelevant data augmentation based on random style transfer, called STRAP, was proposed by researchers and outperformed stain normalization.²⁰⁶ Similar to style transfer,²¹⁵ proposes stain transfer which allows one to virtually generate multiple types of staining from a single stained WSI.

3.3. Cohort selection, scale, and challenges

The data used to create/train CPath CAD tools can greatly impact the performance and success of the tool. Curating the ideal dataset, and thus selecting the ideal set of WSIs for the development of a CAD tool is a nontrivial task. Several works suggest that datasets for deep learning in CPath should include a large quantity of data with a degree of variation and artifacts in the WSIs.^62,172 Some works also recommend the inclusion of difficult or rarely diagnosed cases; other works indicate that inclusion of extremely difficult cases may decrease the performance of advanced models.^172,216

A study highlighting the results of the 2016 Dataset Session at the first annual Conference on Machine Intelligence in Medical Imaging outlines several key attributes to create an ideal medical imaging dataset,²¹⁷ including: having a large amount of data to achieve high performance on the desired task, quality ground truth annotations, and being reusable for further efforts in the field. While the scope of this conference did not include CPath, many of the points made regarding medical imaging datasets are also relevant to the development of CPath datasets. The session also outlined the impact that class imbalances can have on ML models, an issue also prevalent in CPath as healthy or benign regions often outnumber diseased regions by a significant margin.²¹⁸

Our survey of past works in the literature reveals some trends in CPath datasets. Currently, the majority of datasets presented in the literature for CAD tool development are small-scale datasets,⁶² using a small number of images, and/or images from a small number of pathology laboratories. Examples of these smaller datasets include a dataset with 596 WSIs (401 training, 195 testing) from four centres for breast cancer detection²¹⁹ and the Breast Cancer Histology (BACH2018) dataset, which has 500 ROI images (400 training, 100 testing) and 40 WSIs (30 training, 10 testing).²²⁰ Although curating a dataset from fewer pathology laboratories may be simpler, these smaller scale datasets may not be able to effectively generalize to data from other pathology centres.^199,120 An example of this can be seen in which data from different pathology centres are clustered disjointly in a t-distributed stochastic neighbor embedding (t-SNE) representation demonstrated in.¹⁷² Another alternative was proposed in²²¹: using a swarm learning technique multiple AI models were trained on different small data sets separately and then unified into one central model.

Additionally, stain variations, slide artifacts, and variation of disease prevalence may sufficiently shift the feature space such that a deep learning model may not sustain high performance on unseen data in new settings.^120,222 As artifacts in WSIs are inevitable, with some artifacts, such as ink mark-up on glass slides, being an important part of the pathology workflow,²²³ the ability of CAD tools to become robust to these artifacts through exposure to a diverse set of images is an important consideration.

Compared to the number of studies conducted on small-scale datasets, relatively fewer studies have been performed using large-scale, multi-centre datasets.^62,224,172 One study uses over 44,715 WSIs from three organ types, with very little curation of the WSIs for multi-instance learning detailed in.⁶² Stomach and colon epithelial tumors were classified using 8,164 WSIs in.²²⁴ A similar study uses 13,537 WSIs from three laboratories to test a machine learning model trained on 5,070 WSIs and achieves high performance.⁶²

Despite some advancements, there exist major barriers to using such large, multi-centre datasets in CAD development. Notably, for strongly supervised methods of learning, an immense amount of time is needed to acquire granular ground truth annotations on a large amount of data.²²⁴ To combat this, some researchers have implemented weakly-supervised learning by harvesting existing slide level annotations to forego the need for further annotation.⁶² Additionally, it may be difficult to aggregate data from multiple pathology centres due to regulatory, privacy, and attribution concerns, despite the improvements that diverse datasets offer. Section 5 discusses model architectures and training techniques that harness curated datasets of various annotation levels.

Dataset Availability In general computer vision, progress can be tracked by the increasing size and availability of datasets used to train models, e.g. ImageNet grew from 3.2 million images and 5000 classes in 2009 to 14 million images and 21,000 classes in 2021.²²⁵ We infer a similar trend in dataset growth and availability indicates progress in CPath. In our survey of over 700 CPath papers, we determine the current landscape by noting the dataset(s) used in work, along with dataset details such as the organ(s) of interest, annotation level, and stain type, tabulating the results into Table 9.11 of the supplementary materials, with summarized findings from Table 9.11 are shown in Fig. 6.

From Fig. 6 we can clearly see that the majority of datasets used for research developments in computational pathology are privately sourced or require additional registration/request. With organs represented in a small number of datasets, such as the liver, thyroid, brain, etc, having a smaller proportion of freely accessible datasets as compared to the Breast, Colon, or Porstate. This can be problematic when trying to create CAD tools for cancers in these organs due to a lack of accessible data. We additionally note that although data sets requiring registration/request for access can be easily accessible, as in the case of Breast Cancer Histopathological Database (BreakHis)²²⁶ being used in multiple works,227, 228, 229 the need for registration presents a barrier to access as requests may go unanswered or take much time to review.

In our categorization of CPath datasets, we find that a few prominent datasets have been released publicly for use by the research community. Many such datasets are made available through grand challenges in computational pathology,²³⁰ such as the CAMELYON16 and CAMELYON17 challenges for breast lymph node metastases detection,^61,231,232 and the Gland Segmentation in Colon Histology Images Challenge (GLaS) competition for colon gland segmentation in conjunction with Medical Image Computing and Computer Assisted Intervention (MICCAI) 2015.^233,234 Notable amongst publicly available data repositories is the cancer genome atlas (TCGA),²³⁵ a very large-scale repository of WSI-data containing many organs and diseases, along with covering a variety of stain types, magnification levels, and scanners. Data collected from TCGA has been used in a large number of works in the literature for the development of CAD tools.^200,236,237 As such, TCGA represents an essential repository for the development of computational pathology. While patient confidentiality is a general concern when compiling and releasing a CPath dataset, large-scale databases such as TCGA prove that it is possible to provide relatively unrestricted data access without compromising patient confidentiality. Further evaluating public source datasets, it seems that the majority of them use data extracted from large data repositories, such as TCGA, without specifying the IDs of the images used, which provides a challenge in comparing datasets or CAD tool performance across works. However, there are a few datasets that are exceptions to that phenomenon.^64,238, 239, 240

Fig. 6 also provides some insights on the dataset breakdown by organ, stain type, and annotation level. Per organ, it can be seen that the breast, colon, prostate/ovary, and lung tissue datasets are amongst the most common, understandably since cancer occurrence in these regions is the most frequent⁹⁰–complying with cancer statistics findings in 9.5. Multi-organ datasets are the other most common type, where we have designated a dataset to be multi-organ if it compiles WSIs from several different organs. To note, multi-organ datasets are especially useful for the development of generalized image analysis tools in computational pathology. The annotation level provided in the datasets did not indicate any pattern across most organs.

Dataset Bias It is also important to note the potential for bias in datasets that may influence the ability of any deep learning algorithm to generalize on unseen data.^241,242 This problem is a prevalent issue in general machine learning applications,243, 244, 245, 246 and CPath is not immune to it. The survey review in²⁴⁷ reviews a large number of other examples in machine learning that exhibit such bias, both from a dataset-standpoint and an algorithm-standpoint.

Such a lack of generalizability in CPath can impact the ability of machine learning models trained on biased data to meet the needs of patients. As noted in,²⁴¹ minority groups may be disproportionately negatively impacted if care is not taken in curating a diverse dataset that adequately reflects the relevant demographics for the problem to be solved.

Several works have delved into the issue of dataset bias in CPath specifically.^248,249 A notable example is in,²⁴⁹ where the study was able to demonstrate that deep learning models trained on WSIs from TCGA were able to infer the organization that contributed the slide sample. Notably, some features, such as genetic ancestry, patient prognosis, and several key genomic markers were significantly correlated with the site the WSI was provided from. As the vast majority of data in TCGA is acquired from 24 origin centers,²⁴⁸ such site-specific factors may impact the ability of a DL model to perform well on patient data from different sites.

As discussed previously, having a large set of diverse data may help to mitigate generalization issues.^120,199,241 Additionally, the study²⁴⁹ makes the suggestion that training data should be from separate sites than validation data, and that per-site performance of a model should be reported when validating a model. In doing this, the robustness of the model to site-specific variation, including both stain and demographic related variation, can be evaluated.

4.1. Supervised annotation

In contrast to general computer vision, computer scientists do not have expert-level knowledge of histopathology and thus they are not as efficient at generating annotations or labels of pathology images. Further, labels cannot be easily obtained by outsourcing the task to the general public. As a result, pathologists must be leveraged to obtain labels at some stage of the data collection and curation process, and in many annotation pipelines the first step involves recruiting the help of pathologists for their expertise in labelling.

Obtaining Expert Domain Knowledge The knowledge of pathologists is essential in the development of accurate ground truth annotations–a process most commonly completed by encircling ROI.²¹⁹ However, there are studied instances of inter-observer variance between pathologists when determining a diagnosis.250, 251, 252 As obtaining the most correct label is essential when training a model for CAD, this issue must be addressed and a review of the data by several pathologists can result in higher quality ground truth data as compared to that of a single pathologist. As a result, most datasets are curated by involving a group of pathologists in the annotation process. If there exists a disagreement between the expert pathologists on the annotation of a ground truth, one of several methods is usually employed to rectify the discrepancy. A consensus can be reached on the annotation label through discussion amongst pathologists, as is done in the triple negative breast cancer (TNBC-CI) dataset,²⁵³ the Breast Cancer Surveillance Consortium (BCSC) dataset²⁵⁴ and the minimalist histopathology image analysis dataset (MHIST) dataset.²⁵⁵ Alternatively, images, where disagreements occur, can be discarded, as is done in some works.^256,257 Further, the disagreement between annotators can be recorded to determine the difficulty level of the images, as is done in the MHIST dataset.²¹⁶ This extra metadata aids in the development of CAD tools for analysis.

Pathologists can also be involved indirectly in dataset annotation. Both the Multi-organ nuclear segmentation dataset (MoNuSeg)²⁵⁸ and ADP¹⁶⁷ have non-expert labelers annotate their respective datasets. A board-certified pathologist is then tasked with reviewing the annotations for correctness. Alternatively, some researchers have employed a pathologist in performing quality control on WSIs for curating a high-quality dataset with minimal artifacts.^87,259 To enable the large scale collection of accurate annotated data, Lizard²⁶⁰ was developed using a multi-stage pipeline with several significant “pathologist-in-the-loop” refinement steps.

Existing pathological reports, along with the metadata that comes from public large-scale databases like TCGA, can also be leveraged as additional sources of task-dependent annotations without the use of further annotation. For example, TCGA metadata was used to identify desirable slides in,²⁶ while pathological diagnostic reports were used for breast ductal carcinoma in-situ grading in.²⁶¹

To note, there are some tasks where manual annotation by pathologists can be bypassed altogether. For instance, IHC was applied to generate mitosis figure labels using a Phospho-Histone H3 (PHH3) slide-restaining approach in,²⁶² while immunofluorescence staining was used as annotations to identify nuclei associated with pancreatic adenocarcinoma.²⁶³ These works parallel the techniques that pathologists often use in clinical practice, such as the use of IHC staining as a supplement to H$\&$E stained slides for difficult to diagnose cases.²⁶⁴ They demonstrate high performance on their respective tasks wherein the top-performing models on the Tumor Proliferation Assessment Challenge 2016 (TUPAC16)²⁶⁵ dataset were achieved.²⁶² Importantly, these techniques still utilize supervision, albeit weakly, by leveraging lab techniques that have been developed and refined to identify the desired regions visually.

Ground-Truth Diagnostic Information Understanding different annotation levels and their impact on the procedural development of ML pipelines is an important step in solving tasks within CPath. There are five possible levels of annotation, in order of increasing granularity (from weakly-supervised to fully-supervised): patient, slide, ROI, patch, and pixel. Fig. 7 overviews the benefits and limitations of each level. For additional information regarding each annotation level please refer to Appendix A.8.

Picking the Annotation Level Selecting an annotation level depends largely on the specific CPath task being addressed, as shown in Fig. 8. For example, segmentation tasks tend to favor pixel-level annotations as they require precise delineation of a nucleus or tissue ROI. Conversely, disease diagnosis tends to favor datasets with ROI-level annotations, as diagnosis tasks are predominantly associated with the classification of diseased tissue, the higher-level annotations may provide a sufficient level of detail and context for this task.²⁶⁶

Fig. 8 shows that tasks that use stronger supervision are more likely to be used in CAD tool model development. However, due to the high cost of pixel-level annotation, fully supervised annotations are challenging to develop. Even patch-based annotations often require the division and analysis of a WSI into many small individual sub-images resulting in a similar problem to pixel-based annotations.^63,212 In contrast, WSI data is most often available with an accompanying slide-level pathology report regarding diagnosis thus making such weakly labeled information at the WSI level significantly more abundant than ROI, patches, or pixel-level data.^267,268 Different levels of annotation can be leveraged together, as demonstrated by a framework to use both pixel and slide level annotations to generate pseudo labels in.²⁶⁹ Additionally, it is common in CPath to further annotate the slide-level WSIs on an ROI or patch level structure.^{32,85,270,271}

Active Learning Tools Active learning annotation tools bridge the gap between the need for highly supervised labels and the current abundance of less informative annotations. Such works seek to ease the annotation process by using computational approaches to assist the human annotator. For example, in,¹⁶⁸ a platform was developed for creating nuclei and gland segmentation ground truth labels quickly and efficiently. A convolutional neural network (CNN), trained on similar cohort data, was used to segment nuclei and glands with different mouse actions.¹⁶⁸ Alternatively, Awan et al.¹⁶⁹ presented the HistoMapr^TM platform to assist in diagnosis and ground truth data collection. Through this tool, a pathologist selects one of several proposed classes for each given ROI, thus mitigating the need for hand-drawn annotations or manual textual input.¹⁶⁹ Similarly, an active learning model called the Human-Augmenting Labeling System (HALS)²⁷² was developed to increase data efficiency by guiding annotators to more informative slide sections. Quick Annotator (QA)²⁷³ is another tool which provides an easy-to-use online interface for annotating ROIs and was designed to create a significant improvement in the annotation efficiency of histological structures by several orders of magnitude. There are other active learning annotation tools proposed for different applications in computer vision that can be investigated for use in the pathology datasets. Such examples include methods to produce object segmentation masks for still images^274,275 as well as video.²⁷⁶ One notable example is DatasetGAN²⁷⁵; the model is proposed as a training data creator, and it is shown that the model can produce segmentation masks with a small number of labelled images in the training data. While these systems are for general computer vision, they may be adoptable in computational pathology, and would facilitate the necessary relationship between pathologists and computer scientists in the development of CAD tools. As such, they may prove to be a valuable contributor to the CAD system development workflow.

Tissue-Class and Disease Complexity Much of the current CPath research operates under the umbrella of supervised learning tasks, and correspondingly uses labeled data to develop automated CAD tools. We refer to supervised learning to include a diverse spectrum of annotation i.e. weak-supervision (e.g. patient-level) all the way to strong-supervision (e.g. pixel-level). Classes within a dataset can be task-dependent, for example as shown in Table 9.11 of the supplementary material, datasets primarily used for segmentation such as MoNuSeg²⁵⁸ and CPM-17²⁷⁷ have classes for each annotated pixel indicating the presence or absence of nuclei. However, classes need not be task-dependent; datasets such as CAMELYON16²³¹ outline metastases present in WSIs that can be used for a variety of applications, including disease detection²³¹ and segmentation tasks.²⁷⁸

The current paradigm for dataset compilation in computational pathology, particularly for disease detection and diagnosis, treats different disease tissue types as separate independent classes. For example, BreakHis divides all data into benign/malignant breast tumours.²²⁶ At the ROI level, GLaS divides colon tissue into five classes: healthy, adenomatous, moderately differentiated, moderately-to-poorly differentiated, and poorly differentiated.²³³ So far, this approach to class categorization has resulted in high-performing CAD tools.^{80,83,84,228,229,279} However, the treatment of different disease tissue types as an independent class is perceived differently in computer vision domain where the representation learning of normal objects is done differently compared to anomalies. A similar synergy can be found by differentiating healthy tissue classes from diseased ones and one should be mindful about defining meaningful tissue ontology for annotation and labeling.

4.2. Optimum labeling workflow design

This section focuses on the steps required for compilation of a CPath dataset which is broken into three main sub-tasks: Data Acquisition, Data Annotation, and Data Management, as per Fig. 9. Each sub-task is discussed below with reference to its individual components in the hierarchical structure in Fig. 9.

Data Acquisition Database compilation starts from data acquisition. When collecting data, it is vital that there are large amounts of data,²⁸⁰ along with having sufficient diversity.^62,172 Specifically, diversity in CPath data occurs in multiple ways, such as staining methods, tissue types and regions, laboratory processes, and digital scanners. We advise that CPath researchers consult expert pathologists on the diversity of data required for various tasks. Ideally, all the data acquired in pathology should be perfect without any irregularity and artifacts. However, some level of artifacts and irregularity are unavoidable and introducing realistic artifacts that are representative of real-world scenarios into the data increases the robustness and generalizability of CAD tools.

Data Annotation After collecting sufficient data, the next task is annotation of the data. Data annotation is a costly process in both time and money, thus a budget and schedule should always be established when generating labels. There are often various approaches for annotating different structures,²⁸¹ so a specific labelling taxonomy should be defined a priori. As mentioned previously, annotation should involve expert pathologists due to the domain knowledge requirement and importance of label correctness. A table of commonly used commercially available annotation software for annotating different slide formats are show in Table 2, along with their compatible image formats which is important to note when trying to build compatible and accessible datasets.

Once the ontology of class-definitions is defined (in collaboration with expert pathologists), there will be two ways to generate labels or annotations in general: domain expert labelling or non-expert labelling. The domain expert labelling refers to having pathologists annotate data that they are specialized at, which is labor-extensive. On the other hand, non-expert labelling can use crowdsourcing techniques to generate weak labels or have non-experts, such as junior pathologists or students, label the data. This process is cheaper and quicker, but it may be harder to maintain the same level of quality as domain expert labeling.²⁸¹ Regardless of the labelling methodology used, labels generated from both should be validated. Finally, to determine the sufficiency of label quantity, one should consider the balance between the number of classes, representation size of each class, and complexity of class representation. Techniques from active-learning could be also leveraged to compensate for lack of resource management as well as maintain the quality of labeling as discussed above.

Data Management Data management is an important aspect of any dataset creation process, and is the one that is most likely to be overlooked. Proper data management should have considerations for reusability, medical regulations/procedures, and continuous integration and development.

Reusability can be broken down into detailed documentation of the data, accessible and robust hosting of data, and consideration for image standards. Poor cross-organizational documentation can lead to missing metadata, ultimately resulting in discarding entire datasets.²⁸² Adherence to an established image standard, such as DICOM, can help resolve some of these issues in reusability. Medical regulations/procedures can be broken down into the construction of a Research Ethics Board (REB) and proper consideration for whom is curating the data. Through incentives for data excellence for medical practitioners, the issue of misaligned priorities between data scientists, domain experts, and field partners can be resolved.²⁸² To ensure that models used on actual patients remain relevant and hidden errors do not propagate, continuous integration/development (CI/CD) must be implemented. These systems must include at least two components, a method to audit predictions from the model, as well as a way to refine the training data accounting for discrepancies found through auditing. Several algorithms deployed in high-risk environments, including medical diagnosis, proved to only work well when data was updated after initial deployment.^283,282 Throughout the data management process, consultation with domain experts is a vital step in ensuring the success of data compilations.²⁸⁴

5.1. Classification architectures

In CPath, general classification architectures are the most prevalent due to their straightforward applicability to a wide range of tasks including tissue subtype classification, disease diagnosis and detection (more details in Section 2 and Fig. 4). Architectures commonly used for natural images, in particular CNNs, are widely adopted for CPath. To maximize model performance, it is a common approach to pre-train the model on large datasets like ImageNet before subsequent fine-tuning them to perform well for the specific CPath task, a task known as transfer learning.^{64,86,87,206,250,261,267,}287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312 Transfer learning for CPath allows for: 1) improved generalizability, particularly for tasks with limited amount of data; and 2) improved ease in finetuning a model compared to training from scratch.³¹³

Graph Convolutional Neural Networks (GCN)^314,315 is an alternative architecture that can be used to improve the learning of context-aware features across the WSI. GCNs typically consist of nodes representing elements and edges defining relationships between nodes. In,³¹⁶ a GCN was defined on a WSI, where the nodes represent patches and edges represent the connections among patches; this work obtained remarkable results on cancer prognosis task outperforming the SOTA in four out of five cancer types.³¹⁶

Vision Transformers (ViT),³¹⁷ have recently emerged as a direct application of Transformer models³¹⁸ to the image domain. In ViT, images are sub-patched and flattened into a 1D embedding along with a positional encoding which is then classified by an MLP head. By using the positional encoding, the model’s attention mechanism can focus computation on the most relevant areas of the image. ViT models have been applied with great success to CPath tasks, especially in conjunction with pre-trained CNN models.³¹⁹ We refer the reader to a comprehensive survey of transformer methods in medical image analysis for more details.³²⁰

General classification architectures are also commonly used as a foundation for novel architectural designs. For example, Squeeze-and-Excitation (SE) modules were introduced to reduce the number of parameters in ResNet and DenseNet blocks while maintaining high accuracy.^321,229 A fully-connected conditional random field (CRF) was incorporated on top of a CNN encoder to improve performance while maintaining the same level of computational complexity.³²² Lastly, patch sampling and pooling were used with AlexNet to perform slide-level disease diagnosis and segmentation.¹⁴²

Finally, in order to achieve superior performance, many researchers often rely on ensemble or multi-stage techniques which combine the predictive power or feature extraction abilities of multiple models to form a final output. These approaches have shown performance improvements compared to traditional single model classifiers.323, 324, 325, 326, 327, 328 However, this often comes at the expense of higher computational requirements.

5.2. Segmentation architectures

Segmentation is widely used in CPath, as shown in Fig. 4, and enable localizing the area of interest at the pixel level.³²⁹ U-Net was initially developed for neuronal structure segmentation in electron microscopy image stacks,³²⁹ but has become one of the most common architectures for segmentation in CPath.^{152,205,207,237,273,278,304,323,}330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349 U-Net has an encoder-decoder structure: an encoder to contract features spatially and a decoder to expand them again to capture semantically related context and generate pixel-level predictions.³²⁹ The U-Net model has been used to segment nuclei for creating a novel dataset with unsupervised learning,²³⁷ but it should be noted that this process also relies on the Mask R-CNN framework and pathologists for quality-checking purposes.

Another common approach for segmentation is to use fully convolutional networks (FCNs),^{80,84,198,258,270,}349, 350, 351, 352, 353, 354, 355 customized architectures constructed by combining multiple components of various architectures, or introducing new components to pre-existing architectures.^{84,168,234,259,266,269,300,309,343,344,}356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369 For example, one work used a custom CNN to predict whether each pixel was benign or malignant, while a second CNN was used to refine the initial prediction through probability fusion.⁸⁴

5.3. Object detection architectures

In this section, we specifically focus on architectures that are used for object detection in CPath, where bounding boxes are predicted around regions of interest. A major CPath application for object detection is mitosis detection with the primary goal for counting the number of mitosis instances. To this end, a large number of studies has been dedicated to this application.^{199,200,218,262,337,343,}370, 371, 372, 373, 374, 375, 376, 377, 378, 379 Object detection has been additionally applied for nuclei,^77,380, 381, 382, 383, 384, 385 colorectal gland^80,341,386 and glomeruli detection387, 388, 389; however, it can also be applied to the detection of a variety of histopathological objects including tumor-infiltrating lymphocytes³⁹⁰ or keratin pearls.³⁶⁴

In CPath, object detection employs a combination of pre-existing off-the-shelf architectures and customized neural networks to perform object detection tasks, as shown in Fig. 10. A model called CircleNet, which uses a deep layer aggregation network as a backbone, was proposed to detect round objects.³⁸⁸ Their approach involves using an anchor-free “center point localization” framework in order to output a heatmap with center points followed by a conversion into a bounding circle for the detection of kidney glomeruli. A multi-stage deep learning detection model based on Fast R-CNN was proposed in.³⁷² First, a modified Fast R-CNN generated region proposals, then a ResNet-50 model eliminated false positives, and a Feature Pyramid Network detected mitosis in sparsely annotated images using a ResNet backbone.³⁷⁸

5.4. Multi-task learning

Multi-task models are individual models predicting for multiple tasks at once (e.g. classification and segmentation), as defined in Section 2. Multi-task learning (MTL) can be beneficial over independent task learning because sharing representations between related tasks can create more generalizable representations and encourage the task heads to make logically consistent predictions. This type of model, however, is uncommon in CPath, as it requires annotating multiple tasks for each image.^{32,82,120,377,}391, 392, 393 We discuss some of these papers in further detail below.

In one work, a ResNet-50 backbone followed by independent decoders (a pyramid scene parsing network for segmentation and a fully-connected layer for classification) was used to solve 11 different tasks (4 segmentation based and 7 classification based).¹²⁰ With significantly less computation, the MTL model achieved comparable or better results to single task learning in classification, but comparatively worse results in segmentation. Similarly, in,⁸² a ResNet-50 with two parallel branches to perform segmentation and classification, was able to achieve comparable results on both tasks through an MTL approach.

While the results are impressive, there is still work to be done in this field. One work found that model performance may be sensitive to the number and type of tasks used during training.³⁹¹ If the tasks are unrelated, this could deteriorate the performance compared to a single-task setting. How to weigh different task objectives and select optimal tasks to be trained together is yet an active area of research.^394,395 MTL represents an interesting field of research in CPath as it may reduce the necessity to train multiple deep neural networks to perform different tasks.

5.5. Multi-modal learning

As opposed to multi-task networks where multiple tasks are learned simultaneously, the multi-modal approach involves using network input features from multiple domains/modalities at once.³⁹⁶ In the case of CPath, modalities can be represented as pathologists’ reports, gene expression data, or even WSI images. Most commonly, immunohistochemistry (IHC) stains alongside the H&E stain to better visualize specific proteins.397, 398, 399 As a result, models can learn better unified/shared latent representations which capture correlations from multiple indicators, since some information may not be captured by individual indicators.⁴⁰⁰ This approach can be viewed as adding hand-crafted features to boost performance. While the use of deep learning normally implies using learned features to replace hand-crafted ones, using hand-crafted features can nonetheless improve performance compared to strictly deep learning approaches when data is limited.⁴⁰¹ Indeed, many works have obtained best performance by combining manual and learned features.^371,383,402 This was demonstrated in the case of mitotic cell classification when an ensembled classifier model using hand-crafted features set a new record for the MITOS-ATYPIA 2014 challenge with an F-score of $86\%$_.⁴⁰³ However, where data is plentiful, CNNs alone can outperform all other hand-crafted features. In the same MITOS-ATYPIA 2014 challenge, the previous record was broken this way with a new F-score of $96\%$_.⁴⁰⁴ Although one cannot compare these two works directly as they use different classifier heads and dataset balancing methods, one can argue that the optimal choice of approaches from deep learning, classical ML, and different modalities should depend on the situation. Multi-modal approaches are gaining traction in CPath for specific problems, especially where useful additional data is available.^405,406 For example, gene expression data and WSI images are often combined to improve cancer prognosis prediction.^407,408

5.6. Vision-language models

Following its successful use in the natural image domain, vision-language data (consisting of histopathology images paired with relevant natural language text) is becoming increasingly prominent in CPath. Whether it be the development of foundational models⁴⁰⁹ extending to CPath, or fine tuning state-of-art large large models for use in downstream tasks,410, 411, 412 leveraging the semantic information embedded in the natural language data is becoming more evidently beneficial. It was only recently that foundational language models advanced enough to become useful in CPath, and this has triggered an explosion of interest into building models at the intersection of visual and language information. At the moment, language data is primarily used to address Multi-Instance Learning, although this is still an extremely new field and we anticipate that future works will surely address more advanced tasks (see Section 7.4 for further discussion).

5.7. Sequential models

Recurrent Neural Networks (RNNs) are typically used in tasks with temporally-correlated sequential data, such as speech or time series.⁴¹³ Since RNNs consider the past through the hidden state, they are suited for handling contextual information. While images are the default data format in CPath (and hence poorly-suited for RNNs), some works opt to combine RNNs with CNNs as a feature extractor,^{62,88,414,213,224,257,259,}415, 416, 417, 418, 419, 420 most commonly by aggregating patches or processing feature sequences.^{62,224,259,418,421} Another application of RNNs is to consider spatial relations between patches, which can be lost after extracting from the slide.^213,257

A particularly exciting use of RNNs is in deciding which region within an image should be examined next.^417,422 In the “Look, Investigate, and Classify” 3-stage model, an long short-term memory (LSTM) was used to classify the ROI cropped from the current patch and predict the next region to be analyzed, and achieved good performance while only using $15\%$ of pixels from the original image.⁴¹⁷ Similarly, an LSTM network was used to better predict ROIs by treating state features similar to time-series data, thus identifying only relevant examples to use for training.⁴²² And an LSTM transformer with “Feature Aware Normalization” (FAN) units for stain normalization was used in parallel with a VGG-19 network.³¹⁸ More recently, transformers using attention mechanisms have been used to allow parallelization and better sequence translation compared to older RNNs or LSTM networks.⁴²³

5.8. Synthetic data and generative models

With annotated data difficult to obtain in CPath, especially with granular labels (see Section 3.3 for more discussion), this is problematic for training generalizable models. Hence, generating synthetic data from a controlled environment (either via simulation or a trained model) for augmenting the available training set of annotated data shows much promise. Originally developed for visual style transfer in general computer vision, generative models learn to create novel instances of samples from a given data distribution - they form the dominant approach in CPath.

Initial works primarily utilized Generative Adversarial Networks (GANs) for patch synthesis,^330,424, 425, 426 stain normalization,^206,208, 209, 210^,427,428, style transfer,^206,429, 430, 431 and various other tasks.²¹² One unsupervised pipeline relied on a non-GAN model to create an initial patch that was refined by a GAN.³³⁰ In another work, one CycleGAN generated tumor images and another non-tumor, in order to train a classification network.⁴³² One work used neural image compression to learn the optimal encoder to map image patches to spatially consistent feature vectors.²¹² Another work first classified bone marrow cell representations and then used an unsupervised GAN to generate more instances from each cluster.⁴³³ A self-supervised CycleGAN was also used for stain normalization, and shown to improve model performance in subsequent detection and segmentation tasks.²¹⁰ Similarly, a CycleGAN pipeline was applied to perform artificial IHC re-staining.⁴³⁴ Recent works in GANs attempt to model spatial awareness of tissues and improve the realism of the generated samples.⁴³⁵

Lately, diffusion models have become the SOTA in general computer vision and now produce far more semantically plausible and noise-free images than GANs. These improvements promise to make synthetic data finally accepted by pathologists and the broader CPath community as reliable training data and significantly improve the generalizability of models trained on them.⁴³⁶

5.9. Multi-instance learning (MIL) models

Multi-instance learning (MIL) involves training from data that is labelled as high-level bags consisting of numerous unlabelled instances. In the context of CPath, these labelled bags often represent annotated slides of far more unlabelled patch instances.⁴³⁷ As labels at the WSI level are much easier to obtain (and hence more prevalent) than patch-level annotations, MIL has been applied to CPath by a significant number of papers.62, 63, 64^{,267,268,308,316,418,}437, 438, 439, 440, 441, 442, 443, 444, 445, 446, 447, 448, 449, 450, 451, 452, 453, 454, 455, 456 Since both utilize coarser annotations for training on massive images, MIL is similar to weakly-supervised learning. However, weak supervision predicts at a finer level (e.g. pixel segmentation from labelled patches) than the provided annotation while MIL prediction is typically at the same level.

One notable work used a two stage approach to first encode patches with a CNN from a slide into feature vectors and then pass the most cancer-likely ones to an slide-level classification RNN. A similar work first detected abnormal regions in the WSI before adaptively fusing the instance-level features with an importance coefficient.⁴³⁹ Adding additional instance-specific attributes tends to improve MIL performance. One work applied a nuclei grading network to provide a cell-level prediction for each patch, and demonstrate this out-performs hand-crafted cell features for overall slide classification.³¹⁹ Recent works explore the morphological and spatial relationships between instances, which conforms with pathologist diagnostic intuitions and have demonstrably improved performance, especially with unbalanced data.⁴⁵⁶

As not all instances are equally relevant to the bag label, many works focus on building attention mechanisms to adaptively focus on more relevant instances. One work used such a mechanism to highlight regions of interest and improve localization relative to other SOTA CNNs.⁴³⁷ MIL models can be improved by considering multi-scale information: one work notably used embeddings from different magnification levels and self-supervised contrastive learning to learn WSI classifiers.²⁶⁸ Some works explicitly encode the patient-slide-patch hierarchy in the attention mechanism,^457,458 with one work using a cellular graph for top-down attention.⁴⁵⁹ Graph Neural Networks (GNNs) have been explored to leverage intra- and inter-cell relationships, enabling cancer grading,⁴⁶⁰ classification,⁴⁶¹ and survival prediction.^462,463 These hierarchy- and morphology-aware models are the current SOTA and pave the way for future improvements.

One persistent challenge with using MIL in CPath, compared to natural image computer vision, is the lack of large-scale WSI datasets.⁴⁶⁴ One recent work addressed issues related to small sample cohorts by splitting up large bags (and their labels) into smaller ones through pseudo-bags.⁴⁶⁵

5.10. Contrastive self-supervised learning for few-shot generalization

The idea of using contrastive learning (CL) for self-supervised learning (SSL) dates back to 2005, yet only recently gained momentum in CPath.^268,306,466, 467, 468, 469, 470, 471 By using a contrastive loss, a feature embedding is learned to ensure similar (positive) examples are close in vector space, while dissimilar (negative) examples are distant.^466,467 Contrastive learning is an attractive approach for CPath because when used as self-supervision for few-shot learning,^268,306,468 it does not require labelling the massive self-supervision image set but only labelling the small subset used for training on the downstream task, an approach that has recently achieved SOTA performance in a wide array of tasks in CPath.³⁰⁶ SimCLR was originally proposed to learn representations invariant to different augmentation transforms (such as crop, noise) for natural images,⁶⁶ and when applied to CPath, was found to match or outperform SOTA supervised techniques.³⁰⁶ Self-supervised pre-training has been shown to perform best against fully-supervised pre-training when applied to small but visually-diverse datasets.³⁰⁶ Recent works have focused on transferring the self-supervised representations to the downstream task more intelligently: through latent space transfers,⁴⁷² with an awareness of the patient-slide-patch hierarchy,^473,474 or with semi-supervised pseudo-label guidance.⁴⁷⁵

5.11. Novel CPath architectures

In this section, we discuss papers that made significant changes to the model design or completely designed an architecture from scratch for CPath tasks.^{64,277,357,361,382,}476, 477, 478, 479 Typically, model architectures are adapted from the natural image domain and minor changes applied for CPath tasks, rather than being designed from scratch for CPath directly. Unfortunately, general computer vision architectures typically require large computational resources not necessarily available in clinical settings and are prone to overfitting on smaller CPath training sets.⁴⁷⁶

More importantly, CPath tasks often comprise of multiple specialized sub-tasks not addressed by common architectures - in such cases, CPath-specific architectures perform better. “PlexusNet” achieved SOTA performance with significantly fewer parameters⁴⁷⁶ and “Hover-Net” used a three-branched architecture for nuclei classification and instance segmentation.²⁷⁷ Path R-CNN similarly used one branch to generate epithelial region proposals and another to segment tumours.³⁸²

In other cases, custom architectures are designed to obtain better performance with respect to certain metrics^{85,295,338,377,388} or improve computational efficiency and speed,¹⁹⁶ where since model inference can be a bottleneck for WSI processing. To automate architecture design, neural architecture search (NAS) is often used. This is an umbrella term covering evolutionary algorithms (EA), deep learning (specifically Reinforcement Learning), and gradient-based NAS searches. There are two approaches to EA: (1) Neuroevolution, which more generally optimizes at the neuron-level to find optimal weights, and (2) Evolutionary-Algorithms based NAS (EANAS), which searches for optimal combinations of mid-sized neural network blocks and conducts training after this architectural search.^480,481 In CPath, reinforcement learning-based NAS has designed models for cancer prediction which were found to train faster, have fewer parameters, and perform comparably with manually designed models.⁴⁸² Another work demonstrated that a significantly smaller model can outperform existing SOTA models on a variety of CPath tasks using an adaptive optimization strategy.⁴⁷⁹

We hypothesize that NAS has yet to be explored significantly in CPath due to the lack of annotated data (see Section 3.3) and its relative recency as a research area. According to the “no free lunch theorems”,^483,484 no single model can perform best on all tasks. However, computationally efficient but performant models are crucial for CPath applications, and NAS is the most promising approach to computationally design such architectures without manual engineering.

5.12. Model comparison

The various model architectures and types discussed above can and should be compared on common benchmarks to determine the best models for a given task.⁴⁸⁵ Numerous papers have conducted such benchmarking work on CNNs. One work comparing GoogLeNet, AlexNet, VGG16, and FaceNet on breast cancer metastasis classification found that deeper networks (i.e. GoogLeNet) predictably performed better.¹⁹⁹ Another work found that using ResNet-34 with a custom gradient descent performed best.³⁸⁷ Finally, VGG-19 performed best in colorectal tissue subtype clasification, showing that deeper SOTA networks do not necessarily perform better universally. Which CNN performs best depends on the task, the nature of the data, the metrics used, training time, hyperparameters, and/or hardware constraints.

Likewise, third parties have organized “grand-challenges” to facilitate the fair comparison of different techniques on a common CPath task and dataset. In some cases, SOTA CNNs achieve the best results, such as the adapted GoogLeNet that obtained the highest AUC²³¹ and the AlexNet that achieved highest accuracy⁴⁸⁶ for breast cancer detection in the CAMELYON16 challenge. Likewise, SqueezeNet, which is an existing SOTA network performs best in colorectal tissue subtype classification.⁴⁸⁷ On the contrary, the best performing models for mitosis detection in the TUPAC16¹⁹⁹ and MITOS12³³⁸ challenges both relied on custom CNN architectures. For breast cancer diagnosis, a novel Hybrid CNN achieved the best results in the BACH18 (ICIAR18) dataset⁴⁸⁸ while the two teams achieving the best classification accuracy in the BreakHis dataset used differing approaches: one directly used ResNet-50²⁹¹ and the other used an ensemble of VGG networks.⁴⁸⁹ For nuclei segmentation on the Kumar-TCGA dataset, a novel framework using ResNet and another existing model achieved the highest F1-score.³⁵⁷ Lastly, a custom CNN achieved the best results for gladn segmentation on the GLaS dataset.¹⁶⁸

However, as mentioned in Section 3.3, many grand challenges use private datasets or even extract data from larger public repositories without referencing the original WSIs used. Furthermore, benchmark datasets address different tasks and lack standardization. As models that are hyper-optimized to for specific sets of data continue to be released, the lack of more standardized benchmark datasets and model comparison studies make it impossible to systematically compare new models against existing ones or assess their robustness in clinical settings, thus impeding model development in CPath.

6.1. Clinical validation

Within the domain of CPath, clinical validation is essential for substantiating the decisions produced by deep learning models so that they are more readily accepted by the medical community. Generally, acceptable clinical criteria are determined by authoritative professional guidelines, consensus, or evidence-based sources. However, in CPath, prediction results are generated by the computer scientists and engineers who build the model, and may not be completely aware of where their work fits into the clinical pathology workflow–the clinical implications of this arrangement are often unknown.¹⁶⁹ By incorporating pathologist expertise, clinical validation can better align the technical work with clinical objectives.

Despite the importance of this step for real-world deployment, very few works have performed clinical validation with expert pathologists. We identify three prominent types of clinical validation in the CPath literature: (1) direct performance comparison of CAD tools with pathologists on a similar task, (2) impact of CAD tool assistance on pathologist performance, and (3) pathologist validation of CAD tool outputs. Each topic is further discussed in the sections alongside notable results.

Direct Performance Comparison with Pathologists To validate the benefits of deep learning methods, it is desirable that they equal or surpass the performance humans to gain the trust of pathologists in their decisions and their willingness to use them as a second opinion.⁴⁹⁰ With this in mind, many papers directly compared their models with pathologists in tasks such as prognosis and diagnosis.

One study on cancer detection found that the top computational models from the CAMELYON16 challenge out-performed the 11 pathologists with a two-hour time constraint and performed similarly to the expert pathologist without a time constraint.²³¹ This suggests that deep learning models could be particularly useful in clinical scenarios with excessive numbers of time-critical cases to diagnose. Similarly, for tissue subtype classification, another study performed similarly to, or slightly better than individual pathologists. The proposed model agreed with all pathologists $66.6\%$ of the time and agreed with two-thirds of pathologists $76.7\%$ of the time.⁷⁵ An additional study claimed their deep learning model outperformed pathologists without gynecology-specific training in ovarian carcinoma classification.¹¹⁹ This pushes the idea that CAD predictions can be used as a second opinion due to the potential for human error by individual pathologists.

One paper on diagnosis²⁵⁶ demonstrated that deep learning models can correctly classify images that even individual pathologists failed to correctly identify. However, another paper found that $50\%$ of the examples misclassified by their model were also misclassified by at least one pathologist.⁴⁹¹ This suggests that deep learning models can aid pathologists in decision-making, but as they tend to achieve a specificity and sensitivity similar to pathologists, they must be applied cautiously to avoid reinforcing the biases or errors of individual pathologists.

Deep learning models for prognosis have been shown to achieve performance similar to or better than experts as well.^238,289,381 In one study, the best model for renal clear cell carcinoma classification achieved $83\%$ accuracy, outperforming the inter-pathologist accuracy of $80\%$_.²⁸⁹ This shows that deep learning models and pathologists may perform similarly on patient prognosis.

Overall, AI approaches are not perfect but have approached expert-level ability in a variety of tasks. Deep learning could play an important role as a second opinion and in democratizing the knowledge distilled from many pathologists to other pathology centres. Specifically, deep learning models appear to be best used as a tool to enhance the pathologist workflow, and could provide aid in making quick decisions with high accuracy.¹⁴

Impact of CAD Tool Assistance Much of CPath research is conducted under the assumption that the resulting AI tools will be intuitive, usable, and beneficial to pathologists and patients. However, CAD tools that are developed without feedback from pathologists could fail to integrate into a realistic pathologist workflow or impact the most significant diagnostic tasks. Thus, a valuable validation experiment is to compare and comprehend the performance of expert pathologists in clinical tasks before and after being given the assistance of a CAD tool.

In one study, a CAD system called Paige Prostate Alpha leveraged a weakly-supervised algorithm to highlight patches in a WSI with the highest probability of cancer.⁶² When used by pathologists, the model significantly improved sensitivity, average review time, and accuracy over unaided diagnosis. Likewise, another study using the LYNA algorithm examined the performance of six pathologists on breast cancer tumor classification before and after being able to see the LYNA-predicted patch heatmaps. The results indicate using LYNA substantially improved sensitivity, average review time, and the subjective “obviousness” score for all breast cancer types.^72,27

These studies suggest that integrating CAD tools into the clinical workflow will greatly improve pathologist efficiency. However, there is a general lack of research on the impact of CAD tools on pathology efficiency. Such studies would shed more light on the impact of CAD tools and identify approaches for implementation in clinical settings.

6.2. FDA regulations

Despite the ongoing development of CAD tools in CPath and its potential for triaging cases and providing second opinions, the regulations regarding this technology pose an obstacle to the testing and deployment of these devices. The FDA currently provides three levels of clearance on AI/ML-based medical technologies: 510(k) clearance, premarket approval, and the De Novo pathway. While one source lists 64 AI/ML-based medical solutions that are currently FDA-approved or cleared, none of these are in the field of CPath.⁴⁹² A few companies, such as Paige AI, hold the 510(k) clearance for their digital pathology image viewer; however, an automated diagnostic system has yet to be approved. This may indicate a reluctance to change, and the lack of clarity in the process of FDA approval has prevented numerous impactful technologies from being deployed. There is a need for collaboration between researchers, doctors, and governmental bodies to establish a clear pathway for these novel technologies to be validated and implemented in clinical settings.

7.1. Contrastive self-supervised learning becomes mainstream

Data annotation for CPath is a persistent problem - it is easy to collect large amounts of visual data but much harder to annotate them. Transfer learning can help but it is difficult to transfer a model trained on one dataset to generalize to another. Whereas past efforts focused on carefully engineered methods, the recent development of contrastive self-supervised learning⁶⁶ means that it has become the mainstream approach in CPath.^268,306,468 Not only does it utilize the massive amounts of unlabelled images typically available in CPath, but it also as a result only requires finetuning on a small set for the downstream task. We anticipate that this will lead to the development of general-use foundational models in the future to perform the most common CPath tasks, as more pathology images are collected and models become more advanced.

7.2. Prediction becoming increasingly high-level

We noticed that recent research works are increasingly addressing higher level prediction tasks than before. Whereas patch classification^267,287, 288, 289, 290 or pixel segmentation^237,329, 330, 331, 332, 333 was formerly mainstream, these problems appear to have been largely solved, and now there is far more research into higher-level problems dominate, such as multiple-instance learning.^62,437,457, 458, 459 As computational methods continue to improve, it is natural that they are applied not merely as attention aids for pathologists (i.e. at the pixel or patch level), but furthermore are used to make intelligent slide- and patient-level decisions on their own. Indeed, they promise to vastly improve pathologist efficiency when used with human pathologists in the loop to validate the automated decisions, especially when paired with modern natural language capabilities.

7.3. Spatial and hierarchical relationships receiving attention

Inspired by the approach taken for natural image computer vision, the mainstream approach in CPath currently requires breaking up large WSIs into smaller patches and perceiving them independently (see Fig. 7). However, this ignores the spatial relationships between cells and tissues or between the patches and their parent slides in histopathology images, which are often relevant or even crucial when making decisions. Many works have recently found success in explicitly encoding an awareness of these inter-cell relationships^460,462,463 and the patch-slide-patient hierarchy,457, 458, 459 especially using Graph neural networks, but these suffer from higher latency than conventional CNNs. We anticipate future works will seek to speed up GNNs for tasks where spatial and hierarchical relationships are important and continue developing hierarchy-aware attention for MIL techniques.

7.4. Vision-language models for explainable predictions

One persistent problem in CPath has been developing models that can explain their decisions for human validation. One obvious route is to develop models that produce natural language output (and even converse with the human user to explain their decisions), but until recently, this would have required collecting massive amounts of pathology text paired with images. With foundational vision-language models widely available and able to generalize to great effect in the natural image domain,493, 494, 495 recent works have shown that they perform excellently when applied with minimal re-training to CPath images.410, 411, 412 Further advances require collecting more pathology-specific data, but we anticipate that crowd sourcing of public pathology annotations will become mainstream and this will lead to the development of foundational vision-language models. As natural language capabilities continue improving, we also anticipate that synoptic report automation will become feasible and reinforcement learning from human feedback (RLHF)⁴⁹⁶ will become common for improving CPath language models.

7.5. Synthetic data now realistic enough

Whereas one way to combat the difficulty of annotating CPath data is to develop models that require fewer annotations, another trend is to generate more annotated data for training. Whereas concerns were previously raised about their realism, new advances in generative image models have now been leveraged to produce realistic histopathology images and pixel-accurate annotations simultaneously. However, current works are limited by specific tissues, organs, diseases,^{212,330,432,433} or stains^210,434 and are limited by their inability to easily expand to other histopathology content. We note that generating synthetic data via game engines and 3D model assets is a recent trend in the natural image domain,497, 498, 499 but visual modelling of histopathology entities is little explored. We anticipate that future works will attempt to improve synthetic histopathology image generation by: (1) attempting to create generative models that can generalize to a broad variety of histopathology images and (2) create simulation software to generate realistic histopathology images without learned models.

8.1. CPath as anomaly detection

Typically in computer vision, the various classes represent distinct normative entities, such as airplanes or bears.^500,501 There exist abundant “normal” samples and potentially few “anomalous” samples, which are considered data points significantly dissimilar to the majority within a given class.⁵⁰² These anomalies are not only out of distribution from the samples in a dataset, there is also a lack of consensus on understanding anomalous representations as effectively identifying anomalies requires ML models to learn a feature space encompassing all “normal” samples within each class.⁵⁰²

In other words, in general computer vision, each class cannot simply be considered as an anomalous version of any other class. However, in CPath, since each class is often considered a different disease state on a single tissue type, diseased classes are essentially extensions of the “normal” healthy class into the “anomalous” zone. From a pathologist’s perspective, similar to the general computer vision approach, the curriculum learning process of a resident pathologist first involves training on histology and gaining a mastery of normal tissue identification, and then train on diseased tissues, so they are able to flag the sample as anomalous and follow up with possible diagnosis.

In light of this, it may be illuminating to approach the problem from an anomaly detection viewpoint: provided a model has sufficient variety of healthy tissue, any anomalies must then be diseased. The output of such an anomaly detection algorithm is dependent on the task at hand. One source describes several meaningful output types that may be produced⁵⁰²: an anomaly score which describes how anomalous a sample is and a binary label indicating whether a sample is normal or anomalous. If only identifying anomalous samples is enough, a binary classification procedure may be sufficient. However, if it is necessary to identify the particular stage of progression of a disease type, then a more granular approach in assigning some anomaly score may be more appropriate as explored in a previous work.⁵⁰³ This work found that the confidence score in tissue classification was inversely correlated with disease progression, thus the confidence score may act as a proxy for an anomaly score. Theoretically, such approaches may better replicate the behaviour of pathologists. While several works have used an anomaly detection approach on medical image data outside of CPath,504, 505, 506 few works tackle the problem for WSI data in CPath.

8.2. Leveraging existing datasets

As mentioned in Section 3.3 of this paper, a minority of datasets in CPath are available to be freely used by the public. Additionally, the level of annotations varies for each dataset. However as can be noted in Table 9.11 of the supplementary material, for prominent public datasets such as CAMELYON16, CAMELYON17, GlaS, BreakHis, and TCGA, there is far more available data annotated at the slide level as opposed to more granular predictions. For example, considering breast datasets, there are 399 WSIs annotated at the Slide and ROI levels in CAMELYON16⁵⁰⁷ and 1399 WSIs annotated at the Patient, Slide, and ROI levels in CAMELYON17,⁶¹ in contrast, the TCGA-BRCA dataset contains $1163$ diagnostic slides and $1978$ tissue slides that are accompanied with labels at the Patient and Slide levels and diagnostic reports with labels for tissue features and tumor grades.²³⁵

The lack of publicly available datasets with granular annotations is a major challenge in CPath. To address this lack some training data, techniques have been proposed to efficiently obtain labels, such as an active deep learning frameworks that use a small amount of labelled data to suggest the most pertinent unlabelled samples to be annotated.⁵⁰⁸ Alternatively, other works propose models to synthetically create WSI patches, usually with the use of GANs. For example, Hou et al.³³⁰ introduced an unsupervised pipeline that was capable of synthesizing annotated data at a large scale, noting that even pathologists had difficulty distinguishing between real and synthesized patches. However, despite these promising results, the issue of acquiring accurate and large datasets remains a prevalent issue within CPath.

Generally, tasks such as tissue classification or gland segmentation require labels at the ROI, Patch, or Pixel levels. However, existing data annotated at the patient and slide levels can be used for these tasks by leveraging weakly supervised techniques such as MIL,^64,437 or by learning rich representations using self-supervised techniques such as DINO^65,509 and contrastive learning³⁰⁶ that can be used in downstream tasks. Specifically, work is being done to develop training methodologies and architectures that are more data efficient for patient- and slide-level annotations, such as CLAM, which is a MIL technique that is used to train a performant CPath model with as little as $25\%$ of the training data to get over $0.9$ AUC.⁶⁴ Another recent work used self-supervised learning on WSIs without labels to train a hierarchical vision transformer and used the learned representations to fine-tune for cancer subtyping tasks. This finetuned model outperformed other SOTA methods that used supervised learning methods on both the full training set and when all models used only $25\%$ of the training set. These examples demonstrate a recent trend in the application of weakly and self-supervised learning techniques to leverage pre-existing and available data with weak labels, showcasing that a large amount of granular labels are not necessarily required for achieving SOTA performance. We urge researchers in the CPath field to follow this trend and focus on how to leverage existing weakly labelled datasets, especially to learn rich representations as a pre-training step for learning on smaller strongly labelled datasets.

8.3. Creating new datasets

Although we mention the availability of many datasets and comment on how to leverage this existing data, there is still a need for new CPath datasets that address overlooked clinical and diagnostic areas. Therefore, creation of new CPath datasets should focus on addressing two main goals: (1) tasks that are not addressed adequately by existing datasets and (2) accumulating as large a dataset as possible with maximal variety.

Regarding the first goal, there are still organs, diseases, and pathology tasks without freely available data or sufficient annotations to develop CAD tools. For example, in Fig. 6, we see that whereas breast tissue datasets are abundant, there are few public datasets for the brain and none for the liver. Collecting and releasing datasets for these organs would have significant impact in enabling further works focusing on these applications. Further, analysis of specific organ synoptic reports can guide CPath researchers to build CAD tools to identify or discriminate the most impactful diagnostic parameters. In the case of the prostate, which is discussed in Section 2.5, the synoptic report requires distinguishing IDC from PIN and PIA as it correlates to high Gleason scores. This is important, as high-grade PIN is a cancer precursor requiring follow-up sessions for screenings. These parameters are identified and noted in the report by the pathologist and factor into the final diagnosis and grading. Thus, collecting annotated datasets for such parameters can be crucial to developing CAD tools that are relevant to clinical workflows and can enrich learned representations.

The second goal is concerned with the scaling laws of deep learning models with respect to the amount of data available and their application to diverse clinical settings. As seen in the general computer vision domain, larger datasets tend to improve model performance, especially when used to learn rich model representations through pre-training that can be used for downstream tasks such as classification and semantic segmentation.⁵¹⁰ Additionally, ensuring that datasets capture the underlying data distribution and thus sufficiently encompass the test distribution has been shown to be especially important in the medical domain.⁵¹¹ For CPath, this means ensuring a dataset captures the expected variations in tissue structure, disease progression, staining, preparation artifacts, scanner types, and image processing. Collecting a sufficiently large dataset continues to be problematic, however, so recent works have focused on using crowd sourcing to annotate histopathology data posted publicly on Twitter and YouTube,^409,411 a practice that is similar to that commonly used for natural images.

8.4. Pre- and post-analytical CAD tools

In recent years, advances in image analysis, object detection, and segmentation have motivated new approaches to support the analytical phase of the clinical workflow, especially in the two steps where CAD tools could significantly increase efficiency and accuracy: (1) specimen registration and (2) pathology reports. This need is highlighted by a study determining that the pre-analytical and post-analytical phases (as shown in Fig. 2 account for up to $77\%$ of medical errors in pathology.⁵¹² Likewise, Meier et al. classify $14\%$ of medical errors as diagnostic errors, with an even smaller proportion being misinterpreted diagnoses in their study.⁵¹³ Other authors attribute approximately $15-25\%$ of diagnostic errors to slide interpretation.514, 515, 516, 517, 518 These results reinforce the need for CPath applications that address more than just the analytical phase.⁵¹⁹ Considering post-analytical step of compiling a pathology report, a few natural language processing efforts have been used to analyze completed pathology reports,520, 521, 522 extract primary site codes from reports,⁵²³ and generate of captions or descriptive texts for WSI patches.⁴¹⁵ However, to the best of our knowledge, there are no works that reliably extract clinical data from service requests and electronic medical records to automatically generate synoptic or text reports. Developing such a tool that could explicitly identify the most significant parameters for its decisions would directly improve clinical workflow and increase the interpretability of the results at the same time. We encourage the field of CPath to expand its efforts in creating tools for the pre- and post-analytical steps in order to reduce the large proportion of clinical errors attributed to those phases, and suggest some potential applications in Fig. 2.

8.5. Multi domain learning

Despite being particularly well-suited for CPath, multi-domain learning (MDL) is still a relatively unexplored topic. MDL aims to train a unified architecture that can solve many tasks (e.g. lesion classification, tumour grading) for data coming from different domains (e.g. breast, prostate, liver). During inference, the model receives an input image and the corresponding domain indicator and is able to solve the corresponding task for the given domain. There are two reasons that make MDL attractive for CPath. The first is that the additional information from a source domain (coming from a related organ such as the stomach) can be informative for improving performance in the target domain (e.g. colon). By sharing representations between related domains, the model is enabled to generalize to other domains. The second motivation is to alleviate the data sparsity problem where one domain has a limited number of labeled data. Through MDL, the domain with limited data can benefit from the features that are jointly learned with other related tasks/domains.^524,525

8.6. Federated learning for multi-central CPath

Data-driven models require a large amount of data to yield strong performance. In CPath, this requires incorporating diverse datasets with varying tissue slide preparations, staining quality, and scanners. An obvious solution to train such models is to accumulate the data from multiple medical centers into a centralized repository. However, in practice, data privacy regulations may not permit such data sharing between medical institutions, especially between countries. A possible solution lies in privacy-preserving training algorithms, such as federated learning,^526,527 which can make use of decentralized data from multiple institutions while maintaining data privacy. In federated learning, training starts with a generic machine learning model in a centrally located server. But instead of transferring data to a centralized server for training, copies of the model are sent to individual institutions for training on their local data. The learning updates are encrypted and sent to the central server and then aggregated across the institutions. Ming Lu et al.⁴⁴⁷ demonstrated the feasibility and effectiveness of applying federated, attention-based weakly supervised learning for general-purpose classification and survival prediction on WSIs using data from different sites. Using such algorithms for CPath can facilitate cross-institutional collaborations and can be a viable solution for future commercial solutions that need to continuously augment and improve their ML models using decentralized data.

8.7. CPath-specific architecture designs

Many deep learning architectures are not designed for CPath specifically, which raises a serious question about the optimality of using “borrowed” architectures from general computer vision. For instance,⁴⁷⁶ notes that traditional CV architectures may not be well suited for CPath due to a large number of parameters that risk overfitting. Additionally, the field of pathology has much domain-specific knowledge that should be taken into account before choosing an ML model. For example, under varying magnifications different morphological patterns are captured, from cellular-level details to tissue architecture features.³²⁸ Naively applying an architecture without considering such details could discard key visual information and lead to deteriorated performance.

Unlike natural images, WSIs exhibit translational, rotational, and reflective symmetry⁵²⁸ and CNNs for general vision applications do not exploit this symmetry. The conventional approach to overcome this issue is to train the model with augmented rotations and reflections, but this increases training time and does not explicitly restrict CNN kernels to exploit those symmetries. Rotation-equivariant CNNs, which are inherently equivariant to rotations and reflections were introduced for digital pathology,⁵²⁸ significantly improving over a comparable CNN on slide level classification and tumor localization. Similarly, Lafarge et al.³⁷⁶ designed a group convolution layer leveraging the rotational symmetry of pathology data to yield superior performance in mitosis detection, nuclei segmentation, and tumor classification tasks. These results motivate the application and further research of rotation-equivariant models for CPath.

In general, we note that the SOTA computer vision architectures used in computational pathology have tended to lag behind those used for natural images by a couple of years. This delay in knowledge propagation from the mainline computer vision research in natural images may be due to the data-centric nature of the CPath field. As data labelling is specialized and expensive to conduct, annotating more data or clever training tweaks to finetune established architectures is more attractive than developing advanced, specialized architectures. However, we recommend that CPath researchers should still use the most powerful relevant models available for the simple reason that they tend to perform best given the computational resources available. While computational efficiency is generally not as important during training, it is imperative at inference time if models are to be run in real-time on medical devices with limited computational resources.

8.8. Digital and computational pathology adoption

Despite the numerous advantages to the clinical workflow and applications offered by using digital pathology and CPath, the adoption of digital pathology remains the first barrier to clinical use. A major reason for adoption hesitancy is the common opinion that digital slide analysis is an unnecessary step in a pathologist’s workflow which has been refined over decades to produce reproducible and robust diagnoses without digitization.^9,18, 19, 20 In terms of clinical efficiency, studies have shown mixed results, with two finding that digitization actually decreased efficiency (by increasing turnaround time by $19\%$).^529,530 However, another study demonstrated a clear increase in productivity and reduction of turnaround-time.²⁴ One of the co-authors (B.N.) has implemented digital pathology at a public tertiary institution, which began as a pilot study over three years including three experienced academic pathologists and showed that digitization reduced turnaround time by $18\%$ for biopsies and $25\%$ for resections, and increased case output by $17\%$. These trial results led to all pathologists not retiring within two years to transition to a digital pathology workflow in 2019. Due to the varied nature of results and outcomes in studies analyzing the effectiveness of digital pathology there is more work to be done to have a multi-institution and lab analysis for more general and concrete results.

A major factor in the adoption of digital or computational pathology practices is the source of funding and the pay structure of pathologists. A few cost-analysis studies show that the transition to digital pathology becomes financially advantageous in 2 years, with savings projected to be up to $5M after 5 years in a sizeable tertiary center.^9,531, 532, 533 The financial impact will also be viewed differently in public vs private healthcare settings. Public healthcare is primarily limited by funding and universal access to healthcare whereas for private lab networks improvements in processes and services are directly linked to the prospects of obtaining additional contracts and increased profitability. However, studies considering multiple institutions and funding settings are still required to fully characterize the financial impact compared to clinical benefit. Additionally, on an individual pathologist level, compensation structures can affect buy in for implementation. For example, at our co-authors’ B.N. and V.Q.T institution, a fee-for-service structure is used to compensate pathologists thus an increase in throughput and productivity has a direct correlation to increased pay. We propose that this fee-for-service model contributes to the widespread embracement of DP at this institution. In contrast, pathologists in a salary-based environment are paid based based on a combined package of services which includes diagnostics, research, teaching, administration, quality control, etc. An increase in clinical productivity would technically not benefit them directly, as it would translate to a high number of rendered diagnostics over the same amount of time.

Integration CPath into the clinical workflow is relatively understudied as few papers have actually deployed, or performed clinical validation of their results. Works in this area have either proposed methods to deploy their models in the clinic or developed tools to enable the use of their research in the clinic.^287,490,534 However, as a primary goal of CPath is the use of CAD tools in clinical settings, more works should consider how to integrate models and tools into the clinical workflow, especially in conjunction with expert clinicians.

8.9. Institutional challenges

Several institutional challenges may affect the implementation of CPath tools, and similar challenges in implementing digital pathology workflows at medical institutions have been well-described by many studies.535, 536, 537, 538, 539 As noted by multiple studies considering the digital transition of pathology laboratories,535, 536, 537, 538, 539 the importance of a common shared goal and frequent communication between the involved parties is necessary to successfully deploy a digital system. These lessons are likely extendable in the context of CPath and CAD development as well. Specifically, Cheng et al.⁵³⁸ reported on their experiences and lessons learned as a 7-point-based system to efficiently deploy a digital pathology system in a large academic center. We believe similar systematic approaches will need to be developed to implement CPath applications in a clinical setting.

Another institutional challenge concerns the regulatory oversight at the departmental, institutional, accrediting agencies, pathology association, state/provincial, and federal levels. Regulatory measures underlying WSI scanners are well established, as well as the technical and clinical validation of their use.540, 541, 542 On the other hand, patient confidentiality, ethics, medical data storage regulations, and data encryption laws are equally, if not more, time-consuming and intensive to comply with. These issues can be mitigated by deploying a standardized digital pathology system throughout multiple institutions at the state/provincial level. For example, our co-author (B.N.) has obtained governmental approval and funding to distribute a set of digital pathology systems throughout the province’s public anatomical pathology laboratories. Similarly, a unified set of standards for processing and digitizing slides, along with unifying storage and access to WSIs for research use in collaborative efforts is paramount in moving forward in both the development and implementation of CAD systems.

8.10. Clinical alignment of CPath tasks

Researchers in the CPath field must ensure that the CAD tools they create are clinically relevant and applicable to pathology so that effort and resources are not allocated towards extraneous or clinically irrelevant tasks. For example, certain CADs have been proposed to facilitate case triaging and reduce turnaround time for critical diagnoses.72, 73, 74^,543,544 However, several regulatory agencies in pathology aim for $90\%$ of cases to be completed within a timeframe of 72 hours for signing-out resection specimens and up to 48 hours for biopsies.^545,546 In this context, triaging becomes extraneous, as signing out cases faster than 48-72 hours has no clinical impact. However, in the context of an institution operating at longer turnaround times or struggling to keep up with the caseload, this method could be lifesaving. Alternatively, identifying mitotic figures and counting positive Ki-67 nuclei are appreciated tools already in use in multiple digital pathology settings, despite these tools being seldom applied to the large caseload proportion of most practicing pathologists.

As noted previously, the overall number of pathologists in the USA has decreased $17\%$ from 2007 to 2017 and caseloads have increased by $41.7\%$_.⁵⁴⁷ This trend places further emphasis of developing CAD tools towards specific challenges encountered by pathologists and where sub-specialists may not be readily available. For example, a large consortium generated a prostate cancer CAD that achieved a $86.8\%$ concordance with expert genitourinary pathologists,⁵⁴⁸ a significant breakthrough for healthcare settings where prostate biopsies are not signed out by sub-specialists. Additionally, targeting specific diagnoses with high rates of medical errors and inter-observer variance, notably in dermatological, gynecological, and gastrointestinal pathology, should be prioritized and integrated into practice quickly to support patient care.⁵⁴⁹ Finally, advanced CAD tools capable of diagnosing features out of reach by conventional pathology could have a great impact. For example, identifying the origin of metastases from morphological cues on the WSIs without added IHC⁴⁴⁸ or CADs capable of calculating the exact involvement of cancer on a biopsy core for prognostic purposes.⁵⁴⁸

8.11. Concluding remarks

Bringing pathologists and computer scientists together and initiating meaningful collaborations with shared gains between all parties is likely the most efficient path forward for CPath and CAD integration. Opportunities to facilitate collaborations should be promoted by parties such as the Pathology Innovation Collaborative Community and the Digital Pathology Association. Furthermore, we encourage involved pathologists and computer scientists to communicate and collaborate on studies towards the common goal of providing patients with fast, reproducible, and high-quality care.

A.1. Clinical pathology workflow

Sample Collection Tissue samples are categorized into two general types a) small specimens, normally obtained to diagnose disease and guide subsequent treatment and b) large specimens, surgically removed to treat the disease after diagnosis. Small specimen biopsies are performed by different specialties in different settings, which can vary from family doctors sampling skin lesions to head and neck specialists performing panendoscopic biopsies under anesthesia. Based on the type of sample required and its originating site, small specimen samples are obtained by different methods: 1) core biopsies, 2) cytological specimens, 3) small excisions, and 4) pincer biopsies. In contrast, large specimens are mostly resections performed by surgeons for treatment purposes once the diagnosis has already been obtained. Large specimens are significantly more complex than small specimens and require a high-level of expertise to process before reaching the microscopic interpretation step.

Accessioning The patient-care team fills out request forms which are tagged to the pathology specimen and sent along with the sample to the pathology department to enter the specimen and patient information into the laboratory information system (LIS). Depending on the institution, the LIS can then be linked to the electronic medical records (EMR) system to populate basic demographic data, and to a slide tracking system (STS) to locate and time each event in pathology. Ensuring that accessioning is done correctly is essential, as specimen mix-ups or incorrect data entry from request forms are a large source of errors in the worfklow.⁵¹²

Specimen Preparation Most samples arrive at the pathology lab having already been preserved in a 4% formalin solution. Other preservation media are used for specific pathology objectives; notably using Roswell Park Memorial Institute medium (RPMI) to preserve cells for flow cytometry and glutaraldehyde preservation for electronic microscopy.⁵⁵⁰ Sometimes, the samples are sent to the pathology lab without a preservation medium, and in this “fresh” state the tissue will undergo cold ischemia changes until the pathologist can assess the sample, perform specimen preparation, and fix the tissue in the preservation media. Indeed, each of these events has an impact on the quality of the tissue, which in turn significantly affects the quality of the final WSI. Specifically, large specimens are almost always processed by a pathologist who will open the container, take basic measurements, and open the organ to ensure uniform formalin penetration.

At some medical centers, intra-operative consultations for resection samples are processed in a frozen section procedure, which allows for more rapid diagnosis of the tissue specimens while trading off diagnostic accuracy when compared to fixation in formalin.^551,552 Frozen section samples will be rapidly frozen after the proceeding grossing step, therefore they arrive in the lab unprocessed and remain unprocessed until after grossing, undergoing the aforementioned cold ischemia changes in the meantime.

Grossing Once the basic specimen preparation has occurred, the tissue is analyzed by the pathology team without the use of a microscope; a step called grossing. Smaller specimens are often grossed by technicians who fill out a template-based description notably highlighting the number, size, and appearance of the fragments. After grossing, the tissue fragments are placed into tissue cassettes for final fixation. Grossing larger specimens is much more complex, and is usually performed by pathologists, pathologist-assistants, medical residents, or fellows who have undergone extensive training. The grossing starts by orienting the specimen according to the surgical procedure. By cross-referencing the clinical findings and the EMR reports, the operator will localize the disease, locate the pathological landmarks, describe these landmarks, and measure the extent of the disease. Specific sampling of these landmarks is performed, and these samples are then put into cassettes for the final fixation.

Final Tissue and Slide Preparation For non-frozen section samples, the final fixation step will accrue all the samples put into cassettes and will add a last phase of fixation in formalin. Afterwards, the tissue cassettes are removed from formalin and inserted into a tissue processor which dehydrates the tissue through alcohol gradients, subsequently replacing the liquid with melted paraffin wax. These samples are removed from the tissue processor and placed into a mold filled with paraffin which solidifies into a block. A technician then cuts the blocks with a microtome into $4\mathit{\mu m}$ slices and places them onto positively charged slides. Slides are then deparaffinized, rehydrated by an inverse alcohol gradient, and stained with pigments such as Hematoxylin and Eosin (H&E), or further processed for ancillary testing by immunohistochemistry. Once these steps are complete, the tissue is dehydrated, cleared, and mounted with a coverslip. Note that significant variations will affect the final quality of the slides based on the performance of the prior steps, the experience of the technician who cut the slides, and batch effects from the reagents which are often reused for multiple slides prior to being replaced–known as cross-contamination.

Frozen section samples are rapidly frozen using a freezing medium such as liquid nitrogen, dry ice, or isopentane.^{174,175,551,552} After freezing, the tissue is cut using a microtome and fixed immediately, most often with formalin.^174,175 Slides are then stained and covered with a glass coverslip and stored at $-80$°C.¹⁷⁵

Slide Scanning and Image Processing The stained slides are cleaned of excess mounting media and scanned with a whole-slide scanner. They are then loaded onto the image management software, which has previously been linked to the LIS and the STS.⁵⁵³

Interpretation After a slide is processed and prepared, a pathologist views the slide to analyze and interpret the sample. The approach to interpretation varies depending on the specimen type. Interpretation of smaller specimens is focused on diagnosis of any disease. Analysis is performed in a decision-tree style approach to add diagnosis-specific parameters, e.g. esophagus biopsy $\to$ type of sampled mucosa $\to$ presence of folveolar-type mucosa $\to$ identify Barrett’s metaplasia $\to$ identify degree of dysplasia. Once the main diagnosis has been identified and characterized, the pathologist sweeps the remaining tissue for secondary diagnoses which can also be characterized depending on their nature. Larger specimens are more complex and focus on the focus is on characterizing the tissue and identifying unexpected diagnoses beyond the prior diagnosis from a small specimen biopsy. Microscopic interpretation of large specimens is highly dependent on the quality of the grossing and the appropriate detection and sampling of landmarks. Each landmark (e.g., tumor surface, tumor at deepest point, surgical margins, lymph node in mesenteric fat) is characterized either according to guidelines, if available, or according to the pathologists’ judgment. After the initial microscopic interpretation additional deeper cuts (“levels”), special stains, immunohistochemistry, and/or molecular testing may be performed to hone the diagnosis by generating new material or slides from the original tissue block.

Pathology Report The pathologist synthesizes a diagnosis by aggregating their findings from grossing and microscopic examination in combination with the patient’s clinical information, all of which are included in a final pathology report. The classic sections of a pathology report are patient information, a list of specimens included, clinical findings, grossing report, microscopic description, final diagnosis, and comment. The length and degree of complexity of the report again depends on the specimen type. Small specimen reports are often succinct, clearly and unambiguously listing relevant findings which guide treatment and follow-up. Large specimen reports depend on the disease, for example, in cancer resection specimens the grossing landmarks are specifically targeted at elements that will guide subsequent treatment.

In the past, pathology reports had no standardized format, usually taking a narrative-free text form. Free text reports can omit necessary data, include irrelevant information, and contain inconsistent descriptions.⁶⁹ To combat this, synoptic reporting was introduced to provide a structured and standardized reporting format specific to each organ and cancer of interest.^69,70 Over the last 15 years, synoptic reporting has enabled pathologists to communicate information to surgeons, oncologists, patients, and researchers in a consistent manner across institutions and even countries. The College of American Pathologists (CAP) and the International Collaboration on Cancer Reporting (ICCR) are the two major institutions publishing synoptic reporting protocols. The parameters included in these protocols are determined and updated by CAP and ICCR respectively to remain up-to-date and relevant for diagnosis of each cancer type. For the field of computational pathology, synoptic reporting provides a significant advantage in dataset and model creation, as a pre-normalized set of labels exist across a variety of cases and slides in the form of the synoptic parameters filled out in each report. Additionally, suggestion or prediction of synoptic report values are a possible CPath application area.

A.2. Diagnostic tasks

Here we provide some examples of diagnostic tasks where CPath has been applied, for the reader to understand the variety of diagnostic problems that CPath can be used to address.

Detection A machine learning framework for detecting cancerous tissue regions and predicting scan-level diagnosis is proposed in,²⁶⁶ wherein thresholding and statistical analysis used to abstain from making a decision in uncertain cases.In contrast to directly predicting the presence of cancers, feature-focused detection tasks can be highly useful in patient diagnosis and treatment planning. For example, identifying microsatellite instability is a crucial factor in determining if immunotherapy will be effective on a patient, and deep learning methods were shown to be effective at detecting microsatellite instability in.²⁸⁷ Similarly, the detection of fibrous regions in liver WSIs is a precursor step to liver tumor classification and a computational approach to detection was demonstrated in.⁵⁵⁴ Furthermore, the first automatic detection algorithm for keratin pearls, which are valuable biomarkers for oral squamous cell carcinoma grading is presented in.³⁶⁴ Future research into automated detection methods for similar cancer biomarkers could be a valuable step towards developing AI-based pathologist support tools. As an example, lymphocytes, a type of white blood cell, can be detected and quantified to assess the overall health of the immune system. However, manually detecting these cells is a time-consuming task and pathologists rarely identify and count lymphocytes. Thus several computational approaches, including open source tools such as QuPath⁵⁵⁵ and deep learning based approaches, are used to provide lymphocyte counts to pathologists.²⁰⁵ Likewise, counting nuclei can contribute towards diagnoses, however, nuclei detection is a difficult task because of the large variations in the shape of different types of nuclei, such as nuclear clutter, heterogeneous chromatin distribution, and irregular and fuzzy boundaries. Addressing these issues, for example, spatially constrained context-aware correlation filters with hierarchical deep features extracted from multiple layers of a pre-trained network were proposed to accurately detect nuclei in.³⁸⁵

Tissue Subtype Classification Deep neural networks have been shown to be effective at extracting molecular tumor features from histopathology images, opening new avenues for deep learning applications in computational pathology.⁵⁵⁶ As an extension of the tissue subtype classification task, ML models are often able to identify important correlations between tissue structures and disease. Work on nuclei classification suggests that features regarding the nuclear inner texture are most relevant for high classification accuracy.⁵⁵⁴ Additionally, a classifier discovered unique chromatin patterns associated with specific types of thyroid follicular lesions in.⁵⁵⁷ The potential discovery of similar associations makes tissue subtype classification a relevant task to pursue. Another work presented a computational pathology framework that can localize well-known morphological features on WSIs without the need for spatial labels for each feature using attention-based multiple-instance learning on WSI classification.⁶⁴ This method outperforms standard weakly-supervised classification algorithms and is adaptable to independent test cohorts, biopsy/resection samples, and varying tissue content. Additionally, the co-representation learning for classification (CoReL) framework is proposed in³⁷⁹ to improve state-of-the-art classification performance for nuclei classification, mitosis detection, and tissue type classification with less data.³⁷⁹

Disease Diagnosis As stated in 2, disease diagnosis can be considered a fine-grained classification problem which subdivides the general positive disease class into finer disease-specific labels based on the organ and patient context. Under this paradigm, research tends to be focused on maximizing performance for reliable clinical applications.^415,490,558 Recently, works have begun implementing different emulations of pathologist behaviour in their proposed models. For instance, multi-scale receptive fields were proposed for use in networks to simulate the pathologist viewing process of slides at varying zoom levels.³²⁸ Alternatively, weighted slide-level features were used to classify Barett’s esophagus and esophageal adenocarcinoma, similar to a pathologist assessing the overall impact of various cancer biomarkers.²⁸⁸ To emulate how pathologists isolate and focus on salient regions of the slide, the concept of visual attention can be applied to identify the most important regions of tissue slides, thus ignoring diagnostically-irrelevant image regions.^417,422,437 Such methods indicate a positive step towards the clinical implementation of AI-based CAD tools by reinforcing and emulating tested methodologies in pathology. Further, differential diagnoses in complicated cases of metastatic tumors and cancer of unknown primary (CUPs) can require many clinical tests to narrow a differential diagnosis, and a method called Tumor Origin Assessment via Deep Learning (TOAD) is introduced as an assistive tool to assign a differential diagnosis.³⁰² This work uses digitized H&E slides of tumors with known primary origins to train a model with transfer learning and weakly supervised multitask learning to simultaneously identify the tumorous or metastatic regions and predicts the site of origin.

Segmentation Segmentation CAD tools can capture characteristics of individual glands, nuclei, and tumor regions. The wide generalizability of this task to various disease types makes it a particularly suitable tool for computational pathology, on which many studies have been conducted.^277,330, 331, 332^,350 For example, models that use segmentation to determine nuclear characteristics including size and shape can help pathologists distinguish between various cell types and consequently, disease severity.^207,234 In,³⁴⁹ a generalized deep learning-based framework was proposed which uses a sequence of novel techniques forin the preprocessing, training, and inference steps which in conjunction improve the efficiency and the generalizability of model. Similarly, a new framework for WSI analysis in colonoscopy pathology, including lesion segmentation and tissue diagnosis was developed and includes an improved U-Net with a VGG net as the backbone, as well as two training and inference schemes to address the challenge of high resolution images analysis.³⁰⁴

There are also some instances of segmentation in different organs. For example, an interactive segmentation model was proposed in which the user-provided squiggles guide the model toward semantic segmentation of tissue regions.³⁶⁷ Also, they proposed four novel techniques to automatically extract minimalistic and human-drawn-like guiding signals from Ground Truth (GT) masks so that they can be used during the model’s training. Similarly for the eye, macular edema (ME) is a common disease where analyzing the fluid lesions is a critical stage of the diagnostic process. The optical coherence tomography (OCT) technique can potentially investigate three fluid types and a novel pipeline for segmentation of the three types of fluid lesions in OCT was proposed in.³⁵⁵ They presented a multi-layer segmentation to detect the ROI and presented an FCN architecture with attention gate (AG) and spatial pyramid pooling (SPP) module to improve the feature extraction. To predict cellular composition from images, ALBRT is proposed in,⁴⁷⁰ using contrastive learning to learn a compressed and rotation-invariant feature representation which first detects the presence of different cell types in an image patch and then provides cell counts for each type. Another novel deep learning model was developed for simultaneous nuclei instance segmentation in.³⁶⁹ The model is based on an encoder-decoder architecture design that performs nuclei segmentation by predicting the distance of pixels from their nuclei centers along with the nuclei probability masks and predicts nuclei classes when nuclei type annotations are available. Another work in nuclei segmentation is the hard-boundary attention network (HBANet), which identifies hard-boundaries between nuclei, a difficult problem due to overlapped nuclei.⁵⁵⁹ It presents a background weaken module (BWM) to improve the model’s attention to the foreground, and integrates low-level features containing more detailed information into deeper feature layers. Furthermore, a gradient-based boundary adaptive strategy (GS) is designed to generate boundary-weakened data as extra inputs and train the model in an adversarial manner. Finally, segmentation has also been applied to delineate tumorous tissue regions for a variety of cancer types, such as breast,^278,560 colorectal,^142,363 and prostate cancer.^476,561 Such works assist in the efficient isolation of tumor tissue, which is a crucial task for making accurate disease predictions.

A.3. Prognosis

Prognostic models must predict the likely development of a disease based on patient features. For instance, a prognostic model was developed by adjusting a tumor microenvironment-based spatial map with clinical variables such as patient age, gender, health history, and cancer stage.⁵³⁴ This multi-domain data analysis approach is advanced by another work, which uses both histopathological image data and cancer genomic data in their novel deep learning framework.^477,562 In,⁵⁶³ the authors discussed the correlation between platelets and other haematological measures to cancer by assessing patient status and considering the patient features in the primary care dataset, such as age and sex. They demonstrate the model performance with the plot of survival analysis per age group for platelets. Experiments on the publicly available TCGA data demonstrates that prognostic accuracy was maximized when both forms of data were simultaneously considered. Merging information from multiple WSIs of a patient allowed a hybrid aggregation network (HANet), consisting of a self-aggregation module and a WSI-aggregation module, to predict survival risks.⁵⁶⁴

A.4. Prediction of treatment response

Oral epithelial dysplasia (OED) segmentation is critical for early identification and effective treatment and HoVer-Net+ is a model to simultaneously perform nuclear instance segmentation (and classification) and semantic segmentation of epithelial layers based on H$\&$E stained histopathology slides of the oral mucosa.³⁶⁵ This model achieves the state-of-the-art performance in both tasks ($0.839$ dice score) and is the first method for simultaneous nuclear instance segmentation and semantic tissue segmentation.

A.5. Cancer statistics

Cancer remains the leading cause of global mortality in 2020, claiming nearly 10 million lives or approximately 1 in 6 deaths.⁵⁸³ The grey circle in Fig. 12 illustrates the prominence of Breast, Prostate, Colon, and Rectum, as well as Lung cancers, which collectively account for half of all diagnosed cases. The mounting volume of pathology cases poses a significant challenge in clinical workflows. This underscores the pivotal role of computational pathology in streamlining processes, aiding pathologists in coping with overwhelming workloads. Notably, certain cancers not only exhibit high prevalence but also contribute substantially to the overall mortality rates. Lung cancer, for instance, represents approximately $12\%$ of cancer cases in the United States, with its prognosis falling within the lowest range, as depicted in Fig. 12 ($0-30\%$). Computational pathology proves instrumental, particularly in lung cancer, by facilitating classification and prognosis tasks due to distinct variations among its types in terms of presentation, prognosis, and treatment strategies. Conversely, there are less prevalent cancers like liver cancer characterized by poor prognoses. CPath’s ability to compile specialized datasets for such cancers not only aids pathologists but also supports clinicians in devising personalized treatment plans. Understanding disease statistics and severity is paramount when designing a Computer-Aided Diagnosis (CAD) tool in Computational Pathology (CPath) or curating datasets. By factoring in disease prevalence, mortality rates, and severity across various cancer types, CAD tools can be optimized to prioritize detection, prognosis, and treatment planning for the most prevalent and severe cases, aligning computational pathology advancements with the urgent needs of patients and healthcare practitioners.

A.6. Whole slide imaging

Generally, a WSI scanning device is composed of four major components¹⁸⁰: (1) a light source; (2) a slide stage; (3) an objective lens; and (4) a digital camera. In order to produce a WSI that is in focus, which is especially important for CPath works, appropriate focal points must be chosen across the slide either using a depth map or by selecting arbitrarily spaced tiles in a subset.¹⁸⁷ Once focal points are chosen, the image is scanned by capturing tiles or linear scans of the image, these individual components are then stitched together to form the full image known as the big flat TIFF image.^180,187 To reduce the area needed to be scanned, a segmentation algorithm can be used within the scanner to separate tissue regions from extraneous background regions.¹⁹⁸ Additionally, slide can also be scanned at various magnification levels depending on the downstream task and analysis required. The vast majority of WSIs are scanned at $20\times$ ($\sim 0.5\mathit{\mu m}$/pixel) or $40\times$ ($\sim 0.25\mathit{\mu m}$/pixel) magnification as these are the most useful in practice for general pathologist.¹⁸⁰

WSI Storage and Standards

WSIs are in giga-pixel dimension format.^30,188 For instance a tissue in $1\mathit{cm}\times 1\mathit{cm}$ size scanned $@0.25\mathit{\mu m}$/pixel resolution can produce a $4.8$GB image (uncompressed) with a $\mathrm{50,000}\times \mathrm{50,000}$ pixels. Due to this large size, hardware constraints may not support viewing entire WSIs at full resolution.¹⁸⁹ Therefore, WSIs are most often stored in a tiled format, so that only the viewed portion of the image (tile) is loaded into memory and rendered.¹⁸⁹ Additionally, to support efficient zooming, WSIs are stored in a pyramidal structure, where higher levels of the pyramid represent lower magnification levels. The highest level of the pyramid is generally a very low resolution thumbnail snapshot of the WSI, while the lowest is the full resolution image (i.e. big-flat TIFF). In this way, different zoom levels can be captured and displayed in a WSI format efficiently–artificially replicating different zooming levels from optical microscopy.^189,193 Additionally, WSIs can be compressed before storage to reduce their filesize, often using JPEG, JPEG 2000, or LZW algorithms, which can reduce an image size by more than seven times.¹⁸⁰ Alongside the WSI, metadata regarding patient, tissue specimen, scanner, and WSI information is stored for reference.^30,188,190 Due to their clinical use and importance, it is important to develop effective storage solutions for these WSI data files and metadata, allowing for robust data management, querying of WSIs, and efficient data retrieval.^191,192

To develop CPath CAD tools in a widespread and general manner, a standardized format for WSIs and their corresponding metadata is essential.¹⁸⁸ However, there is a general lack of standardization for WSI formats outputted by various scanners, as shown in Table 1, especially regarding metadata storage. The Digital Imaging and Communications in Medicine (DICOM) standard provides a framework for biomedical image format and data management and has been extended to the CPath field through Supplement 145.^190,193 Some research has shown that the use of the DICOM standard allows for efficient data access and greater interoperability between different centres and different CPath-related devices.¹⁸⁸ However there is currently a lack of widespread adoption,^{30,188,190,584} reflected in Table 1 where only two recorded scanners are DICOM-compliant. Notably, with regards to metadata, DICOM provides a systematic format detailing a variety of relevant medical information, consistent with DICOM standards in other medical imaging fields.¹⁹⁰ The expansion through Supplement 145 also adds pyramid-tiling for WSIs, a format that is directly beneficial to creation of CPath and CAD tools. While many scanners have adopted the pyramid-type scheme for image data, they have not fully adopted the DICOM image format, outputting in either TIFF, BigTIFF format, or in TIFF-derivatives such as SVS or GTIFF. While the TIFF format allows for semi-structured metadata,¹⁸⁹ the consistency in metadata structure offered by DICOM is an advantage over the former.¹⁹⁰

Apart from storage format, a general system for storing and distributing WSIs is also an important pillar for CPath. Whereas in other medical imaging fields such as radiology, images are often stored in a picture archiving and communications systems (PACS) in a standardized DICOM format, with DICOM storage and retrieval protocols,¹⁸⁹ the need for standardization persists in pathology for WSI storage solutions. Few works have proposed solutions to incorporate DICOM-based WSIs in a PACS, although some research has successfully implemented a WSI PACS consistent using the DICOM standard using a web-based service for viewing and image querying.¹⁸⁹

A.7. Organs and diseases

The following Appendix section is a supplement to Section 2.5. Details are provided for several notable works in CPath for the organ types listed in the subsection.

Breast

• •

  A subset of breast-focused research studies the correlation between tissue morphology and molecular differences. Molecular testing can often be preferred over tissue morphology assessments when selecting breast cancer treatments as it provides objective and reproducible disease classifications. For instance, the connection between epithelial patterns with various molecular predictions and heatmaps was used to clearly visualize this correlation in.⁴²⁹

• •

  In³⁴² a proposed U-NET based architecture called piNET is used for cell detection and classification in order to calculate the proliferation index (PI) of the Ki67 biomarker. The network classifies cells as Ki67+/Ki67- and uses this classification to calculate the PI. The architecture is able to achieve a PI accuracy of $85.2\%$, higher than the accuracies of other models.³⁴²

• •

  A two-stage CNN, one for patch-level feature extraction and the other for classification, was proposed and achieved $95\%$ accuracy for classifying normal, benign, in situ carcinoma, and invasive carcinomas.⁵⁸⁵

• •

  In,⁵⁸⁶ several well-known models (DenseNet121, ResNet50, VGG16, and Xception), are compared with their own CNN model named lightweight convolutional neural network (LCNN) for the detection of breast cancer metastasis to axillary lymph nodes (ALN).

• •

  Further research has proposed an automated patient-level tumor segmentation and classification system that takes full use of diagnosis information hints from pathologists.³⁶⁶ A multi-level view DeepLabV3+ (MLV-DeepLabV3+) was created to investigate the differentiating aspects of cell characteristics between tumor and normal tissue for tumor segmentation. Furthermore, expert segmentation models were chosen and merged using Pareto-front optimization to mimic expert consultation and provide a flawless diagnosis.

Prostate

• •

  Most works focus on prostate aim to classify cancer based on Gleason scoring. To aid in that effort, a U-Net model for object semantic segmentation is created in,³⁴⁷ with the goal of precisely labeling each pixel in an image as belonging to either foregrounds, which may contain glands, or background.

• •

  A method is proposed in³⁸² that trains a model on both epithelial cell detection and Gleason grade prediction tasks to achieve better performance in both tasks than models trained on either of the tasks alone. Further, some works have investigated epithelial cell detection to explore data augmentation and stain/color normalization techniques^199,204 which demonstrates the importance of epithelial cell features as an indicator for prostate cancer detection.

• •

  Work in⁵⁸⁷ focused on classifying glands, gland boundary regions, and stroma. The authors opted for two classic classifiers: support vector machines (SVM) and Convolutional Neural Networks (CNNs), finding the SVM to perform best by offering high accuracy and good indicators of regions which are present with high probability. The output of the SVM classifier could help pathologists locate existing glands, saving them a significant amount of time from actively searching for them.

Lung

• •

  To further aid diagnosis, prognosis, and treatment decisions, a novel method for nuclei detection and characterization is introduced in²³⁴ using an unsupervised autoencoder network to learn without the use of any annotations. The unsupervised autoencoder is used to construct a CNN that only requires $5\%$ of training data to generate comparable results to the SOTA on supervised lymphocyte and nuclei tasks, thereby reducing the need for extremely large annotated datasets.

• •

  In,⁴⁹¹ the work not only classifies adenocarcinoma and squamous cell carcinoma, but also predicts the 10 most commonly mutated genes in adenocarcinoma. The findings indicate the presence of genotype-phenotype correlations for lung cancer tissues, and paves the way for cancer classification and mutation predictions of other types of less common lung cancers.

Colon and Rectum

• •

  An interesting detection application for MSI is implemented in.²⁸⁷ In this work, ResNet-18 is used to predict MSI from H&E slides with an AUC of $0.84$, although performance was reduced with tissue samples from different ethnicities. As MSI requires extensive additional testing which is not always performed,²⁸⁷ the study highlights the applicability of deep learning in detecting this important prognosis indicator.

• •

  The work in⁴²⁸ presents a weakly-supervised model named the Slide Level Annotation Model (SLAM) based on ShuffleNet⁵⁸⁸ that can be trained to detect genetic/molecular changes, including MSI or BRAF mutation, in colorectal WSIs. The results show improvement over SOTA models, and a visualization heatmap is generated which allow for improved result interpretability and analysis.

Bladder

• •

  In,⁴¹⁵ bladder cancer grade classification is explored using a large dataset of 915 WSIs which ends up outperforming 17 pathologists by an average of $10\%$. The model in this study has been integrated into an end-to-end diagnostic tool that provides interpretable cellular-level ROI visualization and natural language descriptions of histology slides. However, there was also a diagnostic disagreement of $23\%$ between the system and pathologists, which could hinder the diagnostic process and consequently limit the overall productivity.

Kidney

• •

  For effective patient prognosis, tissue microarray analysis is typically used to identify biomarkers. Currently, this process is time-consuming and prone to error, especially due to the heterogeneity of nuclei. A random forest classifier was proposed to more efficiently detect cancerous nuclei in MIB-1 stained tissue micro-array spots and predict the survival rate for renal cell carcinoma (RCC) patients in.³⁸¹ The results show that there is a significant difference in survival times for patients with high and low proliferating tumors, and further state MIB-1 staining as a key prognostic factor for the survival chance of RCC.

• •

  To aid in determing donor organ acceptance in kidney transplants,⁸⁴ uses frozen kidney sections as input data to identify the percentage of glomerulosclerosis.

Brain

• •

  The achieved state-of-the-art SOTA classification accuracy on the 2014 MICCAI Grand Challenge dataset was achieved in.⁷⁶ However, on completely unseen datasets, performance varied from $84\%$ to $93\%$. This shows the complexity of the diseases; low grade glioma (LGG) versus glioblastoma multiforme (GBM) classification is not a trivial task. Not only do their appearances vary in pathological samples, the diagnosis is often made from a few distinct features in a small slide region.⁷⁶

Liver

• •

  A notable work for liver cancer classification evaluates a pathologist’s performance using a liver cancer diagnostic tool for the diagnosis of HCC and cholangiocarcinoma.⁴⁹⁰ Despite lower performance than pathologists, the tool’s decisions directly affected pathologists’ decisions. Correct model predictions increased pathologists’ average accuracy, while incorrect predictions lowered average accuracy. Furthermore, pathologists frequently consulted the model’s predictions for difficult cases. This confirms the potential use of deep learning models as an AI diagnostic tool to provide knowledgeable second opinions.

Lymph Nodes

• •

  One of the earliest histopathology challenges, ICPR2010, targets lymphocyte and centroblast counting.⁵⁸⁹

• •

  The importance of the lymph nodes in cancer diagnosis is notably addressed in the CAMELYON16 and CAMELYON17 Challenges, in which participants classify lymph node metastases.²⁶⁴ Metastases in breast regional lymph nodes are classified based on size: micrometastasis, macrometastasis, and isolated tumour cells (ITCs). The ITC classification accuracy was less than $40\%$ for all top teams. This indicates that there was extreme difficulty in detecting ITCs, most likely due to the small size and variability. These results suggest that further improvements can be made by introducing more true positives of ITC data, or incorporating IHC stain information as an additional layer of information to improve detection robustness.

Organ Agnostic

• •

  Nuclei segmentation of epithelial, inflammatory, fibroblast and miscellaneous tissues is performed across seven different organs in.²⁷⁷ The method attempts to generalize across a large variety of datasets for increased usability and scalability in a clinical setting.

A.8. Ground truth labelling and annotation

Patient-level Annotation Patient-level annotations assign a single label to a single patient and come from case reports that can address multiple WSIs from a primary organ site. In addition to the WSIs, the Laboratory Information System (LIS) may also contain additional metadata, diagnostic information, and analytical or synoptic report information.⁵⁹⁰ Notably, the LIS can store specimen type, molecular and genetic tests, patient medical history, and clinical variables such as the patient’s age and gender.^591,592

Slide-level Annotation Slide-level annotations designate labels for a single WSI, which encompasses diagnosis and cancer information.^268,302 In comparison to the patient-level, this level of focus provides a more precise tissue location for the provided diagnosis.^316,450,593

ROI-level Annotation ROI-level annotation identifies regions within a slide that can be of either diagnostic or analytical relevance to a pathologist. Regions themselves can be designated using two methods: (1) bounding boxes^169,213,219 or (2) pixel-wise masks that are augmented on the WSIs.^61,220,594 Importantly, each ROI is considered to be a single class,⁵⁹⁵ but the labels represent more detailed tissue structures providing more specific and detailed diagnostic information than patient and slide levels,⁶⁴ ultimately being more applicable in disease diagnosis tasks.³⁶⁵

Patch-level Annotation Patch-level annotation is done on mosaic tiles (usually in a square shape) extracted from the WSI/ROI with a given field of view (FOV). Most deep-learning models are trained at the patch level, which contains anatomical structures of tissues and cells. Patches are either single-labeled or multi-labeled according to the taxonomical labeling workflow.³⁸⁰ One key aspect for patch-level annotation is determining the optimum FOV to encompass enough tissue classes,⁵⁹⁶ as considering smaller or bigger FOV can provide different advantages, as demonstrated in Fig. 7 for the patch-level.

Patch-Size Selection The choice of the patch size is limited by the computational complexity of the hardware that is used for training CAD tools. For instance, the majority of deep learning pipelines accept image sizes of less than 300$\times$300 pixels.^64,468,597, 598, 599^, The size of the FOV needs to be determined such that acceptable levels of morphological tissues will be covered within that patch. Accordingly, the pixel resolution is determined given a certain FOV and patch size. Given the above factors, if a larger FOV is required, then pixel resolution is limited which translates to information loss. In comparison, if higher pixel resolution is required, then the FOV will be limited accordingly which may exclude cellular/architectural relevance pertaining to the underlying class representation.³⁷⁹ To mitigate this tradeoff, larger image dimensions are required which consequently increases the computational power required for patch processing (e.g. high RAM GPU memory or parallelized multi-GPU processing).⁵⁹⁶

Pixel-level Sizing Pixel-level annotation requires labelling each pixel as a specified class. In this level, features are simple to extract and sufficient for describing the images as they encompass color and texture information.⁶⁰⁰ However, there is a lack of biological interpretability as the other levels of annotation more appropriately describe characteristics of the cellular and tissue structures.²⁶⁹ A solution based on human-interpretable image features can include histological knowledge and expert annotations that can describe different cell anatomies such as the stroma, the nuclei of the cells, and the size and shape of tumor regions, as well as the texture of the tissues and the location of tumor-infiltrating lymphocytes.⁶⁰¹

Pixel-wise masks are differentiated from pixel-level annotations in that when an ROI mask of this type is tiled into multiple instances (i.e. patches), each sample is considered a single class. The ROI-level is in contrast to the pixel-level annotations, wherein the latter is defined to include all annotations where each patch can contain several class types.

A.9. Surveyed datasets

A.10. Organ overview

For each paper recorded in the literature, a collection of information about their specific goal was collected. This information was categorized by organ and arranged into a table, the organs being: Basal/Epithelium, Bladder, Brain, Mouth/Esophagus, Breast, Liver, Lymph Node, Prostate/Ovary, Kidney, Lung, Pancreas, Thyroid, Stomach/Colon. Below will be the explanation of each column in Table 9.11:

• 1.

  References: Reference number of the paper that involved the specified task.

• 2.

  Tasks: Specific target goal that the work wanted to achieve, this range from different types of detection, classification, and segmentation to prognosis and diagnosis.

• 3.

  Disease Specification: Describes the pathology of the target goal of the paper.

• 4.

  Methods: Define the different machine learning methods used to achieve the proposed target task of the paper.

A.11. Technicalities by task

For each paper recorded in the literature, a collection of information on the Neural Network architectures used was organized and categorized by its specific task. It was found that across the majority of papers, the following five tasks were the most prevalent: Detection, Disease Diagnosis, Segmentation, WSI Processing, and Patient Prognosis. At the end of the table, an Other Task section was added to attach other works that don’t follow the selected tasks. Below will be the explanation of each column in Table 9.11:

• 1.

  References: Reference number of the paper that involved the specified task.

• 2.

  Tasks Specification: Describes the pathology of the target goal of the paper.

• 3.

  Architecture: Defines the different Neural Network architectures used to achieve the proposed target task of the paper.

• 4.

  Datasets: Name of the datasets used for the specified task (see Table 9.11 for information on datasets).