Over the past decade, several urban visual analytics systems and tools have been proposed to tackle a host of challenges faced by cities, in areas as diverse as transportation, weather, and real estate. Many of these tools have been designed through collaborations with urban experts, aiming to distill intricate urban analysis workflows into interactive visualizations and interfaces. However, the design, implementation, and practical use of these tools still rely on siloed approaches, resulting in bespoke systems that are difficult to reproduce and extend. At the design level, these tools undervalue rich data workflows from urban experts, typically treating them only as data providers and evaluators. At the implementation level, they lack interoperability with other technical frameworks. At the practical use level, they tend to be narrowly focused on specific fields, inadvertently creating barriers to cross-domain collaboration. To address these gaps, we present Curio, a framework for collaborative urban visual analytics. Curio uses a dataflow model with multiple abstraction levels (code, grammar, GUI elements) to facilitate collaboration across the design and implementation of visual analytics components. The framework allows experts to intertwine data preprocessing, management, and visualization stages while tracking the provenance of code and visualizations. In collaboration with urban experts, we evaluate Curio through a diverse set of usage scenarios targeting urban accessibility, urban microclimate, and sunlight access. These scenarios use different types of data and domain methodologies to illustrate Curio&rsquo;s flexibility in tackling pressing societal challenges. Curio is available at <a=href"urbantk.org/curio.">urbantk.org/curio</a>. 
Index Terms - Urban analytics, urban data, spatial data, dataflow, provenance, visualization framework, visualization system.

Using Curio to build a dataflow to analyze the shadow impact of a proposed building in Boston.

Curio: A Dataflow-Based Framework for Collaborative Urban Visual Analytics

Unsupervised video anomaly detection (UVAD) aims to detect abnormal events in videos without any annotations. It remains challenging because anomalies are rare, diverse, and usually not well-defined. Existing UVAD methods are purely data-driven and perform unsupervised learning by identifying various abnormal patterns in videos. Since these methods largely rely on the feature representation and data distribution, they can only learn salient anomalies that are substantially different from normal events but ignore the less distinct ones. To address this challenge, this paper pursues a different approach that leverages data-irrelevant prior knowledge about normal and abnormal events for UVAD. We first propose a new normality prior for UVAD, suggesting that the start and end of a video are predominantly normal. We then propose normality propagation, which propagates normal knowledge based on relationships between video snippets to estimate the normal magnitudes of unlabeled snippets. Finally, unsupervised learning of abnormal detection is performed based on the propagated labels and a new loss re-weighting method. These components are complementary to normality propagation and mitigate the negative impact of incorrectly propagated labels. Extensive experiments on the ShanghaiTech and UCF-Crime benchmarks demonstrate the superior performance of our method. 
Funding: NSF award ECCS-2400900

Learning Anomalies with Normality Prior for Unsupervised Video Anomaly Detection

Deep learning approaches have made significant success in single-view 3D reconstruction, but they often rely on expensive 3D annotations for training. Recent efforts tackle this challenge by adopting an analysis-by-synthesis paradigm to learn 3D reconstruction with only 2D annotations. However, existing methods face limitations in both shape reconstruction and texture generation. This paper introduces an innovative Analysis-by-Synthesis Transformer that addresses these limitations in a unified framework by effectively modeling pixel-to-shape and pixel-to-texture relationships. It consists of a Shape Transformer and a Texture Transformer. The Shape Transformer employs learnable shape queries to fetch pixel-level features from the image, thereby achieving high-quality mesh reconstruction and recovering occluded vertices. The Texture Transformer employs texture queries for non-local gathering of texture information and thus eliminates incorrect inductive bias. Experimental results on CUB-200-2011 and ShapeNet datasets demonstrate superior performance in shape reconstruction and texture generation compared to previous methods. 
Funding: NSF award CNS-1828265, NSF award ECCS-2400900

Analysis-by-Synthesis Transformer for Single-View 3D Reconstruction

In the biomedical domain, taxonomies organize the acquisition modalities of scientific images in hierarchical structures. Such taxonomies leverage large sets of correct image labels and provide essential information about the importance of a scientific publication, which could then be used in biocuration tasks. However, the hierarchical nature of the labels, the overhead of processing images, the absence or incompleteness of labeled data, and the expertise required to label this type of data impede the creation of useful datasets for biocuration. From a multi-year collaboration with biocurators and text-mining researchers, we derive an iterative visual analytics and active learning strategy to address these challenges. We implement this strategy in a system called BI-LAVA—Biocuration with Hierarchical Image Labeling through Active Learning and Visual Analytics. BILAVA leverages a small set of image labels, a hierarchical set of image classifiers, and active learning to help model builders deal with incomplete ground-truth labels, target a hierarchical taxonomy of image modalities, and classify a large pool of unlabeled images. BI-LAVA&rsquo;s front end uses custom encodings to represent data distributions, taxonomies, image projections, and neighborhoods of image thumbnails, which help model builders explore an unfamiliar image dataset and taxonomy and correct and generate labels. An evaluation with machine learning practitioners shows that our mixed human-machine approach successfully supports domain experts in understanding the characteristics of classes within the taxonomy, as well as validating and improving data quality in labeled and unlabeled collections. 
CCS Concepts: Human-centered computing -> Visual analytics; Computing methodologies -> Machine learning algorithms

Interaction during the case study: COVID-19 Image Modality Labeling.

BI-LAVA: Biocuration with Hierarchical Image Labeling through Active Learning and Visual Analytics

Over the past decade, there has been a significant increase in the development of visual analytics systems dedicated to addressing urban issues. These systems distill intricate urban analysis workflows into intuitive, interactive visual representations and interfaces, enabling users to explore, understand, and derive insights from large and complex data, including street-level imagery, street networks, and building geometries. Developing urban visual analytics systems, however, is a challenging endeavor that requires considerable programming expertise and interaction between various multidisciplinary stakeholders. This situation often leads to monolithic and isolated prototypes that are hard to reproduce, combine, or extend. Concurrently, there has been an increase in the availability of general and urban-specific toolkits, frameworks, and authoring tools that are open source and abstract away the need to implement low-level visual analytics functionalities. This paper provides a hierarchical taxonomy of urban visual analytics systems to contextualize how they are usually designed, implemented, and evaluated. We develop this taxonomy across three distinct levels (i.e., dimensions, categories, and tags), juxtaposing visualization with analytics, data, and system dimensions. We then assess the extent to which current open-source toolkits, frameworks, and authoring tools can effectively support the development of components tailored to urban visual analytics, identifying their strengths and limitations in addressing the unique challenges posed by urban data. In doing so, we offer a roadmap that can guide the effective employment of existing resources and chart a pathway for developing and refining future systems. 
Keywords: Visual Analytics, Visualization Toolkits, Visualization Grammars, Visualization Authoring, Urban Visual Analytics

Assessing the Landscape of Toolkits, Frameworks, and Authoring Tools for Urban Visual Analytics Systems

September 2023 marks the 50th anniversary of the Electronic Visualization Laboratory (EVL) at University of Illinois Chicago (UIC). EVL&rsquo;s introduction of the CAVE Automatic Virtual Environment in 1992, the first widely replicated, projection-based, walk-in, virtual-reality (VR) system in the world, put EVL at the forefront of collaborative, immersive data exploration and analytics. However, the journey did not begin then. Since its founding in 1973, EVL has been developing tools and techniques for real-time, interactive visualizations - pillars of VR. But EVL&rsquo;s culture is also relevant to its successes, as it has always been an interdisciplinary lab that fosters teamwork, where each person&rsquo;s expertise contributes to the development of the necessary tools, hardware, system software, applications, and human interface models to solve problems. Over the years, as multidisciplinary collaborations evolved and advanced scientific instruments and data resources were distributed globally, the need to access and share data and visualizations while working with colleagues, local and remote, synchronous and asynchronous, also became important fields of study. This paper is a retrospective of EVL&rsquo;s past 50 years that surveys the many networked, immersive, collaborative visualization and VR systems and applications it developed and deployed, as well as lessons learned and future plans. 
Article appears in the journal PRESENCE: Virtual and Augmented Reality, in the Special Issue on Immersive Visualization Laboratories - Past, Present and Future, edited by S. Su and W. Sherman, <a href="https://direct.mit.edu/pvar/issue/volume/33">https://direct.mit.edu/pvar/issue/volume/33</a>. 
Article copyrighted by Massachusetts Institute of Technology (MIT). 
DOI: <a href="https://doi.org/10.1162/pres_a_00421">https://doi.org/10.1162/pres_a_00421</a>

EVL faculty, students, and collaborators 1975.

Electronic Visualization Laboratory&rsquo;s 50th Anniversary Retrospective: Look to the Future, Build on the Past

Digital twin models are of high interest to Head and Neck Cancer (HNC) oncologists, who have to navigate a series of complex treatment decisions that weigh the efficacy of tumor control against toxicity and mortality risks. Evaluating individual risk profiles necessitates a deeper understanding of the interplay between different factors such as patient health, spatial tumor location and spread, and risk of subsequent toxicities that can not be adequately captured through simple heuristics. To support clinicians in better understanding tradeoffs when deciding on treatment courses, we developed DITTO, a digital-twin and visual computing system that allows clinicians to analyze detailed risk profiles for each patient, and decide on a treatment plan. DITTO relies on a sequential Deep Reinforcement Learning digital twin (DT) to deliver personalized risk of both long-term and short-term disease outcomes and toxicity risk for HNC patients. Based on a participatory collaborative design alongside oncologists, we also implement several visual explainability methods to promote clinical trust and encourage healthy skepticism when using our system. We evaluate the efficacy of DITTO through quantitative evaluation of performance and case studies with qualitative feedback. Finally, we discuss design lessons for developing clinical visual XAI applications for clinical end users. 
Index Terms: Medicine; Machine Learning; Application Domains; High Dimensional data; Spatial Data; Activity Centered Design

Overview of DITTO.

DITTO: A Visual Digital Twin for Interventions and Temporal Treatment Outcomes in Head and Neck Cancer

Though the object detection performance on standard benchmarks has been improved drastically in the last decade, current object detectors are often vulnerable to domain shift between the training data and testing images. Domain adaptation techniques have been developed to adapt an object detector trained in a source domain to a target domain. However, they assume that the target domain is known and fixed and that a target dataset is available for training, which cannot be satisfied in many real-world applications. To close this gap, this paper investigates fully test-time adaptation for object detection. It means to update a trained object detector on a single testing image before making a prediction, without access to the training data. Through a diagnostic study of a baseline self-training framework, we show that a great challenge of this task is the unreliability of pseudo labels caused by domain shift. We then propose a simple yet effective method, termed the IoU Filter, to address this challenge. It consists of two new IoU-based indicators, both of which are complementary to the detection confidence. Experimental results on five datasets demonstrate that our approach could effectively adapt a trained detector to various kinds of domain shifts at test time and bring substantial performance gains. Code is available at https://github.com/XiaoqianRuan1/IoU-filter. Code is available at https://github.com/ XiaoqianRuan1/IoU-filter. 
Funding:: The COMPaaS DLV project (NSF award CNS-1828265)

Fully Test-time Adaptation for Object Detection

Unsupervised semantic segmentation (USS) aims at partitioning an image into semantically meaningful segments by learning from a collection of unlabeled images. The effectiveness of current approaches is plagued by difficulties in coordinating representation learning and pixel clustering, modeling the varying feature distributions of different classes, handling outliers and noise, and addressing the pixel class imbalance problem. This paper introduces a novel approach, termed Imbalance-Aware Dense Discriminative Clustering (IDDC), for USS, which addresses all these difficulties in a unified framework. Different from existing approaches, which learn USS in two stages (i.e., generating and updating pseudo masks, or refining and clustering embeddings), IDDC learns pixel-wise feature representation and dense discriminative clustering in an end-to-end and self-supervised manner, through a novel objective function that transfers the manifold structure of pixels in the embedding space of a vision Transformer (ViT) to the label space while tolerating the noise in pixel affinities. During inference, the trained model directly outputs the classification probability of each pixel conditioned on the image. In addition, this paper proposes a new regularizer, based on the Weibull function, to handle pixel class imbalance and cluster degeneration in a single shot. Experimental results demonstrate that IDDC significantly outperforms all previous USS methods on three real-world datasets, COCO-Stuff-27, COCO-Stuff-171, and Cityscapes. Extensive ablation studies validate the effectiveness of each design. Our code is available at https://github.com/MY-LIU100101/IDDC. Our code is available at https://github.com/MY-LIU100101/IDDC. 
Keywords: Unsupervised semantic segmentation · Imbalance-Aware Dense Discriminative Clustering · End-to-end training ·
Deep clustering 
Funding: The COMPaaS DLV project (NSF award CNS-1828265)

Imbalance-Aware Discriminative Clustering for Unsupervised Semantic Segmentation

Studies have proven the effectiveness of mindfulness and forest-immersion therapy as an option for stress reduction. However, what happens when an individual is seeking stress relief but the outdoors is unavailable due to unpredictable weather or even a lack of accessibility? The purpose of this research project is to observe the effects of guided scenic meditation in a virtual reality environment on physiological stress. Additionally, it performs a platform comparison by deploying the same meditative experience and testing its effectiveness using a room-scale immersive environment and a desktop computer, as desktops ($400 - $1000) are far more affordable and widely available compared to room-scale immersive environments ($1 million+). Using each platform, twenty participants were asked to engage with a meditative application that simulates the sights, sounds, and smells of a forest environment. Based on the objective data collected, the participants experienced a decrease in both heart rate and blood pressure, demonstrating the effectiveness of the simulation in reducing physiological stress. However, the room-scale immersive environment proved to be more successful in reducing the heart rate than the desktop computer. Furthermore, according to the participants, their subjectively perceived stress levels decreased, resulting in a far more relaxed state than before. The results showcase that a virtual simulation of a real environment actively contributes to stress reduction and can be used in cases when the outdoors is unavailable. The platform comparison demonstrates that the product of the research - a guided scenic meditation application that simulates a forest - is accessible and does not require a million-dollar room-scale display in order to be effective in reducing stress.

The guided scenic meditation application deployed to the CAVE2, pictured on the left, and the desktop, pictured on the right.

The Effects of Virtual Reality Environments on Physiological Stress: A Platform Comparison Between Room-Scale Displays and Desktop Computers (MS Thesis Project)

This project was performed during the COVID-19 pandemic. It presents a visual computing framework (MOTIV) for analyzing moral rhetoric on social media around controversial topics such as Stay-at-Home orders. Using Moral Foundation Theory, it proposes a methodology for deconstructing and visualizing the when, where, and who behind each of these moral dimensions as expressed in microblog data. The project was developed in collaboration with experts from language processing, communications, and causal inference. Our approach integrates microblog data with multiple sources of geospatial and temporal data, and leverages unsupervised machine learning (generalized additive models) to support collaborative hypothesis discovery and testing. 
Abstract: We present a visual computing framework for analyzing moral rhetoric on social media around controversial topics. Using Moral Foundation Theory, we propose a methodology for deconstructing and visualizing the when, where, and who behind each of these moral dimensions as expressed in microblog data. We characterize the design of this framework, developed in collaboration with experts from language processing, communications, and causal inference. Our approach integrates microblog data with multiple sources of geospatial and temporal data, and leverages unsupervised machine learning (generalized additive models) to support collaborative hypothesis discovery and testing. We implement this approach in a system named MOTIV. We illustrate this approach on two problems, one related to Stay-at-home policies during the COVID-19 pandemic, and the other related to the Black Lives Matter movement. Through detailed case studies and discussions with collaborators, we identify several insights discovered regarding the different drivers of moral sentiment in social media. Our results indicate that this visual approach supports rapid, collaborative hypothesis testing, and can help give insights into the underlying moral values behind controversial political issues. 
Supplemental Material: <a href="https://osf.io/ygkzn/?view_only=6310c0886938415391d977b8aae8b749">https://osf.io/ygkzn/?view_only=6310c0886938415391d977b8aae8b749</a>

MOTIV visualization of moral framing on social media in 2020.

MOTIV: Visual Exploration of Moral Framing in Social Media

Sunlight and shadow play critical roles in how urban spaces are utilized, thrive, and grow. While access to sunlight is essential to the success of urban environments, shadows can provide shaded places to stay during the hot seasons, mitigate heat island effect, and increase pedestrian comfort levels. Properly quantifying sunlight access and shadows in large urban environments is key in tackling some of the important challenges facing cities today. In this paper, we propose Dee Umbra, a novel computational framework that enables the quantification of sunlight access and shadows at a global scale. Our framework is based on a conditional generative adversarial network that considers the physical form of cities to compute high-resolution spatial information of accumulated sunlight access for the different seasons of the year. We use data from seven different cities to train our model, and show, through an extensive set of experiments, its low overall RMSE (below 0.1) as well as its extensibility to cities that were not part of the training set. Additionally, we contribute a set of case studies and a comprehensive dataset with sunlight access information for more than 100 cities across six continents of the world. Deep Umbra is available at <a href="https://urbantk.org/shadows">https://urbantk.org/shadows</a>. 
Index Terms - Urban computing, Urban analytics, Sunlight access, Shadow, Generative adversarial networks.

Example where shadows are cast across tiles.

Deep Umbra: A Generative Approach for Sunlight Access Computation in Urban Spaces

e-JUST - Environmental Justice using Urban Scalable Toolkit, a state-of-the-art GIS-based web platform, is being developed to inform local and regional decision-making regarding air quality and heat in the Chicago metropolitan area, utilizing Earth science and high-resolution climate model outputs. e-JUST contributes to mitigating risks associated with extreme heat and air pollution by improving decision-making related to heat and air quality stressors in the Chicago metropolitan region. e-JUST technology architecture involves ReactJS, Vega-Lite, and Urban Toolkit's functionalities. e-JUST integrates a variety of meteorological and air quality datasets. Among these are NASA&rsquo;s remote sensing observations, such as GOES 16 and ECOSTRESS, that measure surface temperatures at 100 m and hourly intervals. The environmental data layers include high-resolution (1 km and higher) outputs from urban climate models related to temperature, NO2 and PM2.5, ozone, and other pollutants. To account for demographic data, we use the 2020 US Census. Our data fusion approaches include further downscaling environmental data layers to street scale using machine learning algorithms for spatiotemporal interpolation, enabling more accurate predictions at scales relevant to communities. e-JUST&rsquo;s statistical view includes a heat matrix that illustrates correlations between several environmental and socioeconomic variables at a grid point in time, as well as a scatterplot that allows users to compare observed data and forecasts of two different variables. Overall, this toolkit provides information for the near past and near-real-time forecasting to aid in decision-making and emergency management preparedness for Chicago communities. Additionally, our platform is designed to provide users with easy access to data, enabling them to view and analyze data immediately. The e-JUST toolkit is anticipated to benefit a wide audience, ranging from the general public to academic researchers.

e-JUST - Environmental Justice using Urban Scalable Toolkit

Similarity detection seeks to identify similar, but distinct items over multivariate datasets. Often, similarity cannot be defined computationally, leading to a need for visual analysis, such as in cases with ensemble, computational, patient cohort, or geospatial data. In this work, we empirically evaluate the effectiveness of common visual encodings for multivariate data in the context of visual similarity detection. We conducted a user study with 40 participants to measure similarity detection performance and response time under moderate scale (16 items) and large scale (36 items). Our analysis shows that there are significant differences in performance between encodings, especially as the number of items increases. Surprisingly, we found that juxtaposed star plots outperformed superposed parallel coordinate plots. Furthermore, color-cues significantly improved response time, and attenuated error at larger scales. In contrast to existing guidelines, we found that filled star plots (Kiviats) outperformed other encodings in terms of scalability and error.

Similarity detection over multivariate data using five visual encodings (clockwise order from top left): Kiviat diagrams, Lines-Only glyphs, Color-cue Kiviat diagrams, Parallel Coordinate Plots, and Color-cue Parallel Coordinate Plots.

Kiviat Defense: An Empirical Evaluation of Visual Encoding Effectiveness in Multivariate Data Similarity Detection

Biocuration is the process of analyzing biological or biomedical articles to organize biological data into data repositories using taxonomies and ontologies. Due to the expanding number of articles and the relatively small number of biocurators, automation is desired to improve the workflow of assessing articles worth curating. As figures convey essential information, automatically integrating images may improve curation. In this work, we instantiate and evaluate a first-in-kind, hybrid image+text document search system for biocuration. The system, MouseScholar, leverages an image modality taxonomy derived in collaboration with biocurators, in addition to figure segmentation, and classifiers components as a back-end and a streamlined front-end interface to search and present document results. We formally evaluated the system with ten biocurators on a mouse genome informatics biocuration dataset and collected feedback. The results demonstrate the benefits of blending text and image information when presenting scientific articles for biocuration. 
Index Terms - document search, biocuration

MouseScholar interface.

MouseScholar: Evaluating an Image+Text Search System for Biocuration

Rayleigh-Benard (RB) convection occurs in a fluid layer when there is a temperature difference between the top and bottom boundaries, resulting in the spontaneous formation of convective motion.  In this study, a direct numerical simulation has been performed aimed at elucidating the impact of introducing dispersed droplets into a single-phase flow on the heat transfer dynamics within this flow.  The dimensionless parameters considered in this study encompass a Prandtl number of 4, a Weber number of 6000, and a Rayleigh number of 10^8.  Dispersed droplets were incorporated over a range spanning from 0 to 50%.

Effects of employing liquid-liquid emulsions on heat transfer within a turbulent Rayleigh-Benard convection

In the realm of Computational Fluid Dynamics (CFD), the demand for memory and computation resources is extreme, necessitating the use of leadership-scale computing platforms for practical domain sizes. This intensive requirement renders traditional checkpointing methods ineffective due to the significant slowdown in simulations while saving state data to disk. As we progress towards exascale and GPU-driven High-Performance Computing (HPC) and confront larger problem sizes, the choice becomes increasingly stark: to compromise data fidelity or to reduce resolution. To navigate this challenge, this study advocates for the use of in situ analysis and visualization techniques. These allow more frequent data &ldquo;snapshots&rdquo; to be taken directly from memory, thus avoiding the need for disruptive checkpointing. We detail our approach of instrumenting NekRS, a GPU-focused thermal-fluid simulation code employing the spectral element method (SEM), and describe varied in situ and in transit strategies for data rendering. Additionally, we provide concrete scientific use cases and report on runs performed on Polaris, Argonne Leadership Computing Facility’s (ALCF) 44 Petaflop supercomputer and Jülich Wizard for European Leadership Science (JUWELS) Booster, Jülich Supercomputing Centre’s (JSC) 71 Petaflop High Performance Computing (HPC) system, offering practical insight into the implications of our methodology.

Side view visualization of a RBC case.

Scaling Computational Fluid Dynamics: In Situ Visualization of NekRS using SENSEI

SAGE3, the newest and most advanced generation of the Smart Amplified Group Environment, is an open-source software designed to facilitate collaboration among scientists, researchers, students, and professionals across various fields. This tutorial aims to introduce attendees to the capabilities of SAGE3, demonstrating its ability to enhance collaboration and productivity in diverse settings, from co-located office collaboration to remote collaboration to both at once, with diverse displays, from personal laptops to large-scale display walls. Participants will learn how to effectively use SAGE3 for brainstorming, data analysis, and presentation purposes, as well as installation of private collaboration servers and development of custom applications. 
CCS CONCEPTS: Human-centered computing, Collaborative and social computing systems and tools; Open source software; Visualization systems and tools; Interactive systems and tools. 
KEYWORDS: Large Displays, Space to Think, Collaboration, Visualization, Data Science, Data Analysis, Computational Narratives

SAGE3 for Interactive Collaborative Visualization, Analysis, and Storytelling

Applications in immersive environments have gained popularity for training, learning, and recreational tasks. Due to the increasing availability of sensors and data-capturing devices, research is extending the use of immersive environments to support visual analytics processes, including sensemaking and strategic immersion for interaction and task completion. In this work, we propose Personal Situated Analytics (PSA) framework to embed users into recorded meetings with support for multiple degrees of immersion in the Reality-Virtuality spectrum. Our proposed framework encompasses various stages such as tracking, data capturing, data cleaning, data synchronization, prototype building, and deploying the final product to end-user hardware. We evaluate this framework on a data analysis scenario between human subjects and a conversational AI agent. Our pilot study (n=12) using this framework compares user experiences when using two different devices: Hololens2 and Quest2.

A user embedded in a conversation using Personal Situated Analytics in Virtual Reality as virtual avatars.

Using Personal Situated Analytics (PSA) to Interpret Recorded Meetings

Personalized head and neck cancer therapeutics have greatly improved survival rates for patients, but are often leading to understudied long-lasting symptoms which affect quality of life. Sequential rule mining (SRM) is a promising unsupervised machine
learning method for predicting longitudinal patterns in temporal data which, however, can output many repetitive patterns that are difficult to interpret without the assistance of visual analytics. We present a data-driven, human-machine analysis visual system
developed in collaboration with SRM model builders in cancer symptom research, which facilitates mechanistic knowledge discovery in large-scale, multivariate cohort symptom data. Our system supports multivariate predictive modeling of post-treatment symptoms
based on during-treatment symptoms. It supports this goal through an SRM, clustering, aggregation back end, and a custom front
end to help develop and tune the predictive models. The system also explains the resulting predictions in the context of therapeutic
decisions typical in personalized care delivery. We evaluate the resulting models and system with an interdisciplinary group of modelers and head and neck oncology researchers. The results demonstrate that our system effectively supports clinical and symptom research. 
Index Terms: Temporal Data; Life Sciences; Mixed Initiative Human-Machine Analysis; Data Clustering and Aggregation

Longitudinal symptom analysis and prediction for head and neck patients (ICC treatment).

Roses Have Thorns: Understanding the Downside of Oncological Care Delivery Through Visual Analytics and Sequential Rule Mining

While cities around the world are looking for smart ways to channel new advances in data collection, management, and analysis to address their day-to-day problems, the complex nature of urban issues and the overwhelming amount of available structured and unstructured data have posed significant challenges in translating these efforts into actionable insights. In the past few years, urban visual analytics tools have significantly helped tackle these challenges. When analyzing a feature of interest, an urban scientist must transform, integrate and visualize different thematic (e.g., sunlight, flows) and physical (e.g., buildings, sidewalks) layers, oftentimes across multiple spatial and temporal scales. However, integrating and analyzing these layers requires expertise in different fields, increasing development time and effort. This makes the entire visual data exploration and system implementation hard for experienced programmers and also sets a high entry barrier for urban experts outside of computer science. With this in mind, in this paper, we present the Urban Toolkit (UTK), a flexible and extensible visualization framework that enables the easy authoring of web-based visualizations through a new high-level grammar specifically built with common urban use cases in mind. In order to facilitate the integration and visualization of different urban data, we also propose the concept of knots to merge thematic and physical urban layers. We evaluate our approach through use cases and a series of interviews with experts and practitioners from different domains, including urban accessibility, urban planning, architecture, and climate science. UTK is available at urbantk.org. 
Index Terms: Urban visual analytics, Urban analytics, Urban data, Visualization toolkit.

UTK abstracts spatial joins through the use of knots.

The Urban Toolkit: A Grammar-based Framework for Urban Visual Analytics

Weather forecasting is essential for decision-making and is usually performed using numerical modeling. Numerical weather models, in turn, are complex tools that require specialized training and laborious setup and are challenging even for weather experts. Moreover, weather simulations are data-intensive computations and may take hours to days to complete. When the simulation is finished, the experts face challenges analyzing its outputs, a large mass of spatiotemporal and multivariate data. From the simulation setup to the analysis of results, working with weather simulations involves several manual and error-prone steps. The complexity of the problem increases exponentially when the experts must deal with ensembles of simulations, a frequent task in their daily duties. To tackle these challenges, we propose ProWis: an interactive and provenance-oriented system to help weather experts build, manage, and analyze simulation ensembles at runtime. Our system follows a human-in-the-loop approach to enable the exploration of multiple atmospheric variables and weather scenarios. ProWis was built in close collaboration with weather experts, and we demonstrate its effectiveness by presenting two case studies of rainfall events in Brazil. 
Index Terms - Weather visualization, Ensemble visualization, Provenance management, WRF visual setup

Building, managing, and analyzing a Weather Research and Forecasting (WRF) model ensemble using PROWIS to study an extreme rainfall event in Maricá (RJ, Brazil) in April 2022.

ProWis: A visual approach for building, managing, and analyzing weather simulation ensembles in runtime

New web technologies have enabled the deployment of powerful GPU-based computational pipelines that run entirely in the web browser, opening a new frontier for accessible scientific visualization applications. However, these new capabilities do not address the memory constraints of lightweight end-user devices encountered when attempting to visualize the massive data sets produced by today&rsquo;s simulations and data acquisition systems. In this paper, we propose a novel implicit isosurface rendering algorithm for interactive visualization of massive volumes within a small memory footprint. We achieve this by progressively traversing a wavefront of rays through the volume and decompressing blocks of the data on-demand to perform implicit ray-isosurface intersections. The progressively rendered surface is displayed after each pass to improve interactivity. Furthermore, to accelerate rendering and increase GPU utilization, we introduce speculative ray-block intersection into our algorithm, where additional blocks are traversed and intersected speculatively along rays as other rays terminate to exploit additional parallelism in the workload. Our entire pipeline is run in parallel on the GPU to leverage the parallel computing power that is available even on lightweight end-user devices. We compare our algorithm to the state of the art in low-overhead isosurface extraction and demonstrate that it achieves 1.7×–5.7× reductions in memory overhead and up to 8.4× reductions in data decompressed.

Interactive full-resolution isosurface visualization of the 2048×2048×1920 Richtmyer-Meshkov (R-M) data set in the browser.

Speculative Progressive Raycasting for Memory Constrained Isosurface Visualization of Massive Volumes

Using immersive environments for training, education, and recreation is becoming increasingly popular among individuals. In recent years these environments have also been used to facilitate data analytics processes involving interaction and task completion. In this work, we discuss Personal Situated Analytics (PSA) a cross-platform framework to embed users into recorded conversations with support for multiple degrees of immersion in the Reality-Virtuality spectrum. Our development pipeline consisted of several stages: tracking, data capturing, data cleaning, data synchronization, prototype building, and deploying the final product to end-user hardware. We evaluate PSA by analyzing data from a recorded meeting involving human participants and a visual conversational AI agent. Our pilot study (n = 12) using this framework compares user experiences for MR and VR environments. 
Index Terms: Immersive Analytics, Situated Analytics, Visualization techniques, Human-centered computing

PSA is an immersive MR/VR framework for experiencing and analyzing recorded conversations and events. We see two participants represented as virtual avatars engaged in a conversation.

PSA: A Cross-Platform Framework for Situated Analytics in MR and VR

Estimating the human posture from an image or a video is a fundamental task in computer vision. It not only enhances other vision tasks like action recognition, person re-identification, and virtual try-on but also facilitates many real-world applications including robotics, healthcare, sports, and retail. An effective and efficient human pose estimation system can enable robots to acquire skills from demonstrations, support physical therapists in diagnosing and rehabilitating patients, assist sports analysts and coaches in tracking and training athletes, and empower retailers to establish employee-free stores. Thanks to the advancement of deep learning and the availability of large-scale datasets, the performance of state-of-the-art human pose estimation methods has drastically improved in recent years. These approaches can accurately estimate postures in daily activities and sports. However, several challenges persist. For example, (1) it remains challenging to estimate postures that occur rarely or are entirely absent in the training data; (2) handling complex scenarios, such as crowded environments, motion blur, low-light conditions, and occlusions, is still a formidable task; (3) there is a growing need to develop efficient models that can estimate human poses in real-time or on low-power devices; (4) exploring novel applications for human pose estimation that can bring societal benefits or transform industries is an exciting avenue of research. This special issue presents a collection of high-quality papers on human pose estimation and its applications. They cover a wide range of topics, from fundamental human pose models with improved accuracy and efficiency to applications in human behavior understanding and assessment, sign language recognition, and multi-camera calibration. Through this special issue, we would like to open the path toward more discussion between practitioners and researchers in this very important yet challenging field of human pose estimation.

Guest editorial: special issue on human pose estimation and its applications

Semi-Supervised Temporal Action Localization (SS-TAL) aims to improve the generalization ability of action detectors with large-scale unlabeled videos. Albeit the recent advancement, one of the major challenges still remains: noisy pseudo labels hinder efficient learning on abundant unlabeled videos, embodied as location biases and category errors. In this paper, we dive deep into such an important but understudied dilemma. To this end, we propose a unified framework, termed Noisy Pseudo-Label Learning, to handle both location biases and category errors. Specifically, our method is featured with (1) Noisy Label Ranking to rank pseudo labels based on the semantic confidence and boundary reliability, (2) Noisy Label Filtering to address the class-imbalance problem of pseudo labels caused by category errors, (3) Noisy Label Learning to penalize inconsistent boundary predictions to achieve noise-tolerant learning for heavy location biases. As a result, our method could effectively handle the label noise problem and improve the utilization of a large amount of unlabeled videos. Extensive experiments on THUMOS14 and ActivityNet v1.3 demonstrate the effectiveness of our method. The code is available at github.com/kunnxia/NPL.

Learning from noisy pseudo labels for semi-supervised temporal action localization

* Authors: 
Belcaid, M., Leigh, J., Theriot, R., Kirshenbaum, N., Tabalba, R., Rogers, M. University of Hawai’i at Manoa, Honolulu, HI 
Johnson, A., Brown, M., Renambot, L., Long, L., Nishimoto, A., University of Illinois Chicago, Chicago, IL 
North C., Harden, J., Virginia Tech, Blacksburg, VA 
Translational software research bridges the gap between scientific innovations and practical applications, driving impactful societal advancements. However, developing such software is challenging due to interdisciplinary collaboration, technology adoption, and post-funding sustainability. This article presents the experiences and insights of the Scalable Adaptive Graphics Environment (SAGE) team, which has spent two decades developing translational, cross-disciplinary, collaboration tools to benefit computational science research. With a focus on SAGE and its next-generation iterations, we explore the inherent challenges in translational research, such as fostering cross-disciplinary collaboration, motivating technology adoption, and ensuring post-funding product sustainability. We also discuss the roles of funding agencies, policymakers, and academic institutions in promoting translational research. Although the journey is fraught with challenges, the societal impact and satisfaction derived from translational research underscore its significance in the broader scientific landscape. This article aims to encourage further conversation and the development of effective models for
translational software projects.

The process that defines domain science, computer science (CS), and sustainable translational research.

Reflecting on the Scalable Adaptive Graphics Environment Team’s 20-Year Translational Research Endeavor in Digital Collaboration Tools

Recent advances in precision medicine for treating Oropharyngeal cancer (OPC) have led to significant improvements in survival for patients. However, OPC survivors often are left with severe side effects that result from damage to vital tissues in the head and neck caused by secondary radiation received during treatment. These side effects can persist for months after treatment cessation, and diminish the patients’ quality of life. Existing guidelines for radiation therapy plans identify dose limits to individual organs with the goal to avoid or minimize permanent tissue damage and their related toxicity. However, existing tools fail to account for either the three-dimensional nature of dose distributions or the fact that critical organs often work as a system that can have multiple failure points. 
We present an unsupervised approach for stratifying patients based on three-dimensional dose distributions to multiple organs and evaluate their association with long-term patient-reported symptoms. We apply this method to predicting severe patient-reported symptoms 6 months after ceasing radiation therapy for OPC patients. For actionable interpretability, we introduce a rule-mining method that simplifies our strata into a set of dose thresholds, and demonstrate that these thresholds outperform existing dose treatment guidelines. 
Keywords: Radiation therapy; Oropharynx cancer; Head and Neck cancer; Stratification; Symptom Burden; Quality of Life

Results of a chi-squared test between covariates and membership in each set of clusters for each outcome.

Multi-Organ Spatial Stratification of 3-D Dose Distributions Improves Risk Prediction of Long-Term Self-Reported Severe Symptoms in Oropharyngeal Cancer Patients Receiving Radiotherapy: Development of a Pre-Treatment Decision Support Tool.

Motivation: Figures in biomedical papers communicate essential information with the potential to identify relevant documents in biomedical and clinical settings. However, academic search interfaces mainly search over text fields. 
Results: We describe a search system for biomedical documents that leverages image modalities and an existing index server. We integrate a problem-specific taxonomy of image modalities and image-based data into a custom search system. Our solution features a front-end interface to enhance classical document search results with image-related data, including page thumbnails, figures, captions, and image-modality information. We demonstrate the system on a subset of the CORD-19 document collection. A quantitative evaluation demonstrates higher precision and recall for biomedical document retrieval. A qualitative evaluation with domain experts further highlights our solution’s benefits to biomedical search. 
Availability and implementation: A demonstration is available at <a href="https://runachay.evl.uic.edu/scholar">https://runachay.evl.uic.edu/scholar</a>. Our code and image models can be accessed via <a href="https://github.com/uic-evl/bio-search">https://github.com/uic-evl/bio-search</a>. The dataset is continuously expanded. 
Image caption: Search interface displaying the results of a boolean query. (A) The color-coded legend displays the image modalities in the first level of the taxonomy. (B) The search bar, with a keyword box showing a boolean query, a time filter, and an image modalities filter limiting results to documents containing \textit{radiology} or \textit{fluorescence microscopy} figures. (C) The metadata of a result includes the document title, year, venue, authors, and the hits on the abstract and full text. Below is the count of figures and modalities in the document. (D) The enhanced document surrogate shows the image content, including the page thumbnail, extracted figure, subfigures and modalities, and caption. 
<a href="https://doi.org/10.1093/bioadv/vbad095">https://doi.org/10.1093/bioadv/vbad095</a>

Search interface displaying the results of a boolean query.

Enhancing biomedical search interfaces with images

Objective: Evaluate the effectiveness of machine learning tools that incorporate spatial information such as disease location and lymph node metastatic patterns-of-spread, for prediction of survival and toxicity in HPV+ oropharyngeal cancer (OPC). 
Materials & methods: 675 HPV+ OPC patients that were treated at MD Anderson Cancer Center between 2005 and 2013 with curative intent IMRT were retrospectively collected under IRB approval. Risk stratifications incorporating patient radiometric data and lymph node metastasis patterns via an anatomically-adjacent representation with hierarchical clustering were identified. These clusterings were combined into a 3-level patient stratification and included along with other known clinical features in a Cox model for predicting survival outcomes, and logistic regression for toxicity, using independent subsets for training and validation. 
Results: Four groups were identified and combined into a 3-level stratification. The inclusion of patient stratification in predictive models for 5-yr Overall survival (OS), 5-year recurrence free survival, (RFS) and Radiation associated dysphagia (RAD) consistently improved model performance measured using the area under the curve (AUC). Test set AUC improvements over models with clinical covariates, was 9 % for predicting OS, and 18 % for predicting RFS, and 7 % for predicting RAD. For models with both clinical and AJCC covariates, AUC improvement was 7 %, 9 %, and 2 % for OS, RFS, and RAD, respectively. 
Conclusion: Including data-driven patient stratifications considerably improve prognosis for survival and toxicity outcomes over the performance achieved by clinical staging and clinical covariates alone. These stratification generalize well to across cohorts, and sufficient information for reproducing these clusters is included. 
Keywords: Head & Neck cancer, Statistical and research methods, Machine learning, Radiation oncology, Health Informatics, Imaging, Risk stratification, Oropharyngeal cancer, Clustering, Radiomics

Average lymph node involvement and clinical staging categories of the 3 derived lymph node + radiomics clusters across both the training and validation datasets.

Spatially-aware clustering improves AJCC-8 risk stratification performance in oropharyngeal carcinomas

Honorable Mention in the EuroVis Best Paper Category 
Exploratory analysis of the chemical space is an important task in the field of cheminformatics. For example, in drug discovery research, chemists investigate sets of thousands of chemical compounds in order to identify novel yet structurally similar synthetic compounds to replace natural products. Manually exploring the chemical space inhabited by all possible molecules and chemical compounds is impractical, and therefore presents a challenge. To fill this gap, we present ChemoGraph, a novel visual analytics technique for interactively exploring related chemicals. In ChemoGraph, we formalize a chemical space as a hypergraph and apply novel machine-learning models to compute related chemical compounds. It uses a database to find related compounds from a known space and a machine learning model to generate new ones, which helps enlarge the known space. Moreover, ChemoGraph highlights interactive features that support users in viewing, comparing, and organizing computationally identified related chemicals. With a drug discovery usage scenario and initial expert feedback from a case study, we demonstrate the usefulness of ChemoGraph. 
Keywords: chemical space exploration, cheminformatics, multipartite graphs, data visualization 
CCS Concepts: Applied computing → Chemistry 
<iframe width="560" height="315" src="https://www.youtube.com/embed/kVpz6TDjQMw" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

An overview of ChemoGraph.

ChemoGraph: Interactive Visual Exploration of the Chemical Space

Most real-world networks are both dynamic and multivariate in nature, meaning that the network is associated with various attributes and both the network structure and attributes evolve over time. Visualizing dynamic multivariate networks is of great significance to the visualization community because of their wide applications across multiple domains. However, it remains challenging because the techniques should focus on representing the network structure, attributes, and their evolution concurrently. Many real-world network analysis tasks require the concurrent usage of the three aspects of the dynamic multivariate networks. In this paper, we analyze current techniques and present a taxonomy to classify the existing visualization techniques based on three aspects: temporal encoding, topology encoding, and attribute encoding. Finally, we survey application areas and evaluation methods; and discuss challenges for future research. 
CCS Concepts Human-centered computing → Graph drawings

Overview of the taxonomy of DMVN visualization techniques along with illustrations of the categories.

The State of the Art in Visualizing Dynamic Multivariate Networks

Developing applicable clinical machine learning models is a difficult task when the data includes spatial information, for example, radiation dose distributions across adjacent organs at risk. We describe the co-design of a modeling system, DASS,to support the hybrid human-machine development and validation of predictive models for estimating long-term toxicities related to radiotherapy doses in head and neck cancer patients. Developed in collaboration with domain experts in oncology and data mining, DASS incorporates human-in-the-loop visual steering, spatial data, and explainable AI to augment domain knowledge with automatic data mining. We demonstrate DASS with the development of two practical clinical stratification models and report feedback from domain experts. Finally, we describe the design lessons learned from this collaborative experience. 
CCS Concepts 
Human-centered computing→Scientific visualization; Computing methodologies→Machine learning; Applied computing→Life and medical sciences

DASS interactive model building for head/neck cancer.

DASS Good: Explainable Data Mining of Oncology Imaging and Toxicity Data

Methane the main component of natural gas is also a potent greenhouse gas that requires robust monitoring, so that scientists, educators, and the general public can get better informed of its effect(s) on local/regional air quality and current short-term mitigations for climate change. However, current data visualizations for methane, and other scientific data, are either static 2-D graphs/plots or animated through using videos/gifs. These data visualization approaches can be limiting in that they lack interactiveness for an end user to gain exploratory insights by comparing and contrasting different areas of a dataset. This report thereby showcases how to build a functional and interactive visualization as the solution to the previously mentioned limitation. 
Keywords: Interactive Scientific Data Visualization, Climate Data Visualization, Methane Visualization

This figure shows the respective volumetric/spheric plots for 07/02/2021 at 1000 hPa. Figure 2:

Visualizing the NOAA CarbonTracker-CH4 3-D Mole Fraction (Pressure Corrected) Dataset

Background: Existing studies have reported that forest therapy utilizing sensory stimuli increases the activity of the parasympathetic nervous system and decreases the activity of the sympathetic nervous system. 
Purpose: To observe the effects of guided scenic meditation in a virtual reality environment on physiological stress. 
To perform a platform comparison by deploying the same meditative experience and testing its effectiveness using a room-scale display and a desktop computer. 
Significance: Rather than using a headset, the study uses a room-scale display - the CAVE2 Hybrid-Reality Environment - to test the effects of virtual reality environments on stress reduction. 
The platform comparison intends to showcase that if proven effective, the product is accessible, as desktop computers are far more affordable ($400 - $1000) and widely available relative to room-scale displays ($1 million+).

Fig 1. On the left is the CAVE2 version of the application. Fig 2. On the right is the desktop version of the application.

The Effects of Virtual Reality Environments on Physiological Stress: A Platform Comparison Between Room-Scale Displays and Desktop Computers (poster)

In the past, existing studies have proven the effectiveness of forest therapy as an option for stress reduction, but what happens when an individual is seeking stress relief but the outdoors is unavailable due to unpredictable weather or even a lack of accessibility? The purpose of this research project is to observe the effects of guided scenic meditation in a virtual reality environment on physiological stress. Additionally, it intends to perform a platform comparison by deploying the same meditative experience and testing its effectiveness using a room-scale display and a desktop computer, as the latter ($400 - $1000) is far more affordable and widely available relative to the former ($1 million+). Using both platforms, ten participants were asked to engage with a Unity-built meditative application that simulates the sights, sounds, and smells of a forest environment. Based on the data collected, the participants experienced a decrease in both heart rate and blood pressure, demonstrating the effectiveness of the simulation in reducing physiological stress. Similarly, according to the participants, their subjectively perceived stress levels decreased, resulting in a far more relaxed state than before. The results showcase that a virtual simulation of an environment actively contributes to stress reduction and can be used in cases when the outdoors is unavailable. The platform comparison demonstrates that the product
of the research - a guided scenic meditation application that simulates a forest - is accessible and does not require a million-dollar room-scale display in order to be effective in reducing stress.

The forest model developed in Unity.

The Effects of Virtual Reality Environments on Physiological Stress: A Platform Comparison Between Room-Scale Displays and Desktop Computers

While cities around the world are increasingly promoting streets and public spaces that prioritize pedestrians over vehicles, significant data gaps have made pedestrian mapping, analysis, and modeling challenging to carry out. Most cities, even in industrialized economies, still lack information about the location and connectivity of their sidewalks, making it difficult to implement research on pedestrian infrastructure and holding the technology industry back from developing accurate, location-based Apps for pedestrians, wheelchair users, street vendors, and other sidewalk users. To address this gap, we have designed and implemented an end-to-end open-source tool - TILE2NET - for extracting sidewalk, crosswalk, and footpath polygons from orthorectified aerial imagery using semantic segmentation. The segmentation model, trained on aerial imagery from Cambridge, MA, Washington DC, and New York City, offers the first open-source scene classification model for pedestrian infrastructure from sub-meter resolution aerial tiles, which can be used to generate planimetric sidewalk data in North American cities. TILE2NET also generates pedestrian networks from the resulting polygons, which can be used to prepare datasets for pedestrian routing applications. The work offers a low-cost and scalable data collection methodology for systematically generating sidewalk network datasets, where orthorectified aerial imagery is available, contributing to over-due efforts to equalize data opportunities for pedestrians, particularly in cities that lack the resources necessary to collect such data using more conventional methods. 
Keywords: Pedestrian network extraction, Semantic segmentation, Automated network generation, Pedestrian infrastructure

Model results showing detected sidewalk, crosswalk and footpath centerlines in a) Boston and Cambridge, b) Manhattan and parts of Brooklyn, c) Wash- ington, DC. The maps are shown at the same scale for comparison.

Mapping the walk: A scalable computer vision approach for generating sidewalk network datasets from aerial imagery

Weakly-supervised Video Anomaly Detection (WVAD) aims to detect abnormal events in videos given only video-level labels for training. Recent methods relying on multiple instance learning (MIL) and self-training achieve good performance, but they tend to focus on learning easy abnormal patterns while ignoring hard ones, e.g., unusual driving trajectory or over-speeding driving. How to detect hard anomalies is a critical but largely ignored problem in W-VAD. To tackle this challenge, we propose a novel framework, termed Abnormal Ratios guided Multi-phase Self-training (ARMS), for W-VAD. It includes a new abnormal ratio-based MIL (AR-MIL) loss and a new multi-phase self-training paradigm. The AR-MIL loss guides the learning of hard anomalies by enforcing a minimum ratio of abnormal snippets in an abnormal video and no abnormal snippets in a normal video. Our multi-phase self-training paradigm sequentially performs bootstrapping, hard anomalies mining, and adaptive self-training so as to address pseudo labeling on easy anomalies, detect hard anomalies, and setting adaptive abnormal ratios for different videos in a unified framework. Experimental results on three benchmark datasets, i.e., ShanghaiTech, UCF-Crime, and XD-Violence, show that ARMS outperforms all previous state-of-the-art methods and has a great advantage in detecting hard anomalies. 
Index Terms: Anomaly detection, weakly-supervised video anomaly detection, multiple instance learning.

Abnormal Ratios Guided Multi-Phase Self-Training for Weakly-Supervised Video Anomaly Detection

Contrails are condensation trails generated from emitted particles by aircraft engines, which perturb Earth’s radiation budget. Simulation modeling is used to interpret the formation and development of contrails. These simulations are computationally intensive and rely on high-performance computing solutions, and the contrail structures are not well defined. We propose a visual computing system to assist in defining contrails and their characteristics, as well as in the analysis of parameters for computer-generated aircraft engine simulations. The back end of our system leverages a contrail-formation criterion and clustering methods to detect contrails&rsquo; shape and evolution and identify similar simulation runs. The front-end system helps analyze contrails and their parameters across multiple simulation runs. The evaluation with domain experts shows this approach successfully aids in contrail data investigation. 
Index Terms: Scalar Field Data, Physical & Environmental Sciences, Mathematics, Feature Detection, Tracking & Transformation 
<a href="http://dx.doi.org/10.1109/TVCG.2022.3209356">http://dx.doi.org/10.1109/TVCG.2022.3209356</a>

Contrail structure groups and evolution over time.

Visual Analysis and Detection of Contrails in Aircraft Engine Simulations

It has often been observed, informally and in academic literature, that speech directed toward individuals who are hard of hearing is naturally modified by talkers in an attempt to accommodate the listeners. These modifications take numerous forms, from the easily expressed &ldquo;louder&rdquo; and/or &ldquo;slower&rdquo; to the highly technical &ldquo;vowel space expansion&rdquo; and many others. There are also speech modifications made in numerous other circumstances, not only for hard of hearing listeners, but for listeners in noisy backgrounds, second language listeners, and many others, known under the collective (is sometimes too-broad) umbrella as clear speech. There is, however, an asymmetry inherent to all these talker-listener scenarios, which is that talkers have only the broadest guidance at best (&ldquo;Louder!&rdquo; &ldquo;Slower!&rdquo;) from their listener partners, which they may simply ignore. 
With an interest in developing new and practical interventions for the elderly and hard of hearing, this dissertation takes as its research questions: First, the feasibility of the production of a signal processing system which allows real-time modification of audio signals. Second, the tolerance and effectiveness of partially emulating clear speech, by using signal processing techniques to modify (specifically, to slow) the rate of received speech signals. And finally, the effectiveness of placing the control of speech rate directly into the hands of listeners, a capability that has never been possible prior to electronic mediation. 
This dissertation is organized as follows: 
Chapter One is a survey of both the circumstances which prompt these spontaneous speech modifications, as well as of the details of the modifications which are made. 
Chapter Two presents an overview of the various methods of changing the rate of speech including the simple but ineffective technique of re-sampling, as well as several frequency domain techniques. These systems were intended for use on pre-existing, rather than live, signals. 
Chapter Three presents the details of several incarnations of this real-time system, including a single-talker system implemented on a laptop computer, presented in [1] and used directly (with two laptops for two talkers) in the study presented in [3]; a two-talker wireless system using Android smartphones and local network connectivity presented at [2]; and an unpublished system using a central laptop base station, to which Android smart phones could
connect even when hundreds of miles apart. This chapter contains details required for the implementation of both live signals, and live control of the playback rate of those signals. Finally, this system includes the design details of systems used in the three user studies of Chapters Five, Six, and Seven. 
Chapter Four presents a user study designed to examine the effects of using real-time temporal modification software in a conversational setting, previously published in [3]. The key element of the study design was the use of a scored Diapix test [7] which was designed to elicit spontaneous conversational speech between two study participants without the involvement of or prompting by researchers. The main results of this study are (i) that at modest amounts of stretching (40% additional playback time, with lengthy silences not stretched) the speech patterns of subjects are not changed to a statistically significant degree, and (ii) under those same circumstances, there is no statistical evidence of changes in performance on the Diapix test. 
Chapter Five presents the first of three user studies focusing on the little-studied subject of the interaction of user choice and user control over received speech rates in adverse conditions. This study was based off the preexisting QuickSIN test [8], designed to test and characterize a listener&rsquo;s ability to understand short sentences of speech played against a background of noise. In the first part of this study, users were asked to listen to several short sentences each played against multiple levels of background noise. For each level of background noise, the subjects were asked to use a computer interface to set the rate of the speech to whatever rate they believed was most helpful in understanding the foreground speech. (The background babble did not change rate.) These preferred expansion rates were recorded for later use and analysis. In the second part of the study, subjects were asked to listen to and immediately repeat back sentences at those same noise conditions, both with speech modified to their preferred expansion rates, and unmodified. The subjects’ repetition of those sentences was also recorded for analysis. 
Our results included statistically significant differences in received speech rate at opposite ends of the noise condition, with subjects requesting slower speech in the presence of increase noise; as well as an overall belief that slowing speech was beneficial to subjects’ performance. However, analysis of the subjects&rsquo; ability to repeat was in fact mildly degraded at moderate to high noise conditions and was not improved at any noise condition. This work was published at Interspeech 2018 [4]. 
Chapter Six presents the second of the three user control studies. In this study, non-native students with recent TOEFL scores were recruited as test subjects. After familiarization with a graphical interface, test subjects were asked to participate in six comprehension tasks, each of which was a lengthy (several minutes long) audio track of a TOEFL test to which they had not previously been exposed. During three of these tasks, the subjects were given
the ability to change the rate of playback as they saw fit, and asked to do so in whatever fashion they believed would best help them understand the audio passage. The other three comprehension tasks were presented unmodified, without a control interface. After each task, the subjects were immediately given a short multiple choice quiz about the contents of the previous audio passage. Subjects&rsquo; behavior (i.e., the rate changes specified) and quiz responses were collected and analyzed. 
Our analysis of the data shows that a considerable majority of the subjects used speech slowing in all three of their trials (roughly 2/3) and an overwhelming majority used the technique in at least one of their trials (80%), but did not reveal a statistically significant improvement or degradation in listening comprehension. There is also a weak correlation between lower TOEFL scores and more slowing. This work was published at Interspeech 2019 [5]. In the work described above, the algorithms employed stretch speech uniformly - if set to expand an audio track by 40%, all sounds, including all parts of speech all background noises, and even silences, are expanded by the same rate. For large expansion factors, the speech so produced sounds somewhat unnatural, because this is not how talkers spontaneously produce slow speech. With this in mind, Chapter Seven returns to the topic of speech in noise, but puts even more precise control in the hands of test subjects: A relatively simple neural network was developed to classify individual phonemes into six broad phonemic categories within an audio track. A study was also designed around this tool which asked subjects to listen to short sentences in noise, and then expand or contract the audio only for particular phonemes (e.g., first modify vowels, then modify fricatives, etc.) Following this, all modifications were applied simultaneously, and subjects were asked to listen to modified and unmodified/control sentences in three levels
of background noise and asked to transcribe the sentences into a computer interface. 
Our analysis found either no statistically significant improvement or degradation (at one noise level) or statistically significant degradation (at two noise levels) of intelligibility while using this technique. 
Finally, Chapter Eight presents overall conclusions, with additional discussion of the entire work and of directions for future studies.

Unwrapped Schematic of Cochlea

Audio Expansion Techniques For Real-Time Communication And Intelligibility Enhancement

Cluster schedulers are crucial in high-performance computing (HPC). They determine when and which user jobs should be allocated to available system resources. Existing cluster scheduling heuristics are developed by human experts based on their experience with speciﬁc HPC systems and workloads. However, the increasing complexity of computing systems and the highly dynamic nature of application workloads have placed tremendous burden on manually designed and tuned scheduling heuristics. More aggressive optimization and automation are needed for cluster scheduling in HPC. In this work, we present an automated HPC scheduling agent named DRAS (Deep Reinforcement Agent for Scheduling) by leveraging deep reinforcement learning. DRAS is built on a hierarchical neural network incorporating special HPC scheduling features such as resource reservation and backﬁlling. An efﬁcient training strategy is presented to enable DRAS to rapidly learn the target environment. Once being provided a speciﬁc scheduling objective given by the system manager, DRAS automatically learns to improve its policy through interaction with the scheduling environment and dynamically adjusts its policy as workload changes. We implement DRAS into an HPC scheduling platform called CQGym. CQGym provides a common platform allowing users to ﬂexibly evaluate DRAS and other scheduling methods such as heuristic and optimization methods. The experiments using CQGym with different production workloads demonstrate that DRAS outperforms the existing heuristic and optimization approaches by up to 50%. 
Index Terms - High-performance computing, cluster scheduling, deep reinforcement learning, job starvation, backﬁlling, resource reservation, OpenAI Gym 
<a href="https://ieeexplore.ieee.org/abstract/document/9894371">https://ieeexplore.ieee.org/abstract/document/9894371</a>

DRAS overview. DRAS agent (at the bottom) represents the scheduling agent; the environment (at the top) comprises the rest of the system, including job wait queue and HPC cluster.

DRAS: Deep Reinforcement Learning for Cluster Scheduling in High Performance Computing

BACKGROUND: Acute lung injury and the acute respiratory distress syndrome (ARDS) are characterized by pulmonary inflammation, reduced endothelial barrier integrity and filling of the alveolar space with protein-rich edema fluid and infiltrating leukocytes. Animal models are critical to uncovering the pathologic mechanisms of this devastating syndrome. Intravital imaging of the intact lung via two photon intravital microscopy has proven a valuable method to investigate lung injury in small rodent models through characterization of inflammatory cells and vascular changes in real-time. However, respiratory motion complicates the analysis of these time series images and requires selective data extraction to stabilize the image. Consequently, analysis of individual alveoli may not provide a complete picture of the integrated mechanical, vascular and inflammatory processes occurring simultaneously in the intact lung. To address these challenges, we developed a web browser-based visualization application named Alveolus Analysis to process, analyze and graphically display intravital lung microscopy data. RESULTS: The designed tool takes raw temporal image data as input, performs image preprocessing and feature extraction offline, and visualizes the extracted information in a web browser-based interface. The interface allows users to explore multiple experiments in three panels corresponding to different levels of detail: summary statistics of alveolar/neutrophil behavior, characterization of alveolar dynamics including lung edema and inflammatory cells at specific time points, and cross-experiment analysis. We performed a case study on the utility of the visualization with two members or our research team and they found the tool useful because of its ability to preprocess data consistently and visualize information in a digestible and informative format. CONCLUSIONS: Application of our software tool, Alveolus Analysis, to intravital lung microscopy data has the potential to enhance the information gained from these experiments and provide new insights into the pathologic mechanisms of inflammatory lung injury. 
<a href="https://github.com/uic-evl/AlveolusAnalysis">Visits the code repository.</a>

Alveolus Analysis: A Web Browser-based Tool to Analyze Lung Intravital Microscopy

Inaccessible urban infrastructure creates and reinforces systemic exclusion of people with disabilities and impacts public health, physical activity, and quality of life for all. To improve the design of our cities and to enable more equitable policies and location-centric
technology designs, we need new data collection techniques, data standards, and accessibility-infused analytic tools and interactive maps focused on the quality, safety, and accessibility of pathways, transit ecosystems, and buildings. In this workshop, we bring together leading experts in human mobility, urban design, disability, and accessible computing to discuss pressing urban access challenges across the world and brainstorm solutions. We invite contributions from practitioners, transit officials, disability advocates,
and researchers.

The Future of Urban Accessibility for People with Disabilities: Data Collection, Analytics, Policy, and Tools

Background. Personalized radiotherapy can improve treatment outcomes of head and neck cancer (HNC) patients, where currently a &rsquo;one-dose-fits-all&rsquo; approach is the standard. The aim was to establish individualized outcome prediction based on multi-institutional international &ldquo;big data&rdquo; to facilitate risk-based stratification of HNC patients. 
Methods. The data of 4611 HNC radiotherapy patients from three academic cancer centers was split into 4 cohorts: a training (n=2241), independent test (n=786), and external validation cohorts 1 (n=1087) and 2 (n=497). Tumor- and patient-related clinical variables were considered in a machine learning pipeline to predict overall survival (primary endpoint) and local and regional tumor control (secondary endpoints); serially, imaging features were considered for optional model improvement. Finally, patients were stratified into high, intermediate, and low risk groups. 
Results. Performance score, AJCC8th stage, pack-years, and Age were identified as predictors for overall survival, demonstrating good performance in both the training cohort (c-index=0.72 [95% CI, 0.66-0.77]) and in all three validation cohorts (c-indices: 0.76 [0.69-0.83], 0.73 [0.68- 0.77], and 0.75 [0.68-0.80]). Excellent stratification of HNC patients into high, intermediate, and low mortality risk was achieved; with 5-year overall survival rates of 17-46% for the high-risk group compared to 92-98% for the low-risk group. The addition of morphological image feature further improved the performance (c-index=0.73 [0.64-0.81]). These models are integrated in a clinic-ready interactive web-interface: <a href="https://uic-evl.github.io/hnc-predictor/">https://uic-evl.github.io/hnc-predictor/</a>. 
Conclusions. Robust model-based prediction was able to stratify HNC patients in distinct high, intermediate and low mortality risk groups. This can effectively be capitalized for personalized radiotherapy, e.g., for tumor radiation dose escalation/de-escalation. 
See <a href+"https://www.sciencedirect.com/science/article/pii/S095980492200805X">Science Direct</a> for more information.

Head and Neck Cancer Predictive Risk Estimator to Determine Control and Therapeutic Outcomes of Radiotherapy (HNCPREDICTOR)

Sidewalks play a pivotal role in urban mobility of everyday life. Ideally, sidewalks provide a safe walkway for pedestrians, link public transportation facilities, and equip people with routing and navigation services. However, there is a scarcity of open sidewalk data, which not only impacts the accessibility and walkability of cities but also limits policymakers in generating insightful measures to improve the current state of pedestrian facilities. As one of the most famous crowdsourced data repositories, OpenStreetMap (OSM) could aid the lack of open sidewalk data to a large extent. However, completeness and quality of OSM data have long been a major issue. In this paper, we offer a preliminary study on the availability and trustworthiness of OSM sidewalk data. First, we compare OSM
sidewalk data coverage in over 50 major cities in the United States. Then, we select three major cities (Seattle, Chicago, and New York City) to further analyze the completeness of sidewalk data and its features, and to compute a trustworthiness index leveraging historical OSM sidewalk data.

Coverage of sidewalks and roads with sidewalk info in Chicago, Seattle, and New York City (from left to right). Seattle OSM has comparably higher percentage of sidewalk geometries and roads with sidewalk information mapped than the other two cities.

Crowdsourcing and Sidewalk Data: A Preliminary Study on the Trustworthiness of OpenStreetMap Data in the US

Contrails are condensation trails generated from emitted particles by aircraft engines, which perturb Earth&rsquo;s radiation budget. Simulation modeling is used to interpret the formation and development of contrails. These simulations are computationally intensive and rely on high-performance computing solutions, and the contrail structures are not well defined. We propose a visual computing system to assist in defining contrails and their characteristics, as well as in the analysis of parameters for computer-generated aircraft engine simulations. The back-end of our system leverages a contrail-formation criterion and clustering methods to detect contrails&rsquo; shape and evolution and identify similar simulation runs. The front-end system helps analyze contrails and their parameters across multiple simulation runs. The evaluation with domain experts shows this approach successfully aids in contrail data investigation. 
Index Terms: Scalar Field Data, Physical & Environmental Sciences, Mathematics, Feature Detection, Tracking & Transformation 
 <iframe width="420" height="315"src="https://www.youtube.com/embed/mFllS2wQW8I">
</iframe>

Run analysis.

Recent technological innovations have led to an increase in the availability of 3D urban data, such as shadow, noise, solar
potential, and earthquake simulations. These spatiotemporal datasets create opportunities for new visualizations to engage experts
from different domains to study the dynamic behavior of urban spaces in this under-explored dimension. However, designing 3D
spatiotemporal urban visualizations is challenging, as it requires visual strategies to support analysis of time-varying data referent
to the city geometry. Although different visual strategies have been used in 3D urban visual analytics, the question of how effective
these visual designs are at supporting spatiotemporal analysis on building surfaces remains open. To investigate this, in this paper we first contribute a series of analytical tasks elicited after interviews with practitioners from three urban domains. We also contribute a quantitative user study comparing the effectiveness of four representative visual designs used to visualize 3D spatiotemporal urban data: spatial juxtaposition, temporal juxtaposition, linked view, and embedded view. Participants performed a series of tasks that required them to identify extreme values on building surfaces over time. Tasks varied in granularity for both space and time dimensions. Our results demonstrate that participants were more accurate using plot-based visualizations (linked view, embedded view) but faster using color-coded visualizations (spatial juxtaposition, temporal juxtaposition). Our results also show that, with increasing task complexity, plot-based visualizations perform better in preserving efficiency (time, accuracy) compared to color-coded visualizations. Based on our findings, we present a set of takeaways with design recommendations for 3D spatiotemporal urban visualizations for researchers and practitioners. Lastly, we report on a series of interviews with four practitioners, and their feedback and suggestions for further work on the visualizations to support 3D spatiotemporal urban data analysis. 
Index Terms - Visualization, urban analytics, urban data, spatiotemporal data, empirical evaluation.

Urban surface data from different domains: (a)flooding simulation analysis-environment, (b)view impact analysis-urban planning, (c)building design performance analysis-urban design, (d)solar potential & sunlight analysis-urban design/architecture

A Comparison of Spatiotemporal Visualizations for 3D Urban Analytics

Powerful detectors at modern experimental facilities routinely collect data at multiple GB/s. Online analysis methods are needed to enable the collection of only interesting subsets of such massive data streams, such as by explicitly discarding some data elements or by directing instruments to relevant areas of experimental space. Thus, methods are required for configuring and running distributed computing pipelines-what we call flows-that link instruments, computers (e.g., for analysis, simulation, artificial intelligence [AI] model
training), edge computing (e.g., for analysis), data stores, metadata catalogs, and high-speed networks. We review common patterns associated with such flows and describe methods for instantiating these patterns. 
We present experiences with the application of these methods to the processing of data from five different scientific instruments, each of which engages powerful computers for data inversion, model training, or other purposes. We also discuss implications of such methods for operators and users of scientific facilities. 
<a href="https://www.cell.com/patterns/fulltext/S2666-3899(22)00231-8">https://www.cell.com/patterns/fulltext/S2666-3899(22)00231-8</a>

Linking scientific instruments and computation: Patterns, technologies, and experiences

Understanding functional correlations between the activities of neuron populations are vital for the analysis of neuronal networks. Analyzing large-scale neuroimaging data obtained from hundreds of neurons simultaneously poses significant visualization challenges. We developed V-NeuroStack, a novel network visualization tool to visualize data obtained using calcium imaging of spontaneous activity of neurons in a mouse brain slice as well as in vivo using two-photon imaging. V-NeuroStack creates 3D time stacks by stacking 2D time frames for a time-series dataset. It provides a web interface to explore and analyze data using both 3D and 2D visualization techniques. Previous attempts to analyze such data have been limited by the tools available to visualize large numbers of correlated activity traces. V-NeuroStack&rsquo;s 3D view is used to explore patterns in dynamic large-scale correlations between neurons over time. The 2D view is used to examine any timestep of interest in greater detail. Furthermore, a dual-line graph provides the ability to explore the raw and first-derivative values of activity from an individual or a functional cluster of neurons. V-NeuroStack can scale to datasets with at least a few thousand temporal snapshots. It can potentially support future advancements in in vitro and in vivo data-capturing techniques to bring forth novel hypotheses by allowing unambiguous visualization of massive patterns in neuronal activity data. See <a href="https://onlinelibrary.wiley.com/doi/10.1002/jnr.25139">https://onlinelibrary.wiley.com/doi/10.1002/jnr.25139</a> 
Keywords: 3D visualization, calcium imaging, dynamic network, neural network, spatio-temporal data

V-NeuroStack application with a loaded sample dataset from brain slice data.

V-NeuroStack: Open-source 3D time stack software for identifying patterns in neuronal data

Hummingbird: is a modern, innovative performance merging live theater and interactive virtual reality by bringing a group of active participants into a shared space for a live performance. The performance premiered as part of Chicago&rsquo;s Tony Award-winning Goodman Theatre New Stages Festival showcasing innovative and ground-breaking theater works in December 2021. This project bridges art, science and live theater through a collaborative research effort between computer science and design faculty and students at the University of Illinois Chicago (UIC) Electronic Visualization Laboratory (EVL) and Chicago theater directors, actors, videographers and producers. Hummingbird&rsquo;s story celebrates courage and coming of age through the eyes of a gutsy teen who must outsmart her mother&rsquo's narcissistic boss and survive dangerous new technology in a live, immersive adventure. Hummingbird extends traditional live theater and makes virtual reality art accessible to a broader audience, demonstrating how virtual reality can transform theatrical storytelling. 
CCS CONCEPTS 
Applied computing - Arts and humanities; Performing arts; Human-centered computing - Collaborative and social computing; Collaborative and social computing theory, concepts and paradigms; Computer-supported cooperative work. 
KEYWORDS 
Virtual reality, Storytelling, Theater, Multi-user, Performance

&ldquo;Hummingbird&rdquo; being performed at EVL during pilot performances summer of 2021

Hummingbird: A Collaborative Live Theater and Virtual Reality Adventure

Probability distributions are omnipresent in data analysis. They are often used to model the natural uncertainty present in real phenomena or to describe the properties of a data set. Designing efficient visual metaphors to convey probability distributions is, however, a difficult problem. This fact is especially true for geographical data, where conveying the spatial context constrains the design space. While many different alternatives have been proposed to solve this problem, they focus on representing data variability. However, they are not designed to support spatial analytical tasks involving probability quantification. The present work aims to adapt recent non-spatial approaches to the geographical context, in order to support probability quantification tasks. We also present a user study that compares the efficiency of these approaches in terms of both accuracy and usability. 
Keywords: spatial visualization; uncertainty visualization

A choropleth map showing the number of taxi trips in each neighborhood of Midtown Manhattan (NYC) with each neighborhood having an associated monthly distribution of number of taxi trips, as highlighted for the regions of Midtown Center and Kips Bay.

A Comparative Study of Methods for the Visualization of Probability Distributions of Geographical Data

In today&rsquo;s Big Data era, data scientists require modern workflows to quickly analyze large-scale datasets using complex codes to maintain the rate of scientific progress. These scientists often rely on available campus resources or off-the-shelf
computational systems for their applications. Unified infrastructure or over-provisioned servers can quickly become bottlenecks for specific tasks, wasting time and resources. Composable infrastructure helps solve these problems by providing users with new ways to increase resource utilization. Composable infrastructure disaggregates a computer&rsquo;s components - CPU, GPU (accelerators), storage and networking - into fluid pools of resources, but typically relies upon infrastructure engineers to architect individual machines. Infrastructure is either managed with specialized command-line utilities, user interfaces or specification files. These management models are cumbersome and difficult to incorporate into data-science workflows. We developed a high-level software API, Composastructure, which, when integrated into modern workflows, can be used by infrastructure engineers as well as data scientists to reorganize composable resources on demand. Composastructure enables infrastructures to be programmable, secure, persistent and reproducible. Our API composes machines, frees resources, supports multi-rack operations, and includes a Python module for Jupyter Notebooks. 
Keywords: distributed systems, testbed implementation and deployment, composable infrastructure, deep learning, visualization, infrastructure as code

Moving from Composable to Programmable

Composable infrastructure holds the promise of accelerating the pace of academic research and discovery by enabling researchers to tailor the resources of a machine (e.g., GPUs, storage, NICs), on-demand, to address application needs. We were first introduced to composable infrastructure in 2018, and at the same time, there was growing demand among our College of Engineering faculty for GPU systems for data science, artificial intelligence / machine learning / deep learning, and visualization. Many purchased their own individual desktop or deskside systems, a few pursued more costly cloud and HPC solutions, and others looked to the College or campus computer center for GPU resources which, at the time, were scarce. After surveying the diverse needs of our faculty and studying product offerings by a few nascent startups in the composable infrastructure sector, we applied for and received a grant from the National Science Foundation in November 2019 to purchase a mid-scale system, configured to our specifications, for use by faculty and students for research and research training. 
This paper describes our composable infrastructure solution and implementation for our academic community. Given how modern workflows are progressively moving to containers and cloud frameworks (using Kubernetes) and to programming notebooks (primarily Jupyter), both for ease of use and for ensuring reproducible experiments, we initially adapted these tools for our system. We have since made it simpler to use our system, and now provide our users with a public facing JupyterHub server. We also added an expansion chassis to our system to enable composable co-location, which is a shared central architecture in which our researchers can insert and integrate specialized resources (GPUs, accelerators, networking cards, etc.) needed for their research. 
In February 2020, installation of our system was finalized and made operational and we began providing access to faculty in the College of Engineering. Now, two years later, it is used by over 40 faculty and students plus some external collaborators for research and research training. Their use cases and experiences are briefly described in this paper. Composable infrastructure has proven to be a useful computational system for workload variability, uneven applications, and modern workflows in academic environments. 
Keywords: composable infrastructure, deep learning, visualization, resource management, workload management, user workflow, composable co-location, infrastructure as code

Composable Infrastructures for an Academic Research Environment: Lessons Learned

The Electronic Visualization Laboratory at the University of Illinois Chicago acquired a 24 compute node, 64 GPU composable infrastructure compute cluster, COMPaaS DLV - Composable Platform as a Service: Instrument for Deep Learning & Visualization in 2019.  Since then 40+ users have designed and run computations on COMPaaS.  To track cluster utilization, evaluate hardware
performance, and simply analyze application resource utilization of this experimental composable infrastructure system we set out to
collect metrics on the hardware and software infrastructure.  Users deploy their applications on the cluster using Kubernetes, a container orchestration platform that enables users to deploy applications on the cluster without the need to have complete knowledge of the underlying hardware and with minimal assistance from system administrators.  Following this, we utilized tooling already adapted for Kubernetes-based clusters to collect hardware and software metrics.  The metrics provided by this tooling is useful for system administrators in evaluating cluster performance and utilization, as well as useful for users in determining their application&rsquo;s performance and discovering bottlenecks.

GPU Power Consumption

Monitoring COMPaaS

In today&rsquo;s Big Data era, data scientists require modern workflows to quickly analyze large-scale datasets using complex codes to maintain the rate of scientific progress. These scientists often rely on available campus resources or off-the-shelf computational systems for their applications. Unified infrastructure or over-provisioned servers can quickly become
bottlenecks for specific tasks, wasting time and resources. Composable infrastructure helps solve these problems by providing users with new ways to increase resource utilization. Composable infrastructure disaggregates a computer&rsquo;s components – CPU, GPU (accelerators), storage and networking - into fluid pools of resources, but typically relies upon infrastructure engineers to
architect individual machines. Infrastructure is either managed with specialized command-line utilities, user interfaces or specification files. These management models are cumbersome and difficult to incorporate into data-science workflows. We
developed a high-level software API, Composastructure, which, when integrated into modern workflows, can be used by infrastructure engineers as well as data scientists to reorganize composable resources on demand. Composastructure enables infrastructures to be programmable, secure, persistent and reproducible. Our API composes machines, frees resources, supports multi-rack operations, and includes a Python module for Jupyter Notebooks. 
Keywords - distributed systems, testbed implementation and deployment, composable infrastructure, deep learning, visualization, infrastructure as code

Composastructure System Architecture

Composable infrastructure holds the promise of accelerating the pace of academic research and discovery by enabling researchers to tailor the resources of a machine (e.g., GPUs, storage, NICs), on-demand, to address application needs. We were first introduced to composable infrastructure in 2018, and at the same time, there was growing demand among our College of Engineering faculty for GPU systems for data science, artificial intelligence / machine learning / deep learning, and visualization. Many purchased their own individual desktop or deskside systems, a few pursued more costly cloud and HPC solutions, and others looked to the College or campus computer center for GPU resources which, at the time, were scarce. After surveying the diverse needs of our faculty and studying product offerings by a few nascent startups in the composable infrastructure sector, we applied for and received a grant from the National Science Foundation in November 2019 to purchase a mid-scale system, configured to our specifications, for use by faculty and students for research and research training. 
This paper describes our composable infrastructure solution and implementation for our academic community. Given how modern workflows are progressively moving to containers and cloud frameworks (using Kubernetes) and to programming notebooks (primarily Jupyter), both for ease of use and for ensuring reproducible experiments, we initially adapted these tools for our system. We have since made it simpler to use our system, and now provide our users with a public facing JupyterHub server. We also added an expansion chassis to our system to enable composable co-location, which is a shared central architecture in which our researchers can insert and integrate specialized resources (GPUs, accelerators, networking cards, etc.) needed for their research. 
In February 2020, installation of our system was finalized and made operational and we began providing access to faculty in the College of Engineering. Now, two years later, it is used by over 40 faculty and students plus some external collaborators for research and research training. Their use cases and experiences are briefly described in this paper. Composable infrastructure has proven to be a useful computational system for workload variability, uneven applications, and modern workflows in academic environments. 
Keywords - composable infrastructure, deep learning, visualization, resource management, workload management, user workflow, composable co-location, infrastructure as code

COMPaaS Hardware Racks

The annual incidence of head and neck cancers (HNC) worldwide is more than 550,000 cases, with around 300,000 deaths each year. However, the incidence rates and disease-characteristics of HNC differ between treatment centers and different populations, due to undetermined reasons, which may or not include socioeconomic factors. The multi-faceted and multi-variate nature of the data in the context of the emerging field of health disparities research makes automated analysis impractical. Hence, we present a visual analysis approach to explore the health disparities in the data of HNC patients from two different cohorts at two cancer care centers. Our approach integrates data from multiple sources, including census data and city data, with custom visual encodings and with a nearest
neighbor approach. Our design, created in collaboration with oncology experts, makes it possible to analyze the patients’ demographic, disease characteristics, treatments and outcomes, and to make significant comparisons of these two cohorts and of individual patients. We evaluate this approach through two case studies performed with domain experts. The results demonstrate that this visual analysis approach successfully accomplishes the goal of comparing two cohorts in terms of different significant factors, and can provide insights into the main source of health disparities between the two centers. 
Index Terms: Biomedical and medical visualization - Design studies - Hypothesis forming

A tale of two head and neck cancer centers, University of Illinois and MD Anderson Cancer Center.

A Tale of Two Centers: Visual Exploration of Health Disparities in Cancer Care

Pose guided person image generation means to generate a photo-realistic person image conditioned on an input person image and a desired pose. This task requires spatial manipulation of the source image according to the target pose. However, convolutional neural networks (CNNs) are inherently limited to geometric transformations due to the fixed geometric structures in their building modules, i.e., convolution, pooling and unpooling, which cannot handle large motion and occlusions caused by large pose transform. This paper introduces a novel two-stream context-aware appearance transfer network to address these challenges. It is a three-stage architecture consisting of a source stream and a target stream. Each stage features an appearance transfer module, a multi-scale context module and two-stream feature fusion modules. The appearance transfer module handles large motion by finding the dense correspondence between the two-stream feature maps and then transferring the appearance information from the source stream to the target stream. The multi-scale context module handles occlusion via contextual modeling, which is achieved by atrous convolutions of different sampling rates. Both quantitative and qualitative results indicate the proposed network can effectively handle challenging cases of large pose transform while retaining the appearance details. Compared with state-of-the-art approaches, it achieves comparable or superior performance using much fewer parameters while being significantly faster. 
Keywords: Person image generation, Appearance transfer, Multi-scale context, Efficient image generation 
Relevant Funding: The COMPaaS DLV project (NSF award CNS-1828265) 
<a href="https://doi.org/10.1016/j.patcog.2021.108451">https://doi.org/10.1016/j.patcog.2021.108451</a>

Exploiting appearance transfer and multi-scale context for efficient person image generation

While designing sustainable and resilient urban built environment is increasingly promoted around the world, significant data gaps have made research on pressing sustainability issues challenging to carry out. Pavements are known to have strong economic and environmental impacts; however, most cities lack a spatial catalog of their surfaces due to the cost-prohibitive and time-consuming nature of data collection. Recent advancements in computer vision, together with the availability of street-level images, provide new opportunities for cities to extract large-scale built environment data with lower implementation costs and higher accuracy. In this paper, we propose CitySurfaces, an active learning-based framework that leverages computer vision techniques for classifying sidewalk materials using widely available street-level images. We trained the framework on images from New York City and Boston and the evaluation results show a 90.5% mIoU score. Furthermore, we evaluated the framework using images from six different cities, demonstrating that it can be applied to regions with distinct urban fabrics, even outside the domain of the training data. CitySurfaces can provide researchers and city agencies with a low-cost, accurate, and extensible method to collect sidewalk material data which plays a critical role in addressing major sustainability issues, including climate change and surface water management. 
Keywords: Sustainable built environment; Surface materials; Urban heat island; Semantic segmentation; Sidewalk assessment; Urban analytics; Computer vision 
ISSN 2210-6707 
<a href="https://www.sciencedirect.com/science/article/pii/S2210670721008933">https://www.sciencedirect.com/science/article/pii/S2210670721008933</a>

Predictions of the model on the held-out test set. Fine details and boundaries of objects like poles, plants, wooden sticks, and fire hydrants are very precisely predicted.

CitySurfaces: City-scale semantic segmentation of sidewalk materials

In the last 20 years, extreme weather-related events like floods, landslides, droughts, and wildfires have caused the death of 1.23 million people and a loss of 2.97 trillion dollars. Studies show that low and lower-middle-income countries are the most impacted ones given the lack of investment in disaster risk management. To reduce the impact of these events, weather researchers have been developing numerical weather models that inform public agencies about the impending extreme events in advance. Despite being powerful tools, these models can suffer from several sources of uncertainty, ranging from the approximation of micro-scale physical processes to the location-dependent calibration of parameters, which is especially critical in developing countries. To minimize uncertainty effects, researchers generate several different weather scenarios to compose an ensemble of simulations that typically are inspected using manual, laborious, and error-prone approaches. In this paper, we propose an interactive visual analytics system, called X-Weather, developed in close collaboration with weather researchers from Brazil. Our system contributes a set of statistics and probability-based visualizations that allows the assessment of extreme weather events by effortlessly navigating through and comparing ensemble members. We demonstrate the effectiveness of the system through two case studies analyzing tragic events that happened in the mountain region of Rio de Janeiro in Brazil. 
Keywords: Visual analytics, Weather visualization, Ensemble visualization 
ISSN 0097-8493 
<a href="https://www.sciencedirect.com/science/article/pii/S0097849322000073">https://www.sciencedirect.com/science/article/pii/S0097849322000073</a>

The X-Weather interface.

Visualizing simulation ensembles of extreme weather events

Although cancer patients survive years after oncologic therapy, they are plagued with long-lasting or permanent residual symptoms, whose severity, rate of development, and resolution after treatment vary largely between survivors. The analysis and interpretation of symptoms is complicated by their partial co-occurrence, variability across populations and across time, and, in the case of cancers that use radiotherapy, by further symptom dependency on the tumor location and prescribed treatment. We describe THALIS, an environment for visual analysis and knowledge discovery from cancer therapy symptom data, developed in close collaboration with oncology experts. Our approach leverages unsupervised machine learning methodology over cohorts of patients, and, in conjunction with custom visual encodings and interactions, provides context for new patients based on patients with similar diagnostic features and symptom evolution. We evaluate this approach on data collected from a cohort of head and neck cancer patients. Feedback from our clinician collaborators indicates that THALIS supports knowledge discovery beyond the limits of machines or humans alone, and that it serves as a valuable tool in both the clinic and symptom research. 
keywords: Cancer, Visualization, Medical Treatment, Data Visualization, Encoding, Principal Component Analysis, Neck, Temporal Data, Application Motivated Visualization, Life Sciences, Mixed Initiative Human Machine Analysis

Acute vs. late phase analysis via THALIS.

THALIS: Human-Machine Analysis of Longitudinal Symptoms in Cancer Therapy

SAGE2 (Scalable Amplified Group Environment) is the de-facto platform to support group work on wide-band display environments. Unlike most web applications, the SAGE environment, due to the nature of its collaborative model, needs a nuanced handling of security aspects. This paper details the security requirements of SAGE2, the Identity and Access Control model that was developed to address those requirements, and the details of the Identity and Access Management system that the SAGE team implemented based on this new model. Further, we present a comparison of this new system with some of the popular collaboration platforms to highlight the uniqueness of SAGE2 integrated with this new Identity and Access Management system. 
Keywords: security, access control, collaborative environment, wide-band displays

Examples of SAGE2 wide-band displays

Securing Collaborative Work in Wide-band Display Environments

Figures within biomedical articles present essential evidence of the relevance of a publication in a curation workflow. In particular, visual cues of the image modality or experimental methods can help expert curators identify relevant papers from an increasing number of publications. Automating the identification of these content-bearing images can thus be helpful in computer-assisted curation. However, the paucity of labeled datasets and the specialized training required to label such images hinder the development of such tools. To address this problem, we present the design of ANIMO, a labeling system that integrates extraction and segmentation tools to ease the annotation burden. We first introduce two taxonomies of image modalities and experimental methods, derived in collaboration with curators. On the back-end of the system, we process batches of documents and create a labeling task per document. At the front-end, expert curators can access these tasks through a web interface and access the article of interest. We describe the evaluation of this system by a group of biocurators, and the human factor lessons learned from this interdisciplinary experience. 
doi: 10.1109/BIBM52615.2021.9669898

Front-end interface for labeling a document showing a taxonomy adapted for the CORD-19 dataset.

ANIMO: Annotation of Biomed Image Modalities

The spread of the SARS-CoV-2 virus and its contagious disease COVID-19 has impacted countries to an extent not seen since the
1918 flu pandemic. In the absence of an effective vaccine and as cases surge worldwide, governments were forced to adopt measures to inhibit the spread of the disease. To reduce its impact and to guide policy planning and resource allocation, researchers have been developing models to forecast the infectious disease. Ensemble models, by aggregating forecasts from multiple individual models, have been shown to be a useful forecasting method. However, these models can still provide less-than-adequate forecasts at higher spatial resolutions. In this paper, we built COVID-19 EnsembleVis, a web-based interactive visual interface that allows the assessment of the errors of ensembles and individual models by enabling users to effortlessly navigate through and compare the outputs of models considering their space and time dimensions. COVID-19 EnsembleVis enables a more detailed understanding of uncertainty and the range of forecasts generated by individual models. 
Index Terms: Human-centered computing; Visualization; Visualization application domains; Visual analytics

COVID-19 EnsembleVis: Visual Analysis of County-level Ensemble Forecast Models

There is a steadily growing interest in leveraging ecosystems of digital devices that go beyond a single desktop for collaborative visual data analysis and exploration. This new thrust of multi-device interfaces supports new models for complex collaboration scenarios, and have great potential to support analysts in their data analysis by utilizing each device’s capabilities. However, there are some challenges inherently associated with visual data analysis in multi-device environments (MDE). This dissertation investigates how the analytical process occurs in multi-user multi-device environments to provide a theoretical understanding of collaborative exploratory visual data analysis and better inform the design of visualization tools. First, I touched on the challenges of designing cross-device visualization tools by introducing the design and implementation of a multi-device system for collaborative visual data analysis that enables cross-device visualization sharing and simultaneous interaction. Then, through an exploratory user study, I evaluated strategies of exploratory visual data analysis in a collaborative multi-user multi-device environment. I synthesized a two-level characterization of the analysis structure from observed analysis behaviors. I observed that subjects navigate the data space in three identified exploration patterns and the analysis was primarily depth-oriented. In addition, the cost of deciding what to explore next &ldquo;Gulf of Goal Formation&rdquo; is higher in collaborative settings due to short-term memory and the recency effect. I hypothesized that visualizing the dimensions search space would increase the breadth of the analysis and reduce the decision cost. Using a between-groups study, I evaluated the effect of revealing information about what dimension&rsquo;s data space coverage(s) were investigated and what were left. The results indicate that visualizing dimensions search space increases the breadth of the analysis and reduces the decision cost by positively affecting the rate of views generation.

This research investigates the design of visualization tools for collaborative exploratory visual data analysis in multi-device environments.

Supporting Collaborative Exploratory Visual Data Analysis in Multi-device Environments

Data exploration stands to benefit from environments that permit users to examine and juxtapose many views of data, particularly views that present diverse selections of data values and attributes. Large, high-resolution environments are capable of showing many related views of data, but efficiently creating and displaying visualizations in these environments presents significant challenges. In this dissertation, I will present my research on &ldquo;multi-view data exploration interactions&rdquo; that enable users to create sets of views with coherent data value and attribute variations, through multi-modal speech and mid-air pointing gestures in large display
environments. This work enables users to rapidly and efficiently generate sets of views in support of multi-view data exploration tasks, organize these views in coherent collections, and operate on sets of views collectively, rather than individually, to efficiently reach large portions of the &lsquo;data and attribute space&rsquo;. I will present three contributions: 1) an observational study of data exploration in a large display environment with speech and mid-air gestures, 2) &lsquo;Traverse&rsquo;, an interaction technique for data exploration, based on this study, which uses natural language to create and pivot sets of views, and 3) &lsquo;Ditto&rsquo;, a multi-modal speech and mid-air pointing gesture interactive environment, which utilizes the multi-view data exploration technique, in
large display environments.

Study setting with the remote mediator in an adjacent room generating views using Tableau and dropping them to the user&rsquo;s display environment using SAGE2.

Interaction with Multiple Data Visualizations Through Natural Language Commands

Aircraft engines emit particulates that alter the chemical composition of the atmosphere and perturb the Earth’s radiation budget by creating additional ice clouds in the form of condensation trails called contrails. We propose a multi-scale visual computing system that will assist in defining contrail features with parameter analysis for computer-generated aircraft engine simulations. These simulations are computationally intensive and rely on high-performance computing (HPC) solutions. Our multi-linked visual system seeks to help in the identification of the formation and evolution of contrails and in the identification of contrail-related spatial features from the simulation workflow.

Visual interface of aircraft engine simulation analysis.

Parameter Analysis and Contrail Detection of Aircraft Engine Simulations

The graph convolutional network (GCN) has recently achieved promising performance of 3D human pose estimation (HPE) by modeling the relationship among body parts. However, most prior GCN approaches suffer from two main drawbacks. First, they share a feature transformation for each node within a graph convolution layer. This prevents them from learning different relations between different body joints. Second, the graph is usually defined according to the human skeleton and is suboptimal because human activities often exhibit motion patterns beyond the natural connections of body joints. To address these limitations, we introduce a novel Modulated GCN for 3D HPE. It consists of two main components: weight modulation and affinity modulation. Weight modulation learns different modulation vectors for different nodes so that the feature transformations of different nodes are disentangled while retaining a small model size. Affinity modulation adjusts the graph structure in a GCN so that it can model additional edges beyond the human skeleton. We investigate several affinity modulation methods as well as the impact of regularizations. Rigorous ablation study indicates both types of modulation improve performance with negligible overhead. Compared with state-of-the-art GCNs for 3D HPE, our approach either significantly reduces the estimation errors, e.g., by around 10%, while retaining a small model size or drastically reduces the model size, e.g., from 4.22M to 0.29M (a 14.5 X reduction), while achieving comparable performance. Results on two benchmarks show our Modulated GCN outperforms some recent states of the art. 
<a href="https://github.com/ZhimingZo/Modulated-GCN">Download Code</a> 
Funding: The COMPaaS DLV project (NSF award CNS-1828265)

Modulated Graph Convolutional Network for 3D Human Pose Estimation

The latest developments of machine learning (ML) and deep learning (DL) algorithms have paved the way to effectively analyze the atomic structure of chemically-complex materials. In this work, we present a DL model built upon a fully convolutional neural network (FCN) to resolve the random elements distribution of complex PtNiPdCoFe high-entropy alloys (HEAs) represented in the scanning transmission electron microscopy (STEM) images at atomic resolution. The objective of the proposed neural network is to learn through semantic seg-mentation the non-linear correlations between the pixels&rsquo; intensities of STEM images and the number of atoms of different constituent elements in the atomic columns (i.e., column heights) in the HEA&rsquo;s structure. We demonstrate that our DL model is capable of correctly estimating the column heights or with an error up to 1 atom for the majority of the columns in the HEA structures represented in the simulated STEM images used to train and test the network. This establishes a sufficiently high level of confidence in the estimation of the element distribution in experimental images. The predicted distributions in different STEM images of nanoparticles reveal inhomogeneous fluctuations with local aggregations in the elemental atomic fractions within the columns. The most pronounced aggregation is displayed by Pt, which is the largest and most electronegative element in the synthesized HEA material. The proposed DL method is beneficial for an in-depth characterization of the structural properties of HEAs and multielement 3D materials in general. 
Keywords: Deep learning, High entropy alloys, STEM images, Column heights, Elements distribution

Example of atomic model of HEA characterized by a nearly equiatomic composition of Pt, Ni, Pd, Co and Fe.

Deep learning for mapping element distribution of high-entropy alloys in scanning transmission electron microscopy images

Support vector regression (SVR) is particularly beneficial when the outcome and predictors are nonlinearly related. However, when many covariates are available, the method&rsquo;s flexibility can lead to overfitting and an overall loss in predictive accuracy. To overcome this drawback, we develop a feature selection method for SVR based on a genetic algorithm that iteratively searches across potential subsets of covariates to find those that yield the best performance according to a user-defined fitness function. We evaluate the performance of our feature selection method for SVR, comparing it to alternate methods including LASSO and random forest, in a simulation study. We find that our method yields higher predictive accuracy than SVR without feature selection. Our method outperforms LASSO when the relationship between covariates and outcome is nonlinear. Random forest performs equivalently to our method in some scenarios, but more poorly when covariates are correlated. We apply our method to predict donor kidney function 1 year after transplant using data from the United Network for Organ Sharing national registry.

Results of simulation 1 setting with 300 total covariates X, 15 of them truly associated with the outcome Y.

Feature Selection for Support Vector Regression Using a Genetic Algorithm

Exploring large virtual environments, such as cities, is a central task in several domains, such as gaming and urban planning. VR systems can greatly help this task by providing an immersive experience; however, a common issue with viewing and navigating a city in the traditional sense is that users can either obtain a local or a global view, but not both at the same time, requiring them to continuously switch between perspectives, losing context and distracting them from their analysis. In this paper, our goal is to allow users to navigate to points of interest without changing perspectives. To accomplish this, we design an intuitive navigation interface that takes advantage of the strong sense of spatial presence provided by VR. We supplement this interface with a perspective that warps the environment, called UrbanRama, based on a cylindrical projection, providing a mix of local and global views. The design of this interface was performed as an interactive process in collaboration with architects and urban planners. We conducted a qualitative and a quantitative pilot user study to evaluate UrbanRama and the results indicate the effectiveness of our system in reducing perspective changes, while ensuring that the warping doesn&rsquo;t affect distance and orientation perception. 
Index Terms: Virtual Reality, VR Navigation, Cylindrical deformation. 
<a href="https://youtu.be/M8BFZnxq-Qg">Watch the video on YouTube.</a>

The UrbanRama mode in our urban VR system deforms the city allowing the user to obtain both a local as well as a global perspective of the urban environment.

UrbanRama: Navigating Cities in Virtual Reality

Motivation: Biomedical research findings are typically disseminated through publications. To simplify access to domain-specific knowledge while supporting the research community, several biomedical databases devote significant effort to manual curation of the literature - a labor intensive process. The first step toward biocuration requires identifying articles relevant to the specific area on which the database focuses. Thus, automatically identifying publications relevant to a specific topic within a large volume of publications is an important task toward expediting the biocuration process and, in turn, biomedical research. Current methods focus on textual contents, typically extracted from the title-and-abstract. Notably, images and captions are often used in publications to convey pivotal evidence about processes, experiments and results. 
Results: We present a new document classification scheme, using both image and caption information, in addition to titles-and-abstracts. To use the image information, we introduce a new image representation, namely Figure-word, based on class labels of subfigures. We use word embeddings for representing captions and titles-and-abstracts. To utilize all three types of information, we introduce two information integration methods. The first combines Figure-words and textual features obtained from captions and titles-and-abstracts into a single larger vector for document representation; the second employs a meta-classification scheme. Our experiments and results demonstrate the usefulness of the newly proposed Figure-words for representing images. Moreover, the results showcase the value of Figure-words, captions and titles-and-abstracts in providing complementary information for document classification; these three sources of information when combined, lead to an overall improved classification performance. 
Availability and implementation: Source code and the list of PMIDs of the publications in our datasets are available upon request. 
<a href="
https://doi.org/10.1093/bioinformatics/btab331">https://doi.org/10.1093/bioinformatics/btab331</a>

The process for converting figures into their corresponding Figure-words. The set of figures extracted from the biomedical document (left). The corresponding Figure-word with their vector representations indicating the types of comprising panel (right).

Utilizing Image and Caption Information for Biomedical Document Classification

Collaborative Virtual Environments have been increasingly used in supporting group work. Several collaborative systems have come up in recent years as a way of meeting the ever changing needs of group work, with asymmetric setups being the latest addition to a diverse collection of systems. In recent years, we see attempts at multi-domain collaboration. Research on interdisciplinary collaboration reveals a need for supporting a hybrid of tools and representations that are familiar to users of different domains. 
In this dissertation, I present a prototype implementation of a Hybrid Collaborative Virtual Environment that combines multiple representations to facilitate multi-domain collaboration. Preliminary evaluation of the prototype showed that hybridization of different representations can lead to role formation due to dependency on representations, when users are limited to interacting with a single representation. Integrating shared views and shared interaction into this Hybrid Collaborative Virtual Environment with the help of a large shared display environment can help solve the representational dependency issue. Shared views and shared interactions
enable users to freely interact with any of the different representations and also eases communication amongst them. I evaluate this integration by means of a formal study having a within group design: A group of three participants perform collaborative tasks using the Hybrid Collaborative Virtual Environment with and without the integration of shared views and shared interactions. I discuss the results of the study and show how shared views and shared interaction improves the collaborative experience.

Hybrid Collaborative Virtual Environments

Patient-Reported Outcome (PRO) surveys are used to monitor patients&rsquo; symptoms during and after cancer treatment. Late symptoms refer to those experienced after treatment. While most patients experience severe symptoms during treatment, these usually subside in the late stage. However, for some patients, late toxicities persist negatively affecting the patient&rsquo;s quality of life (QoL). In the case of head and neck cancer patients, PRO surveys are recorded every week during the patient&rsquo;s visit to the clinic and at different follow-up times after the treatment has concluded. In this paper, we model the PRO data as a time-series and apply Long-Short Term Memory (LSTM) neural networks for predicting symptom severity in the late stage. The PRO data used in this project corresponds to MD Anderson Symptom Inventory (MDASI) questionnaires collected from head and neck cancer patients treated at the MD Anderson Cancer Center. We show that the LSTM model is effective in predicting symptom ratings under the RMSE and NRMSE metrics. Our experiments show that the LSTM model also outperforms other machine learning models and time-series prediction models for these data. 
CCS Concepts: Computing methodologies; Neural networks 
<stong>Keywords: Long Short-Term Memory (LSTM), Patient Reported Outcomes (PRO), Late Toxicity, Symptom Severity Prediction

Validation RMSE (a) and NRMSE (b) scores for the prediction of individual symptoms 6Wk and 12Mo after treatment.

Predicting late symptoms of head and neck cancer treatment using LSTM and patient reported outcomes

To improve risk prediction for oropharyngeal cancer (OPC) patients using cluster analysis on the radiomic features extracted from pre‑treatment Computed Tomography (CT) scans. 553 OPC Patients randomly split into training (80%) and validation (20%), were classified into 2 or 3 risk groups by applying hierarchical clustering over the co‑occurrence matrix obtained from a random survival forest (RSF) trained over 301 radiomic features. The cluster label was included together with other clinical data to train an ensemble model using five predictive models (Cox, random forest, RSF, logistic regression, and logistic‑elastic net). Ensemble performance was evaluated over the independent test set for both recurrence-free survival (RFS) and overall survival (OS). The Kaplan–Meier curves for OS stratified by cluster label show significant differences for both training and testing (p val < 0.0001). When compared to the models trained using clinical data only, the inclusion of the cluster label improves AUC test performance from .62 to .79 and from .66 to .80 for OS and RFS, respectively. The extraction of a single feature, namely a cluster label, to represent the high‑dimensional radiomic feature space reduces the dimensionality and sparsity of the data. Moreover, inclusion of the cluster label improves model performance compared to clinical data only and offers comparable performance to the models including raw radiomic features.

Publications

Curio: A Dataflow-Based Framework for Collaborative Urban Visual Analytics

Learning Anomalies with Normality Prior for Unsupervised Video Anomaly Detection

Analysis-by-Synthesis Transformer for Single-View 3D Reconstruction

BI-LAVA: Biocuration with Hierarchical Image Labeling through Active Learning and Visual Analytics

Assessing the Landscape of Toolkits, Frameworks, and Authoring Tools for Urban Visual Analytics Systems

Electronic Visualization Laboratory’s 50th Anniversary Retrospective: Look to the Future, Build on the Past

Contact