A Bounded Measure for Estimating the Benefit of Visualization (Part II): Case Studies and Empirical Evaluation
<p>The London underground map (<b>right</b>) is a deformed map. In comparison with a relatively more faithful map <b>(left</b>), there is a significant amount of information loss due to many-to-one mappings in the deformed map, which omits some detailed variations among different connection routes between pairs of stations (e.g., distance and geometry). One common rationale is that the deformed map was designed for certain visualization tasks, which likely excluded the task for estimating the walking time between a pair of stations indicated by a pair of red or blue arrows. In one of our experiments, when asked to perform such tasks using the deformed map, some participants with little knowledge about London or London Underground performed these tasks well. Can information theory explain this phenomenon? Can we quantitatively measure relevant factors in this visualization process?</p> "> Figure 2
<p>Major temperate scales proposed in history. Different lines show instances used as observation points, some of which became major reference points. Note: “Celsius* 1742” indicates the original scale proposed by Anders Celsius, while “Celsius 1743” indicates the revised Celsius scale used today that was proposed by Jean-Pierre Christin. The Newton scale is not linearly related to the others (shown as dash lines).</p> "> Figure 3
<p>Three alphabets illustrate possible metro maps (letters) in different grid resolutions. Increasing the resolution enables the depiction of more reality, while reducing the resolution compels more abstraction.</p> "> Figure 4
<p>An example scenario with two states <span class="html-italic">good</span> and <span class="html-italic">bad</span> has a ground truth PMF <math display="inline"><semantics> <mrow> <mi>Q</mi> <mo>=</mo> <mo>{</mo> <mn>0.8</mn> <mo>,</mo> <mn>0.2</mn> <mo>}</mo> </mrow> </semantics></math>. From the output of a biased process that always informs users that the situation is <span class="html-italic">bad</span>. Five users, LD, DF, RG, UC, and OC, have different knowledge and thus different divergence. The five candidate measures return different values of divergence. We would like to see which sets of values are more intuitive. The illustration on the top-right shows two transformations of the alphabets and their PMFs, one by the misleading communication and the other by the reconstruction. The bar chart shows the divergence values calculated by each candidate measure, while the four parallel coordinate plots (PCPs) show the values of <math display="inline"><semantics> <mrow> <msub> <mi mathvariant="script">H</mi> <mo movablelimits="true" form="prefix">max</mo> </msub> <mi mathvariant="script">D</mi> </mrow> </semantics></math> (divergence scaled by <math display="inline"><semantics> <mrow> <mi>S</mi> <msub> <mi>E</mi> <mo movablelimits="true" form="prefix">max</mo> </msub> </mrow> </semantics></math>), benefit, <math display="inline"><semantics> <msub> <mi>K</mi> <mi>υ</mi> </msub> </semantics></math> (impact of knowledge against relying solely on visual information), and <math display="inline"><semantics> <msub> <mi>K</mi> <mi>ψ</mi> </msub> </semantics></math> (against random guess).</p> "> Figure 5
<p>An example scenario with four data values: A, B, C, and D. Two processes (one correct and one biased) aggregated them to two values AB and CD. Users CG, CU, CB attempt to reconstruct [A, B, C, D] from the output [AB, CD] of the correct process, while BG, BS, and BM attempt to do so with the output from the biased processes. The bar chart shows the divergence values of the six users computed using the five candidate measures. The illustration on the right shows two transformations of the alphabets and their PMFs, one by the correct or biased process (pr.) and the other by the reconstruction. The bar chart shows the divergence values calculated by each candidate measure, while the four PCPs show the values of <math display="inline"><semantics> <mrow> <msub> <mi mathvariant="script">H</mi> <mo movablelimits="true" form="prefix">max</mo> </msub> <mi mathvariant="script">D</mi> </mrow> </semantics></math> (i.e., divergence scaled by <math display="inline"><semantics> <mrow> <mi>S</mi> <msub> <mi>E</mi> <mo movablelimits="true" form="prefix">max</mo> </msub> </mrow> </semantics></math>), benefit, <math display="inline"><semantics> <msub> <mi>K</mi> <mi>υ</mi> </msub> </semantics></math> and <math display="inline"><semantics> <msub> <mi>K</mi> <mi>ψ</mi> </msub> </semantics></math>. The values for Cvi and Bvi correspond to <math display="inline"><semantics> <msub> <mi>R</mi> <mi>correct</mi> </msub> </semantics></math> and <math display="inline"><semantics> <msub> <mi>R</mi> <mi>biased</mi> </msub> </semantics></math>, respectively.</p> "> Figure 6
<p>A volume dataset was rendered using the maximum intensity projection (MIP) method, which causes curved surfaces of arteries to appear rather flat. Posing a question about a “flat area” in the image can be used to tease out a viewer’s knowledge that is useful in a visualization process. This example was first described in Part I of this two-part paper [<a href="#B4-entropy-24-00282" class="html-bibr">4</a>] for demonstrating the role of human knowledge in dealing with information loss due to many-to-one mappings in such a visualization image. Similar to <a href="#entropy-24-00282-f003" class="html-fig">Figure 3</a> (<a href="#sec3-entropy-24-00282" class="html-sec">Section 3</a>) in this part, the example was used in Part I to illustrate the difficulty to interpret the unboundedness of the KL-divergence when considering a binary alphabet <math display="inline"><semantics> <mrow> <mi mathvariant="double-struck">A</mi> <mo>=</mo> <mo>{</mo> <mi mathvariant="italic">curved</mi> <mo>,</mo> <mi mathvariant="italic">flat</mi> <mo>}</mo> </mrow> </semantics></math> with maximum entropy of 1 bit.</p> "> Figure 7
<p>For the survey question shown in <a href="#entropy-24-00282-f006" class="html-fig">Figure 6</a>, our survey of 10 participants returned 8 answers for A, 1 for B, 0 for C, and 1 for D. Among them, more knowledgeable participants (referred to as experts) returned 3 answers for A and 1 for B, and none for C or D. We consider two possible ground truth PMFs. <math display="inline"><semantics> <mrow> <msub> <mi>Q</mi> <mn>1</mn> </msub> <mo>=</mo> <mrow> <mo>{</mo> <mn>0.1</mn> <mo>,</mo> <mn>0.878</mn> <mo>,</mo> <mn>0.002</mn> <mo>,</mo> <mn>0.02</mn> <mo>}</mo> </mrow> </mrow> </semantics></math> is based on our observations of photographs of arteries, and <math display="inline"><semantics> <mrow> <msub> <mi>Q</mi> <mn>2</mn> </msub> <mo>=</mo> <mrow> <mo>{</mo> <mn>0.75</mn> <mo>,</mo> <mn>0.25</mn> <mo>,</mo> <mn>0.0</mn> <mo>,</mo> <mn>0.0</mn> <mo>}</mo> </mrow> </mrow> </semantics></math> is based on the experts’ survey results. The top four PCPs show the values of <math display="inline"><semantics> <mrow> <msub> <mi mathvariant="script">H</mi> <mo movablelimits="true" form="prefix">max</mo> </msub> <mi mathvariant="script">D</mi> </mrow> </semantics></math>, benefit, <math display="inline"><semantics> <msub> <mi>K</mi> <mi>υ</mi> </msub> </semantics></math>, and <math display="inline"><semantics> <msub> <mi>K</mi> <mi>ψ</mi> </msub> </semantics></math> calculated based on <math display="inline"><semantics> <msub> <mi>Q</mi> <mn>1</mn> </msub> </semantics></math>, while the bottom four PCPs are measured based on <math display="inline"><semantics> <msub> <mi>Q</mi> <mn>2</mn> </msub> </semantics></math>. In addition, we also consider five other groups that make a random guess or always answer A, B, C, or D.</p> "> Figure A1
<p>A visual analytics workflow features a general trend of alphabet compression from left (World) to right (Tasks). The potential distortion compares at an information space reconstructed based on the output with the original input information space. When we place different processes (i.e., (<b>a<sub>1</sub></b>,<b>a<sub>2</sub></b>,<b>b</b>–<b>d</b>)), in the workflow, we can appreciate that statistics, algorithms, visualization, and interaction have different levels of alphabet compression, potential distortion, and cost.</p> "> Figure A2
<p>Estimating the benefit of visualization and knowledge impact in relation to the survey result of Question 5 (<a href="#entropy-24-00282-f006" class="html-fig">Figure 6</a>).</p> "> Figure A3
<p>A survey for collecting data that reflects the use of some knowledge in viewing two types of London underground maps.</p> "> Figure A4
<p>London underground survey: question sheet 1 (out of 3).</p> "> Figure A5
<p>London underground survey: question sheet 2 (out of 3).</p> "> Figure A6
<p>London underground survey: question sheet 3 (out of 3).</p> "> Figure A7
<p>The average time used by surveyees for answering each of the 12 questions. The data does not indicate any significant advantage of using the geographically-deformed map.</p> "> Figure A8
<p>The original table of numerical values for the text in the main paper.</p> "> Figure A9
<p>The PCPs of the data in <a href="#entropy-24-00282-f0A8" class="html-fig">Figure A8</a>.</p> ">
Abstract
:1. Introduction
- Reviewed the related work that prepared for this cost–benefit measure, provided the measure with empirical evidence, and featured the application of the measure.
- Identified a shortcoming of using the Kullback–Leibler divergence (KL-divergence) in the cost–benefit measure and demonstrated the shortcoming using practical examples.
- Presented a theoretical discourse to justify the use of a bounded measure for finite alphabets.
- Proposed a new bounded divergence measure, while studying existing bounded divergence measures.
- Analyzed nine candidate measures using seven criteria reflecting desirable conceptual or mathematical properties, and narrowed the nine candidate measures to six measures.
- We report several case studies for collecting practical instances to evaluate the remaining candidate measures.
- We demonstrate the uses of the cost–benefit measurement to estimate the benefit of visualization in practical scenarios and the human knowledge used in the visualization processes.
- We report the discovery of a new conceptual criterion that a divergence measure is a summation of the entropic values of its components, which is useful in analyzing and visualizing empirical data.
- Finally, we bring the multi-criteria decision analysis (MCDA) in Parts I and II together and offer a recommendation to revise the information-theoretic measures proposed by Chen and Golan [3].
2. Related Work
2.1. Measurement Science
2.2. Metrics Development in Visualization
2.3. Measurement in Empirica Experiments
3. Overview, Notations, and Problem Statement
3.1. Brief Overview
3.2. Mathematical Notations
- The positions of the two stations are fixed on each grid and there is only one path between the red station and the blue station.
- As shown on the top-right of Figure 3, only horizontal, and diagonal path-lines are allowed.
- When one path-line joins another, it can rotate by up to .
- All joints of path-lines can only be placed on grid points.
3.3. Problem Statement
4. Evaluation Methodology and Criteria
- (a)
- P is close to a uniform PMF, while the ground truth Q is dissimilar to a uniform PMF—This suggests that the users may not have adequate knowledge and may have been making random guesses. In such a case, their task performance would lead to a PMF similar to .
- (b)
- P is close to a PMFthat characterizes the available visual information while the ground truth Q differs fromnoticeably—This suggests that the users may not have adequate knowledge and may have been reasoning about the options in entirely based on what is depicted visually. In such a case, their performance would result in a PMF similar to .
- (c)
- P is close to the ground truth Q, while Q differs fromandnoticeably—This suggests that the users may have been able to make the perfect combination of the available visual information and their knowledge. In such a case, their task performance could lead to a PMF similar to the ideal PMF Q.
5. Synthetic Case Studies
5.1. Synthetic Case S1
- LD—The user has a little doubt about the output of the process, and decides the letter of bad 90% of the time, and the letter of good 10% of the time, i.e., with PMF .
- FD—The user has a fair amount of doubt, with .
- RG—The user makes a random guess, with .
- UC—The user has adequate knowledge about but under-compensates it slightly, with .
- OC—The user has adequate knowledge about but over-compensates it slightly, with .
5.2. Synthetic Case S
- CG makes random guess, .
- CU has useful knowledge, .
- CB is highly biased, .
- BG makes guess based on , .
- BS makes a small adjustment, .
- BM makes a major adjustment, .
5.3. An Extra Conceptual Criterion
6. Experimental Case Studies
6.1. Volume Visualization (Criterion R)
6.2. London Underground Map (Criterion R)
Benefit for: | ||||||
spot on | 0.287 | |||||
close | 0.033 | |||||
wild guess |
7. Conclusions
Supplementary Materials
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
Abbreviations
BG | Biased process in conjunction with faithful Guess |
BM | Biased process in conjunction with a Major adjustment in decision |
BS | Biased process in conjunction with a Small adjustment in decision |
CB | Correct process in conjunction with Biased reasoning |
CG | Correct process in conjunction with Random guess |
CU | Correct process in conjunction with Useful knowledge |
FD | a Fair amount of Doubt |
KL | Kullback–Leibler |
LD | Little Doubt |
MCDA | Multi-Criteria Decision Analysis |
ML | Machine Learning |
PCP | Parallel Coordinates Plot |
PMF | Probability Mass Function |
OC | Over-Compensate |
RG | Random Guess |
UC | Under-Compensate |
Appendix A. Explanation of the Original Cost-Benefit Measure
Appendix A.1. An Information-Theoretic Measure for Cost-Benefit Analysis
- Alphabet Compression (AC) measures the amount of entropy reduction (or information loss) achieved by a process. As it was noticed in [3], most visual analytics processes (e.g., statistical aggregation, sorting, clustering, visual mapping, and interaction) feature many-to-one mappings from input to output, hence losing information. Although information loss is commonly regarded as harmful, it cannot be all bad if it is a general trend of VA workflows. Thus, the cost–benefit ratio makes AC a positive component.
- Potential Distortion (PD) balances the positive nature of AC by measuring the errors typically due to information loss. Instead of measuring mapping errors using some third party metrics, PD measures the potential distortion when one reconstructs inputs from outputs. The measurement takes into account humans’ knowledge that can be used to improve the reconstruction processes. For example, given an average mark of 62%, the teacher who taught the class can normally guess the distribution of the marks among the students better than an arbitrary person.
- Cost (Ct) of the forward transformation from input to output and the inverse transformation of reconstruction provides a further balancing factor in the cost–benefit ratio in addition to the trade-off between AC and PD. In practice, one may measure the cost using time or a monetary measurement.
Appendix A.2. An Information-Theoretic Reasoning about Why Visualization Is Useful
Appendix B. How Tasks and Users Are Featured in the Cost-Benefit Ratio?
- The appropriateness depends on many attributes of a task, such as the selection of variables in the data and their encoded visual resolution required to complete a task satisfactorily, and the time allowed to complete a task.
- The appropriateness depends also on other factors in a visualization process, such as the original data resolution, the viewer’s familiarity of the data, the extra information that is not in the data but the viewer knows, and the available visualization resources.
- The phrase creates a gray area as to whether information loss is allowed or not, and when or where one could violate some principles such as those principles in [64].
Surveyee’s ID | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
Questions with (Correct Answers) and [Database] in Brackets | S1 | S2 | S3 | S4 | S5 | S6 | S7 | S8 | P9 | P10 |
1. Use of different transfer functions (D), [Carp] | (D) | (D) | (D) | (D) | (D) | c | b | (D) | a | c |
2. Use of translucency in volume rendering (C), [Engine Block] | (C) | (C) | (C) | (C) | (C) | (C) | (C) | (C) | d | (C) |
3. Omission of voxels of soft tissue and muscle (D), [CT head] | (D) | (D) | (D) | (D) | b | b | a | (D) | a | (D) |
4. sharp objects in volume-rendered CT data (C), [CT head] | (C) | (C) | a | (C) | a | b | d | b | b | b |
5. Loss of 3D information with MIP (B, a), [Aneurysm] | (a) | (B) | (a) | (a) | (a) | (a) | D | (a) | (a) | (a) |
6. Use of volume deformation (A), [CT head] | (A) | (A) | b | (A) | (A) | b | b | (A) | b | b |
7. Toenails in non-photo-realistic volume rendering (B, c), [Foot] | (c) | (c) | (c) | (B) | (c) | (B) | (B) | (B) | (B) | (c) |
8. Noise in non-photo-realistic volume rendering (B), [Foot] | (B) | (B) | (B) | (B) | (B) | (B) | a | (B) | c | (B) |
9. Knowledge about 3D medical imaging technology | 4 | 3 | 4 | 5 | 3 | 3 | 3 | 3 | 2 | 1 |
10. Knowledge about volume rendering techniques | 5 | 5 | 4–5 | 4 | 4 | 3 | 3 | 3 | 2 | 1 |
All Participants | Experts | The Rest | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Question | A | B | C | D | A | B | C | D | A | B | C | D | |
1. (Carp) | numbers: | 1 | 1 | 2 | 6 | 0 | 0 | 0 | 4 | 1 | 1 | 2 | 2 |
probability: | 0.10 | 0.10 | 0.20 | 0.60 | 0.00 | 0.00 | 0.00 | 1.00 | 0.17 | 0.17 | 0.33 | 0.33 | |
2. (Engine Block): | numbers: | 0 | 0 | 1 | 9 | 0 | 0 | 0 | 4 | 0 | 0 | 1 | 5 |
probability: | 0.00 | 0.00 | 0.10 | 0.90 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.17 | 0.83 | |
3. (CT head) | numbers: | 2 | 2 | 0 | 6 | 0 | 0 | 0 | 4 | 2 | 2 | 0 | 2 |
probability: | 0.20 | 0.20 | 0.00 | 0.60 | 0.0 | 0.0 | 0.0 | 1.00 | 0.33 | 0.33 | 0.00 | 0.33 | |
4. (CT head) | numbers: | 2 | 4 | 3 | 1 | 1 | 0 | 0 3 | 0 | 1 | 4 | 0 | 1 |
probability: | 0.20 | 0.40 | 0.30 | 0.10 | 0.25 | 0.00 | 0.75 | 0.00 | 0.17 | 0.67 | 0.00 | 0.17 | |
5. (Aneurism) | numbers: | 8 | 1 | 0 | 1 | 3 | 1 | 0 | 0 | 5 | 0 | 0 | 1 |
probability: | 0.80 | 0.10 | 0.00 | 0.10 | 0.75 | 0.25 | 0.00 | 0.00 | 0.83 | 0.00 | 0.00 | 0.17 | |
6. (CT head) | numbers: | 5 | 5 | 0 | 0 | 3 | 1 | 0 | 0 | 2 | 4 | 0 | 0 |
probability: | 0.50 | 0.50 | 0.00 | 0.00 | 0.75 | 0.25 | 0.00 | 0.00 | 0.33 | 0.67 | 0.00 | 0.00 | |
7. (Foot) | numbers: | 0 | 5 | 5 | 0 | 0 | 1 | 3 | 0 | 0 | 4 | 2 | 0 |
probability: | 0.00 | 0.50 | 0.50 | 0.00 | 0.00 | 0.25 | 0.75 | 0.00 | 0.00 | 0.67 | 0.33 | 0.00 | |
8. (Foot) | numbers: | 1 | 8 | 1 | 0 | 0 | 4 | 0 | 0 | 1 | 4 | 1 | 0 |
probability: | 0.10 | 0.80 | 0.10 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 | 0.17 | 0.67 | 0.17 | 0.00 |
Appendix C. Survey Results of Useful Knowledge in Volume Visualization
- Ground truth PMF .
- If one always answers A: .
- If one always answers B: .
- If one always answers C: .
- If one always answers D: .
- Survey results (all): .
- Survey results (expert): .
- Survey results (rest): .
- Survey results (all): .
- Survey results (expert): .
- Survey results (rest): .
Appendix D. Survey Results of Useful Knowledge in Viewing London Underground Maps
Surveyee’s ID | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Questions | P1 | P2 | P3 | P4 | P5 | P6 | P7 | P8 | P9 | P10 | P11 | P12 | Mean | |
Q1: | answer (min.) | 8 | 30 | 12 | 16 | 20 | 15 | 10 | 30 | 20 | 20 | 20 | 30 | 19.25 |
time (sec.) | 06.22 | 07.66 | 09.78 | 11.66 | 03.72 | 04.85 | 08.85 | 21.12 | 12.72 | 11.22 | 03.38 | 10.06 | 09.27 | |
Q2: | answer (min.) | 15 | 30 | 5 | 22 | 15 | 14 | 20 | 20 | 25 | 25 | 25 | 20 | 19.67 |
time (sec.) | 10.25 | 09.78 | 06.44 | 09.29 | 12.12 | 06.09 | 17.28 | 06.75 | 12.31 | 06.85 | 06.03 | 10.56 | 09.48 | |
Q3: | answer (min.) | 20 | 45 | 10 | 70 | 20 | 20 | 20 | 35 | 25 | 30 | 20 | 240 | 46.25 |
time (sec.) | 19.43 | 13.37 | 10.06 | 09.25 | 14.06 | 10.84 | 12.46 | 19.03 | 11.50 | 16.09 | 11.28 | 28.41 | 14.65 | |
Q4: | answer (min.) | 60 | 60 | 35 | 100 | 30 | 20 | 45 | 35 | 45 | 120 | 40 | 120 | 59.17 |
time (sec.) | 11.31 | 10.62 | 10.56 | 12.47 | 08.21 | 07.15 | 18.72 | 08.91 | 08.06 | 12.62 | 03.88 | 24.19 | 11.39 | |
Q5: | time 1 (sec.) | 22.15 | 01.75 | 07.25 | 03.78 | 14.25 | 37.68 | 06.63 | 13.75 | 19.41 | 06.47 | 03.41 | 34.97 | 14.29 |
time 2 (sec.) | 24.22 | 08.28 | 17.94 | 05.60 | 17.94 | 57.99 | 21.76 | 20.50 | 27.16 | 13.24 | 22.66 | 40.88 | 23.18 | |
answer (10) | 10 | 10 | 10 | 9 | 10 | 10 | 10 | 10 | 9 | 10 | 10 | 10 | ||
time (sec.) | 06.13 | 28.81 | 08.35 | 06.22 | 09.06 | 06.35 | 09.93 | 12.69 | 10.47 | 05.54 | 08.66 | 27.75 | 11.66 | |
Q6: | time 1 (sec.) | 02.43 | 08.28 | 01.97 | 08.87 | 05.06 | 02.84 | 06.97 | 10.15 | 18.10 | 21.53 | 03.00 | 07.40 | 08.05 |
time 2 (sec.) | 12.99 | 27.69 | 04.81 | 10.31 | 15.97 | 04.65 | 17.56 | 16.31 | 20.25 | 24.69 | 15.34 | 20.68 | 15.94 | |
answer (9) | 9 | 10 | 9 | 9 | 4 | 9 | 9 | 9 | 8 | 9 | 9 | 9 | ||
time (sec.) | 07.50 | 06.53 | 04.44 | 16.53 | 19.41 | 05.06 | 13.47 | 07.03 | 12.44 | 04.78 | 07.91 | 16.34 | 10.12 | |
Q7: | time 1 (sec.) | 17.37 | 08.56 | 01.34 | 03.16 | 08.12 | 01.25 | 21.75 | 15.56 | 02.81 | 07.84 | 02.22 | 46.72 | 11.39 |
time 2 (sec.) | 17.38 | 13.15 | 02.34 | 03.70 | 08.81 | 02.25 | 22.75 | 26.00 | 17.97 | 10.37 | 03.18 | 47.75 | 14.64 | |
answer (7) | 7 | 7 | 7 | 7 | 6 | 7 | 7 | 7 | 6 | 7 | 7 | 7 | ||
time (sec.) | 07.53 | 06.34 | 03.47 | 03.87 | 02.75 | 04.09 | 02.16 | 04.94 | 26.88 | 05.31 | 06.63 | 12.84 | 07.23 | |
Q8: | time 1 (sec.) | 12.00 | 08.50 | 06.09 | 02.88 | 08.62 | 14.78 | 19.12 | 08.53 | 12.50 | 10.22 | 12.50 | 20.00 | 11.31 |
time 2 (sec.) | 13.44 | 10.78 | 23.37 | 09.29 | 13.03 | 36.34 | 23.55 | 09.50 | 13.53 | 10.23 | 32.44 | 22.60 | 18.18 | |
answer (6) | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | ||
time (sec.) | 02.62 | 05.94 | 02.15 | 04.09 | 04.94 | 07.06 | 07.50 | 04.90 | 04.37 | 04.53 | 05.47 | 09.43 | 05.25 | |
Q9: | answer (P) | P | P | P | P | P | P | P | P | P | P | P | P | |
time (sec.) | 35.78 | 02.87 | 07.40 | 13.03 | 06.97 | 52.15 | 13.56 | 02.16 | 08.13 | 09.06 | 01.93 | 08.44 | 13.46 | |
Q10: | answer (LB) | LB | LB | LB | LB | LB | LB | LB | LB | LB | LB | LB | LB | |
time (sec.) | 05.50 | 03.13 | 12.04 | 14.97 | 07.00 | 26.38 | 11.31 | 03.38 | 06.75 | 07.47 | 06.50 | 09.82 | 09.52 | |
Q11: | answer (WP) | WP | WP | WP | WP | WP | WP | WP | WP | WP | WP | WP | WP | |
time (sec.) | 06.07 | 05.35 | 07.72 | 05.00 | 04.32 | 23.72 | 05.25 | 03.07 | 10.66 | 05.37 | 02.94 | 17.37 | 08.07 | |
Q12: | answer (FP) | FP | FP | FP | FP | FP | FP | FP | FP | FP | FP | FP | FP | |
time (sec.) | 05.16 | 02.56 | 11.78 | 08.62 | 03.60 | 19.72 | 11.28 | 03.94 | 20.72 | 01.56 | 02.50 | 06.84 | 08.19 | |
live in metro city | >5 yr | >5 yr | mths | 1–5 yr | >5 yr | 1–5 yr | weeks | >5 yr | 1–5 yr | >5 yr | mths | mths | ||
live in London | >5 yr | >5 yr | mths | 1–5 yr | 1–5 yr | mths | mths | mths | mths | mths | mths | mths |
Surveyee’s ID | ||||||
---|---|---|---|---|---|---|
Questions | P13 | P14 | P15 | P16 | Mean | |
Q1: | answer (min.) | 15 | 20 | 15 | 15 | 16.25 |
time (sec.) | 11.81 | 18.52 | 08.18 | 07.63 | 11.52 | |
Q2: | answer (min.) | 5 | 5 | 15 | 15 | 10.00 |
time (sec.) | 11.10 | 02.46 | 13.77 | 10.94 | 09.57 | |
Q3: | answer (min.) | 35 | 60 | 30 | 25 | 37.50 |
time (sec.) | 21.91 | 16.11 | 10.08 | 22.53 | 17.66 | |
Q4: | answer (min.) | 20 | 30 | 60 | 25 | 33.75 |
time (sec.) | 13.28 | 16.21 | 08.71 | 18.87 | 14.27 | |
Q5: | time 1 (sec.) | 17.72 | 07.35 | 17.22 | 09.25 | 12.89 |
time 2 (sec.) | 21.06 | 17.00 | 19.04 | 12.37 | 17.37 | |
answer (10) | 10 | 8 | 10 | 10 | ||
time (sec.) | 04.82 | 02.45 | 02.96 | 15.57 | 06.45 | |
Q6: | time 1 (sec.) | 35.04 | 38.12 | 11.29 | 07.55 | 23.00 |
time 2 (sec.) | 45.60 | 41.32 | 20.23 | 40.12 | 36.82 | |
answer (9) | 9 | 10 | 9 | 8 | ||
time (sec.) | 03.82 | 13.57 | 08.15 | 34.32 | 14.97 | |
Q7: | time 1 (sec.) | 01.05 | 02.39 | 09.55 | 11.19 | 06.05 |
time 2 (sec.) | 02.15 | 05.45 | 09.58 | 13.47 | 07.66 | |
answer (7) | 10 | 6 | 7 | 7 | ||
time (sec.) | 01.06 | 01.60 | 02.51 | 14.06 | 04.81 | |
Q8: | time 1 (sec.) | 08.74 | 26.14 | 20.37 | 15.01 | 17.57 |
time 2 (sec.) | 16.50 | 30.55 | 27.01 | 17.91 | 22.99 | |
answer (6) | 6 | 6 | 6 | 6 | ||
time (sec.) | 09.30 | 03.00 | 02.11 | 04.94 | 04.48 | |
Q9: | answer (P) | P | P | P | P | |
time (sec.) | 05.96 | 09.38 | 04.56 | 05.16 | 06.27 | |
Q10: | answer (LB) | LB | LB | LB | LB | |
time (sec.) | 12.74 | 07.77 | 01.30 | 09.94 | 07.94 | |
Q11: | answer (WP) | WP | WP | WP | WP | |
time (sec.) | 09.84 | 04.43 | 03.39 | 07.18 | 06.21 | |
Q12: | answer (FP) | FP | FP | FP | FP | |
time (sec.) | 06.22 | 10.46 | 06.78 | 05.10 | 07.14 | |
live in metro city | never | days | days | days | ||
live in London | never | days | days | days |
References
- Klein, H.A. The Science of Measurement: A Historical Survey; Dover Publications: Mineola, NY, USA, 2012. [Google Scholar]
- Stevens, S.S. On the Theory of Scales of Measurement. Science 1946, 103, 677–680. [Google Scholar] [CrossRef] [PubMed]
- Chen, M.; Golan, A. What May Visualization Processes Optimize? IEEE Trans. Vis. Comput. Graph. 2016, 22, 2619–2632. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Chen, M.; Sbert, M. A Bounded Measure for Estimating the Benefit of Visualization (Part I): Theoretical Discourse and Conceptual Evaluation. Entropy 2022, 24, 228. [Google Scholar] [CrossRef]
- Chen, M. A Short Introduction to Information-Theoretic Cost-Benefit Analysis. arXiv 2021, arXiv:2103.15113. [Google Scholar]
- Boslaugh, S.; Watters, P.A. Statistics in a Nutshell: A Desktop Quick Reference Paperback; O’Reilly: Akron, OH, USA, 2008. [Google Scholar]
- Pedhazur, E.J.; Schmelkin, L.P. Measurement, Design, and Analysis: An Integrated Approach; Lawrence Erlbaum Associates: Mahwah, NJ, USA, 1991. [Google Scholar]
- Tal, E. Measurement in Science. In The Stanford Encyclopedia of Philosophy; Zalta, E.N., Ed.; Stanford University: Palo Alto, CA, USA, 2020; ISSN 1095-5054. [Google Scholar]
- Schlaudt, O. Measurement. Online Encycl. Philos. Nat. 2020, 5, 11. [Google Scholar] [CrossRef]
- Newton, I. Scala graduum caloris. Philos. Trans. 1701, 22, 824–829. [Google Scholar]
- Grigull, U. Newton’s temperature scale and the law of cooling. Wärme Stoffuübertrag 1984, 18, 195–199. [Google Scholar] [CrossRef]
- Wikipedia. Newton Scale. Available online: https://en.wikipedia.org/wiki/Newton_scale (accessed on 20 November 2021).
- Behrisch, M.; Blumenschein, M.; Kim, N.W.; Shao, L.; El-Assady, M.; Fuchs, J.; Seebacher, D.; Diehl, A.; Brandes, U.; Pfister, H.; et al. Quality Metrics for Information Visualization. Comput. Graph. Forum 2018, 37, 625–662. [Google Scholar] [CrossRef]
- Bertini, E.; Tatu, A.; Keim, D. Quality Metrics in High-Dimensional Data Visualization: An Overview and Systematization. IEEE Trans. Vis. Comput. Graph. 2011, 17, 2203–2212. [Google Scholar] [CrossRef]
- Tufte, E.R. The Visual Display of Quantitative Information; Graphics Press: Cheshire, CT, USA, 1986. [Google Scholar]
- Cui, Q.; Yang, J.; Ward, M.; Rundensteiner, E. Measuring Data Abstraction Quality in Multiresolution Visualizations. IEEE Trans. Vis. Comput. Graph. 2006, 12, 709–716. [Google Scholar] [CrossRef]
- Johansson, J.; Cooper, M. A Screen Space Quality Method for Data Abstraction. Comput. Graph. Forum 2008, 27, 1039–1046. [Google Scholar] [CrossRef]
- Friedman, J.H.; Tukey, J.W. A Projection Pursuit Algorithm for Exploratory Data Analysis. IEEE Trans. Comput. 1974, 23, 881–890. [Google Scholar] [CrossRef]
- Tukey, J.W.; Tukey, P. Computer Graphics and Exploratory Data Analysis: An Introduction. In Proceedings of the 6th Annual Conference and Exposition: Computer Graphics, Dallas, TX, USA, 14–18 April 1985; pp. 773–785. [Google Scholar]
- Bertini, E.; Santucci, G. Quality Metrics for 2D Scatterplot Graphics: Automatically Reducing Visual Clutter. In International Symposium on Smart Graphics; Butz, A., Krüger, A., Olivier, P., Eds.; Springer: Berlin/Heidelberg, Germany, 2004; Volume 3031, pp. 77–89. [Google Scholar]
- Wilkinson, L.; Anand, A.; Grossman, R. Graph-theoretic Scagnostics. In Information Visualization, IEEE Symposium; IEEE Computer Society: Washington, DC, USA, 2005; pp. 157–164. [Google Scholar]
- Sips, M.; Neubert, B.; Lewis, J.P.; Hanrahan, P. Selecting Good Views of High-dimensional Data using Class Consistency. Comput. Graph. Forum 2009, 28, 831–838. [Google Scholar] [CrossRef]
- Tatu, A.; Albuquerque, G.; Eisemann, M.; Schneidewind, J.; Theisel, H.; Magnor, M.; Keim, D. Combining Automated Analysis and Visualization Techniques for Effective Exploration of High-dimensional Data. In Proceedings of the IEEE Symposium on Visual Analytics Science and Technology, Atlantic City, NJ, USA, 2–13 October 2009; pp. 59–66. [Google Scholar]
- Tatu, A.; Bak, P.; Bertini, E.; Keim, D.; Schneidewind, J. Visual Quality Metrics and Human Perception: An Initial Study on 2D Projections of Large Multidimensional Data. In Proceedings of the International Conference on Advanced Visual Interfaces, Bari, Italy, 7–10 June 2010; pp. 49–56. [Google Scholar]
- Dasgupta, A.; Kosara, R. Pargnostics: Screen-Space Metrics for Parallel Coordinates. IEEE Trans. Vis. Comput. Graph. 2010, 16, 1017–1026. [Google Scholar] [CrossRef]
- Peng, W.; Ward, M.O.; Rundensteiner, E. Clutter Reduction in Multi-dimensional Data Visualization using Dimension Reordering. In Proceedings of the IEEE Symposium on Information Visualization, Austin, TX, USA, 10 October 2004; pp. 89–96. [Google Scholar]
- Rosenholtz, R.; Li, Y.; Nakano, L. Measuring Visual Clutter. J. Vis. 2007, 7, 1–22. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Yu, C.P.; Samaras, D.; Zelinsky, G.J. Modelling visual clutter perception using proto-object segmentation. J. Vis. 2014, 14, 4. [Google Scholar] [CrossRef]
- Filonik, D.; Baur, D. Measuring Aesthetics for Information Visualization. In Proceedings of the IEEE Information Visualization, Atlantic City, NJ, USA, 11–16 October 2009; pp. 579–584. [Google Scholar]
- Jänicke, H.; Chen, M. A Salience-based Quality Metric for Visualization. Comput. Graph. Forum 2010, 29, 1183–1192. [Google Scholar] [CrossRef]
- Bernard, J.; Steiger, M.; Mittelstädt, S.; Thum, S.; Keim, D.A.; Kohlhammer, J. A survey and task-based quality assessment of static 2D color maps. In Visualization and Data Analysis; International Society for Optics and Photonics: Bellingham, WA, USA, 2015. [Google Scholar]
- Mittelstädt, S.; Jäckle, D.; Stoffel, F.; Keim, D.A. ColorCAT: Guided Design of Colormaps for Combined Analysis Tasks. In Proceedings of the Conference on Visualization (EuroVis), Sardinia, Italy, 25–29 May 2015; pp. 115–119. [Google Scholar]
- Mittelstädt, S.; Keim, D.A. Efficient Contrast Effect Compensation with Personalized Perception Models. Comput. Graph. Forum 2015, 34, 211–220. [Google Scholar] [CrossRef] [Green Version]
- Gramazio, C.C.; Laidlaw, D.H.; Schloss, K.B. Colorgorical: Creating discriminable and preferable color palettes for information visualization. IEEE Trans. Vis. Comput. Graph. 2017, 23, 521–530. [Google Scholar] [CrossRef]
- Cleveland, W.S.; McGill, R. Graphical perception: Theory, experimentation, and application to the development of graphical methods. J. Am. Stat. Assoc. 1984, 79, 531–554. [Google Scholar] [CrossRef]
- Isenberg, T. Evaluating and Validating Non-photorealistic and Illustrative Rendering. In Image and Video-Based Artistic Stylisation; Rosin, P., Collomosse, J., Eds.; Springer: London, UK, 2013; pp. 311–331. [Google Scholar]
- Isenberg, T.; Neumann, P.; Carpendale, S.; Sousa, M.C.; Jorge, J.A. Non-Photorealistic Rendering in Context: An Observational Study. In Proceedings of the 4th International Symposium on Non-Photorealistic Animation and Rendering, Annecy, France, 5–7 June 2006; pp. 115–126. [Google Scholar]
- Cole, F.; Sanik, K.; DeCarlo, D.; Finkelstein, A.; Funkhouser, T.; Rusinkiewicz, S.; Singh, M. How Well Do Line Drawings Depict Shape? In Proceedings of the ACM SIGGRAPH Conference, New Orleans, LA, USA, 3–7 August 2009; p. 28. [Google Scholar]
- Mandryk, R.L.; Mould, D.; Li, H. Evaluation of Emotional Response to Non-Photorealistic Images. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Non-Photorealistic Animation and Rendering, Vancouver, BC, Canada, 5–7 August 2011; pp. 7–16. [Google Scholar]
- Liu, Z.; Li, Z. Impact of schematic designs on the cognition of underground tube maps. ISPRS—Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2016, 49, 421–423. [Google Scholar] [CrossRef] [Green Version]
- Hong, S.; Yoo, M.J.; Chinh, B.; Han, A.; Battersby, S.; Kim, J. To Distort or Not to Distort: Distance Cartograms in the Wild. In Proceedings of the CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada, 21–26 April 2018; pp. 1–12. [Google Scholar]
- Tam, G.K.L.; Kothari, V.; Chen, M. An analysis of machine- and human-analytics in classification. IEEE Trans. Vis. Comput. Graph. 2017, 23, 71–80. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Kijmongkolchai, N.; Abdul-Rahman, A.; Chen, M. Empirically measuring soft knowledge in visualization. Comput. Graph. Forum 2017, 36, 73–85. [Google Scholar] [CrossRef]
- Wall, E.; Agnihotri, M.; Matzen, L.; Divis, K.; Haass, M.; Endert, A.; Stasko, J. A heuristic approach to value-driven evaluation of visualizations. IEEE Trans. Vis. Comput. Graph. 2018, 25, 491–500. [Google Scholar] [CrossRef] [PubMed]
- Saket, B.; Srinivasan, A.; Ragan, E.D.; Endert, A. Evaluating interactive graphical encodings for data visualization. IEEE Trans. Vis. Comput. Graph. 2017, 24, 1316–1330. [Google Scholar] [CrossRef]
- Sacha, D.; Stoffel, A.; Stoffel, F.; Kwon, B.C.; Ellis, G.; Keim, D.A. Knowledge generation model for visual analytics. IEEE Trans. Vis. Comput. Graph. 2014, 20, 1604–1613. [Google Scholar] [CrossRef]
- Chen, M.; Ebert, D.S. An ontological framework for supporting the design and evaluation of visual analytics systems. Comput. Graph. Forum 2019, 38, 131–144. [Google Scholar] [CrossRef]
- Shannon, C.E. A mathematical theory of communication. Bell Syst. Tech. J. 1948, 27, 379–423. [Google Scholar] [CrossRef] [Green Version]
- Cover, T.M.; Thomas, J.A. Elements of Information Theory; John Wiley & Sons: Hoboken, NJ, USA, 2006. [Google Scholar]
- Chen, M.; Gaither, K.; John, N.W.; McCann, B. Cost–benefit analysis of visualization in virtual environments. IEEE Trans. Vis. Comput. Graph. 2019, 25, 32–42. [Google Scholar] [CrossRef] [Green Version]
- Ishizaka, A.; Nemery, P. Multi-Criteria Decision Analysis: Methods and Software; John Wiley & Sons: Hoboken, NJ, USA, 2013. [Google Scholar]
- Lin, J. Divergence measures based on the Shannon entropy. IEEE Trans. Inf. Theory 1991, 37, 145–151. [Google Scholar] [CrossRef] [Green Version]
- Roettger, S. The Volume Library. Available online: http://schorsch.efi.fh-nuernberg.de/data/volume/ (accessed on 20 November 2021).
- Nagy, Z.; Schneide, J.; Westerman, R. Interactive Volume Illustration. In Proceedings of the Vision, Modeling and Visualization, Erlangen, Germany, 20–22 November 2002. [Google Scholar]
- Correa, C.; Silver, D.; Chen, M. Feature aligned volume manipulation for illustration and visualization. IEEE Trans. Vis. Comput. Graph. 2006, 12, 1069–1076. [Google Scholar] [CrossRef]
- Wu, Y.; Qu, H. Interactive Transfer Function Design Based on Editing Direct Volume Rendered Images. IEEE Trans. Vis. Comput. Graph. 2007, 13, 1027–1040. [Google Scholar] [CrossRef] [PubMed]
- Jung, Y. Instantreality 1.0. Available online: https://doc.instantreality.org/tutorial/volume-rendering/ (accessed on 20 November 2021).
- Max, N.; Chen, M. Local and global illumination in the volume rendering integral. In Scientific Visualization: Advanced Concepts; Hagen, H., Ed.; Schloss Dagstuhl: Wadern, Germany, 2010. [Google Scholar]
- Chen, M. Cost-Benefit Analysis of Data Intelligence—Its Broader Interpretations. In Advances in Info-Metrics: Information and Information Processing across Disciplines; Chen, M., Dunn, J.M., Golan, A., Ullah, A., Eds.; Oxford University Press: Oxford, UK, 2020. [Google Scholar]
- Streeb, D.; El-Assady, M.; Keim, D.; Chen, M. Why visualize? Untangling a large network of arguments. IEEE Trans. Vis. Comput. Graph. 2019, 27, 2220–2236. [Google Scholar] [CrossRef] [PubMed]
- Chen, M.; Edwards, D.J. ‘Isms’ in Visualization. In Foundations of Data Visualization; Chen, M., Hauser, H., Rheingans, P., Scheuermann, G., Eds.; Springer: Berlin/Heidelberg, Germany, 2020. [Google Scholar]
- Dasgupta, A.; Chen, M.; Kosara, R. Conceptualizing visual uncertainty in parallel coordinates. Comput. Graph. Forum 2012, 31, 1015–1024. [Google Scholar] [CrossRef] [Green Version]
- Viola, I.; Chen, M.; Isenberg, T. Visual Abstraction. In Foundations of Data Visualization; Chen, M., Hauser, H., Rheingans, P., Scheuermann, G., Eds.; Springer: Berlin/Heidelberg, Germany, 2020. [Google Scholar]
- Kindlmann, G.; Scheidegger, C. An Algebraic Process for Visualization Design. IEEE Trans. Vis. Comput. Graph. 2014, 20, 2181–2190. [Google Scholar] [CrossRef] [PubMed]
- Chen, M. The value of interaction in data intelligence. arXiv 2018, arXiv:1812.06051. [Google Scholar]
- Kullback, S.; Leibler, R.A. On information and sufficiency. Ann. Math. Stat. 1951, 22, 79–86. [Google Scholar] [CrossRef]
- Chen, M.; Trefethen, A.; Banares-Alcantara, R.; Jirotka, M.; Coecke, B.; Ertl, T.; Schmidt, A. From data analysis and visualization to causality discovery. IEEE Comput. 2011, 44, 84–87. [Google Scholar] [CrossRef]
A Summary of the Empirical Scores Obtained of the Four Case Studies | |||||||
---|---|---|---|---|---|---|---|
Criteria | |||||||
S1: | order | adequate | adequate | excellent | excellent | good | good |
benefit | adequate | good | excellent | good | excellent | good | |
knowledge | excellent | excellent | excellent | excellent | excellent | excellent | |
score | 1 | 2 | 5 | 4 | 4 | 3 | |
order | excellent | excellent | good | excellent | adequate | good | |
benefit | adequate | excellent | good | adequate | good | adequate | |
knowledge | excellent | excellent | excellent | excellent | excellent | excellent | |
score | 3 | 5 | 3 | 3 | 2 | 2 | |
order | excellent | excellent | good | excellent | inadequate | good | |
benefit | excellent | excellent | excellent | excellent | excellent | excellent | |
knowledge | excellent | excellent | good | good | adequate | good | |
score | 5 | 5 | 3 | 4 | 0 | 3 | |
order | excellent | excellent | excellent | excellent | adequate | excellent | |
benefit | adequate | adequate | adequate | excellent | adequate | adequate | |
knowledge | excellent | excellent | excellent | excellent | excellent | excellent | |
score | 3 | 3 | 3 | 5 | 1 | 3 | |
Empirical Subtotal: | 12 | 15 | 14 | 16 | 7 | 11 | |
Combining All Scores Obtained from the Conceptual and Empirical Evaluation | |||||||
Criteria | |||||||
Conceptual Subtotal [4]: | 30 | 30 | 28 | 30 | 26 | 29 | |
Empirical Subtotal: | 12 | 15 | 14 | 16 | 7 | 11 | |
Componentization (extra criterion): | 5 | 1 | 5 | 5 | 5 | 5 | |
Total without the extra criterion: | 42 | 45 | 42 | 46 | 33 | 40 | |
Total with the extra criterion: | 47 | 46 | 47 | 51 | 38 | 45 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Chen, M.; Abdul-Rahman, A.; Silver, D.; Sbert, M. A Bounded Measure for Estimating the Benefit of Visualization (Part II): Case Studies and Empirical Evaluation. Entropy 2022, 24, 282. https://doi.org/10.3390/e24020282
Chen M, Abdul-Rahman A, Silver D, Sbert M. A Bounded Measure for Estimating the Benefit of Visualization (Part II): Case Studies and Empirical Evaluation. Entropy. 2022; 24(2):282. https://doi.org/10.3390/e24020282
Chicago/Turabian StyleChen, Min, Alfie Abdul-Rahman, Deborah Silver, and Mateu Sbert. 2022. "A Bounded Measure for Estimating the Benefit of Visualization (Part II): Case Studies and Empirical Evaluation" Entropy 24, no. 2: 282. https://doi.org/10.3390/e24020282