This document outlines a usability inspection lab where students will learn to evaluate interactive system design through expert analysis. The lab has students apply heuristic evaluation to analyze the design of a website. They will independently use design heuristics to identify usability issues, then aggregate their findings to compile a report and discuss improvements with the design team. The goal is for students to understand how to effectively evaluate design with expert feedback.
This document outlines a usability inspection lab where students will learn to evaluate interactive system design through expert analysis. The lab has students apply heuristic evaluation to analyze the design of a website. They will independently use design heuristics to identify usability issues, then aggregate their findings to compile a report and discuss improvements with the design team. The goal is for students to understand how to effectively evaluate design with expert feedback.
This document outlines a usability inspection lab where students will learn to evaluate interactive system design through expert analysis. The lab has students apply heuristic evaluation to analyze the design of a website. They will independently use design heuristics to identify usability issues, then aggregate their findings to compile a report and discuss improvements with the design team. The goal is for students to understand how to effectively evaluate design with expert feedback.
This document outlines a usability inspection lab where students will learn to evaluate interactive system design through expert analysis. The lab has students apply heuristic evaluation to analyze the design of a website. They will independently use design heuristics to identify usability issues, then aggregate their findings to compile a report and discuss improvements with the design team. The goal is for students to understand how to effectively evaluate design with expert feedback.
Purpose The purpose of this lab is to learn the technique of evaluating interactive system design through expert analysis where experts give their feedback directly, based on their expertise or a set of design heuristics.
Outcomes After completing this lab, students will be able to apply this technique to evaluate interactive system design. They will have a good insight about: ▪ How to evaluate design with experts. What are the different phases involved? ▪ When to get design critique from experts. ▪ Benefits and drawbacks of evaluation with experts
Introduction In this lab, we are going to apply an evaluation technique called heuristic evaluation to evaluate the design of an interactive system. Heuristic Evaluation was created by Jakob Nielsen and colleagues (Section 9.3.2, Human–Computer Interaction, Third Edition, Alan Dix et al), about twenty years ago. The basic idea of heuristic evaluation is to provide a set of people — often other stakeholders on the design team or outside design experts — with a set of design heuristics or principles, and ask them to use those to look for problems in our design. Each of them independently walks through a variety of tasks using our design to look for the bugs. Doing independently is important since different evaluators are going to find different problems. At the end of the process, they’re going to get back together and talk about what they found. This “independent first, gather afterwards” is how we get a “wisdom of crowds” benefit in having multiple evaluators. One of the reasons that we’re talking about this early in the lab is that it is a technique that you can use, either on a working user interface or on sketches of user interfaces. So, heuristic evaluation works really well in conjunction with paper prototypes and other rapid, low fidelity techniques enabling us to get our design ideas out quick and fast.
When to evaluate with experts
We can get peer critique really at any stage of your design process, but let’s highlight few that can be particularly valuable.
1- First, it’s really valuable to get peer critique before user testing, because that helps you not waste your users on stuff that’s just going to get picked up automatically. 2- The rich qualitative feedback that peer critique provides can also be really valuable before redesigning your application, because what it can do is it can show you what parts of your app you probably want to keep, and what are other parts that are more problematic and deserve redesign.
CCS356 –Lab Manual 1
3- Third, sometimes, you know there are problems, and you need data to be able to convince other stakeholders to make the changes. And peer critique can be a great way, especially if it’s structured, to be able to get the feedback that you need, to make the changes that you know need to happen. 4- And lastly, this kind of structured peer critique can be really valuable before releasing software, because it helps you do a final sanding of the entire design, and smooth out any rough edges.
Benefits and drawbacks of evaluation with experts
Heuristic evaluation can often be a lot faster. It takes just an hour or two for an evaluator. In heuristic evaluation, the results come pre-interpreted because your evaluators are directly providing you with problems and things to fix. Now conversely, experts walking through your system can generate false positives that wouldn’t actually happen in a real environment.
To explain how the evaluation takes place, let us consider NTS’s website (www.nts.org.pk) and apply this technique to evaluate it.
Activity 1: First evaluation phase
The first evaluation generally takes up to couple of hours, depending on the nature and complexity of the software. The evaluators use the software freely to gain a feel for the methods of interaction and the scope. It is also important to get all of your evaluators up to speed, during this phase, on what the story is behind your software — any necessary domain knowledge they might need — and tell them about the scenario that you’re going to have them step through.
During this activity, we are going to walk through the NTS website just to get used to its flow, to get feel of it, and to get the idea of features it offers.
Activity 2: Second evaluation phase
In the second evaluation phase, the evaluators independently carry out another run- through, whilst applying the chosen heuristics. Now which heuristics should we use? Nielsen’s ten heuristics are a fantastic start, and we can augment those with anything else that’s relevant for our domain. The evaluators would focus on individual elements, look for the heuristics violations and document them. Each documented violation must include the issue, its severity rating, the heuristic that it violates, and a description of exactly what the problem is. Given below is the severity rating system Jakob Nielson created and we may use the same.
0 - don’t agree that this is a usability problem
1 - cosmetic problem 2 - minor usability problem 3 - major usability problem; important to fix 4 - usability catastrophe; imperative to fix Violation 1 Issue: Unable to figure out “How did I get here?” and “How do I get back to where I came from? CCS356 –Lab Manual 2 Severity: 3
Heuristic violated: Show system status, User control and freedom
Description: Figure 3.1 shows one of the pages visited on the website. The system provides no clue to user to figure out his/her way through the site, and to find the way back if he/she accidentally clicks on the wrong link. There are no breadcrumbs which offer freedom to users to "jump" to previous categories in the sequence without using the Back key, other navigation bars, or the search engine. There is no highlighting of top menu item to make user aware the menu category current page belongs to.
Figure 3.1: Information Page about Law Practitioners Test on NTS Website
Violation 2 Issue: Inconsistent color scheme
Severity: 1
Heuristic violated: Standard and consistency
Description: Figure 3.2 shows two different pages visited on the website. Both follows different color scheme leading to inconsistent design.
Violation 3 Issue: NTS logo is not clickable on many pages.
Severity: 2
Heuristic violated: Standard and consistency, Flexibility and efficiency
Description: The NTS logo, though clickable when on home page (www.nts.org.pk), is not clickable on many pages.
CCS356 –Lab Manual 3
Figure 3.2: Screenshot of Two Different Pages of NTS Website
Activity 3: Aggregation Once every evaluator has performed the evaluation individually and independently, next step is to get together, aggregate the findings and compile a report, of all distinct usability issues, that can be presented and discussed with the design team. Since multiple evaluators may find same problems, so rather than presenting a redundant set of identified problems, it would be better to include them once in the report. As a designer, we should ask evaluators to include in this report is the Issues-Evaluators matrix (Figure-3), where rows represent evaluators and columns represent the issues identified. This matrix would help designers in differentiating between a very good evaluator and a not very good evaluator.
CCS356 –Lab Manual 4
Figure 3.2: Issues-Evaluators Matrix Activity 4: Debriefing session with the design team Finally, after all evaluators have gone through the interface, listed their problems, and combined them in terms of the severity and importance, next step is to debrief with the design team. This is a nice chance to be able to discuss issues in the user interface and qualitative feedback, and suggest improvements on how these problems can be addressed. In this debrief session, it can be valuable for the development team to estimate the amount of effort that it would take to fix one of these problems. For example, voiolation1 in activity 3 is easy to fix. Conversely, violation 1 is major usability issue which takes a more effort, but its importance will lead you to fix it. There could be other things where the importance relative to the cost involved but just don’t make sense to deal with right now. This debrief session can be a great way to brainstorm future design ideas, as we’ve got all the stakeholders in the room, and while the ideas, about what the issues are with the user interface, are fresh in their minds.
Home Activities The student should work in a group of 3-5 for the following activities. Activity 2 and Activity 3 must be performed individually while Activity 1 and Activity 4 in group.
Activity 1: Choice of System to be Evaluated
Take NTS website as an example and find the usability problems.
Activity 2: First evaluation phase (Individually and
independently) During this activity, each group member will walk through the NTS website just to get used to its flow, to get feel of it, and to get the idea of features it offers.
Activity 3: Second evaluation phase (Individually and
independently) During this activity, each group member will use Jakob Nielson’s ten design heuristics and look for the heuristics violations and document them. Each documented violation must
CCS356 –Lab Manual 5
include the issue, its severity rating, the heuristic that it violates, and a description of exactly what the problem is.
Activity 4: Aggregation (In group)
For this activity, all the group member will get together, aggregate the findings and compile a report that would contain all distinct usability issues and the Issues-Evaluators matrix.
Assignment Deliverables Students need to submit a report containing following items:
▪ Small description of the chosen system
▪ Issues identified, by each of the group members. Each documented issue must highlight: o Issue o Severity o Heuristics violated o Description (may include screenshots for better understanding)
▪ List of all the unique issues found and Issues-Evaluators Matrix, where rows represent evaluators and columns represent the usability issues
