College of Computing and Digital Media Dissertations

Date of Award

Fall 11-12-2021

Degree Type


Degree Name

Doctor of Philosophy (PhD)


School of Computing

First Advisor

Adam Steele, PhD

Second Advisor

Xiaowen Fang, PhD

Third Advisor

Peter Hastings, PhD

Fourth Advisor

Kevin Buffardi, PhD


Heuristic evaluation (HE) is one of the most widely used usability evaluation methods. The reason for its popularity is that it is a discount method, meaning that it does not require substantial time or resources, and it is simple, as evaluators can evaluate a system guided by a set of usability heuristics. Despite its simplicity, a major problem with HE is that there is a significant gap in the quality of results produced by expert and novice evaluators. This gap has made some scholars question the usefulness of the method as they claim that the evaluation results are a product of the evaluator’s experience rather than the method itself.

In response, the goal of this thesis is to bridge the gap between expert and novice evaluators. Based on interviews with 15 usability experts, which focused on their experience with the method, the difficulties they faced when they were novices, and how they overcame such difficulties, it presents a comprehensive protocol called Coherent Heuristic Evaluation (CoHE). This step-by-step protocol guides novice evaluators from the moment they decide to conduct an evaluation until the submission of their evaluation report.

This protocol was verified by conducting an experiment to observe the difference between novices using the CoHE protocol and novices using Nielsen’s 10 usability heuristics without the guidance. The experiment involved 20 novices performing two sessions; the first was an understanding session where the novices read and understood the heuristics and the second was an inspecting session where they inspected a system. The findings show that, while evaluators take more time to read and evaluate a system using CoHE, they tend to identify more problems. The experiment also demonstrated that CoHE can improve the thoroughness, effectiveness, and f-measure of evaluation. However, the validity of CoHE was comparable to that of HE.



To view the content in your browser, please download Adobe Reader or, alternately,
you may Download the file to your hard drive.

NOTE: The latest versions of Adobe Reader do not support viewing PDF files within Firefox on Mac OS and if you are using a modern (Intel) Mac, there is no official plugin for viewing PDF files within the browser window.