Usability metrics: Knowing what to assess
By Arathi Sethumadhavan
In my last blog post, I wrote about the common usability misconceptions that Philip Kortum identified in his latest book. In this blog, I will discuss the usability metrics that the author has identified and provide my impressions on the suitability of these metrics to evaluate products.
Effectiveness
–Effectiveness is the ability of a user to perform a task.
–Was the user able to complete the task without any difficulties? Did the user struggle initially but was finally able to complete the task? Was there a use error but did the user recover without any help?
–During a usability evaluation, researchers typically capture errors of omission, errors of commission, out of sequence errors, and close calls.
Effectiveness can be a useful metric to understand user interactions with a plethora of products from consumer products to medical devices. For example, is the user able to use a microwave to warm his or her food? Were the buttons on the panel confusing or misleading? Just last week, I saw a new colleague give up warming up his food because he could not figure out which series of buttons needed to be selected in the microwave in the office kitchenette to warm his food.
Effectiveness is also widely used to assess user interactions with medical devices. In studies involving medical devices, users are given an opportunity to use the device independently without any guidance from the study moderator. User performance is observed by the moderator and performance on tasks are measured. Moderator categorizes user performance into categories such as successful, resolved with a close call or incorrect. Users are probed after completing all study scenarios to gain a better understanding of the reasons why they struggled to use or were unable to use the device features.
Efficiency
–Efficiency refers to the amount of effort spent by the user to complete a task.
— Longer task completion times are an indication that the user interface poses difficulties to the user.
Efficiency can serve as an excellent metric with a variety of products, especially in situations where the user is under time pressure. For example, how long does it take a user to operate a kiosk in a train station to buy a ticket, when she knows that the next train departs in 5 minutes? This metric can be valuable in safety critical domains as well. For example, how long does it take an air traffic controller to take over manual control following the failure of an automated aid?
Time on task can be very useful in comparative usability studies when a product development team is trying to decide between multiple design concepts or platforms. For example, we conducted a comparative usability study where we compared the dosing behaviors of caregivers and acquaintances of diabetic patients with glucagon nasal powder (a product of Locemia) and an injectable glucagon (product of Eli Lilly). We found that users were able to administer glucagon nasal powder more quickly (Efficiency) and with a higher success rate (Effectiveness) than injectable glucagon. Based in part on the results from this study, Eli Lilly acquired worldwide rights to the intranasal glucagon developed by Locemia, a lucrative partnership for both companies, giving them a distinct competitive edge.
Satisfaction
— In addition to efficiency and effectiveness, user satisfaction is another metric that can provide insights into users likes and dislikes and general impressions about the product. For example, the “after scenario questionnaire” assesses user satisfaction after a scenario by querying users on the ease with which they were able to complete tasks in a scenario, with the amount of time it took them to complete the tasks, and with the support information available to complete the tasks.
Another commonly used tool to assess user satisfaction is the System Usability Scale, which is used to assess improvements in user satisfaction after making design modifications to a product. Users provide responses using a 1-5 scale on various factors such as product complexity, product inconsistency, ease of use, need for technical support, learnability, and confidence.
Satisfaction is also assessed in usability studies involving medical devices, where users are asked about their overall experience with a Product. This can provide a lot of insight into users’ likes and dislikes as well as difficulties encountered when using the Product.
However, there are situations where users’ satisfaction scores does not correlate with their performance. This is no surprise because there is a lot of research that shows that users don’t always know what they want and often times choose designs that do not result in the most optimal performance. In these situations, it becomes very important to rely on multiple metrics to make product design decisions. If the product in question is a medical device, then it is more important to go with a design that results in fewer use events even if users are less satisfied by it. This does not mean that user satisfaction is not as important as effectiveness and efficiency. An ideal product is certainly one that a user can learn and use fast without making any errors of omission or commission and that satisfies them, integrates well into their lives, and even excites them.
References
Kortum, P. (2016). Usability assessment: How to measure the usability of products, services, and systems. Santa Monica, CA: Human Factors and Ergonomics Society.
Lewis, J. R. (1995) IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use. International Journal of Human-Computer Interaction, 7:1, 57-78.