Usability Evaluation | part 1

Usability evaluation methods deal with assessing the usability of a product. The main goal of evaluating usability is to define usability problems and obtain measures to make products more usable and pleasant to use. Since usability has already become a large part of the user experience (UX) and most of the methods can also be used to optimize more than just the usability, the following methods can also be called UX research methods.

Because of that, these methods can also be divided into qualitative and quantitative research.

Quantitative
Quantitative UX research is about putting numerical data to problems that can be modified to more usable statistics. Classical quantitative research methods include all types of surveys and polls. Although they started including a small amount of qualitative data, Google Analytics is also a big and important source for quantitative data, especially for digital products.

Qualitative
Qualitative UX research is about observations, user behavior, and the user’s perceptions. It involves collecting non-numerical data like text, audio, video, and images to understand the user experience in detail. This type of research is commonly used in all social sciences. Classical Qualitative research methods include interviews, observations, field studies, and also usability tests. According to Jakob Nielsen and William Bruce Cameron, you should also focus on qualitative research during UX projects because it often delivers more convincing results for less money.

“quantitative studies are usually too narrow to be fruitful and are oftentimes directly misleading.”
– Jakob Nielsen

“Not everything that can be counted counts, and not everything that counts can be counted.”
– William Bruce Cameron

Because there are a lot of different UX research methods and each method has its own unique advantages and disadvantages there is no list of best UX research methods. It always depends on the goal of your usability evaluation, the time and budget you have, the target audience of your product, the stage of the process you are in, and many more factors.

To get a better overview about which UX research you could use for your product, Susan Farrell created the UX research cheat sheet. It should not work as a checklist, it just shows which methods are available along the design process.

source: https://www.nngroup.com/articles/ux-research-cheat-sheet/

Usability Inspection Methods

Additionally to all the testing methods, there are also a number of different inspection methods available. Inspections are done by evaluators, usually usability professionals, and their main goal is to find usability problems in the design. Some inspections methods are also suitable for getting an overview about the usability of an entire system. Like the tests, these methods can also be performed early in the process of the project. Additionally to the following methods, you can also do a Heuristic Estimation but this method is not used very often.

Heuristic Evaluation

The heuristic evaluation is the most informal inspection method. Usability professionals identify where a product follows predefined usability principles (heuristics) and where it doesn’t.

Heuristic evaluations are a quick and lower-cost way to measure the usability of a product before conducting user tests. They should be part of the process and used in conjunction with usability tests. According to a test conducted by Jedd Sauro, heuristics evaluations are able to detect between 30 and 50% of all usability issues, including all severe ones. Because this number also depends on the number of evaluators you should have about 3 to 5 professionals testing it to get the best value for your money.

Heuristics

The most well-known usability heuristics are based on the fifth chapter of Jakob Nielsen’s book “Usability Engineering” from 1993. Despite being nearly 30 years old, these guidelines are well established and still up-to-date. Jakob Nielsen has even slightly reworded them and added some newer examples last year.

But during a heuristic evaluation, a professional evaluator might also use different and more specific heuristic or additional product-related ones. Arnie Lund, Bruce Tognazzini, and Ben Shneiderman have also defined additional sets of heuristics. For modern apps, you can also use Apple’s Human Interface Guideline or Google’s Material Design Guidelines as Usability Heuristics.

Arniel Lunds “Expert Ratings of Usability Maxims”

Bruce Tognazzini’s “First Principles of Interaction Design”

Ben Shneiderman’s “Eight Golden Rules of Interface Design”

source: https://www.nngroup.com/articles/ten-usability-heuristics/

The Process

The process of a heuristic evaluation is divided into three sections: Planning, Executing, and Reviewing.

Planning
Since it is a usability inspection method with clear objectives, you should plan your desired outcome in advance. Setting goals and understanding exactly what needs to be evaluated is one of the most important parts of the planning phase. The evaluators must also be aware of the demographics, needs, motivations, and behaviors of the target audience. Although there are no real users present during the evaluation, the usability professionals should always keep the users and the situation the product is used in mind. During the planning phase, it is also important to choose the number of evaluators – in this case ideally between 3 and 5 to get the most results for as little money as possible. Additionally, you should also define the set of heuristics you want to use for the inspection.

Executing
During this phase, the usability professionals are actually doing the inspection of the interface. They will go through the user flow and analyze all pages based on the previously defined heuristics. This part of the inspection is normally recorded to get the best result out of the evaluation and be able to see exactly where the problems are, what the attempted tasks were, why it is a problem and usually they are also suggesting different ways on how to fix the problems.

Reviewing
After the inspection, it is important that the experts are creating a list with all usability issues and additional findings that should be addressed. Ideally, these usability problems should also be rated and prioritized based on how severe they are. These results are normally presented in the form of a report that describes the process of the inspection, all usability problems, and different suggestions on how to fix them. This report, together with the recordings, should also be the basis for the next steps the project team is going to make.

Cognitive Walkthrough

During a cognitive walkthrough, one or more evaluators work through some tasks and a set of questions from the perspective of the user. The goal is to simulate the user’s problem-solving process. It was originally designed for ticket machines, ATMs, postal kiosks, or other interactive systems where users have little or no training before using the system. Currently, it is mainly used for evaluating the learnability of a system for new and occasional users.

Pluralistic Walkthrough

A pluralistic walkthrough is a group evaluation of a design where users are guided through tasks while designers and developers address their concerns or questions about the interface. This method works best in the very early stages of the design process with a paper prototype or rough wireframes.

Consistency Inspection

Consistency inspections are usually done by designers on larger projects. It is used to ensure that the interface does things the same on all pages. The ultimate goal of the consistency inspection is that it looks like one designer did the whole project and every element that does the same thing also looks and feels the same.

Feature Inspection

The feature inspection emphasizes the importance of functionality in order to achieve good usability. It identifies critical tasks a user would perform while using a system and the features that would be used to perform these tasks. After that process, the features are then evaluated.

Standards Inspection

For a standard inspection, you need an expert on an interface standard that has to inspect the whole interface for compliance with the standards. Good examples for standard inspections of apps include checking if the Apple Human Interface Guidelines or Google Material Design Guidelines are met. Another important standards inspection determines if the product complies with all the ISO standards.

Apple Human Interface Guidelines

Google Material Design Guidelines

ISO Standards

Formal usability inspection

A formal usability inspection combines different elements from the heuristic evaluation and from the cognitive walkthrough. It is a process with six steps and clearly defined roles for every expert on the team.

Competitive Review

A competitive review is a process where a usability expert analyzes a series of related websites or products from competitors or other companies from the same sector. The expert tries to look for strengths, weaknesses, trends and differences and identifies room for improvement for his/her own product. A competitive review is also possible with just a small portion of a website. For example if you are building a webshop you can take a closer look at amazon and their ordering process.