Usability Evaluation | part 2

Usability Testing Methods

Since good usability should be the standard for anything that is designed to be used by humans, it is also important to ensure that the products are actually usable. The more complex a system is, the more work needs to be done. Usability testing is one of the most used methods to improve the usability of a digital system. It focuses on observing users while they are interacting with a system.

“Start earlier than you think makes sense.”
– Steve Krug

When should I start testing?

Based on when you are doing the tests, there are two different types:

Formative Testing
Formative testing happens while the product is still in development. The main goal of this type is to identify and fix problems based on small tests with users. This type of testing is normally repeated multiple times during the development and can start with the first pencil sketches on paper or even a napkin.

Summative Testing
Summative testing starts when the product is nearly finished or already finished. The main goal of this type is to establish a baseline of metrics and identify if the product meets the requirements.

If you are redesigning a system, it is important to start even earlier. Testing an existing product can give you a lot of useful insights and ideas for the redesign. It helps you with identifying the biggest pain points of the current product and which parts of the system are already working well and should be emphasized even more. If there is time left, you should also take a closer look at the systems from your competitors and ideally also test them. Testing them can also give you additional information about what they are doing better and where there is even more potential for your system.

Testing during a redesign (formative testing) is already quite common, but the majority of tests are still done during the end of the process (summative testing). There is no reason why you should not test your first sketches, paper prototypes, and wireframes in addition to the nearly finished prototypes. The more you test, the easier it is to identify problems and also fix them. Testing earlier saves time and money. Usability tests are especially useful after every important design decision.

How many participants do I need?

The number of users you should test with is also connected is the same as with the usability inspections. To identify the most important usability problems you should test with 3 to 5 users. After the fifth user, you already discovered 85% of the usability problems. Identifying the last 15% will take about 3 or 4 times more people than for the first 85%. That’s also the reason why you should stop at 5 people, especially when doing more expensive tests like the moderated usability test. It is much more efficient to do five smaller tests with a few people during the process than one big extensive usability study in the end.

There are a lot of different usability testing methods out there and every one of them has its unique advantages and disadvantages. Which method you should use always depends on the project and the stage you are in. They also cost different amounts of money.

Moderated Usability Test

While moderated tests get you the richest and most detailed qualitative data from real users, they are also resource and cost-intensive. Although moderated tests would also work great with low fidelity prototypes or wireframes because the moderator can give additional explanations, it is better to use high fidelity prototypes for these tests because they also give you a realistic idea of how the users perceive the design.

During these tests, you usually record the screen, voice, and also face of the user. Based on the recordings you should also create a highlight reel with the biggest problems, suggestions from users, important quotes from them, and statements about what the thought would happen next.

Moderated tests can either be done in-person or remotely. But since the moderator plays an important role and should also be able to read the behavior and non-verbal cues of the participants, it is better to do them in person. Remote moderated usability tests are better if you have a limited budget, if you are testing with users from all over the world, or if you are working on a niche product with a very small number of potential users in your area. During the ongoing pandemic, most of the moderated tests were also done remotely. The biggest challenge for moderators during remote tests is reading the participant and finding the balance between letting the user know that you are listening and interrupting them.

Moderator

Since the moderators play an important role during these tests, they should have solid experience in the field of usability testing or user research. Moderators have to work directly with the participants, guide them through the process, reading them the tasks they have to do, and as follow-up questions to get even more information during specific steps. It is also important that the moderator explains to the participants that the system itself is tested and that they cannot do anything wrong.

Participants

The participants have to use the think-aloud method and talk about what they are thinking about while interacting with the system. Ideally, the participants should also be part of the main target audience of the system.

Observers

To get even better results, you can also give the designers and developers the chance to observe these tests so that they get real time information about the usability and design of the system.

source: https://www.nngroup.com/articles/usability-testing-101/

The biggest disadvantage of moderated usability tests is that the preparing, recruiting, scheduling, coordinating, and sessions themselves cost a lot of time and therefore also a lot of money. Because you always need a moderator for the sessions, this method is also not as scalable as the others.

Unmoderated Usability Tests

Unmoderated usability tests work similarly to the moderated tests and are also a good source for qualitative data. The main difference is, that the moderator is not present during the test. But there is still a moderator or researcher involved that provides the task instructions and also watches and analyses the videos of the tests afterward. Since there is no moderator present, it is mostly used to test specific parts of a product, rather than a whole user journey.

The main advantage of this method compared to moderated tests is that there can be 5, 10, 20, 50, or even more tests simultaneously. Another big advantage is that these tests can be conducted anywhere, anytime, and with anybody. That is also the main reason why this method works best for remote tests. Since it can be conducted much faster and requires less time, it is not only more scalable but also cheaper than the moderated tests.

Moderator

Because the moderator is not present during the test, it is also harder to get the “Why” information. But if the unmoderated tests and tasks are prepared carefully, remind the participants to think aloud after every task and additional task-specific questions are asked before the next task, it can be almost as efficient as a moderated test.

Participants

The participants of an unmoderated test also have to use the think-aloud method. But the biggest advantage for the participants is that they can do the tests anytime and anywhere they want if it is done remotely.

Another big advantage of unmoderated tests compared to moderated tests is that there is a smaller chance for bias. Observed people are often acting differently when they are alone. This phenomenon is also called the observer-expectancy effect.

The biggest disadvantage of this method however is that the participants tend to be quieter because there is nobody reminding them of the thinking aloud if they forget it. That’s also the reason why different remote testing tools only pay their participants if they were also doing it correctly. Because there is no moderator present for guiding the participants through the test, it also just works well with high-fidelity prototypes or existing products.

Ideally, both of these testing methods should be used complementary during an iterative design process. You should start with moderated tests during the early phases of the process with a low fidelity prototype or wireframes and when you are finalizing the project you could do an unmoderated test to see how the product performs with real users around the world.