Chapter 1: The Value of User Testing

The significance of user testing is recognized by nearly all UX designers and members of product teams. This concept may seem straightforward, but it is crucial for identifying usability issues in any digital interface, whether an app, website, or software. Observing real users as they navigate the product and report their difficulties is the most effective way to uncover usability flaws.

Every individual user may spot minor glitches or struggle with certain functionalities. Achieving absolute satisfaction from all users is unrealistic; there will always be some who have different preferences. The key is to identify and address the most critical usability concerns.

A common inquiry is regarding the optimal number of users for testing. More testers can lead to increased costs and time for the project. Thankfully, a straightforward answer exists:

“Five users can uncover 90% of usability issues.”

This well-known statement, derived from our educational experiences, serves as a cornerstone of user testing. By engaging five individuals to evaluate your interface, you can identify nearly all significant flaws. This approach fosters collaboration within teams, as developers can expect timely feedback, managers can easily estimate costs, and the product team gains confidence in their findings.

However, this statement has often been misinterpreted and taken out of context. To grasp its true meaning, we must delve into the origins of user testing, particularly the study titled “A Mathematical Model of the Finding of Usability Problems” by Jakob Nielsen in 1993.

Chapter 2: The Mathematical Foundation of Usability Testing

Nielsen aimed to formulate a mathematical equation to determine the number of users needed to discover all usability issues. His hypothesis suggested that as the number of testers increases, the likelihood of uncovering all problems also rises. Each user identifies both common problems (shared by multiple testers) and unique issues (found by a single user). With enough testers, the unique issues will diminish.

The equation he derived is:

P (i) = N(1-(1-L)i)

In this formula:

• P(i) represents the number of identified problems
• N is the total number of problems (usually an unknown value)
• L denotes the rate of encountered problems
• i stands for the number of testers

In Nielsen's study, he set the total number of problems to 20 and established a mean value of 0.31 for the encounter rate. Thus, when using five testers, the model predicts that approximately 85% of usability issues can be detected.

The theoretical basis for the "five users rule" is now clear. However, applying this rule in practice can be challenging. Nielsen highlighted that two factors affect the discovery of usability problems: the user must experience the issue, and the evaluator must recognize it as a problem.

For instance, experienced testers will identify more problems than novices. Therefore, if you're conducting user testing with less experienced individuals, you may need to increase the number of participants.

Additionally, the evaluator must be able to discern when an issue exists. Take Instagram as an example: users may point out that they cannot access notifications from their profile, yet this may not be recognized as a problem by UX designers. Overlooking such issues can lead to significant UI mistakes.

Moreover, user testing serves as a tool—alongside heuristic reviews—to pinpoint usability challenges. Nielsen suggested that using 15 testers would likely reveal nearly all flaws. However, it's more effective to conduct multiple sessions with fewer testers. If five expert users can identify 85% of problems, you can iterate based on their feedback, modify your design, and test again until no new insights emerge.

Chapter 3: Clarifying Misconceptions

So, why is there confusion surrounding the five-user guideline? While it’s essential to utilize experienced testers and conduct multiple sessions, Nielsen’s assertion that five users suffices in many scenarios is still valid.

The rationale behind this recommendation stems from the cost-benefit analysis of user testing. Given the constraints of time and budget in any project, testing with five users often yields the highest efficiency. With the potential to uncover up to 85% of usability issues, implementing changes before launching a product offers substantial value.

User testing costs can range from $2,000 to $10,000, depending on the interface, plus an additional $420 to $2,000 per user. In contrast, the benefits gained from resolving usability issues can amount to between $1,000 and $19,300 per identified problem.

While the advantages of user testing are significant, utilizing this tool effectively and in the right contexts is crucial for maximizing its efficiency. Remember, every UX/UI design technique is grounded in scientific principles that should be understood and respected.

The first video titled "UX Research and Usability Testing - Designer vs. Developer" discusses the roles of designers and developers in the user testing process.

The second video, "Explaining User Research, User Testing and Where it Fits Into the Design Process," provides insights into the importance of integrating user research within design workflows.