Human Factors Engineering

From Plinko to Precision

A Methodical Process to Capture Actionable Root Cause

Plinko is a game featured on The Price is Right daytime game show but what does it have to do with human factors usability studies? Emily Alexander uses Plinko to explain her approach to working through root causing use errors observed during studies.

About Emily Alexander

Emily Alexander is a senior consultant at Fathom Consulting who specializes in human factors engineering. With more than a decade of experience in the medical device industry, she has led the usability testing efforts for numerous devices. She has been part of 700+ individual study sessions and her work has resulted in successful FDA approvals of 18 products and counting. Emily is most passionate about how study moderation directly impacts reporting and analysis of residual risk.

Emily holds a B.S. mechanical engineering from Iowa State University and a M.S. medical device innovation from the University of Minnesota. Prior to her career in human factors, she gained experience in the medical device industry through roles at Medtronic and Abbott, supporting Operational Quality and Design Assurance.

What is Plinko?

Need a refresher on how to play Plinko or you’ve never seen it played? On The Price is Right, contestants are given “Plinko chips” which can be used on the “Plinko board.” The contestant holds one chip at a time at the top of the vertical Plinko pegboard, releasing it to let it fall through the slots in between pegs. Chips land in a slot at the bottom of the board corresponding to a prize amount.

However, every so often, a Plinko chip may become stuck or take an unexpected path to the bottom of the board, not unlike the path of a root cause conversation with a study participant. This engaging analogy helps human factors practitioners visualize root cause conversations with a fresh perspective on a routine aspect of HFE work. We'll explain how to set up the Plinko board for a study and how to play the game to win, regardless of a study being formative or summative.

Before Study

Step 1: Identify your Plinko slots

Identifying possible root causes ahead of time can help you feel more confident as the moderator going into your study. By knowing what use errors you might observe and imagining where the debrief discussion might go, you can ensure you ask more effective questions while the participant is still in the room.

The first step is to define what your root cause categories will be. Using the Plinko board analogy, these categories will be located at the bottom of the board with each root cause representing a separate slot.

There are a few categories that will be possible for most, if not all, devices:

Device design (physical design)
Intended use not intuitive
Not designed for user’s physical needs
Study artifact
Study design not representative
User performed in different manner due to being in a study
Negative transfer
Intentional misuse
User Oops
Lapse
Slip
Mistake

Device design (physical design)
Intended use not intuitive
Not designed for user’s physical needs
Study artifact
Study design not representative
User performed in different manner due to being in a study

Negative transfer
Intentional misuse
User Oops
Lapse
Slip
Mistake

Next, it’s important to consider some root cause categories that are specific to the design you are testing. For example, if the device being tested has a GUI or screen as part of the design, it can be helpful to break this out from the overall device design listed above to better inform design changes later on. In formative testing, a device might be tested prior to an IFU or other labeling being ready, so a moderator would not need to consider labeling specific root causes.

Device-specific categories might include:

Device design (GUI design)
Intended path not intuitive
Distracted by design aspect (color, font, size, etc)
Training
Content was not included in training
Information provided in training was not clear
Labeling
Organization not intuitive
Instructions not clear
Distracted by design aspect (color, font, size, etc)
Images not meeting user’s needs

Device design (GUI design)
Intended path not intuitive
Distracted by design aspect
(color, font, size, etc.)
Training
Content was not included in training
Information provided in training was not clear

Labeling
Organization not intuitive
Instructions not clear
Distracted by design aspect
(color, font, size, etc.)
Images not meeting user’s needs

Step 2: Understand Plinko slots in terms of the device being studied

Your Plinko board is taking shape and the next step in study preparation is to envision how your chips might fall into each slot at the bottom. Review each of the potential root causes and consider what might cause a participant to provide that root cause as it relates to your device and study.

Examples to consider:

What could create negative transfer during the study? Are the study participants being recruited for experience with a similar product or are there other products users are familiar with for a similar procedure?
What aspects of the device design (physical or GUI) were not intuitive to your team when preparing for the study? These are likely to cause participants difficulty as well.

What about the simulated setup could result in study artifact and can it be avoided ahead of time? For a study focused on an at-home injection device that must be stored in fridge, have a mini-fridge as part of study setup. If using a product that requires blood flow, such as a dialysis machine, have simulated blood or colored liquid to differentiate fluids to be more realistic.

What could create negative transfer during the study? Are the study participants being recruited for experience with a similar product or are there other products users are familiar with for a similar procedure?
What aspects of the device design (physical or GUI) were not intuitive to your team when preparing for the study? These are likely to cause participants difficulty as well.

What about the simulated setup could result in study artifact and can it be avoided ahead of time? For a study focused on an at-home injection device that must be stored in fridge, have a mini-fridge as part of study setup. If using a product that requires blood flow, such as a dialysis machine, have simulated blood or colored liquid to differentiate fluids to be more realistic.

Step 3: Review risk documentation for correlation with Plinko slots

It’s important to familiarize yourself with potential use errors that have been identified in the Use Related Risk Analysis (URRA). By pre-evaluating use errors listed in the risk documentation, you can determine which use errors are likely to fall into certain categories and this can help inspire what root cause questions to ask ahead of time.

Additionally, depending on the stage of development, risk mitigations might or might not be identified prior to the study. If risk mitigations have been included in the URRA but not yet studied, these might represent potential root causes for use errors in the study.

During Study

Step 1: Play Plinko by conducting root cause

Your participant has just given you several Plinko chips in the form of observed use errors during the study and it’s time to root cause. If you’ve seen Plinko on The Price is Right, you know the chips might bounce around in unexpected ways. Root cause debrief during a usability study can leave a moderator with a similar bouncing feeling.

The questions you ask during root cause represent the pegs on the Plinko board. When a chip hits the pegs, it changes direction. Do this enough and your Plinko chip will land in a slot at the bottom of the board, the ultimate goal for root cause.

Plinko board with chip partially down board

You likely have a set list of questions that you draw on to start any root cause discussion. In addition to that list, you may want to consider questions which help identify the root cause category slot responsible for the use error being explored. Start with open-ended questions and allow the participant to narrow the focus. Open ended questions allow the participant to lead the discussion, meaning your chip follows a natural path down the Plinko board rather than being forced into a predetermined root cause slot.

The example questions below align with a root cause discussion where something in the labeling needs updating.

Question

Purpose

What would you change about the design to remind you to do [task with use error]?

When the participant points to which part of the design they would change, they are identifying the design aspect that failed them.

‍Participant points to the IFU > Plinko slots narrow to root cause categories related to labeling

What specifically would you change about [the labeling] to remind you to do [task with use error]?

Once the board has been narrowed down to a few slots, it’s time to identify which slot specifically represents the reason the participant made the error. Again, by identifying what they would specifically change, the participant is highlighting the design aspect that failed them.

‍Participant states they couldn’t locate the instruction needed to do [task with use error] > Plinko slot identified: Labeling – Organization not intuitive

Does the way that instruction is written make sense now that you’ve read it?

There are a lot of design aspects like color, font, printed size for the instructions. Are those design aspects okay or do they need to be changed?

Confirm the other related root cause categories are out of play and didn’t influence the observed use error. Asking questions to rule out other root causes is important because, as seen in the real Plinko game, chips can move in unexpected ways and the participant’s provided root cause might shift.

‍Participant confirms the location as the issue > Plinko slots not in play: Labeling – Instructions not clear, Design aspects

If you could move that instruction anywhere, where would you put it so users are reminded to do [task with use error]?

Ask the participant how they would update the design to improve usability for the task. If the participant confirms a design change would have helped them, they have confirmed the root cause category.

‍Participant states where they’d relocate the instruction for [task with use error] > Plinko slot confirmed: Labeling – Organization not intuitive

It’s important to keep an open mind during the study and consider root causes that haven’t been pre-identified. While the root cause categories suggested here will cover most root causes provided by participants in studies, there’s always the chance that a new and unexpected root cause appears.

For that reason, it may be helpful to include a “wild card” slot as a reminder that sometimes participants come out of left field with their responses and provide a root cause that doesn’t align with what you planned pre-study.

Step 2: Confirm the Plinko chip landed

Ask yourself: Why did the participant make this error?

If you can’t answer this question using feedback the participant has provided, your Plinko chip hasn’t yet successfully landed in a slot and you need to continuing asking root cause questions. Remember, Plinko chips are unpredictable so you shouldn’t assume you know where the chip will land. Instead, rely on what the participant is telling you. Their feedback should move the Plinko chip down the board, not your indirect analysis.

If you’re unsure or even think you have identified the root cause, it doesn’t hurt to ask another question. Too often, root cause stops early meaning formative studies end without actionable changes and summative studies end without the details necessary to assess residual risk.

Step 3: Gather residual risk rationale as needed

During a formative study, gathering details to build risk rationale is an optional step and might be skipped if early in the design process. As the device gets further in the product development cycle, and especially during a summative study, it helps to build a case as to why design changes might be avoided.

There are many aspects to consider when building residual risk rationale. The following is meant to be a starting place as it relates to the root cause categories defined here.

Root cause categories

Residual risk exploration

Device design (Physical/GUI)
Training
Negative transfer
User Oops

While the participant made an error associated with one of these root cause categories, it can be helpful to present any supporting labeling for the task.

If the participant did not reference labeling during use, have them redo the task referencing labeling intended to inform users about that step. If they are successful, this indicates the device manufacturer provided adequate instruction on how to correctly complete the task. Discuss with the participant if anything about the labeling could be changed to ensure users read it prior to use. Participants will often state why they chose not to read instructions which can then be evaluated to determine if the manufacturer could have done more to make sure the labeling was reviewed.

If the participant did reference labeling during use, this indicates there could also be an additional root cause associated with labeling for the use error being explored.

Intentional misuse

Unfortunately, users choose to intentionally use products incorrectly even when they understand what they should do. Asking questions to have the participant explain how to correctly use the device or presenting labeling and asking for their interpretation can help build a case that the manufacturer did everything they could to inform the user of correct use, even if the user chose not to.

Labeling

If the participant referenced labeling during the study and indicated it was the reason they had a use error, it’s helpful to understand to what extend they read the labeling. Some participants read every word while others skim and it can be impossible to know how much was read when it’s somewhere in-between. If they did not fully read the labeling and/or read portions without the instruction related to the use error being explored, ask why not. Follow-up questions such as “if you were using this product for the first time in real life, would you have read the instructions more thoroughly?” or “what would you have done if this was real life and [the use error] occurred?” can be helpful to either update the root cause category or build further rationale that the patient/user would have been okay.

Study artifact

For study artifacts related to an unrepresentative study design, ask questions about how the study setup could have been improved. Once the participant highlights what should have been different (example: providing a fridge for a study where the medication is stored cold), ask questions about how that would have impacted their behavior.

For study artifacts where a participant performed in a certain way because they were in a study environment, discussion should focus on how their behavior would differ in a real-life situation.

In either case, if participants know the correct behavior and can attribute that to something missing in the design setup or the fact that they were in a study, residual risk is lowered because they have indicated they would do the correct actions in a real-life environment.

While summative studies are completed with the intent of evaluating a final design, study results might highlight the need to make further changes. Following a summative study, all use errors and associated root causes should be reviewed for necessary design changes.

After Study

If your Plinko chips landed in slots at the bottom of the Plinko board, you’re a winner!

After a Formative Study: Inform Actionable Change

Plinko chips that landed in slots at the bottom of the Plinko board during a formative study represent opportunities for actionable change and improvements, especially before a summative study.

Root cause categories

Post-study action

Device design (Physical/GUI)
Labeling
Training
User Oops

Design change needed

Negative transfer
Intentional misuse
User Oops

Design change may be needed

Study artifact

Study design change needed

After a Summative Study: Inform Residual Risk

Review of the information gathered during the study will either support residual risk rationale in a summative report or highlight the need for further design changes to improve usability. While summative studies are run with the intent of being the final usability study prior to regulatory submission, it’s important to not ignore user feedback, especially if it highlights improvements that could be made to lower residual risk for patients and/or users.

A Plinko chip in a slot at the bottom of a Plinko board

Conclusion

The Plinko methodology presented here is a tool for human factors practitioners and study moderators to work through root causing use errors observed during a human factors study. By using this framework, you will be a winner every time you play Human Factors Plinko.

And remember, "Help control the pet population. Have your pets spayed or neutered."

About Fathom

Fathom is a business and design consulting firm located in Minneapolis, MN. We’re research-driven. Relentlessly collaborative. Flexible by nature. And we’re committed to providing practical, implementable solutions for your toughest business challenges—solutions informed by the people who will be most affected by them.