When it comes to design, especially that of AI-driven products, I try to avoid running usability tests. Usability tests, as they are popularly conducted, are a waste of time and resources and, in the vast majority of cases, fail to create a better product.

Instead, I focus most of my research time and budget on RITE (Rapid Iterative Testing and Evaluation) studies: the only methodology that I've actually experienced in real life yielding more delightful, usable, and successful AI-driven products in less time.

RITE Study vs. Usability Test

Just what is the difference between a usability test and a RITE study?

Usability tests, as they are popularly run, involve testing by 8-10 participants of a fairly elaborate prototype using a set of pre-defined tasks in a laboratory setting. There are no prototype changes during the study and the result of the test is a usability report outlining issues and recommendations.

In contrast,

RITE studies are conducted using 9-12 participants in 3-4 rounds, with 3 people per round. The critical difference is that in between the rounds, the team takes the time to update the prototype and fix the issues discovered during the previous day’s testing. To enable that, I usually employ the simplest possible prototype for the job, usually sticky notes or the simplest possible Figma click-through.

Those practitioners who love traditional usability tests might object that, in essence, a RITE study is just like 3-4 smaller usability tests strung together and that the differences between the two are just semantics.

I disagree.

The difference is where you put your focus.

Let me explain what I mean by comparing RITE studies and usability tests (as they are typically conducted) in the following 3 points:

#1:  RITE studies form the core of the design process. Usability tests are often treated as QA.

In my experience, I find that most companies typically view usability tests as an optional, expensive undertaking. There is a good reason for this: usability tests run by a third-party contractor cost in excess of USD $20,000 per round (plus facility fees, participant fees, prototype creation costs, etc.) For that reason,

This in turn encourages project managers (and other team members) to misunderstand the whole purpose of the exercise and to treat usability tests as some sort of an elaborate QA process.

Testing late in the game is not effective in helping the team bring about a better AI-driven product. The entire point of the usability testing is to improve the design. Instead, usability tests are conducted too late in the process to affect the very thing it is supposed to fix.

While the UI design for AI-driven products is usually fairly simple, most issues that come up in testing are more fundamental, deeper, and wider in scope than a typical non-AI project. For example, the use case might be totally wrong. Or there are not enough data inputs to create a robust model, so AI cannot predict the right variable. Or the AI model is too conservative or too aggressive for real-life scenarios… These are just some of the AI-specific issues that can tank your project. We discussed some of these pitfalls here: https://www.uxforai.com/p/stop-f-cking-up-ai-projects-avoid-these-5-pitfalls 

Unfortunately, by the time the typical usability test is conducted, many of these deeper fundamental issues are already “baked into” the design and can’t be changed.

This is the fundamental drawback of conducting usability tests, and it is one of the main reasons I don’t conduct them in the middle of the design process.

Instead, I conduct light-weight Agile RITE studies.

In contrast to a typical usability test, a RITE study is conducted as early as possible as part of the design process and is not a test. (Even the name is purposefully different: test vs. study.) “Study” implies that something will be learned as part of the process so that the AI-driven product is given a chance to evolve to a better state, even if it involves changing more fundamental aspects, such as the AI model and data, as described in the next section.

#2:  RITE studies demand the simplest appropriate prototypes that change rapidly. Usability tests often mean fancy rigid prototypes.

Typical  usability findings report is presented up the food chain and quoted many times over, encouraging elaborate video-taping and creation of the costly hi-fi prototype because the design is pretty baked at that point and “in case an executive might want to stop by” so the test needs to “look good.” Furthermore, most third-party usability test moderators tend to demand this. The fancy Figma prototype, complete with animated transitions, becomes bloated with features, costly, and complicated to change, losing sight of the primary purpose of the user-centered process: to create an AI-driven product that works.

I rarely build fancy, animated prototypes because the cost/benefit ratio is just too high. Instead, I find that simpler Figma click-throughs or, even better, sticky notes for AI-driven mobile applications (such as the ones we will be building as part of the upcoming UXStrat Workshop on September 9th: https://strat.events/usa/tickets) allow designers to quickly and inexpensively explore multiple design approaches while dispensing with elaborate camera equipment and other gadgets.

Code talks, bullsh*t walks. Until your design is in the code, it’s just a picture.

Your prototype needs to be only fancy enough to provide a creative solution and unblock development.

RITE study participants should be comfortable brainstorming valuable ideas that can be incorporated into the prototype because the design is not yet finalized. A rough prototype also allows many changes to be made on the fly, sometimes immediately after the first participant is done and before the next evaluator has a chance to see it, thus providing a more efficient, tighter iteration process.

The RITE focus is always on the solution. 

#3:  RITE Studies produce solutions. Usability tests produce reports.

Usability tests (as they are typically run) produce reports.  These reports contain vivid descriptions of usability horrors and best practices designed to help designers atone for the horrific transgressions.

The problem is that AI-driven product design is just too young to have much in the case of solid best practices. Instead, the best to be said after a given usability test is that “ChatGPT does it this way” or “AWS does it that way.” Thus

In a rapidly evolving  industry, designers need space to explore creative solutions.

And the best way to do that is to

A RITE study provides creative space for exploration coupled rapid user feedback focused on creating a solution that works. For this reason, I rarely videotape my studies or provide elaborate reports. At most, I create a FigJam/Miro/Mural board showing the design change progression of the prototype during the study, along with a few key insights that we, as a team, used along the way to pivot and arrive at the present improved design.

An important fringe benefit of RITE Studies

As an important fringe benefit,

The RITE approach is inherently Agile, making it a perfect fit for the Agile/SCRUM projects.  And because the RITE methodology is focused on solutions, rather than inflammatory reports, so it tends to be less adversarial: 

Have a cool idea? Let’s try it. Right Now.

When I work with my team, we communicate and brainstorm organically several times a week using the latest insights of the RITE study, so there is little need for a big usability report. The entire team is there — driving and experiencing the cross-functional design process together, in real-time, and everyone on the team is focused on coming up with creative solutions to tough problems.

Not on producing reports.

Usability studies and elaborate reports have their place. But the middle of an AI-driven product design process is not one of them.

When it comes to designing AI-driven products, don’t keep doing what you have always done. 

Instead, do the RITE thing.

Want to see this process in action and practice Rapid Iterative Testing and Evaluation for design on AI-driven products? Join me and Daria for a full-day UX for AI Workshop at UXStrat on September 9th:  https://strat.events/usa/tickets

Hope to see you there!

Greg

P.S. Our fabulous full-day workshop is on September 9th, 2024, at UXStrat in Boulder WILL sell out like our previous workshops at UXStrat 2023, UX Copenhagen, UXLx in Lisbon, and Rosenfeld Media workshop online. 85% of AI-driven projects fail. Don’t let your next project become a statistic. Get the critical UX for AI techniques and insights you will need to succeed in your next AI-driven project: https://strat.events/usa/tickets