(NOTE on the image: I tried Midjourney’s query “like a boss,” and all I got was... Well, what I expected. So, instead, I specifically asked for “like a boss woman” because… Maybe it’s time? For more on WMD Bias in AI: https://www.uxforai.com/p/wmd-white-male-default-bias-ai)

For those UX folks who are endlessly complaining about needing more research, I have bad news for you: in the 20+ years of my UX work, there has never been a company or organization that did enough research for my needs. Until I took the time and effort to actively create a situation where I felt satisfied with the amount of research my team was doing and could, as a result, innovate with confidence and at the velocity I wanted to achieve.

In this article, Daria and I will share a (not-so) secret step-by-step strategy for creating user research opportunities to improve the odds of success for your  AI-driven projects. 

But before we get into it, I have a small favor to ask:

Today is the LAST day to vote. This is it – after today, I will not ask you again!

Thank you – much obliged. 

Whew! With that out of the way, I have a myth to bust, so please stay with me. 

It will be fun, I promise.

Blue Ocean/Red Ocean is Complete Bullsh*t

The way many Product and Business folks interpret the “Red Ocean/Blue Ocean” argument is that if the use case is well known and there are many existing competitors, we should do more user research because we are relying on small differences in functionality to compete with other sharks in the already “bloodied” waters of the “Red Ocean.” Conversely, if we are entering the mythical shark-free balmy waters of the “Blue Ocean,” we don’t need to do any research at all as we are entering a new market or our product is so new as to have no competition. In fact, user research in the case of “Blue Ocean” will be highly misleading as customers do not even understand the value of the amazing new technology you are bringing to the market. The typical story these Product and Business folks tell themselves to justify this bullsh*t is that Henry Ford once said that if he had asked his customers what they wanted, they would have said “a faster horse.” 

Funny thing – as Steve Glaveski points out, Henry actually gave them exactly what they asked for: a faster method of transportation that does not poop everywhere! https://www.collectivecampus.io/blog/henry-fords-customers-didnt-want-a-faster-horse

As we’ve covered throughout the last year of this column, AI is a fundamentally different, orders-of-magnitude technological advancement. AI-driven products are poised to radically change the nature of SaaS software and the relationship between consumer and system. Finally, AI-driven products are trained, not programmed. In other words, AI is the very definition of “Blue Ocean.”

It has to be the right research, of course (more on this below), but research is the only thing we have that reduces risk and ensures confident innovation. 

As Winston Churchill so eloquently said: 

No one (least of all me!) will argue user research is a perfect tool. It’s a pain in the arse in the best of circumstances. It is time-consuming, expensive, and tedious. 

Most people do it wrong. 

However,

Allow me to demonstrate with a little anecdote. 

Once upon a time, I worked with this AI-driven startup, where the Big Boss was quite fond of the “Blue Ocean” metaphor. After we filed a few patents together in the first few weeks, the honeymoon was officially over, and we were on to a hard stretch of endless requirements churn. Whenever the Big Boss would want to do a complete product pivot (every few hours and twice on Sundays), my response would inevitably be, “OK, let me update the mockup and do some user testing.” After a short while, the Big Boss started to respond with a retort that included some clever variation of his new favorite saying: “Mockup-F*ckup.” (You see, like our ex-resident, he was fond of certain types of word games, too.) It quickly became apparent that the Big Boss genuinely did NOT want to hear what the potential customers had to say. He often said as much to the team, I quote: “This is Blue Ocean! I graduated with honors from the very top schools. Therefore, I know better than the customers what they want.”

The first few times, I must admit I was speechless. But after doing some less-than-authorized user testing, it became clear that Big Boss was actually really smart.

I mean, genuinely brilliant, above and beyond.

Unfortunately, as it turned out, he was a bit too smart for his own good: most people had no chance to understand or be able to use what we were building… Without a Ph.D. from one of the best schools. Despite my intense reservations, I communicated these findings to him, carefully packaged in the most factual matter possible, as scientific a set of research findings as I had ever delivered in my entire career. Fortunately for my sanity (but, let’s be honest, somewhat unfortunately for my pocketbook), we mutually decided to part ways.

However, the story did not end there. About two years after I left, I ran into the Big Boss at a conference. To my complete and utter astonishment, he told me that it was unfortunate that we parted… And that he was now going back to my original designs! Predictably and cordially, I wished him the best of luck without me, but my curiosity was piqued: What the hell happened to his Blue Ocean? After reconnecting with a few (very, very few) staff members still remaining from two years ago, I found out why the Big Boss had a change of heart: It turned out that despite having a whole crew of salespeople, they did not sell ANY of his Blue Ocean product. 

Not a single license.

So there you go:

What kind of Research do you need for AI-Driven Projects?

Steve Jobs is reported to have said: 

I agree! Focus groups suck. 

That is why I never did any in my 20+ years.

However, Steve’s words should never be an excuse for not doing any research whatsoever. Quite the opposite! It should be your license to: 

This is why I recommend deprioritizing or outright stopping the following research programs and methods:

1. Focus Groups

2. Research with Synthetic Users (https://www.uxforai.com/p/navigating-the-abyss-the-dark-side-of-synthetic-ai-user-research-tools)

3. Usability Studies (https://www.uxforai.com/p/utm-source-rss-utm-medium-rss-utm-campaign-why-we-dont-do-mobile-usability-tests)

4. Personas

5. Surveys

6. NPS

Instead, for cutting-edge AI-driven projects, invest your time and money into:

1. RITE studies based on click-through prototypes of specific narrow use cases (see https://www.uxforai.com/p/need-storyboard-ai-project-draw for Part 1 of our Storyboarding for AI series)

2. Co-creation exercises

3. Light-weight ethnography and hallway conversations, especially at industry conferences.

4. Roles

5. Vision Prototyping 

6. “Art of the possible” research with developers and data scientists as well as AI-driven POCs.

I will not dwell into these in detail here – write to me if you are interested in learning more about any of these or, better yet:

1. Vote for my UX for AI workshop at the next SXSW: https://panelpicker.sxsw.com/vote/151357. (You have done that already, right? I’m just making sure… Cause today is the last day and all…)

2. Attend my next UX for AI workshop at UXStrat Boulder, CO, on September 9th to practice drawing storyboards and doing other exercises, such as Digital Twin, essential for success on your next AI project. Register here: https://strat.events/usa/tickets 

How much research is enough?

Before you create your research strategy, define the end state: How do you know when you will be successful? (That is generally an excellent question to ask for any endeavor, UX or otherwise. It’s a GREAT question we don’t ask nearly enough!) 

When it comes to AI-driven projects, there are fundamentally after three main goals:

1. Ongoing in-flight tactical delivery of critical initiatives in plenty of time to iterate ahead of development.

2. Ongoing vision prototype guidance for strategic initiatives.

3. Emergency decision support: If plan B is needed (e.g., a critical feature is entirely blocked by an unexpected dev blocker or a miss on the design or planning side), you can confidently design, user-test, and deliver an alternative solution in under 24 hours.

It turns out that for a typical team size of 250-500 total developers and today’s typical below-bare-minimum of 12-15 designers, one research coordinator, and one research assistant, it comes down to:

To accomplish this without killing yourself, you need to put your recruiting on autopilot by creating a Customer Research Panel.

Customer Research Panel: User Research on Autopilot

The best way to have enough research is to have 6-8 one-hour conversations with your customers per week. The easiest way to accomplish this is to put the customer conversations on autopilot. Here’s a detailed step-by-step guide on how to do this.

1. Set up monthly recurring meetings with your “most desirable” customers

In my experience, having monthly conversations with your customers works best. It’s enough time for new prototypes to show up, so it keeps conversations fresh and interesting, while, at the same time, you are not annoying your customers unnecessarily. Assuming monthly meetings, you will need: 8 meetings per week * a meeting every 4 weeks = 32 customers.

Now 32 feels like a lot of customers. It’s true. You will not get there overnight. But if you are diligent and committed, even starting from scratch, you will achieve critical mass after 3-4 months. 

Diligence and Commitment are why you need a dedicated customer-research assistant to pre-screen customers, set up and maintain customer conversations and update records, confirm meetings, cancel or reschedule meetings whenever a customer cannot make it, add and remove customers from your program, etc. Do not underestimate the importance of this task! This is a tedious job that requires skill and tenacity and it forms the backbone of your program. 

2. Figure out what you mean by “Most Desirable” Customers

What you mean by “most desirable” will likely change radically over time, but you have to start somewhere. At first, just begin with customers who complain the most – they will be the most likely to pick up the phone and talk to you if you are willing to listen. Simply let them vent for 30 minutes and record everything they say. Then let them try out some innovative prototypes. They will give you unvarnished, honest feedback. They will also likely come back because they feel valued and cared for and will likely remain loyal customers for years to come. 

Make sure you report your findings to your x-functional team and share the team’s response with the customer's CSM. It's good to let customers vent, but they will want to know whether you're actually doing anything with their feedback. You can work with your CSM/PM to agree on how to communicate with customers about the status of their requests. Make it clear from the start that not all requests can or will be handled. You have to be honest about it, and you also have to show them how their input is actually impacting the product so they don't lose interest or get even angrier than they were in the first place -- or so disillusioned that they drop your product. It’s a balance, but most people react positively when someone is willing to attentively listen to their concerns. So start with listening – it’s a win-win! 

Over time, work with the business and sales to figure out who are the customers in your “sweet spot” and aggressively create relationships with those customers. Leadership support is crucial – you want the entrie organization to be on the same page. Setting up research is the key to long-term success, so you want the leadership to help you secure support from Sales and Account Executive leaders. 

Be proactive in reaching out after securing verbal commitment on the first research call. Let your dedicated research assistant handle the recruiting and scheduling. Avoid having multiple people from your team reaching out at the same time!

3. Dig your well before you are thirsty.

Just so. Make sure your research program delivers a full 360-degree visibility by engaging with internal customers, support engineers, salespeople, and field CTOs willing to give you a piece of their mind. Often, when you have to design and test a plan B approach quickly, internal customers are your best bet because it’s hard to set up more than 1-2 external customer meetings in a single 24 hours, but it’s not unusual to be able to quickly set up 3-5 meetings with your internal customers, provided you have taken the time to build and maintain those relationships. I suggest that to complement your 32 external customers, you will need additional regular monthly meetings with 20-25 internal customers. (NOTE: if you can't create a dedicated call loop with these internal customers, it often works to ask for time in one of their regular team syncs or another meeting that already exists.) Again, having that dedicated research assistant is critical to keeping all these meetings straight.

4. Learn about the People on your panel.

Your dedicated research coordinator should invest the time to learn about the people in your panel and write down critical findings about your 32 external customers and 20-25 internal customers. That includes their role, company, team size, working hours, products they purchased from you, how often they use your system, their family situation, etc… And for G*d’s sake, note how to pronounce their name correctly!

Maintain detailed notes about what you showed to each of your customers the last time you met with them. If they don’t care to hear about product line X but really care about product Y, and complain about product line Z, note that! Then, never bother them with product X design proposals while making sure they are the first to hear about new product Y features and product Z improvements. 

Now, it seems like you need expensive software to do this. Well, it can’t hurt. However, we are not talking about 1000s of customers – so you don’t really need a fancy CRM. You are talking only about a panel of roughly 50 people, thus a simple Google sheet or Excel spreadsheet will do just fine.

You don’t need someone super-senior (although, once again, it can’t hurt.) A high-performing mid-level person of the right temperament and level of dedication will probably do fine. Again, the choice of your research coordinator here will be critical to success. 

5. Establish the Right Process

If you want the right Product to be the outcome of your research, in addition to having the right People on your research panel, the right Process is also critical (this is known as the 3Ps principle). Here’s one way you can manage it:

• Kick off your research for the week with a 10-15 minute Monday morning research standup meeting.

• Briefly discuss who has what new things to research and how long they will need to talk to each customer. 

• Briefly discuss deadlines for project delivery, how big of a change you are testing, and the level of urgency. 

• Briefly discuss who you will be meeting with this week. Prioritize according to the customer’s preferences and what they have already seen and given you feedback on. Don’t show them the same thing again!

• For every customer meeting on the schedule, write down a simple agenda right there in the standup meeting. A typical weekly plan might look something like this:

  • Monday AM: George Stephanolopis, ACME, Inc. DevOps

    • Sally: Project Y drop-down vs. auto-complete (15 min)

    • Sam: Project Z complete report workflow (30 min) URGENT 

    • Greg: AI-driven vision discussion (15 min) OPTIONAL

    • NOTE: No Project X!

  • Monday PM: Stephanie Logan, LevelUp, Inc. Security Specialist

    • Kim: Project X redesign (40 min)

    • Sam: Project Z complete report workflow (20 min) URGENT 

  • Tuesday… Etc.

• Have a weekly meeting with your project team where you share what you found, how you iterated on the design as a result, and what you plan to do next.

6. Train ALL UX Designers to do Research

I never understood the intense desire of some UX managers to pigeonhole UX people into designers and researchers. Not only is this inefficient, but it is just plain impractical in the age of fast, continuous design and development of AI-driven products and squeezed, bare-bones, value-driven UX teams. Sure, some people will be better at one thing or another and gravitate accordingly, but everyone should be able to walk and chew gum at the same time. I split my teams into generalists (UX designers) and Visual/DesignOps designers:

• All generalists should absolutely do their own stunts: perform their research and obtain direct customer feedback on their designs.

• Visual/DesignOps people can also do so optionally but are not required to participate.

• The research coordinator helps tie everything together. 

End of story. 

One particularly pertinent anecdote comes to mind from a F100 retailer I used to work with. I had a very intense research session with a customer where they described how much they hated a particular proposed design change, and instead, we brainstormed and co-created this incredible alternative that not only fit their workflow like a glove but was highly congruent with previous feedback and, as a bonus, provided the company with the innovative patent-worthy idea! When I, elated and inspired, exited the research lab to discuss the uniquely dynamic exchange of ideas culminating in such stunning success…  I found my designer did not bother to attend the session but instead went to eat free bagels. 

No.

Nope.

Watching the proceedings from the one-way mirror (or, even worse, watching the recording afterward!) is not the same as being there. The moments of creation are rare and precious gifts from the gods. The drops of inspiration that sometimes come to us mere mortals are the drops of nectar of wisdom, the divine mead that was rumored to be brewed from the blood of Kvasir (this is one of my favorite Viking stories – Google it sometime.) 

So:

Be there. 

Be present. 

Be ready to receive the inspiration. 

… Or be prepared to go without and be forever confined to mediocrity. 

Sounds harsh? OK then – show me something cool you created without a serious investment of passion, empathy, and effort. 

Go ahead. I’ll wait. 

7. Remain flexible

As mentioned in point 5 above, remain nimble and dynamic and stay alert for new insights. 

Don’t have anything new to show? Ask your customers how things went this week, and let them show you how they solved problems or added value using your product. What did they struggle with? What do they find most useful? Ask them to demonstrate. Watch like a hawk. Ask questions. Brainstorm ideas together. This is not rocket surgery, but it takes time and practice to do well. (https://sensible.com/rocket-surgery-made-easy/)

Have a conflict? Be ready with the rest of your team to jump in with alternative project research. All designers and researchers who are free should attend all of the research sessions and be ready to jump in with questions and designs. (Invite PMs and Dev Leads, too!) Have the research coordinator MC the whole thing and have a Plan B ready to go, just in case things go sideways.

Research is the lifeblood of innovation.

The meat of the UX work. 

Just do it. We just showed you how. Take these ideas and make your way.

Any questions?

Greg Nudelman and Daria Kempka (Contributing Editor)

P.S. Our fabulous full-day workshop is coming up on September 9th, 2024, at UXStrat in Boulder. It WILL sell out like our previous workshops at UXStrat 2023, UX Copenhagen, UXLx in Lisbon, and Rosenfeld Media workshop online. If you found this article useful, don’t miss out on critical learnings you will need in your next project and get your ticket now: https://strat.events/usa/tickets 

P.P.S. And please (this one is absolutely, positively, the last time ever this year we will ask…) remember to vote for Greg’s session at SXSW: https://panelpicker.sxsw.com/vote/151357

You’ve done it this time, right?

Thank you!