User testing is a critical step in launching successful conversational experiences for the design team at this global bank. But testing with static prototypes limited the efficiency of the user testing process and the value of its findings.
User research could only do moderated testing because someone had to guide users through point-and-click mockups. Test sessions were time- and labor-intensive, reducing the feedback available to designers.
What’s more, static prototypes didn’t provide a true representation of the user experience. Designers made decisions based on hypotheticals and “what-ifs” since testers could only imagine what they would do, type, or say.
“How can you do accurate user testing if you don’t have something customers can interact with?” asks Gina Riley, VP, User Experience and Conversation Design Lead. “You’re basing design decisions on what a user might do, not on what they actually did.”
Designers didn’t get real-world feedback until six weeks after a feature was pushed to production, which restricted the speed and frequency of design iterations. In the meantime, the bank risked frustrating customers with a confusing or incomplete assistant experience.
Production-Like Prototypes Transform User Testing
When her team adopted Voiceflow, Gina leveraged the platform’s high-fidelity prototyping capabilities to enhance user testing in two game-changing ways.
First, when a design is ready for user testing, designers simply click a button and Voiceflow builds a dynamic, shareable prototype that simulates a real assistant experience. “It’s as close to someone actually using it in production as you can get,” says Gina.
Second, the user research team uses the interactive prototype to do unmoderated user tests. Giving testers free rein provides a “more accurate read” of user engagement and can uncover more issues with a design’s usability. (They still do moderated testing if it serves the research objective.)
Thanks to these advances, Gina says her team “has seen a huge improvement in the efficiency and accuracy of our research.”
More Efficient, Scalable User Testing
Since adjusting their approach to user testing, test sessions are shorter and less tedious, with researchers observing rather than prompting users. Increased testing efficiency allows research to run more tests, capture more learnings, and evaluate more design options.
For example, in the past, research might have conducted 20 moderated A/B tests to evaluate designs for a new feature. Now, they can easily run 50 unmoderated tests to collect richer data from a larger, more diverse group of participants.
Designers can even update prototypes on the fly based on early insights into how testers interact with the assistant.
“This way of running research really expedites the process,” Gina reports.
Making the Most of NLU Data
Gina’s team takes advantage of Voiceflow’s built-in NLU Manager to increase the accuracy of user testing.
When evaluating a feature enhancement, designers set up a realistic testing experience by creating prototypes with actual NLU data, imported from the production system.
If designers are creating a new feature and no conversation data exists, they can leverage user inputs during testing to build a preliminary NLU model. Gina says this reduces a lot of upfront guesswork because “we get a baseline for the first iteration of a design.”
In both cases, user testing helps optimize the NLU model which, in turn, improves assistant performance. Voiceflow’s exports and integrations allow designers to share production-ready NLU data with data science—an outcome Gina calls “win-win.”
Designing with Data—and Confidence
Testing with Voiceflow prototypes has empowered Gina’s team to adopt a more data-driven design approach.
The platform’s Transcripts functionality automatically records each tester’s interactions with the prototype. These transcripts help designers determine where the assistant successfully helped users and which components of the conversation were confusing or frustrating.
Designers use insights from transcripts and research readouts to improve conversational experiences.
“We’re able to make more informed decisions on our designs,” Gina says, “because the data you get from user testing is more robust and accurate.”
Designers also have detailed data to explain their design choices to product owners and other stakeholders.
Faster Workflow, Better Assistant
Ultimately, Gina believes the new user testing capabilities will help her team deliver better products faster.
The time from starting a design through providing a prototype to user research is half what it once was. In fact, conversation design now has a standing spot on user testing’s monthly calendar because they can turn out prototypes so rapidly with Voiceflow.
In addition, designers get real user feedback early in the design process—well before any enhancement to the assistant goes live. They can iterate designs prior to developer handoff when it’s easier and less costly to make changes. And when a new feature launches, customers enjoy a quality experience backed by accurate user testing.
Best Practices from this Conversational AI Team
- Take a data-driven design approach. Intuition and assumptions don’t cut it anymore. Conversation designers need to use data to improve the customer experience, justify design choices, and elevate the status of a young, but growing discipline.
- Conduct user testing with dynamic prototypes. The best learnings come from real users interacting with real experiences. For accurate, actionable, bias-free feedback test your designs with dynamic, production-like prototypes instead of static, point-and-click mockups.
- Get familiar with NLU data. Learn more about the intents, entities, and utterances that make up your assistant’s NLU model. Understanding NLU management will help you anticipate customer interactions and is key to becoming a more versatile “full stack” conversation designer.
- Observe user tests. There’s no substitute for watching—or hearing—people engage with your assistant. If you see something that needs changing, like an intent to add or a typo to fix, you can update the prototype midstream and keep testing on track.
- Even lean teams can do user testing. You don’t need a user research organization to benefit from user testing. Take advantage of services like UserZoom, Userbrain, or UserTesting to collect quality feedback on your design prototypes.
