The challenge: moving beyond basic chatbots
Turo was using a traditional decision-tree chatbot that had been acquired by Zoom. While their operations team had done an impressive job optimizing the tool, its limitations were becoming increasingly apparent:
- The chatbot could only handle pre-programmed decision trees.
- Users were frustrated by its rigid, mechanical responses.
- It was difficult to measure true contact deflection rates
- The solution couldn't scale with Turo's growing global user base
"We were very limited in terms of capabilities," says Biancamaria. "The chatbot could not handle anything outside of the built-in decision tree."
Finding the right solution: a strategic evaluation
At the beginning of 2023, Benjamin and Eric Comellas, Senior Director of Engineering at Turo, were tasked with analyzing AI opportunities for Turo. Through their evaluation, they identified three distinct categories of potential solutions:
1. Off-the-shelf Chatbot Solutions: These were specialized platforms designed for specific use cases like e-commerce. While easy to implement, they were not flexible enough for Turo's diverse needs. "You can't do anything outside of that specific use case," explains Biancamaria. "It's very constrained."
2. Pure LLM Providers: Platforms like Amazon Bedrock offered direct access to AI models. While providing maximum flexibility, these would require significant upfront investment and engineering resources to build and maintain a complete solution.
3. AI Orchestration Platforms: The middle ground between specialized chatbots and pure LLM providers. These platforms, including Voiceflow, offered flexibility while providing the necessary infrastructure and tools to build quickly.
After extensive evaluation, including discussions with major providers like Google, Turo chose Voiceflow for several key reasons:
- Flexibility with speed: They could leverage existing infrastructure while maintaining the ability to customize and expand beyond just chatbot use cases.
- Quick implementation: The visual interface and support enabled rapid development without requiring investment in additional engineering resources while providing high confidence in time to market.
- Strong partnership: The Voiceflow team provided dedicated support, including bi-weekly meetings and a dedicated Slack channel for rapid problem-solving.
- Future-proof: The extensibility of the platform would allow Turo to easily scale to future use cases, such as automating agent emails.
- Cost-effective: Compared to similar solutions, Voiceflow offered competitive pricing with clearer terms.
"We wanted some flexibility, but not to invest everything right now," explains Biancamaria. "The support from Voiceflow at all layers, from engineering to implementation, really made the difference."
The vision: a proactive customer support agent
The Turo team had ambitious goals for their agent to proactively solve customer support needs in the future – but they knew the most effective implementations take a phased approach.
Rather than rushing to replace their existing system, Turo's team planned to build out their agent in three distinct phases:
1. Informational phase: Build an extremely accurate FAQ agent with low tolerance for hallucination
2. Transactional phase: Connect the agent to user data and enable it to answer specific questions about trips and claims
3. Problem-solving phase: Create an agent that can take actions and proactively assist customers
Their ultimate north star? An AI agent that could say, "I see your flight was delayed. Let me move your trip back an hour or two," without requiring customer intervention.
Aligning the team around an inspirational vision created the buy-in necessary to implement the first phase and get their AI agent project off the ground.
The solution: rapid implementation of phase one
As a proof of concept, Turo chose to launch their initial agent on their "How It Works" page, which is focused on new user education and support. This approach allowed them to validate their AI capabilities while minimizing risk.
"Instead of shipping the first chatbot to replace our existing solution, we shipped it somewhere else in the product," explains Biancamaria. "This allowed us to confirm we could answer questions accurately, that we could escalate when needed, and gain confidence in our approach. Answer accuracy was critical in getting buy-in from other teams, including Customer Support and Legal."
The team moved quickly through seven iterations in two months, following a structured approach:
1. Starting simple: Beginning with basic inquiry and response capabilities based on their knowledge base. The team structured responses in a defined format to enforce step-by-step thinking processes, ensuring responses were grounded in factual knowledge rather than AI-generated assumptions.
2. Iterative improvement: Each iteration added complexity, including prompt refinement and topic validation.
3. Rigorous testing: The team tested the agent’s responses against a suite of 40-50 must-handle questions and validated accuracy, relevance, clarity, and completeness via an LLM scoring system. Thanks to that, the team was able to detect improvement of the overall quality but also regressions.
4. Comprehensive monitoring: The team tracks 37 different metrics related to every aspect of the agent's performance, enabling continuous optimization and improvement.
The team particularly valued Voiceflow's observability features, which provided visibility into the agent's decision-making process. "Being able to see what's happening at every step helps us better understand the agent's behavior," notes Biancamaria. This visibility into agent actions, through features like the debug mode and Prompts CMS, became crucial for maintaining quality and iterating effectively.
Each update built upon lessons from the last, with the team regularly testing with users and refining based on feedback. By the seventh version, they had achieved the reliability and accuracy needed for production deployment.
Technical architecture
Turo’s agent needed to manage a number of different types of information at the same time. To handle this and keep latency low, the team developed a sophisticated system that processes multiple aspects of each interaction simultaneously.
- Parallel analysis: Every user interaction triggers six concurrent processes:
- Question analysis and clarification generation
- Optimized RAG query formulation
- User type determination (host/guest) with confidence scoring
- Geographic context analysis
- Language detection and processing
- Location availability verification
- Response generation:
- Implements distinct style guides for hosts and guests
- Uses structured JavaScript format to enforce logical response construction
- Automatically identifies and integrates relevant hyperlinks
- Includes self-verification and rewriting capabilities
- Generates both responses and UI elements (like buttons) in the user's detected language
- Quality control:
- Real-time response validation -- the agent double-checks its answer and rewrites if it needs adjustment
- Dynamic button creation for improved user experience
- Comprehensive knowledge base integration
- Multilingual capability verification
One of the benefits of working with Voiceflow at the enterprise level is access to staff expertise in agent design and prompt engineering.
“Meeting twice a week with [Voiceflow] definitely helped to remove all the blockers," explains Eric. "The engineering experience was smooth - we could quickly move from a workflow designed by Voiceflow to one that we owned and could refactor ourselves."
The results: achieving 99% response rate
This rapid deployment of their first phase agent helped them meet their criteria for moving on to the next phase. Top priorities for the pilot phase were accuracy and user satisfaction, and it’s safe to say their metrics across those dimensions are impressive:
Accuracy and reliability
- 99% Response Rate with 1% Fallback Rate (cases where the bot cannot provide an answer)
- 90% accuracy
- 2% Escalation Rate to human agents
User satisfaction and engagement
- 82% Bot Satisfaction Rate from user feedback
- 23,000+ successful conversations handled to date
The team noted that a 90% accuracy rate is a realistic goal for an AI Agent because seeking a perfect solution is not the end goal. Achieving similar (or better) results than human agents and focusing on accuracy for blocking points is the objective.
The team also pointed out that the 2% escalation rate is “expected” as the agent was deployed on a prospect-facing page rather than a customer support page, and they look forward to learning more about how the agent handles escalation when they expand into customer support.
That being said, the agent has proven particularly effective at meeting its first objective of new user education, successfully handling common inquiries about booking processes, protection plans, and getting started with Turo. Through comprehensive monitoring of 37 different metrics, the team maintains visibility into every aspect of the agent's performance, enabling continuous optimization and improvement.
Key learnings and best practices
Turo's quick, successful implementation of a proof-of-concept AI agent shows how important it is for enterprises to start with focused use cases while exploring new AI initiatives.
For others considering the same path, keep these insights in mind:
- Set a North Star: The Turo team knew they eventually wanted an agent that could handle complex, proactive support, and that inspiration fueled them as they started at the ground floor
- Start small, scale smart: Begin with a focused use case and expand based on success
- Iterate rapidly: The team went through seven prototypes before reaching their production version
- Know your metrics: Track multiple metrics beyond just satisfaction rates, ideally keeping a dashboard for team alignment and reporting
- Test thoroughly: Combine automated testing with manual verification of responses
- Focus on accuracy: Prioritize response accuracy before expanding into additional features and functionality
“With Voiceflow, we didn't have to focus on how to deploy a RAG, LLM or agent, we just had to consume a simple API to create our knowledge base and then think about general chatbot design and prompting,” says Eric. “Despite the fact that it looked simple, it was necessary to set up an efficient feedback loop with repeated tests (human and automatic) that could measure accuracy, relevance, quality and usefulness. This was key in the first phase of our project and it is what will take us into the next phase of our project.”
Looking ahead
With their initial success, Turo is now positioned to expand their AI capabilities further. Future plans include:
- Replacing their legacy customer support chatbot with the new AI solution
- Expanding the agent's capabilities to handle more complex scenarios
- Integrating deeper transactional capabilities
- Developing proactive problem-solving features
"We knew there was potential, but we weren't exactly sure about the size of it," explains Biancamaria. “We imagined needing some flexibility during the research process. We imagined using Voiceflow for automating agent emails, for example, which would not have been possible with pure chatbot solutions.” Their measured approach and successful implementation with Voiceflow has now given them the confidence to pursue more ambitious AI initiatives.
Want to learn how Voiceflow can help your organization build advanced AI agents? Request a demo today
The challenge: moving beyond basic chatbots
Turo was using a traditional decision-tree chatbot that had been acquired by Zoom. While their operations team had done an impressive job optimizing the tool, its limitations were becoming increasingly apparent:
- The chatbot could only handle pre-programmed decision trees.
- Users were frustrated by its rigid, mechanical responses.
- It was difficult to measure true contact deflection rates
- The solution couldn't scale with Turo's growing global user base
"We were very limited in terms of capabilities," says Biancamaria. "The chatbot could not handle anything outside of the built-in decision tree."
Finding the right solution: a strategic evaluation
At the beginning of 2023, Benjamin and Eric Comellas, Senior Director of Engineering at Turo, were tasked with analyzing AI opportunities for Turo. Through their evaluation, they identified three distinct categories of potential solutions:
1. Off-the-shelf Chatbot Solutions: These were specialized platforms designed for specific use cases like e-commerce. While easy to implement, they were not flexible enough for Turo's diverse needs. "You can't do anything outside of that specific use case," explains Biancamaria. "It's very constrained."
2. Pure LLM Providers: Platforms like Amazon Bedrock offered direct access to AI models. While providing maximum flexibility, these would require significant upfront investment and engineering resources to build and maintain a complete solution.
3. AI Orchestration Platforms: The middle ground between specialized chatbots and pure LLM providers. These platforms, including Voiceflow, offered flexibility while providing the necessary infrastructure and tools to build quickly.
After extensive evaluation, including discussions with major providers like Google, Turo chose Voiceflow for several key reasons:
- Flexibility with speed: They could leverage existing infrastructure while maintaining the ability to customize and expand beyond just chatbot use cases.
- Quick implementation: The visual interface and support enabled rapid development without requiring investment in additional engineering resources while providing high confidence in time to market.
- Strong partnership: The Voiceflow team provided dedicated support, including bi-weekly meetings and a dedicated Slack channel for rapid problem-solving.
- Future-proof: The extensibility of the platform would allow Turo to easily scale to future use cases, such as automating agent emails.
- Cost-effective: Compared to similar solutions, Voiceflow offered competitive pricing with clearer terms.
"We wanted some flexibility, but not to invest everything right now," explains Biancamaria. "The support from Voiceflow at all layers, from engineering to implementation, really made the difference."
The vision: a proactive customer support agent
The Turo team had ambitious goals for their agent to proactively solve customer support needs in the future – but they knew the most effective implementations take a phased approach.
Rather than rushing to replace their existing system, Turo's team planned to build out their agent in three distinct phases:
1. Informational phase: Build an extremely accurate FAQ agent with low tolerance for hallucination
2. Transactional phase: Connect the agent to user data and enable it to answer specific questions about trips and claims
3. Problem-solving phase: Create an agent that can take actions and proactively assist customers
Their ultimate north star? An AI agent that could say, "I see your flight was delayed. Let me move your trip back an hour or two," without requiring customer intervention.
Aligning the team around an inspirational vision created the buy-in necessary to implement the first phase and get their AI agent project off the ground.
The solution: rapid implementation of phase one
As a proof of concept, Turo chose to launch their initial agent on their "How It Works" page, which is focused on new user education and support. This approach allowed them to validate their AI capabilities while minimizing risk.
"Instead of shipping the first chatbot to replace our existing solution, we shipped it somewhere else in the product," explains Biancamaria. "This allowed us to confirm we could answer questions accurately, that we could escalate when needed, and gain confidence in our approach. Answer accuracy was critical in getting buy-in from other teams, including Customer Support and Legal."
The team moved quickly through seven iterations in two months, following a structured approach:
1. Starting simple: Beginning with basic inquiry and response capabilities based on their knowledge base. The team structured responses in a defined format to enforce step-by-step thinking processes, ensuring responses were grounded in factual knowledge rather than AI-generated assumptions.
2. Iterative improvement: Each iteration added complexity, including prompt refinement and topic validation.
3. Rigorous testing: The team tested the agent’s responses against a suite of 40-50 must-handle questions and validated accuracy, relevance, clarity, and completeness via an LLM scoring system. Thanks to that, the team was able to detect improvement of the overall quality but also regressions.
4. Comprehensive monitoring: The team tracks 37 different metrics related to every aspect of the agent's performance, enabling continuous optimization and improvement.
The team particularly valued Voiceflow's observability features, which provided visibility into the agent's decision-making process. "Being able to see what's happening at every step helps us better understand the agent's behavior," notes Biancamaria. This visibility into agent actions, through features like the debug mode and Prompts CMS, became crucial for maintaining quality and iterating effectively.
Each update built upon lessons from the last, with the team regularly testing with users and refining based on feedback. By the seventh version, they had achieved the reliability and accuracy needed for production deployment.
Technical architecture
Turo’s agent needed to manage a number of different types of information at the same time. To handle this and keep latency low, the team developed a sophisticated system that processes multiple aspects of each interaction simultaneously.
- Parallel analysis: Every user interaction triggers six concurrent processes:
- Question analysis and clarification generation
- Optimized RAG query formulation
- User type determination (host/guest) with confidence scoring
- Geographic context analysis
- Language detection and processing
- Location availability verification
- Response generation:
- Implements distinct style guides for hosts and guests
- Uses structured JavaScript format to enforce logical response construction
- Automatically identifies and integrates relevant hyperlinks
- Includes self-verification and rewriting capabilities
- Generates both responses and UI elements (like buttons) in the user's detected language
- Quality control:
- Real-time response validation -- the agent double-checks its answer and rewrites if it needs adjustment
- Dynamic button creation for improved user experience
- Comprehensive knowledge base integration
- Multilingual capability verification
One of the benefits of working with Voiceflow at the enterprise level is access to staff expertise in agent design and prompt engineering.
“Meeting twice a week with [Voiceflow] definitely helped to remove all the blockers," explains Eric. "The engineering experience was smooth - we could quickly move from a workflow designed by Voiceflow to one that we owned and could refactor ourselves."
The results: achieving 99% response rate
This rapid deployment of their first phase agent helped them meet their criteria for moving on to the next phase. Top priorities for the pilot phase were accuracy and user satisfaction, and it’s safe to say their metrics across those dimensions are impressive:
Accuracy and reliability
- 99% Response Rate with 1% Fallback Rate (cases where the bot cannot provide an answer)
- 90% accuracy
- 2% Escalation Rate to human agents
User satisfaction and engagement
- 82% Bot Satisfaction Rate from user feedback
- 23,000+ successful conversations handled to date
The team noted that a 90% accuracy rate is a realistic goal for an AI Agent because seeking a perfect solution is not the end goal. Achieving similar (or better) results than human agents and focusing on accuracy for blocking points is the objective.
The team also pointed out that the 2% escalation rate is “expected” as the agent was deployed on a prospect-facing page rather than a customer support page, and they look forward to learning more about how the agent handles escalation when they expand into customer support.
That being said, the agent has proven particularly effective at meeting its first objective of new user education, successfully handling common inquiries about booking processes, protection plans, and getting started with Turo. Through comprehensive monitoring of 37 different metrics, the team maintains visibility into every aspect of the agent's performance, enabling continuous optimization and improvement.
Key learnings and best practices
Turo's quick, successful implementation of a proof-of-concept AI agent shows how important it is for enterprises to start with focused use cases while exploring new AI initiatives.
For others considering the same path, keep these insights in mind:
- Set a North Star: The Turo team knew they eventually wanted an agent that could handle complex, proactive support, and that inspiration fueled them as they started at the ground floor
- Start small, scale smart: Begin with a focused use case and expand based on success
- Iterate rapidly: The team went through seven prototypes before reaching their production version
- Know your metrics: Track multiple metrics beyond just satisfaction rates, ideally keeping a dashboard for team alignment and reporting
- Test thoroughly: Combine automated testing with manual verification of responses
- Focus on accuracy: Prioritize response accuracy before expanding into additional features and functionality
“With Voiceflow, we didn't have to focus on how to deploy a RAG, LLM or agent, we just had to consume a simple API to create our knowledge base and then think about general chatbot design and prompting,” says Eric. “Despite the fact that it looked simple, it was necessary to set up an efficient feedback loop with repeated tests (human and automatic) that could measure accuracy, relevance, quality and usefulness. This was key in the first phase of our project and it is what will take us into the next phase of our project.”
Looking ahead
With their initial success, Turo is now positioned to expand their AI capabilities further. Future plans include:
- Replacing their legacy customer support chatbot with the new AI solution
- Expanding the agent's capabilities to handle more complex scenarios
- Integrating deeper transactional capabilities
- Developing proactive problem-solving features
"We knew there was potential, but we weren't exactly sure about the size of it," explains Biancamaria. “We imagined needing some flexibility during the research process. We imagined using Voiceflow for automating agent emails, for example, which would not have been possible with pure chatbot solutions.” Their measured approach and successful implementation with Voiceflow has now given them the confidence to pursue more ambitious AI initiatives.
Want to learn how Voiceflow can help your organization build advanced AI agents? Request a demo today
.svg)