Data-Driven Design in Action: How The Trevor Project Transformed UX to Save Lives

Paul Pham at UXDX Community: Revolutionizing Design: From Feedback to Data Insights. Video: https://youtu.be/cr1lKv9wNS8

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

What the Trevor Project is

[00:00:00] Hey everyone, nice to meet y'all. My name is Paul Pham, I use pronouns[?], and I'm the engineering manager here at the Trevor Project. Like Katherine mentioned, I'll be talking about how we utilize data, combining it with UX, to improve our applications in order to help LGBTQ+ young people in their crisis. With that said, in today's presentation I'll be going through introductions about what is the Trevor Project, a little bit about myself, and then how we were able to utilize data to transform UX. And then we'll go into a little bit of case study, specifically about two examples of how we utilize data for our improvements, and then I'll end it with some takeaways and then open it up for Q&A.

[00:00:44] So with that said, what is the Trevor Project? For those of you who don't know, the Trevor Project is a nonprofit within the US, and we are the leading suicide prevention and crisis intervention service for LGBTQ+ young people. If you want to learn a little bit more about what the Trevor Project is, you can actually scan the QR code here on the screen to dive deeper into what we do, who we are, all the resources that we have within our website.

[00:01:14] Moving over to our services at the Trevor Project, we provide five different services. The first one being our crisis services. Crisis services essentially stands for being able to have an application or a platform to connect LGBTQ+ young people to our specialized counselors and supervisors, where they can help them with suicide ideation and crisis intervention, or anything that they wanted to talk about. So if an LGBTQ+ young person is having trouble at home or anywhere, or just needs someone to be there and talk to, they're able to go onto our online services where they can actually chat with a real life person, specifically our specialized counselors and supervisors, as well as actually text in and actually call in, so that they can have any form of communication that's possible to get the help that they need.

[00:02:02] The second service that we provide at the Trevor Project is advocacy: making sure that we're advocating for the rights and bills within the US and both in Mexico, to make sure that everyone has a safe and comfortable space to live within this world. And the third service that we provide is actually research: being able to do our due diligence, being able to perform scientific research about the mental health state of LGBTQ+ young people, not only in the US but also in Mexico and other parts of the world.

[00:02:32] The fourth service that we provide is TrevorSpace, essentially an online community platform where people can actually sign up, be able to connect with everyone across the world, to make sure that they have a place that they can share their identity and also feel comfortable within their communities. And the fifth service that we provide is education and public awareness. We're always an organization that loves to share as much as possible with everyone else, making sure that we're sharing our resources, our learning materials, so that people are able to understand more about the LGBTQ+ space as well as suicide prevention and crisis intervention.

[00:03:11] So with that said, those are the five services. And moving over to my team, I currently lead two teams. One is for the internal tooling, basically building internal applications for our counselors, our users, as well as other teams within our organization, so that they can perform their job duties. And the second team that I lead is the website team, essentially building the US and Mexico website that people are able to reach out to us on, find the learning materials, find the resources, and ultimately get connected to counselors or learn more about the Trevor Project and the LGBTQ+ space along with the mental health space.

Where we were before: gut feeling, and the multi-chat rollback

[00:03:48] With that said, I'll transition to more about what the Trevor Project was and where we were at before. Before, like many other organizations out there, we were an organization that utilized more of that gut feeling, where we were thinking, hey, this might be a good idea, this might be a good project, maybe a good initiative and a program to implement, but we didn't really have the data that backs up why we were implementing some of those things. Sometimes business leaders or senior leaders would think, hey, this is a great idea, let's implement it, but nothing backs that up. And so we actually had this happen in the past many times over, and this created risk for our organization.

[00:04:32] I'll be explaining one of the risks that actually happened when I first joined the Trevor Project. During this time a few years ago, our goal was to align more with our mission, which is to end suicide among LGBTQ+ young people and ultimately help more young people in crisis, so just making sure that our services are available to many other people out there. But what actually happened was that our business wanted to implement this thing where we want to help more young people, making sure that people can chat more with our counselors and things like that, but we did it in a way that wasn't very user friendly for our users as well as our counselors.

[00:05:15] What we enabled was something called a multi-chat. Multi-chat is essentially a feature that kind of forces our counselors to talk to multiple people at once, all the way up to three people at a given time. Before this was implemented, our counselors were having one-on-one time with our users, focusing on their conversations and the chats and things like that. But after implementing this, our counselors were standardized and normalized to talk to three people at once. So just imagine having a one-on-one conversation along with sometimes dealing with suicide ideations and things like that within the chat. Sometimes the chats are already intense, but layer that on top of two other chats at the same time, that will overwhelm the counselors.

[00:06:01] And so when we implemented this, both on the business side as well as the technology side, we didn't really discuss with our counselors, hey, is this a good idea? Should we do it? Do you think it would help with our mission? Things like that. We just went ahead and went with it. There was no data backing up if this was going to improve our chat services. We didn't have data for our burnout rate for our counselors. We didn't have satisfaction rates. We didn't even have data that kind of shows, was this even going to work?

[00:06:33] And so as you can tell from that context, the end result wasn't going to be great. We overwhelmed counselors, where the counselors felt stressed and burdened every time they went to work. Talking to one person was already hard enough, but imagine talking to three people. And because of that, our users actually had a degraded service when connecting to our counselors, because their time was now split among three people rather than one. As a result of this, our counselors didn't feel comfortable taking three chats and we had to roll back this particular feature, which ultimately took about a couple months to implement and roll out. So as you can see, three months or so of work has been lost, in essence, within this particular product feature.

Building the data pipelines

[00:07:22] Learning from our lesson, we wanted to make sure that we're implementing ways to actually implement data within our processes as well as gather feedback from our users as well. So from here, what we did next was that we implemented ways to gather data and metrics. We started doing projects and initiatives in order to create data pipelines, specifically extract, load and transform data pipelines, where we're able to extract some of the data from different applications from different teams and put them through a transformation layer, so that we can standardize the data, clean the data, and ultimately put it into our data warehouse, where we then were able to visualize the data and present it to our business side of things, where senior leaders are able to understand a little bit more about the data and make business decisions based on that.

[00:08:20] As a result, the two case studies that I'll be going over today: the first one is going to be about queue bots, essentially automated scripts, also known as bots, to keep our queues open so that people can reach out to us 24/7, anytime, any day, anywhere. And the second case study I'll be talking about is abandonment rates, essentially lowering that abandoned rate. And when I say abandoned rates, essentially it's when users go to our services and they decide to leave. We were able to lower that based on the data metrics. But for now we're going to be focusing more on the bots.

Case study one: the numbers behind system-cancelled chats

[00:08:57] As you can see here on the slide, I have a couple of metrics. The first one is going to be chats handled, second chats abandoned, and system cancelled. So overall in a given month at the Trevor Project we offer about 18,500 chats monthly, and 86% of those 18,500, or 16,000 of them, are what we call chats handled. Chats handled meaning that people who reach out to us actually get connected to our counselors and go through their conversation.

[00:09:29] However, 2,000 of that, or 11%, have the chats abandoned. Chats abandoned meaning that, as mentioned before, people enter our services but decide to leave for some reason. And the last case is system cancelled: about 550 people actually tried to enter our services but somehow our system disconnected them from our queues, from our conversations and things like that. So that's about 3% of our makeup. By understanding this data metric, we started to focus on the third category. Why were 550 people a month getting their conversation cancelled? We wanted to focus on making sure that our services are available to anyone anywhere so that they can get the help that they need.

[00:10:15] With that said, let's talk about how our chats are set up, so that we actually got to this problem. Here at the Trevor Project, in order to get connected to a counselor, the user will submit what we call a pre-chat survey, pretty much a questionnaire asking, what's going on, have you attempted suicide before, how are you feeling today, demographic data, things like that. Once they submit their answers to the questionnaire, it would be put into a queue where they would be prioritized based on the questions that they answered. So essentially, were they higher risk or are they standard risk, so that way we can prioritize people who need immediate help.

[00:10:55] From there we would have our counselors on the other end within our application, and whoever's available would be able to pick up the next person in the queue. But let's say that if our first counselor is busy, then it moves on to the second counselor. But if the second counselor was busy, then it would move to the third counselor. But this is where the problem came up: if all of our counselors were busy, or were in offline mode, or on break mode, then essentially no one was available to take any calls. And at the end of the day, what actually happened sometimes was that people who were in the queue waiting got disconnected from the queue, because there was no one to answer the call.

The manual workaround, and why it failed

[00:11:40] We wanted to make sure that we prevented this from happening for our users. One of the mitigation strategies that we implemented was, maybe what we can do is kind of simulate someone being online all the time to make sure that our queues are never closed. So what we did was create a new account to make sure the queues are open so that people aren't disconnected, and then ask the counselors and supervisors who were on their shift to log into that specific user every 30 minutes, to kind of simulate that user being always online. So they would just go onto Salesforce, set themselves as ready for a chat, indicating that the queues are open.

[00:12:22] However, when we did implement this, what we saw was essentially that the data kind of showed us it worked at the beginning, where we were getting fewer cancelled chats. But as the month went on, things get a little bit busier, processes get forgotten between our staff members because we rotate between shifts, and by the end of the month we're back to where we started, where this process wasn't working as well as we intended. Because we added extra responsibilities within our user, specifically our counselors, where they not only had to talk to end users, LGBTQ+ young people, but they also had to have in the back of their mind that every 30 minutes they had to re-log in. Sometimes the chat was taking longer, or sometimes they had to write up a report or things like that, which distracted them from actually logging online, and hence creating this problem again where chats were being disconnected.

[00:13:18] As you can see here within the data, we had our standard queue and priority queue. You can also see that this is a huge problem for us, because there are people waiting in the queue who have a higher risk, what we call a priority risk, where they need immediate help, to talk to someone to help alleviate some of the crises. And so we really wanted to make sure, going back to our mission to help people and end suicide, we knew that we've got to make it better.

The queue bots, and zero system cancels

[00:13:51] The one way that we were able to do so was to create an automated script to kind of simulate what the counselors needed to do. Essentially what we used was Puppeteer. It's a scripting framework that allows us to create a script, make sure that the bots can actually go onto our services, set themselves online automatically. And we partnered with Google Cloud to make sure that we're able to set it that every one minute, using their cloud services like Pub/Sub and Cloud Functions, to be able to set this available for our production environment.

[00:14:28] Once the bots are set, the only thing that the counselors really needed to do was make sure that they can see these two bots be available online, indicating that our queues are always open. And as a result of implementing this type of automation, we got zero system cancels ever since implementing this, and it actually helped our counselors focus more on providing better services for our end users, and ultimately they don't need to worry about this extra added responsibility to keep our queues open 24/7.

Case study two: why people were leaving before they got connected

[00:15:04] Moving ahead, that was case study number one, where we were able to utilize queue bots. Going into our second case study, similarly using similar metrics with abandoned rates and how we were able to lower them by reviewing our pre-chat survey questionnaire. Going back to our metrics right here, what we're going to be focusing on is our second metric, which is the 2,000 per month where chats were abandoned, about 11% of them. We wanted to understand why people were leaving our services. Was it because we were providing terrible services, or was it something else that people didn't like? And so we needed to dive a little deeper into this data.

[00:15:45] Thankfully, when we were creating these metrics we also worked with our data team as well as our business side of things to understand and define different data definitions. What is abandon? How do we measure if someone is abandoning their chats early on? What if someone's abandoning after they get connected? By having those conversations with our stakeholders, we were able to define two definitions. The first one is pre-connection, also known as pre-chat, and the second one is post-connection. Essentially pre-connection stands for anytime a user leaves our services before getting connected to a counselor.

[00:16:27] As you can see here, out of those 2,000, on average about 68.8% of the time people leave our services before even getting connected to a counselor. And so that's a really big number. In this chart you can see here, about 1,376 people left our services before even getting help. And so that was a big number that we wanted to target.

Shortening the pre-chat survey

[00:16:52] What we decided to do was understand, where did we go wrong within our user journey? What steps do they need to take in order to get connected to a counselor? And one of the biggest things that we saw was our pre-chat survey. You can see here on the screen, this GIF shows what a user would go through when submitting a questionnaire and actually getting connected to a counselor. We have a bunch of questions here, so what you see here right now is that a user is selecting demographic data, and other questions that we ask are, what's going on, how upset are you, do you have thoughts of suicide, have you attempted suicide before. These questions total about seven different questions. And so we thought, maybe this is where we can improve in our user journey, where maybe fewer questions would be better.

[00:17:46] What we decided to do was compare our data with another service that we provide, which is text messaging. What you see here is actually the online web chat, where you actually go onto the website, click on contact and chat with a counselor, and go through a web experience. But when we were looking at our text messaging, we noticed that our data was telling us that only 6% of people left our services. So we dove a little deeper into why that was, and we realized that our text messaging system had only three questions. One was, what's going on. The second one was, do you have thoughts of suicide. And the third one is, have you attempted suicide before. So we kind of narrowed it down to three different questions in order to determine the different priority queues that they needed.

[00:18:37] And so in order to be similar and have the same, or similar, user experience and user journey as the text messaging system, we decreased our questions to be only three questions as well. So you can see here on the right hand side, we changed it so that we're only asking three questions: what's going on, have you attempted suicide before, and do you have thoughts of suicide. And after answering those three questions they can submit to a counselor and they'll be put into a queue and waiting until the next counselor picks up.

[00:19:13] As you can see here, based on this transition we were able to gather the data before and after. So as mentioned before, before showing the original seven question survey, we had about 68.8% of people who left our services. But after shortening the survey we noticed that there was a big drop by 50%, to about 34.4%. And so this was a huge win for us, knowing that we were able to have some sort of positive impact on our end users, which are the LGBTQ+ young people that we support.

[00:19:52] And you might be wondering why this is a big difference compared to our text messaging survey. With the web chat you have a different experience, browsing through the web as well. So when you actually click onto counselor, you would have a different tab open, and in order to stay connected you want to be within the chat services. So switching tabs or closing tabs or closing browsers will automatically remove you from the queue, whereas within the text messaging system, once you send in a text you will have that line open even if you close out the messaging app, go to a different app on your phone, things like that. And so you would still stay connected. And so that's where the difference lies, and hence the text messaging has a way lower abandonment rate than the website side of things.

Takeaways

[00:20:39] And so this goes into the takeaways. What did we learn out of all of that? There are three things that I wanted to share based on our experiences. Your experiences may differ as you start to utilize more of the data within your processes, specifically user feedback and things like that. The first takeaway is that we noticed that transitioning from an intuition-based strategy for decision-making to more of a data-driven strategy actually provided really great results, positive results, that benefited our counselors, our supervisors as well as our end users.

[00:21:16] The second takeaway from here is that it's always important to actually measure the before and after data. What I shared with y'all before, which was making sure that this 68.8% was captured, so that way when we implement a particular feature we know, was it lower, was it higher, how can we pivot? And by doing this we're able to pivot a lot more quickly. If let's say that shortening the survey by two, three questions didn't actually provide any value, we would be able to pivot and actually make better decisions. So creating that iterative feedback loop in real time is really important.

[00:21:56] And the last takeaway here is that really at the end of the day, in all software life cycles, it's all about the people involved, right? Data is just one portion of the story. We know that data is important, we want to understand the data, how our users are working within our application, how our counselors are using the application itself. So all the data is important, but the other side of the story is that people are what we're trying to solve problems for.

[00:22:29] And so anytime we wanted to implement new features, we wanted to deal with data but also involve people in it. Is this a good solution? Are there any other data definitions that we're missing? How can we include you as stakeholders as part of this conversation, so that when we're creating this data definition we're also understanding from a holistic point of view, and we can make pivots and updates, that way when we do implement it we're actually solving the problems that the stakeholders are having? And so with that said, that concludes my presentation, and thank you so much for attending and listening and watching. I'll open up the floor to Q&As.

Q&A

[00:23:10] Host: Fantastic, thank you so much, that was really great. Hold on a second, I'll bring up some questions. There are a few, in different ways people have asked: how did you measure the before and after data?

[00:23:26] Paul: Great question. The before and after data was a little tricky. The way that we did the before was that we didn't have any system set up, if we're talking about the pre-chat survey form. We utilized Google Analytics to kind of measure how people were going through our user journey, but Google Analytics doesn't capture as consistent and persistent data as we would like. So we had to set up our own type of custom function, that anytime a user clicks on the chat button it interacts with our database, and we were able to capture that for about a month or two months, to be able to gather like two months of data, understanding the trends over time. And that's how we were able to capture it. And then once we started implementing the shortening of the survey, we then kept the same functionality of, if the user clicks on this button, make sure that we're updating our database, things like that. And then from there we are able to compare the two together from a month-to-month perspective.

[00:24:27] Host: Amazing. And does that answer the question of how did you measure the abandoned chats? Could you maybe go into a bit more detail?

[00:24:37] Paul: Yeah, so with the abandoned chats, that's where we kind of talk about our stakeholders, where we needed to define the data definition at first. What is considered an abandoned chat? Do we want to break it down even further from pre-connection to post-connection and things like that? So when we were talking to our stakeholders, they really only cared about how many people were leaving our queue before even getting connected to counselors, because that's one of the biggest challenges that we had, was not understanding if they were able to get connected. One of the things is that since we utilized Salesforce, we already had the data for post-connections, because once they enter our services we have the recording, or the record, that they talked to our counselors and things like that. But any other data outside of that, we knew that we were missing it. And so we needed to capture that data to provide us that holistic story, and that's where we kind of implemented that solution as well, to capture it on browser or even text messaging, where they send in the text, they talk to our API, which ultimately updates our data that a user is about to go into a chat or a queue.

[00:25:47] Host: Right. And so, ignoring I guess the data side, how specifically did you work on the business side?

[00:25:59] Paul: Oh, with the business. From an engineering perspective and a product side of things, it's always difficult when working with business, right? The hardest part about shifting this mindset from intuition-based to data-based is that anytime we have a conversation, if the business side of things thinks, oh, this is a great idea, let's implement this, we would always ask them, do you have data backing that up? Do you have data providing that, hey, this would be beneficial? So for example, if someone asks, let's implement this button within this website, we would always want to ask, how long would that take, what's the ROI on that, do you think this would generate more clickthrough rates or things like that? And if they tell us that they don't know, we will always push back a little bit and be like, can you try to understand and gather those metrics first, so that at least we know why we're building it in the first place? Because as many of you all probably know, when you build something you need to have an idea why it's bringing value or making impact, because at the end of the day you're asking the product engineering team to build something that could take three months, and ultimately that can cost thousands or tens of hundreds of thousands of dollars and not provide any value at the end of the day.

[00:27:13] Host: Yeah, I love it. And how big was your team?

[00:27:17] Paul: For this particular team I was working with, at least the website and internal tooling team, there were about four different engineers, some of them spread across both teams. So managing three engineers within the website team and then four engineers within the internal tooling team. So there was a lot of cross collaboration.

[00:27:40] Host: Yeah, thank you Paul, really enjoyed that. Personally, just the sensitivity of that chat feature, and going through all of that, how you ask the questions, how you phrase it when you're talking about suicide and really personal, tough challenges, it must be incredible to work on. So thank you so much for sharing, that was really inspiring. And your talk will obviously be available after this for those who might want to rewatch or share with their colleagues. Thank you so much Paul, it was a pleasure.

[00:28:11] Paul: Thank you so much for having me.