Leveraging the Power of User Data

Andreea Popescu, Janina Jungiewicz at UXDX Community: Europe East. Video: https://youtu.be/cH9ub7B7-H0

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introductions

[00:00:00] Host: The topic of today is leveraging the power of user data. I encourage you to send some questions in. My goal is to ask Janina and Andreea about all of their experience and time, trying to download it for your benefit, so please do message if you've got any questions that are burning.

[00:00:20] Host: To give you a quick summary, Andreea is a UX mentor mainly for healthcare apps. She currently works for Telekom Romania and has really interesting experience. The things that stuck out to me in your background, Andreea, were the fact that you worked on that UX product, which is a really big thing for the industry at the moment, and more product owners are needing to become UX specialists across the industry, so that's really key there. And then other specialist skills, from usability testing and empathy maps and creating information architecture, which I thought was also really good.

[00:01:06] Host: Janina is working at Codewise, so coming from a multitude of experience working with companies particularly on the marketing side, and she's a product designer there. A psychologist by education, and then, to quote you, a designer out of love. I think what people will get from Janina is very much on the business requirements mixed with technical limitations, and matching that with the user needs. So very much, how do you get both the business and the user aligned, and that's both what we're going to talk about today.

The most effective ways to gather user data

[00:01:45] Host: Jumping in, Janina, if it's okay I'll start with you. We'll talk on data gathering. In your opinion, and with your team, what has been the most effective way of gathering user data and feedback?

[00:02:01] Janina: The most effective way, maybe not for everyone, but in our case, is different approaches to different data. In terms of getting qualitative data, we tend to rely on one-to-one contact with our users, so either it's interviews or even sending over screenshots, or whatever else can work. And then, when getting the quantitative data, our approach is a little bit different. I think this is usual for a lot of people, that we have to have bigger volumes of it for it to be like a guide for us in making decisions.

[00:02:51] Janina: We do this using different analytical tools, measuring mostly some activity in the panel, whenever it's possible, but then also observing certain parts of the product. Things like the payment process, for example, which is something we keep a close eye on, because from the business perspective it's quite important, so that we don't have drops in conversion rates for example. And then also, this is something that I was talking about in my presentation, when it comes to quantitative data, for example measuring how many times a user interacts with certain elements, so that maybe we can make the panel more user-friendly by hiding certain elements or displaying more the ones that are more often used.

[00:03:50] Host: Yeah, that makes sense. I'll go on to qualitative data because that's an interesting one. But Andreea, what's it like for you? Is it quite similar in terms of how you gather data? You've got more complexity in that you've got in-store and physical as well as digital, so talk us through that.

[00:04:10] Andreea: Yes, of course. Before the pandemic time we went pretty often in the stores to actually talk with our customers, because it's very important to get the feeling from them, to understand exactly which are their needs, and also the interaction with the product, the interaction with Telekom products. And right now, during this pandemic, of course we are not going any more into the stores, but we are using a lot of the phone. We are telecommunications, so we are using our minutes, unlimited minutes, so we are calling the customers.

[00:04:47] Andreea: And also for each page we're implementing, we are using a lot the usability testing and user interviews in depth with the user. And of course we are having a dedicated team for gathering all the data from analytics. So somehow it's a mix between data we rely pretty much on. For example, we are having every day the reports from the data team, and also we rely pretty much on the usability testing and the qualitative data.

Hotjar, and what usability testing adds

[00:05:25] Host: Okay. And how specifically have you used that data to get a better understanding of your customer? Could you talk about the usability and the insights there that you've gathered?

[00:05:34] Andreea: Yes, of course. For example, we ran just the last week a usability testing for one of our new features. For example, from Hotjar we understood a few things that users did, but we didn't understand very well exactly why they did some action with that new element in our shopping cart. For that we organized the usability testing, exactly to see how actually the user is interacting with that feature, to understand why they are having blockages, to understand why they are going further, and to understand each action they are taking, the way they are interacting with our tool.

[00:06:23] Andreea: For us this is very, very important. Okay, Hotjar is offering us some data and it's very important, but to have a better understanding on how the things are working, in my opinion the usability testing delivered us this information. So quantitative data is helping us to understand the quantity of everything, but the qualitative data is actually helping us to understand the interaction.

[00:06:56] Host: Yeah, great. And Janina, do you use Hotjar as well, to do the same kind of approach, or do you use any other tools?

[00:07:05] Janina: Yeah, well, to observe the workflow of our users we do use Hotjar. In the past we also had the opportunity to actually observe the workflow on site, when some of our users were visiting us, which was also very interesting. But primarily Hotjar is the tool we use to sort of follow, this is something that Andreea mentioned, the pain points maybe, or the blockages, the places where the user is struggling with the product, or how they interact with it. This is what we mostly use Hotjar for.

Measuring the success of what you ship

[00:07:47] Host: Yeah, really great. And the other piece is, so you've gathered the data, you can look into it. How do you, once you execute, how do you measure the success of that data?

[00:08:00] Janina: Well, to be honest with you, it is very difficult for us to actually measure it, to have it in numbers. Sometimes it's easy. For example, if you do an A/B test on part of the panel, that is easy to measure and easy to sort of test in this way. So I mentioned already the payment process, which is sort of the first contact a user has with the panel. There we can easily A/B test, and then we have precise data because we see the conversion rates.

[00:08:34] Janina: But for the rest of the product it's quite difficult to actually measure the success of different features. Because even if we, for example, fixed something that the users were complaining quite often about, or possibly made some changes, usually it's difficult for us to say that this particular change resulted in, for example, a smaller number of tickets to our support team. We tried to do things like that, but it's often difficult to measure precisely, because the product is quite complex, and often one change may result in some success but we may not always know exactly what the exact success of the feature we added is.

[00:09:31] Host: Yeah, that's interesting, which opens up a lot of questions on team approach, which I'll get to in a moment. But Andreea, from you, how do you measure the success on your end?

[00:09:46] Andreea: Okay, so compared to what Janina said, for us it's a little bit different. For example, I will say again about the feature we tested the last week. Doing the improvement on that feature will help us to get more users in our automated flow, and this is a success because the user right now will have the possibility to go from A to B very, very easy without any other impairment in there.

[00:10:14] Andreea: So the success, for example, for a redesign of a page, the success can be measured from the number of accesses of the page, the time in which the user is staying on that page, and also if we are trying to sell something on that page, because we have also pages just for information and pages designed to sell something. If the redesigned page is one for selling, we measure the success by how many leads we have, how many orders we have on that page, compared with the old page. So we are using pretty much the comparison between pages. We are having some KPIs, so we are having many ways to measure the success of data.

How the teams are structured

[00:11:12] Host: Yeah. In ways it might be — it's never easy, that's the wrong word, but because it's Telekom Romania it's one product. Janina, you're working across multiple separate external companies as well, so I imagine there's extra complexity for you there, and challenges. I will move on now to what you touched on, which is the team and having different ways to look at the outcomes. Often teams are not working with the analytics and they're working in different ways. For you, how is your team structured and how do you feed the user data into them? And Andreea, I will come back to you and ask you the same question.

[00:11:59] Janina: Yeah, for us the sort of separation of quantitative and qualitative data is also reflected in two separate teams. I would say that we have a separate team that's called, I think, business intelligence and automation, I think that's what they are called, and they are often measuring many different things actually, not only the activity in the panel, but also they have a lot of tasks connected to, I don't know, measuring the performance of the product and stuff like this. So they are mostly concerned about data which is numbers, basically.

[00:12:47] Janina: And for the product management team, which I am also a part of, we are more concerned with the qualitative data, which is often just simply user feedback, what they mention, what they talk about, what they complain about, and stuff like this. But because we're two separate teams, it's often quite difficult, and it always comes back to priorities. I think every team thinks of their own, from their own perspective, which I think is perfectly natural, and it's difficult sometimes to join forces at the right time.

[00:13:33] Janina: So for us it has always been a little bit of a challenge to get the right amount of data from them, and then to build a picture of the qualitative data and quantitative data together. So that's basically it. We are sort of separate entities, so we sometimes struggle to paint the picture clearly and let it guide our decisions along.

[00:14:06] Host: And are there any companies that you've looked at that are doing it well in terms of getting that qualitative and quantitative data right, or do you stick with your own team and the processes that work for you?

[00:14:21] Janina: I'm not sure I understand.

[00:14:24] Host: As in, is there any company out there that you would — I mean, the famous one, Spotify guilds, is often mentioned. Is there any best practice that you are aware of, that you keep on top of? Or is it not applicable maybe to your situation?

[00:14:45] Janina: Well, my experience has been that whatever we've tried to do in the past, we have learned from that. We had different strategies to do this, sometimes we relied on their help more, but we never actually tried, for example, restructuring the teams so that one business intelligence specialist was working with us. That never happened up until this point. Maybe that would be a good idea, because in the past we actually did this with marketing. The marketing team wasn't a part of the product team in the past, but now there is one person that represents the marketing in the product team.

[00:15:27] Janina: So I was thinking maybe that would be a solution that we would have, because it comes back to the perspective. We would then have a closer relationship with a person that has a different perspective, that could also teach us something about their process of work. I was thinking this would possibly be a solution.

[00:15:46] Host: Yeah, no, it's a really good one, and it sings to your team and business alignment as well. Andreea, let's talk about you, and how is your team structured?

[00:15:57] Andreea: Yeah, of course. For example, at Telekom we are structured in tribes. We are having a model like, I don't know, data tribe, sales tribe and so on. For example, I'm part of the sales tribe, and in this sales tribe we are very mixed people. We are interacting pretty much, let's say, daily with the data people. In the morning we are finding out exactly what happened the day before with our products, with our pages. So each one in the team, in the whole digital division, is aware about the analytics, what happened on another day.

[00:16:46] Andreea: So we are having developers, we are having UX designers, UI designers, we are working like shared resources, and we are pretty aware about all the data. For example, when we are having a usability testing session, all the other persons involved in that project are aware about the results, are aware about all the stuff that can affect their work. Because for example, after a usability testing session you will have some results, so those results will be improvements, so the people who are working for the project are aware about that. And yeah, so we are having dedicated people for the data in our team.

Getting stakeholder buy-in

[00:17:42] Host: Yeah, no, really good. And I guess in our last panel it came up around team dynamics and different points of view, and I think it's something that could be applicable to everything. But often people go with the approach of, let's look at the data, that's the single source of truth, but in reality that often doesn't work. Andreea, is there anything, are there any complications around team dynamics, are there any challenges there with getting buy-in from any of the stakeholders?

[00:18:17] Andreea: Yes, in the past it happened a lot. Not really at Telekom, but in the past it happened a lot. For example, I had some stakeholders that didn't believe that their solution is not the correct one. And as a specialist I said, okay, it may not fit, because it's not following a pattern, it's not okay for the user, it's not covering the accessibility, and so on. And they said, no, you do not have the right. In that moment, actually what I did, it was to ask the user, to present the product to them, and based on the results, after that I presented to the stakeholder and I said, okay, these are the answers from the user perspective.

[00:19:05] Andreea: If you need to create a product, and if you want to create a product for people, you just need to listen to people, you just need to listen to the intended people, and not your feelings. Sometimes in our work we may fall into subjectivity and say, okay, I like this. But in our work, especially for user experience design, subjectivity should be something that is not with you when you start the work day. You need to be very objective, and your personal preferences should be left out. This is a mandatory thing.

[00:19:52] Andreea: I met people that said, okay, I like it and I don't care what the user cares about or what the user needs, because I liked it. I'm pretty sure many UX designers had this experience, especially in the past, when the user experience domain was something like, okay, not very important, and the research was not a very good part of the process.

[00:20:23] Andreea: But right now I think that, with the whole growth on the technology part, and the whole growth of working from home, and the need of having good tools and the need of having good software, I guess everyone starts seeing the importance of users, and also the data. Because right now, I don't know how it is in other countries, but for example here in Romania we have websites on which you can see every day the statistics, how many people are using the mobile phones, which are the most downloadable applications and so on, which are the features the users are looking for and so on. So it's very important to take into consideration all these statistics, even if they are not done by you. Because I'm pretty sure in startups it's pretty hard sometimes to find the resources to receive and to gather all the data, but there are a lot of resources on the internet you can access and read data about users.

One loud user versus the numbers

[00:21:34] Host: Yeah, that's interesting. Janina, I'll ask you to jump in on your thoughts here, because I'd love to pull out that example you gave of the individual and you brought in the user. Do you think it was the third party that was the key to them changing their mind, or was it a different aspect of that?

[00:21:55] Janina: Do you mean with the example from my presentation?

[00:21:59] Host: No, sorry, not in your presentation. I was saying, Andreea gave an example of a third party coming in, the user, and it was the user, their feedback, to that person that she was having a challenge with, that meant that she finally got it over the line. What I'm trying to ask is, is that the same experience for you? Would you pull it out from your perspective of it being a third party opinion, and maybe that's the way to go, or how has stakeholder management worked for you?

[00:22:38] Janina: Yeah, well, actually this has always been an issue for us, I think, because I think we tend to give more value to one opinion spoken from one user than from a lot of quantitative data. Because this is, I think, like a quality in people, that we believe other people more than we believe numbers. So I think we have struggled with this. I think it's better now because we have a good feedback system, so we are looking at what the feedback is about but also how it looks in numbers, so how often people are mentioning this, how often there is this need for us to build something. But in the past, I think we often did this when someone mentioned a change was needed, and then we were putting our power into that.

[00:23:39] Janina: So I think that's something very tricky, and it can be a trap, especially if you start developing something that's quite big and you sort of go into it because you have the idea that users need it so much, but in reality it was just a couple of them. Then it turns out that no one is actually using it any more, because the couple of people that wanted it are, for example, gone, they are not using the product any more.

[00:24:14] Janina: So I think this is very, very difficult. And when I was listening to Andreea, I sort of had this jealousy a little bit, because of how much data you were actually getting, and I think this is something that we're often missing, so actual numbers that would help us, that would guide us in our decisions.

Prioritizing business and user objectives

[00:24:41] Host: Yeah, it is. Some companies have the luxury of a lot of data, but does that match? There might be a lot of business alignment issues there, Andreea, as well, and team alignment with that. So I think everything has its challenges. Yet really interesting points, and particularly going back to your talk on prioritization. It's not necessarily about striking the balance of prioritizing one user, but how do you strike the balance between one user versus all of that data that you collect? We are very short on time so I have to be selective. I'll ask you on prioritization, what advice would you give to people, aside from looking at your talk, to prioritize both business and user objectives?

[00:25:39] Janina: Well, for me this is going to be a little bit repetitive to the talk, but I think these are the two most important things. One of them is finding other user experience advocates among your team, and finding people that could understand the user perspective and sort of help to motivate others to also see it. I think that's very important. And I also think that getting the data, when it's possible and when it's available, is very important, and I would focus on those two things to move forward with certain changes.

[00:26:21] Janina: Because, as I mentioned in my talk, sometimes you don't really get to have the data that can show that the change you want to make is, for example, going to be good for the user. Sometimes you are not able to show it in numbers, and sometimes it's mostly based on just observing behavior, or people mentioning stuff. So you sort of have to find the balance between getting the data when it's available, and then also going forward with your instincts as well. That is something that is rather challenging for me, but that's something that I'm trying to actively do, due to the data not always being there.

Is there ever enough data?

[00:27:08] Host: Yeah, great. And Andreea, for the data that is almost always there for you, have you any challenges on the other end, that maybe sometimes, as good as data is, you can't read it right and you have to go with gut feel, or with an individual? Have you found anything like that, or what would be your advice there?

[00:27:30] Andreea: No data is enough data, because you do not have enough data. Even if you said, okay, it's an amount of data, but it's not enough. It's not enough because, for example, you're struggling sometimes. For example, having user experience design as a formation, I'm relying a lot on in-depth interviews on that part of the user experience research, so for me that kind of research is never enough. Okay, we are having a lot of data from Hotjar, from Google Analytics, we have people dedicated for receiving and gathering the quantitative data, but for qualitative data we do not have enough. It's never enough.

[00:28:19] Andreea: For example, being a company who is dedicated to a very large amount of people, with different ages, with different demographics, with different education, with different expectations, sometimes it's very hard to get to those people. It was easier when we had the possibility to go in the stores and actually to meet them, to follow them, I don't know, their facial expressions and their intonation and everything. It's easier, because sometimes when you are conducting usability testing they may want or not to show their camera, so you are not able to see how they are behaving.

[00:29:03] Andreea: So in my perspective, if I would say, you laughed about what Janina said, okay, I'm lucky that I'm having data, or I'm having a lot of quantitative data, but I'm not having enough of qualitative data. It's not enough, it's never enough. And of course all this data is helping us to prioritize everything.

[00:29:26] Host: Yeah, it can almost take the person out of it, and that's where maybe people need to make a decision on when it's a good time to move to empathy maps and your usability, those one-to-one interviews. That is all we have time for, I'm going to be cut off. You both are absolutely lovely and it was a real pleasure talking to you, for me twice. Thank you so much, and please, people, follow both Andreea and Janina as well on LinkedIn, and do watch their talks. Thank you again, and I'm going to hand it back to Louise to do the wrap up. Thanks all, bye.