Open Table With Your UXDX SE Asia Speakers

Ananda Nadya, Rory Madden, Marrisse Asuncion at UXDX Community: SE Asia. Video: https://youtu.be/IoZEeP-10cQ

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

How researchers and product managers work together

[00:00:00] Rory: ...trying to get that cross-functional thing working. How I'm going to start the conversation is that there was a survey recently, and what they were doing was trying to see what PMs thought their role was and what UX researchers thought their role was. There was about 80% overlap between what researchers thought they were responsible for and what PMs thought they were responsible for. With that as a starting point, I'd love to hear, and maybe we'll start with Ananda: what is your team structure, and how do researchers and product managers work together?

[00:00:38] Ananda: Okay. I think our research team is really kind of unique, because in the first stage we mostly acted as consultants. Because the number of our members was not much at that time, we could not separate our responsibilities and embed them into a specific product line. For example, in Tokopedia, as an e-commerce platform, we have many squads: those who handle logistics, those who handle promo, those who handle home and search, those who handle flights, et cetera. And we didn't really have many researchers at that time, so we worked together in the team. Every request that came in, we tried to divide based on the availability of the researchers.

[00:01:32] But as the team grew, now we are assigned into the tribes, or maybe the specific product. However, we also still have the responsibility for conducting research that is foundational, which means that it is mandatory. So if there is any request from the tribe that is incoming, we have to argue with them that we really have to balance the time between the ad hoc projects and the foundational projects. Because foundational projects are projects that are already agreed upon by the leaders, which means they are already agreed by the research leaders, product manager leaders, design leaders, so that's why they should be done. As for the ad hoc ones, those will be something that is more discussable between the researcher and the direct stakeholders, and in this case that can include a product manager also.

[00:02:35] I think the interesting thing about research here is that because we are assigned to several tribes, we have this kind of specialty in conducting research that is related to the tribe or the team that we are assigned to. But we can also explore many things, because the foundational projects we are assigned to usually don't depend on our tribe. For example, I am put in the category tribe, so every research related to category will be upon me. But I got this very abstract foundational research regarding impulsive buying, in which I have to find out what makes people shop impulsively. It is done by me, and it is not really related to any category at all. So I think that's the unique thing.

[00:03:28] And we as researchers also work very closely, especially with the product and product design teams: mostly with the product manager, the product manager lead and also the VP of product itself, while also trying to know the implication of the data from the PM team for the product design team, because we want to see the implementations as well. So we stand in between them and we try to communicate things with each other.

[00:04:04] Rory: Great. And I guess, Marrisse, what's your structure like where you work?

[00:04:11] Marrisse: In our company it's the product managers who usually do all of the discovery work, and the discovery work usually includes the research part as well. You do the user interviews, you do FGDs if time allows. We are thinking of hiring people for UX, but it's really mostly the product management team doing the discovery part. That's why I was smiling when you mentioned the overlap.

Foundational research and embedded researchers

[00:04:42] Rory: Excellent. Just back to Ananda. I thought it was a really interesting point there, that you've got some people who have their foundational projects, and then they get the requests coming in, and you mentioned in your maturity that sometimes people are getting assigned to teams. How many researchers are you now, and how many teams roughly would have an embedded researcher, versus that ad hoc stepping in when needed?

[00:05:12] Ananda: I guess we have this kind of KR in which we aim to finish a certain number of foundational projects in a quarter, and it really depends on the availability of the researchers. Maybe of all 20 researchers, maybe 10 will be divided half and half. For example, they will be the ones who are handling foundational work but can also focus on the original research requests from their team. But as you say, not all researchers are assigned to the foundational projects, because we know that foundational projects require more time, and they also require some seniority and expertise, because they're very abstract. So basically they are usually led by more senior researchers, who try to give the opportunity for more junior researchers to chip in also.

[00:06:17] Why we don't really assign all of the team members to conducting foundational research is probably more because of the information gap and also the skill gap. So in order to make more people conduct foundational research in the future, we pair the senior and the junior researchers together in the project.

Training stakeholders to run their own research

[00:06:42] Rory: Great. You mentioned that skill gap and that training piece. Is your pairing just within the research team, or do you also train and pair with people in product teams so that they can do their own research if they need to?

[00:06:57] Ananda: As for that kind of training, it is actually done by any researcher. Usually there are senior researchers who will be responsible for creating the guideline, and then the more junior researchers will be the ones who communicate it to the stakeholders, and also demonstrate the materials and how to conduct the research on their own to the stakeholders. So I think it's more teamwork within the internal research team, in which everyone is responsible for educating the stakeholders.

[00:07:37] And it is something that is kind of mandatory, because we have this kind of project which is called a consulted project, in which the researchers, regardless of seniority, act as someone who oversees the research that is done by the stakeholders from end to end. The end result is not created by the researcher but by the stakeholders, and in order for stakeholders to do so, the research team has to oversee their work and give them feedback on it. And we also iterate on how they conduct it. I think that's probably the answer to it.

[00:08:23] Rory: Excellent, so there's quite a structured process flow there. And Marrisse, on your end, if the product manager is responsible for doing all this research, as well as putting together the roadmap, as well as the stakeholder communication, how do you help? Or is there a structured process for training and upskilling product managers?

[00:08:47] Marrisse: We have different avenues for training. We have what we call Think Paper [?] Friday. On Fridays we don't have any meetings, and we really allocate that time to training. If a product manager wants to learn more about data analytics, for example, or UX research, he or she could just include that in the list of training sessions that we're tracking. Then at the end of the quarter we usually review how each person is doing in terms of the training that they're taking. I think that part really helps. Although, since we don't really have a UX researcher like Ananda, it would also help to get someone who could share their expertise on the matter.

[00:09:43] Rory: And do you have those, some people call them communities of practice, or internal knowledge-sharing communities within the company?

[00:09:54] Marrisse: We're actually starting something called Quipper Product Academy, because we have our own LMS. Through our LMS we're planning to deliver, for example, if I'm a PM, I could create a course related to something that I'd like to share with the rest of the team, and then we're going to deliver it through the LMS. I felt like it was such a nice idea, because you get some element of dogfooding there, and you also get the chance to share what you've learned with the rest of the team. So it's kind of cool.

Getting research out to people

[00:10:31] Rory: That's a great approach. I've heard of a few places trying that kind of general knowledge sharing. Just staying on the topic of knowledge sharing, I'll go back to you, Ananda, on this one. You mentioned that you're doing a huge amount of research, but how do you get that research out to people? How do you make it so that when teams are going and looking at building products, they know, A, that it exists, and B, where to go to find it?

[00:11:02] Ananda: The research team has done a lot of initiatives in order to increase our exposure within the company. As I said in my presentation, we even held these offline events in which we tried to make it fun. We tried to make an exhibition, and when we conducted this, before the pandemic, we went from floor to floor to give people the heads-up about the event that we were going to showcase, and everybody could attend the event, and there would be free food, there would be free games. We tried to make it fun, because we believe that when people know that research is not all about the data, it's not all about proving hypotheses, but also about empathizing with people, it will make it feel more relatable for them.

[00:12:00] And we try to make several initiatives, from the formal ones to the informal ones, since we know that maybe not everyone will be interested if we make things really formal, and some people really like it when we chip in to the informal gatherings. For example, we have this product design town hall, and we also try to share some findings there, so that if people want to know more about the ongoing research they can really ask us.

[00:12:34] Rory: And are there any self-service options for people, or does it rely very much on you pushing out your content to people?

[00:12:43] Ananda: No, it really depends on the context. Sometimes people ask us to collaborate. For example, the product design team wants to hold a town hall, or the product managers want to hold a town hall, and they approach the researchers and say, "Hey, do you have anything that is worth sharing? Because I heard that you are researching this product, and we don't believe the data. So can you tell a story in our town hall, so that people can relate to what we are working towards together?" That way we can actually chip in and influence them in a very indirect manner. We don't really talk about the research report; we talk about how it can fit their agenda. In that example, we want to tell them a story of how things go.

[00:13:39] And if they are interested, they can also request us to do something like this in the future, or if they want to know about the findings, they can reach out to the researcher who is responsible for conducting that research. And maybe if there are many more opportunities in the future, they can really team up together. For example, when I told people about my research, several designers and several business people invited me to an ideation session. From there they were inspired to create a solution together, so that we can really get a grasp on what the feedback means to us and how it can develop into a solution that will solve users' problems.

Avoiding repeated research across teams

[00:14:25] Rory: Great, a nice structure there and an approach. Marrisse, when it's more decentralized, so you're doing your own research in your teams, is there a risk that you might be doing something that another team has already done? How do you share between teams to make sure that you're not repeating the same kind of research?

[00:14:49] Marrisse: That's actually something that we're trying to be better at, because right now each PM goes off on their own and tries to do research for the current feature or initiative that they're working on. One thing that we've been trying to do is to have a central document where we can access each and everyone's findings. We're hoping that it's a step towards sharing what the other PMs are also finding in terms of the discovery work that they're doing.

[00:15:23] Rory: It's a very difficult challenge, because I've tried it. It becomes a product in its own right. You need to be thinking about the users, how they're going to use it, how you update the data, the life cycle. It becomes quite challenging.

Cascading or translating OKRs

[00:15:42] One of the things, I guess, is that both of your talks are very much around coming up with ideas, setting either the roadmap or your foundational research goals. And a very hot topic in the industry at the moment is OKRs and cascading OKRs. What is your view? Maybe we'll start this time with Marrisse. Do you have a concept in your organization of cascading, where the CEO sets some OKRs and then the next team, and the next team down? Or do you translate objectives into something that's relevant for your team?

[00:16:20] Marrisse: What we do is we translate it to what's relevant to our team. We're actually split into different feature squads, so each feature squad has its own KPIs, and all of those KPIs are supposed to support whatever the business goals are, whatever metrics the business thinks it should track for the business to be successful. So you have the business goals, then you have the product goals, and then you have the specific goals and KPIs for each feature squad.

[00:16:57] Rory: And who's responsible for setting, or doing that translation piece?

[00:17:03] Marrisse: Usually the PM works with the EMs and the entire team in their feature squad, and then it's reviewed by the senior PM, to review whether the KPIs set by the feature squads are actually aligned with whatever the goals of the company are.

[00:17:26] Rory: Okay. Just a question: do you have a separation there between a feature squad and a product team? Is one product manager responsible over multiple feature squads?

[00:17:38] Marrisse: Right. In my case I handle the B2C part. It's called Quipper Video Masterclass, and I'm overseeing different feature squads with individual product managers.

[00:17:54] Rory: Okay, so each feature squad has its own product manager, but you're overseeing a number of different feature squads.

[00:18:00] Marrisse: That's right. So I have my KPIs, and each of the feature squads also has KPIs that should support my KPIs, and then my KPIs should support the business goals.

[00:18:14] Rory: One question just on that kind of structure. Your starting point was that everything should have a vision, and from the vision you move on. Is the product manager in each feature squad creating their own vision for their particular subsection, or is there one vision at your level that the other teams follow?

[00:18:38] Marrisse: How we do it now is there's one vision. For example, for Quipper Video Masterclass there's one vision for that product, and then your feature squads just have to make sure that they're tracking the right metrics to support that specific vision.

[00:18:53] Rory: Okay, great. That was an interesting take. If we jump over to Ananda: how does the cascading of objectives work? Because you've mentioned that you have very set objectives within the research team, on particular foundational research that you need to do. Where did those objectives come from, and how do you marry them up with, I'm assuming, the product teams having their own objectives? How do you negotiate that?

Measuring research impact

[00:19:25] Ananda: I think the OKR itself also evolves with the maturity of the team. For example, when we first started, we wanted to count how many projects we had done that year, because we wanted to showcase whether we were able to help a certain number of projects. But as we grow, it's not only about the number of projects itself. It's more about whether our insights are implemented or not, and that can be something that is concrete or something that is more subtle.

[00:20:01] For example, a concrete implementation would be a design that has gone live, and also an education page that has been showcased to sellers or buyers, and also a new business model that has been launched or applied. So that is something concrete, really implemented, in terms of impact.

[00:20:35] But the more subtle one is the one we also aim to go after, because we believe that in the future the objective of the research team is not only helping people get to where they want to be, but also forecasting where they should be going, based on the opportunities that are available for them to go after. So now we are also focusing on whether we can influence the making of the PRD, for example, product requirement documents, and product vision documents. Because within the company we have a division, each year, of what kind of priorities we want to put first. And if there are insights from the research team in the documents that are consumed there, or showcased in the VP presentation, or in the presentation to the higher-ups, we also count that as an impact. Because it means that our abstraction from the findings, being foundational or directional, is used for making a bigger plan than the research itself.

[00:21:49] For example, when we conduct foundational research we want to uncover certain things regarding a topic, but at the end of the day the topic is used in many areas. It can be used by one team, two teams, three teams altogether, and it can lead to multiple implementations that may differ from one to another. So I think the more we want to delve into impact, the more varied the impact is. Now we are also trying to formulate a better understanding of our impact, but so far I think it is mostly divided between concrete impact and strategic impact, like I said before. But we are still working on that.

[00:22:43] Rory: One of the challenges that I've seen, when you were mentioning the concrete examples: often what research finds is a little bit subtle. Somebody might go in with an idea, and research will tell them, well, actually it needs to be a little bit different to what your original thought was. But as soon as you mention it, the person believes it was their idea. They're like, "Oh no, but that's what I meant." But if you hadn't done the research, they would have gone down a different route. So how do you track the ones where, no, we did influence this, this was the original starting point, this is where we are now, we have influenced that? Versus when people will say, and I don't think they're doing it out of a bad place, it's just what they believe, "Oh no, that's what I meant all along."

[00:23:29] Ananda: Okay. I think it is about setting expectations about who did this: whether this is the idea from the researchers, or the idea from the designers or the stakeholders. The ownership of the impact. In my team, after we conduct research, we try to document all of the recommendations based on the user gaps that we identify. For example: users have difficulties with this and they expect this; users have this pain point, so they will prefer to do this. We try to cram it into a document that we can follow up on over time. We have this schedule in which we have to follow up on the implementation of the insights. It can be one month after the research, and we also follow it up again three months after the research, so that we have documentation of the points that we provided them, so that they cannot really try to change what we have written, since we already did this at first and they have known about it.

[00:24:40] If you talk about ownership, I think the most problematic thing is basically that people don't document what they are saying. It makes it really hard for people to be believed by others if they don't have justification that is good enough for people to believe them. But by writing down the points that we want to track, by pointing out where these ideas come from, for example from the user survey, from the internal data, from the interview, from the usability testing, we can know where this originates.

[00:25:21] Or maybe it can be more flexible than that. For example, the design team already has this kind of hypothesis, and it is backed by the results from the usability testing and also the survey. So why not try to own this together? Because the design team has their own justification. For example, they know from the metrics that the design doesn't really perform well, but from the research team we also have this kind of evidence, this kind of percentage that showcases that the reason for users not using that kind of feature is like this. So I think the ownership of the ideas can be really clear, and it is possible for us to even define the ideas together so that we can win together in that matter. Within my team we always emphasize that it's not about who wins. It's more about whether we can do this together, and whether we can be stronger when we work together.

Justifying time for discovery

[00:26:29] Rory: Yeah, that's a great ending point. It's the whole company that wins or loses, rather than the individual. So Marrisse, do you have a similar issue? I've talked to a lot of product managers and they say they never get the time to do the research. It's constantly all about implementation, implementation, implementation. Is there anything that you use to validate and say, look, this is valuable, this is why we need to do that upfront discovery?

[00:27:03] Marrisse: I didn't get the last part, your voice was breaking up.

[00:27:07] Rory: Sorry, I was saying: is there anything that you do to justify why you should spend time doing discovery, when a lot of people are often like, "Oh, just build it, just build it"?

[00:27:21] Marrisse: That's also a common problem, I think, for product managers, especially if you're not in a team-driven roadmap-building culture. If it's a top-down arrangement, what usually happens is there's one person who tells you what he thinks a user needs, like a feature that the user needs, and then as a product manager what you end up doing is just translating those into specs and making sure that you deliver it.

[00:27:58] But if you think about the best practices in product management, discovery is really important, because that person's idea could actually be something that the users didn't actually need. If you don't do the discovery work needed, then you're going to end up spending a lot of time building it. You're going to need to build designs for it, you're going to need to coordinate with the engineers, and then you end up shipping something that the users actually didn't want. So the best way to stress the importance of discovery is to explain that to the people who don't think discovery is important. And if there are concrete examples that you could give, then I think it would also help them realize the importance of discovery.

Failures and what they taught

[00:28:52] Rory: Excellent. I like that approach as well. We're probably down to our last question. Unfortunately Rahel couldn't join us today, but her talk was all about failures and what she's learned from them. So I was wondering, and I'm putting you on the spot now, Ananda, if I start with you: is there anything that you've done where you thought, okay, look, I made some mistakes, but that was a great learning opportunity, because now I know not to do that again? Do you have anything to share with the audience that might be useful for them?

[00:29:28] Ananda: Okay. My failure: I made a mistake when it came to survey blasting. We had this budget for the foundational research, and we had already gotten so many participants for it, but it turned out it was not valid, because the logic that I made, which had been approved by my leader, actually did not let the users access certain key questions. Therefore we had, I don't know, thousands of participants, and we could not use the data, and we had already spent money on that. It was really fatal. It really became a drawback, because it meant that if we wanted to re-source, we had to spend more time and more money, and we also had to tell stakeholders that this was not going to be delivered on time.

[00:30:31] But if I hadn't made that mistake, I wouldn't be where I am now, because from that time I learned more about better planning. What I do to solve that is, actually, I requested more budget than I actually used, so it really works. And I always say this to the other researchers, especially junior researchers: maybe you really have to create several plans, and you have to be flexible about that, since there will be times when things will not go according to plan. And you also have to navigate your mistakes into improving the situation, because people rely on you. In that matter I was the lead researcher, and I was taking sole responsibility for the waste of the participants and the money that we had spent.

[00:31:34] It took a toll on me. But I think what can really support you during a crisis like this is having people who believe in you, and people who can really stand up for you even when you make mistakes, so that you will have the courage to begin again. I am thankful to my friends and also my leaders, who tried to make me become responsible, but not in a way that made me feel down or incapable, but in a way that made me feel like this is something that I should own, and I should figure out how to take care of all this, because it's my responsibility, and there's nothing to be afraid of.

[00:32:21] Rory: Yeah, that's a great example. I've done a few of those myself in my career. You take it, and it hurts so much that you'll never do it again. And Marrisse, do you have an example of something where you've messed up in the past, but that has made you a better product manager today?

[00:32:44] Marrisse: Yeah, there were a lot, but I'll just focus on one. There was this time when we were migrating to a new platform, and of course, since it's a migration, your main worry is: will we be able to migrate all of the data correctly? I was so focused on that part that I didn't do any user interviews. Fortunately we did have some alpha and beta testing, so we were able to course correct early on, before we released the product. But if we hadn't had the alpha and beta tests, I would have shipped something that the users would not be happy about, because it actually didn't align with what they had in mind. What I learned from that is that user interviews are really important, especially if you can do them in the beginning, so you can also consider them as you're planning what the product that you're building will look like.

[00:33:44] Rory: Great. Thank you very much for sharing all of your insights, and not just the failures and your learnings, but also how you're structured, how you share knowledge internally, and what the roles of PM and UX are. That is all we have time for, but I really enjoyed the conversation. If you have any other questions out there, please do share them on Slack, and Ananda and Marrisse will be happy to answer them. Thank you, Ananda, and thank you, Marrisse.

[00:34:12] Ananda: Thank you also, Rory and Marrisse.