Managing Data: Getting Research Applied

Monique Marins, AJ King, Romina Chitic, Ashley May at UXDX EMEA. Video: https://www.youtube.com/watch?v=fwrqVrjc5BQ

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introductions

[00:00:00] AJ: Good morning everyone, how's everyone doing so far today? All good? Still a little bit sleepy, that's fair. I think we're all still a little bit sleepy as well from last night. Sleepy and sleepy, we'll take it as sleepy. Thank you for our amazing introduction. I've never been called The Amazing AJ King before, so I'm going to take that one, I'm going to bank that one. It feels like either a superhero name or maybe like a bad Vegas magician, so you can decide at the end, you can put that in the feedback.

[00:00:30] AJ: I guess we'll get started, the clock's already counting down. Welcome to our panel on managing data, getting research applied. Essentially what we're going to talk about today is not ending up in the situation where we're collecting data for data's sake and then we're not actually doing anything with it. As previously introduced, I'm AJ, I'm a UX researcher at Ocado Group, so we build an end-to-end grocery logistics platform to get supermarkets around the world online and into an e-commerce space. I guess we start at that end with you, Monique.

[00:01:06] Monique: Hi, my name is Monique, I work in the data science space at a company called Trustpilot, where users can share their shopping experience and also businesses can learn from reviews to improve their products and service. I work in an embedded team focused on products, and my mission is to build data products, help stakeholders under decision making, and improve experience for business and end users.

[00:01:41] Romina: Thank you. My name is Romina, I work at SMG, which is the Swiss Marketplace Group. Maybe a bit too loud. I work as a product designer. We work in financial products so we do a lot of very technical research there, and it's a very nice experience. It's a recently formed company and it's quite large, so things are getting a bit complex there as well.

[00:02:11] Ashley: Cool. So my name is Ashley May, I'm a lead UX designer at Trainline. For those of you that don't know, Trainline is the world's leading independent rail booking platform, so our mission is to drive sustainable travel by making it easier for customers to book and search for tickets. We're purely rail and coach only, because that's part of our core tenet in the mission, to drive sustainable travel. And I head up the lifetime value team within the products team at Trainline.

Where does the data go once it arrives?

[00:02:47] AJ: Amazing, welcome everyone. What we're going to start with talking about today is, obviously in all our different functions, data science, product design, UX design, UX research, we're all collecting data regularly. On a daily basis data is coming in, we're kind of inundated with it. Interested in knowing how you go about storing that data. What is step one for you, where does it go once it arrives with you? Maybe we'll start this way down the line, so Ashley.

[00:03:20] Ashley: Thank you. At Trainline we get numerous different sources of data. Obviously we've got government transport data, which is given freely, it's not sensitive, it's related to transit and transport. We also have market-based customer research, so we're active in Europe and the UK primarily. And then we have our customer research team who are seated within our UX team basically. So within our UX team we've got UX designers, researchers, and localization and content UX writers, and the goal of that is obviously to make sure that we've got the right voice for the right market and that we're conducting the right research.

[00:04:08] Ashley: To that end, we have various types of research that we get, so whether it's diary studies, or primary research by holding workshops, which we're able to do again, or if it's online, using tools like User Testing to be able to gather these things. We store those in a research repo, which for us is using Confluence wiki essentially. The purpose of that is to enable us to make sure that the research is visible throughout the developmental process, so that tickets can be linked back to the research for context, and it enables the research team to be able to aggregate all the research and visualize it for the people that need to see it.

[00:04:56] Romina: Okay, so that's pretty straightforward, and I think it's quite similar for us as well. We have a lot of sales data coming in, we have the product teams that work together with researchers to generate their primary research, we have a marketing team that also works on their own to do a separate, let's say, side of the research. Recently we started collecting all of that in Notion, and we're going to see how that works, because it's not always super straightforward. Before that we used to do a lot of presentations from the research team sharing insights and conclusions, which I always thought was super useful and it's something that I enjoy a lot.

[00:05:36] Romina: In our case, because we have quite small teams, it's easy to share the data across the teams and to make sure that the conclusions reach as many of the people in the different departments as possible. For me it's always a focus to try to bring as many people on board into the information that we have, so I'm very human about this. I think it's important that people gain an interest in the research or in the data that we have at our disposal, that they understand why it's valuable, why we even go through the trouble of collecting and drawing conclusions out of that data.

[00:06:14] Romina: I think that once they're involved, even the sales team and everyone else can understand that there's a reason why we have them collecting data, why we need it done a certain way, there's a reason why we are going out there and interviewing users. Everyone from the CPO to the CEO should see the value of that. For me, we have all of these tools and we have all of the documents and we have all of the presentations, but at the end of the day it's about building bridges between these teams and making sure that they all understand the value of what's in front of them. That's what I find to be most helpful for us.

[00:06:51] Monique: For data science it is a bit different. The data as we understand it is, I would say, the raw data, and once we run analysis and we do our research that transforms into insights. In terms of Trustpilot, as a review platform of course we have different sources, we have reviews themselves, we have user engagement points and so on. So depending what our focus area is, and the output, we store that. Documenting is actually mentioned in Confluence pages for example, where it's accessible to everyone. For data science, as it evolves, there's also a little bit of code, and it's important to have that also documented in a repo so we are able to reproduce that if we need to.

[00:07:54] Monique: I think how we share with others really depends on the stakeholder, so what our audience is, and if that's technical or non-technical. Usually we also have the Confluence, that's slightly less technical, then of course the code itself, that's the repository, but we also have presentations. And depending if that's insights coming from live data, we also have dashboards that allow us to monitor insights across time. So I guess it depends, but usually it's a very standard way. Think of the code and the Confluence pages and the dashboards.

Making passive research available across the organization

[00:08:44] AJ: I love that, and I assume that resonates with most people in the room. You've got sort of passive sources, so the things we put on Confluence, the things that sit there, and then we've got our more active, we do presentations, we put that human face on it, we come and bring our insights to the wider group. Thinking for now about the more passive side, I mean we're in a room, it's filled with books. How do you, with so much content in a passive state, how do you make that available to people in your organization and how do you get them access to that, to know what's around?

[00:09:24] Ashley: I don't mind who starts, I can share our learnings I guess. I think for us one of the big changes, I guess for everybody, the global pandemic, was Covid. You went from having the ability to all be in the room and give these kind of wonderful and enriching research presentations, to onboarding new joiners where perhaps their entire time with you is spent remotely and you never meet in person. I think that was daunting, I can imagine it was daunting as a new recruit, and as a team we felt that we needed to scale up our design ops and our onboarding.

[00:10:06] Ashley: To that end it brought us a side benefit, the goal being to create an onboarding package for new joiners. So when the new team cohort joins, they know which department they're going to be working in, and for us we've cherry-picked all of the existing research for them and surfaced it, so they can go through what's already been gleaned, they can look at the insights that are current in the market. And crucially we gave them the time when they joined us to be able to digest all of this. It's not just a bag with a water bottle and some socks, it was a useful repository basically that they can call on, so that when they do get started, and our role was essentially nothing shipped in the first couple of weeks, we want you to be on board and we want you to learn, they can hit the ground running with a much better idea of what the research has already given us.

[00:11:03] Ashley: Furthermore, to the discussion that was going on yesterday, Mike Green was holding a forum on communicating research, and one of the points made, I think it was from a developer, was how do we make this stuff more visible? How do I see it and what can I do with it? When he talked about dashboards, everybody agreed that was a really great visual way of being able to digest research.

[00:11:30] Ashley: One of the nice things I learned from a previous conference, and I thought about it, it was a guy named Martin Reading [?], who used to be the chief product officer at Virgin Atlantic. He introduced this idea of a research museum, which does feel quite suitable given the space we're in. It was this concept of creating a pop-up within your organization in a high traffic area, like the main reception or something, literally having a roller banner and just setting up and making visual the research that you've got from a recent case study. I have to say we've got things like customer lifetime journeys on the wall and stuff, we haven't gone as far as a research museum, but I love the idea of it because it brings that happenstance into it, that you can see the research and new eyes can see it and understand it in ways that they might not have done before.

[00:12:26] AJ: Before we move on, what I love about that, and I've not contemplated before, is having research and data thrown at people on onboarding. So you're saying maybe you don't do the whole research museum thing, as Virgin Atlantic maybe do, but actually the idea that that was in a high foot traffic area. Onboarding is one of the highest foot traffic areas of the business, right? So I think that's great, that you get it in there, get it in early and then build a culture that's based around always looking at data. That's really cool.

[00:12:58] Ashley: Absolutely. And like I say, the crucial factor in that is time. We don't want you to be shipping within the first week. It's great when you do, but it's much better to have an underpinning of some useful research data that you can build on.

[00:13:13] AJ: That's great. Romina, anything to add?

[00:13:16] Romina: We had walls that we built right before Covid started. It worked until it didn't, everyone couldn't see it anymore and then it stopped working. I think if you're in a very well structured team and you have access to all of those repositories and they're well put together, that's wonderful. However, you may find that you work in a company that doesn't have that, which I've done a lot. So what happens when you don't have all of that stuff, when the research isn't structured and you have people being onboarded who don't know what's available and there isn't always someone there to tell them?

[00:13:47] Romina: I encourage people to be the advocates of research. You can be the person who loves research in your team, you can be the one who introduces people who are being onboarded to that kind of stuff. You can organize your own sharing sessions and be the person who wants to speak about a certain topic that has been researched in the past, and when people are being onboarded, be the one who transmits that information further. Because I think people are much more likely to remember it if you address it like that, and to take it seriously, than if you just share what may be an unstructured document or something that doesn't provide you with a lot of clarity. And if it's person to person, even if it's online, people get to ask you questions and they get to get more involved in that. So for me it's like taking the passive research and trying to turn it into active research, or something that's still alive in people's minds. I think that could be really helpful, at least if you're working in a company that's not structured.

[00:14:44] AJ: No, I think that's wonderful, and as humans we transfer so much knowledge through storytelling, that actually taking something passive and moving it into active will engender a level of engagement with it that maybe the passive doesn't get.

[00:15:00] Monique: I think that's quite a tricky question, how to make it available and everyone aware without you manually alerting people, hey, we have that, be aware of that. I think there are a few components of that. As Ashley mentioned, the onboarding part, I think it is not drowning people with data, but it's, okay, this is how you can find things. I think it is maybe the challenge in most places, if we don't have a standardized way that you organize to find things, content, research, then I think it is more difficult.

[00:15:47] Monique: Also I think it is the mindset. It's like the lazy approach: don't try to reinvent the wheel, don't try to do all the research again, let's see if it exists first. So either looking to Confluence pages, a quick search actually can save you a lot of time, and asking people. And I think it is not so enjoyable but it's necessary, documentation is really important, documenting everything. It actually saves a lot of your time, because you can basically replace some meetings with, hey, check out this documentation first, if you have some follow-up questions then we can discuss. From my experience that helps quite a lot.

[00:16:48] AJ: When you say you get lots of questions and then you say, hey, go and check this out, how much of your time, as someone who is senior in your company, who knows where all the data is? I know that I end up having this a lot, people come and go, hey where's this piece of data? How often are you batting away people, how much of your time is spent going, go look over here, go look over here?

[00:17:09] Monique: I think it's also an opportunity to train people: hey, we have this space, you can search. Especially if they are new, I think it is important to spend more time. But also, sharing sometimes a link, I think it's better than having a whole meeting to explain what is in there.

Data decay: how do you know research is still fresh?

[00:17:31] AJ: Makes sense. We can put a slash and then write signpost after our job title as well, for quite a lot of the time. On that, when we signpost people towards data that currently exists, data in previous research, to your point, Monique, let's not reinvent the wheel. If we've done this research, why would we go and do it again? Let's save time, let's go back to it. How, if at all, do you label up data so that we know that it's still fresh? How do you deal with data decay in your organization, so that when you signpost someone towards it you're confident that that piece of data is still relevant, as opposed to something that's expired and we need to do new research?

[00:18:13] Monique: I can start. I think as part of documentation it's quite important for you to list your assumptions, time frame and considerations. I think that is the core of what needs to be in your documentation: your assumptions about the data, your time frame, and the considerations about that in the future. I would start with that, but I'll pass because I'm curious also to see what others have to say.

[00:18:51] AJ: That makes sense.

[00:18:53] Romina: First of all, if we're learning from proper research, we label everything in the sense that we write what the context of the research was, what exactly were we trying to find out, when did that happen, and we put in all of our hypotheses and all of the things that we had in our mind when we started looking into that. I think that just provides you with the context to figure out whether that still applies, whether that's still the case. I think most of the time in our case the people who are actually doing the research, or were involved during the research, are the ones who know if things still apply.

[00:19:28] Romina: I can't say that there's a universal tagging system where everyone can say, hey, this is right now still valid, or it will expire in three months. It would be lovely to do that, maybe, but it's not the case. So the context really helps you a lot. What exactly was being studied, and where and why, and what were we trying to find out? Is that still something that we care about, does that context still exist? And then from that you go on to figuring out whether that information is still valid, whether you should still take it into consideration. But labeling as such, I can't say that I've done that or seen it done very well.

People decay: what happens when the people with the context leave?

[00:20:03] AJ: It's tricky, isn't it, and like all qualitative research there's no right answer for how to do things and where it goes. On that point then, Romina, if the answer is, well, this is fine, whoever's written this documentation up, I'll go talk to them and get some context from them. On the flip side of data decay, what about churn and people decay? You've spoken about people being really important in transferring data. What happens when those people that have the context, to Ashley's point, get pinched by someone else? What do you do in that case?

[00:20:34] Romina: They left, right? I think that's one of the hardest things, because when those people who are the repository of knowledge for your company are leaving, then that can actually be a considerable loss and it can be very hard to get around that. You can do all the documentation you want, if the person who was there and actually cared about that stuff isn't there anymore, no one's going to remember exactly what they tagged, where did they put it, what were they looking into, what were their interests.

[00:21:02] Romina: That's why I think having the research team share their knowledge with as many people as possible is important, because if more people in your company understand the value of that, if they become more invested in it, they will still remember that they got something out of that maybe three months on. And when someone else comes on board they will still be able to say, hey, maybe don't go and start doing this all over again, because our previous researcher or whoever was taking care of this looked into this.

[00:21:28] Romina: I think this information in people's minds, which is hard to keep because we all have too many things to remember, is for us at least invaluable. No documents can replace us. You can always create a repository of, hey, these are the most important things that we think you should still take into consideration and care about when someone else takes over for me, let's say. A researcher or someone in charge can do that, but you still need to always have this person in there who is the one who really truly cares about what's being researched.

[00:21:58] AJ: Yeah, makes sense.

[00:22:00] Ashley: I think I put in my talk, which was recorded, that humans by nature are kind of squashy and a bit weird, and that our processes often then end up being the same. Obviously we're not saying that's the weakest link, because it's all human, it's all engagement, and we need researchers who are skilled and able to ask the questions and surface the research. But for us I think it goes back again to that idea of good housekeeping, of data and of documents, and leaving things better than you found them.

[00:22:40] Ashley: I think the likelihood that you're going to see something through, depending on if it's a multi-year piece of research, is potentially slim, so it's a really useful thing, whether it's creating design documents or organizing research repositories, to leave it better than you found it, and leave it in a way that somebody who's new to the company can absorb it. It's another Covid learning I guess, but that's one thing that we found that's really benefited us.

[00:23:14] AJ: It's kind of interesting. So Romina, you're there, you're like, let's get the people to do it, let's stand on a soapbox and megaphone our way through, tell everyone about the research. Ashley, you're like, they're going to leave, let's put it down in documentation. Monique, sitting on the end as a data scientist whose data is fresh and in dashboards and comes in cleanly on time, I envy you greatly as a qualitative researcher. Where do you think you fit? Are you more team Romina or team Ashley on this one?

[00:23:40] Monique: Thank you. I think it's harder to say. I think that also depends on which stage we are and the state that we're under, research and answering questions. I think it's also a different approach to that. I think there is no answer to it. But I think it's more in the early stages, I think it is important, a mix of documentation and also communication there initially.

[00:24:22] AJ: That's interesting, a really good product answer as well. It depends. It depends is like our go-to answer for everything. I mean, documentation also takes time. You can spend that time documenting or you can spend it sharing.

Getting research applied, not just discussed

[00:24:34] AJ: Romina, when you're there and you want to head out and you want to share your data and everything with it, what is your go-to way to engage and make sure that that research is applied, as opposed to you've just had a chat about it? What is your action to make sure that data is used rather than just sits on a shelf or is passively listened to?

[00:24:55] Romina: I guess it depends who your decision makers are, who you need to get involved in that process. I have on occasion had our CPOs help conduct research, because then they really get interested in it. If you have a CPO do an interview with a user they suddenly start saying, oh wait, this really makes sense, now we should do it, we should definitely do it. So I go a little bit lateral about this, to be honest, it's not just one straightforward way.

[00:25:25] Romina: You can have a lot of good research in there and have the decision makers in your company just ignore it because they don't see the value in it, and for me that's the saddest thing. If you're lucky enough to be in a company where the conclusions of your research are followed through and respected and they are taken seriously, that's wonderful. But if you're not, just understand who those people are who are going to make a decision about your data and about how to apply it, and try to get them on board.

[00:25:55] AJ: Nice. And you know what you were thinking when you did that research, you know what you're trying to figure out, it's just a matter of getting that point across to them. Ashley, Romina makes a great point, documentation takes time. When we've got a million competing priorities, how do you tie that into the workflow that you have to do?

[00:26:17] Ashley: I think for us it goes back to the structure of the team, the fact that we have an integrated research team that we liaise with daily. It feels a lot more collaborative in terms of surfacing what needs to be shared. It's not the researcher's role to be a filter in the process, the idea is that it could be me conducting an interview or it could be them. I think it's a fantastic thing culturally, the idea of getting people out of their core roles.

[00:26:48] Ashley: It's another thing we were discussing yesterday, the idea of when you are in a smaller scale company and there's no research resource. There's obviously ethics that come into play in terms of conducting research, but doing something, showing the willfulness, is better than doing nothing. Whether it's a small scale product or a startup or even an internal product, somebody was talking about internal tools yesterday and that you don't have a customer facing product, so how do you recruit an audience? That's a perfect test bed for something like that.

[00:27:27] Ashley: For example, I used to work on the internal customer services tool at Trainline, and it was great because you had telephone call data, you had NPS scores with post-call questionnaires, but we weren't doing customer or end user research. We just decided that we can do this, this is something we can do. So we recruited our customer service reps in Edinburgh, we had remote calls before remote calls were a thing, with our team at Sitel in India, and we set up sessions and we observed how they used things. It was much more guerrilla and scrappy and probably not formal research and done by the books, but it's the idea, I think, that something is better than nothing, willfulness is better than inaction. And everybody at the end, whether it's in the product team or outside, was much more on board with the outcomes.

[00:28:24] AJ: I love that phrase, it's a little bit scrappy but action is better than inaction in this kind of thing. I kind of agree. I don't know about how your work comes across or how you get thrown problems to solve, but in a lot of cases someone will come to us with something to ask about and so we already, I guess, have buy-in. And then the way we bring people on the journey is essentially, did we give you more confidence? Is our data to give you more confidence? Ultimately that's the final question I'll always ask at the end of a project: with what we presented you, are you more confident to make the decisions you have to go and make? That I don't have to make, it's on you. So did we get you to where you needed to be?

[00:29:07] Monique: I kind of like that. When I think on the use cases, keeping research alive, I think it's also the use cases. If that's to support some decision making, then it's okay, then it's there. And also on the documentation part that you mentioned, and how long you should spend, I guess a lot of people struggle. But I guess a good rule of thumb is proportion to the time you spend to do the research. You don't spend more time documenting it than the time that you actually put in the research. That helps you on the level of details when you're doing things.

Engaging people while the data is being collected

[00:29:51] AJ: Makes sense. I just want to jump on, as a final point, what you were talking about about bringing people along for the ride. I know a few other people have spoken about it over the last couple of days. What are your tactics to get people engaged? Because I guess we're talking about getting research applied. We started thinking at the end of the process, like we've collated all of the data, how does someone now engage with it? But to your point, that's almost too late. How do we engage with them as that data collection is happening? What are your tactics?

[00:30:23] Romina: I mean, if you have departments [?], then it shouldn't be maybe too hard to bring them on board. So probably there's even a step before that, where you have to build these relationships so that people have the confidence to actually go out there with you and do something that for them is maybe not comfortable. Of course there's people like in the sales team, they talk to users all the time, but they might not talk in a way that yields data that's maybe really useful to you. But then you have a lot of other disciplines and a lot of people in your company that never talk to users, they never interact with someone on the other side of the wall, and they just have to do that for the first time. And if they don't trust you, I don't think it's very easy to get them to do that. So first of all you have to build those relationships.

[00:31:10] Romina: And then I guess you can be, if you're really cheeky about it, you can build in all sorts of other things from our biases and research talk. There's some sunk cost effect in there: I invested some time into gathering this data, now I want to actually see what happens with it. You can take advantage of some small things there as well. I'm being cheeky, I know.

[00:31:29] AJ: No, I love that, just turning around to the stakeholder and being like, you've already paid for it, do you want to read it or do you want to leave it on the shelf? That's great. That was brilliant. Monique, where you're collecting data from systems as a data scientist, how do you bring people into the journey earlier on?

[00:31:47] Monique: I guess that's how I approach it, in my talk, about working in embedded teams, it's quite important. From early stages, working with UX and either helping or trying to answer some of the questions through data, I think it is quite important. And it's also nice because I think when you come from a technical background, you put yourself in other people's shoes and try to think how they think, and that's a really good exercise actually. So it avoids you losing yourself in the research. Working in cross-functional teams, I think it's quite helpful.

[00:32:36] Monique: But also, don't spend too much time. As soon as you have some findings, go out there, share, because if you spend some time maybe in a wrong direction, then someone can actually say, no, we're not actually looking at that. So it helps you to focus on the right things. Our PMs or developers, everyone really, at this stage, especially early stage, everyone you can talk to, I think helps you.

[00:33:11] AJ: That's really great, and you're right, the second you have something to share, we should always be sharing it. We're not a vault, it's not data for us to keep, it is there to help us make decisions and move things forward in our product thinking. I also really liked what you mentioned about trying to get into the mindset of, I always try and treat a stakeholder as a user themselves. What are they needing from me? Is the product of research, what is that product, and how will they engage with the data? Because again, at the end of it, it's not data for my sake, it's data for something else.

[00:33:48] AJ: Ashley, you were saying about onboarding. Monique brings everyone along for the ride, Romina's turning around saying, hey, come along, watch some sessions. You're right at the beginning of the process, they've just entered the company onboarding-wise and you're already throwing research at them. Practically, how does that conversation work within everything else in onboarding for you?

[00:34:10] Ashley: I mean, not as a new joiner, but as one of the people that absorbs the research, that may take part, but is also the person who says, how do I know what I need to know? I'm the person who speaks with the researcher and wants to glean that, and then the relationship is making use of the researchers and expertise to be able to get the insights. But that lends itself to our team structure I think, that it's collaborative, but it's using inductive reasoning with the research team to essentially say, we're not looking to come up with a solution, we're just looking to develop an outcome. So making sure that it's not a closed-minded goal to just, is it a go/no go thing, or we're just testing, but to better understand what that outcome is going to be. And then once we have the research and the insights, it's all our jobs to shape that and create the product that solves the problem.

Q&A

[00:35:17] AJ: Amazing. We have a few minutes left if anyone in the audience has any questions for our lovely panel. Someone down at the front here.

[00:35:38] Audience: I know that one topic in the industry that's coming up recently is the democratization of user research and empowering teams to conduct research, because I think in reality you may not have a researcher available to participate in everything you do. Can you guys speak to how you see the democratization of user research across your teams, when you involve a researcher, and how you equip teams to do their own research?

[00:36:04] AJ: Nice. Do you have anyone in particular you'd like to ask that to, or should we just go down the line?

[00:36:15] Romina: We had, at some point, limited resources for research. I've worked in teams where we just didn't have enough time for the researchers to participate in everything or take their time to really be involved with everything. So we did small boot camps with them, where we went through different types of research and they taught some members of our team how they should do that better, how they should structure documents for their research before going in, during the actual research, what's the process. We did a boot camp of research.

[00:36:49] Romina: I guess in that sense you can say that we kind of all leveled up together with them, which just enables you to have more resources to go to when you need to do research and you need to do it in the individual teams. You have more people to go to, they can always check back in with the actual proper researchers and make sure that they're doing things right and they're understanding the conclusions right, because I feel a lot of the time that can get lost. But it seems like less of a cost then, or less of an effort. You don't have to maybe have a huge team of researchers if the researchers that you have on board are actually able to train up a little bit the teams around them. So it's a lower cost in the end for us to do it like that.

[00:37:31] Ashley: I'd say further to the point on costs, we were discussing last night about the inordinate price of some of the research tools that are available, and that's another aspect of democratization I guess, the idea of the scale of a company being shut out of being able to do certain types of research by the limitations of their budgets. I think tooling plays a big part in that. One of the key themes is making use of what you've got. I think we found, whether it's a Google spreadsheet or slides, I'm not suggesting we present back spreadsheets, but just as a tool for gathering data, the idea of making tools that are available to everybody is an important one, so there's not an exclusivity to the process of research.

[00:38:20] AJ: There's 40 on the clock, okay, quick.

[00:38:22] Monique: I think taking advantage of tools, for sure, it saves quite a lot of time. I think coordination across teams as well, in terms of if you're lacking resources. But also if the goal is not proportional to the resources, then I think it's also worth thinking, let's think what we're trying to achieve, because then we're not compatible. So it's a short answer, I hope that's a compliment out there.

[00:38:52] AJ: And from our side, similar to Romina, our PMs are very hungry for this sort of stuff, so we'll run similar sorts of boot camps, asking good questions and going from there. And that is time everyone, so please give a huge round of applause to our lovely panelists.