Boosting Product Team Efficiency by applying research techniques to Developer Experience/Productivity

Max Griffiths, Harri Thomas at UXDX EMEA. Video: https://youtu.be/SNCWDRq1fTA

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introductions

[00:00:08] Harri: Hi everybody, thank you for joining us today folks. We know that we are in competition with another talk happening on the other stage, so we appreciate you making the time for us. Quickly by way of introduction, as Katherine mentioned, my name is Harri Thomas and I lead growth at Great Question. Great Question is an all-in-one UX research tool built for the enterprise, and if you'd like to find out more about us you can come around to the booth and I'd love to give you a quick demo of the platform.

[00:00:31] Max: Hey folks, Max Griffiths, first time up on stage here at UXDX, so great to be here. I work for a company called Thoughtworks. We're a global consulting company, very focused on software engineering practices. Anyone here heard of microservices, put hands up? Anyone heard of data mesh? Yeah, these are some of the terms that we've invented and then published about, and now they're becoming commonplace. Me personally, I work in the engineering effectiveness space, which includes developer experience, hence why I'm here today.

[00:01:05] Harri: Perfect. So today we're going to be talking about enhancing product team efficiency with developer productivity. And as Katherine mentioned, I come from the UX world, I was a UX researcher at Facebook for a number of years, and so I don't know much about developer productivity. So I thought we'd kick off today's talk with Max giving us a bit of a definition of what developer productivity is, and importantly what it isn't as well.

What developer productivity is, and what it isn't

[00:01:30] Max: Yeah, so reflecting on this, for those of you that know a little bit about developer productivity or developer experience, I think developer experience is a much nicer thing for people to be talking about. If you can imagine, if you're not a developer, imagine you are a developer and your boss is coming down to you and saying, right, we're going to start measuring developer productivity. It feels a little bit difficult if you're an engineer. Now suddenly somebody's going to be measuring my productivity with a microscope.

[00:02:01] Max: One of the things that is becoming quite commonplace now is that we shouldn't be measuring developer productivity all the way down to the individual developer. That could go several ways of wrong. You could be looking at measuring people by the amount of code that they're generating, which is a terrible metric.

[00:02:21] Max: So the first thing to think about is, when we're talking about developer productivity, developer productivity engineering, which is quite popular in some of the large tech companies, we're really thinking about what is that ecosystem that the developer or the engineer is working in. And this isn't limited to developers or engineers. You can think about it in terms of product people on the team, you can think about it in terms of QAs or testers on the team, and to an extent, which we'll get into, even designers. But it's really thinking about, what is that ecosystem that they're working in? What are the tools that developers are using? What are the processes that they're using to develop code, to ship code, to understand feedback from customers that might be using that code? And so, technology, people, process, it encompasses all of that, and looking at ways to measure it.

[00:03:06] Max: If you're hearing me saying that we shouldn't be measuring developer productivity, you might ask, then what should we be measuring? A lot of the movement in this space is around measuring waste. So looking for areas of waste in the process, whether that's waiting around, whether that is cognitive load, things like that. So when I think about developer effectiveness, I think about what is the system that they're in, what are the processes that they're using, and then thinking about efficiency and optimization around those.

[00:03:41] Harri: Yeah, absolutely. And one point that you made in the preparation conversation that we had, which I think is really interesting and worth restating, is that for a lot of businesses, outside of the buildings that they're in, developer costs are the number two cost. So it really is something that people should be focused on, in terms of how can we make this very large and expensive line item really work hard for us.

[00:04:01] Max: I think circumstantially as well, we've just come through Covid, we've come through various economic crises, so a lot of my customers are now coming to me saying, Max and Thoughtworks, how can I do more with the same people? I don't have a lot of money right now. Probably everybody can experience a little bit of that. So how do I know that I'm getting the most out of my very costly engineering organization, and if there are areas of waste and I can claw back from that, how should I be investing? How can I tune these processes?

Who is in the room

[00:04:33] Harri: Yeah, absolutely. Okay, great. Well, I think we'll get into a lot of these topics in more detail in a moment, but let's get people's fingers working. So we do have a question for everybody, and if this works correctly it'll be great to learn from people who we've got here, because this will inform some of the questions that we'll inevitably ask next.

[00:04:51] Max: Yeah, from speaking to a few people in the last day or so, there's a lot of UX people here and I think the DX side of it is emerging, which is great, so I'm really happy to be here. So we've seen quite a good mix come through, it's great. I really enjoy knowing the audience when I'm talking to them, because like I said there's so many different little rabbit holes that we can go into.

[00:05:18] Harri: So, a great lot of designers, but a decent number of coders as well. I'm blocking the screen, sorry, I'm getting some sign language happening over there and I didn't know what this meant. This is good. Okay. So just while this is coming up, you mentioned some new terminology for me. As I mentioned, UX research background, you mentioned DX, so developer experience. You went into some definitions earlier about things that it is and isn't, but can you tell me about the emergence of developer experience as a field or as an area of interest within organizations?

[00:05:49] Max: For sure, for sure. I mean, for me I think it probably goes back about 10 years. I think Netflix was one of the first companies that I heard of who started to look at developer relations and having this. So I've always been part of a platform engineering team, or a team that sits in front of cloud engineering and trying to serve out some of those capabilities as self-service products. Again, I use that word product even though it's an internal concept, which I think a lot of people are used to products when it comes to an end customer, but this is new emerging common language now that we're using, developers as a customer.

[00:06:24] Max: And so, following people like Netflix and some of the other big tech giants who have a lot of money, who have large enterprise scale problems, and seeing how they're organizing teams, so not just having a platform engineering team or a cloud engineering team but actually having a layer of developer relations team or developer experience team whose job it is to really listen to developers, really try to understand the problems that they have, and then try to build, whether we call them products or services, whether it's better API design, whether it's better CLI tools, really try to understand that from a developer perspective.

[00:07:00] Max: And it really made me think. I've been in the business of efficiency and CI/CD and DevOps for about 20 years, but only probably in the last six, seven, eight years have I been seeing developers as customers. And I think it really helps reframe this. If you go into an organization like I do quite regularly and they have an old school data center team used to racking and stacking servers, the idea that they would see developers as a customer of those services is quite shocking to them. But using this language has really helped, I think, turn a corner in the industry.

[00:07:31] Harri: Absolutely. And it sounds like that's a cue that they're taking from design ops and research ops, which are now, I wouldn't say mature, but certainly rapidly maturing.

[00:07:40] Max: Yeah. And this thought that design ops and research ops are there as a service to enable and empower the people that they're working with.

[00:07:47] Harri: 100%. Rather than a layer on top, an administrative layer. It's about how can we support these folks, and I think we'll get into that in just a moment. But having a look at who we have here, we've got a really nice even spread, apart from that 3.74% who have said there's something else. So if you're something else and you're not captured here, could you put up your hand please? We've got something else at the back, we've got something else at the front here. Yeah, would you mind sharing with us if we can get the microphone to this person who's got the microphone over here? Come to the front, I think.

[00:08:22] Harri: Sorry, you're answering the question? Yeah, please. What do you do?

[00:08:25] Audience: Yeah, so hi, I'm Mariel, I work at, and I'm in what we call core operations. That is on our operational team, cross cutting across product, technology and UX. And so I work on operational challenges across the entire product, tech and UX org, as well as individually with engineering leadership, UX leadership and product leadership, to help make sure that things run smoothly.

[00:08:51] Harri: So what does a successful quarter look like for you? How do you measure your own success, or how do you know that you've been doing your job well?

[00:09:03] Audience: A lot of it is structured around making sure that we have standard reporting procedures and processes in place, to make sure that we are maturing overall as product and tech org. And so one thing for example is, we started introducing a policy around how we communicate across the org, and monitoring how much emails come in, do we have a strategy in place that we're implementing there around cross-org communications. Another thing was around making sure that we have capacity planning resources to track who works in what team, and that that is maintained and updated. So really, my team developed setting up that system and then maintaining it and training leadership across product, dev and UX in how to use it.

[00:09:43] Harri: Amazing, thank you for that, very comprehensive answer.

[00:09:48] Max: Yeah, great answer, let's give her a clap please. Yeah, well done, love that. So that's awesome, right? I mean, that's where you see companies who are understanding the need to understand some of these organizational channels and pathways that help get work done, and actually going one step further and putting people and a team around it, which I've seen as being a really effective way of bringing attention to the importance of this. And what should we be measuring, how should we be measuring it, what's the purpose of all of this?

[00:10:17] Harri: I think that's so true, and nothing says that a company values a position or a role like putting head count against it. There's a whole lot of investment that goes in around that, and obviously friction that is created when those roles are created. But in your role at Thoughtworks, you're obviously in a consultancy position, how many companies, or what percentage of companies that you're working with would you say have a role similar to that one, where they're thinking about operations, operational efficiency across various different teams and communications within the company?

[00:10:44] Max: In my experience, and I tend to work with not the bleeding edge of startups and some of these companies, like N26 I would probably put in that category, but more on the innovative but larger scale enterprises. So I think that kind of behavior is something that we're actually recommending as Thoughtworks, this idea of enablement teams. If anyone's read the book Team Topologies, fantastic book about these kinds of organizational dynamics and how to set up teams around engineering for success. So yeah, it's something that we recommend. Percentage wise, definitely less than 50, maybe even less than 25 percent of people actually doing this and doing it well.

[00:11:21] Harri: Interesting. So let's see what the audience has around this. So 15% says Max, are we going to be higher or lower than Max's customer base? What's your bet, Max?

[00:11:32] Max: Don't know. I feel like we need to elevate the music when the votes are coming.

[00:11:35] Harri: Well, you can do a little dance for us.

[00:11:36] Max: 15%. I think we've got a pretty cutting edge audience here, so I'd be surprised if it was less than 15%.

[00:11:47] Harri: Come on blue, come on blue. It's like a horse race, isn't it, for those that remember the computer games.

[00:11:55] Max: Yeah, so I think in terms of less about specific percentages, but certainly in terms of the proportions, this would have been a predictor for me anyway, having spoken to a few of the crowd. Like I said at the beginning, it's felt like a very UX heavy conference, I think that's where it's come from, and developer experience is emerging, up and coming. So I'm glad that people haven't heard of developer productivity or developer experience or effectiveness. It gives me purpose today.

[00:12:26] Max: Yeah, so feel free, if you want to know more there's obviously a lot more detail we can go into about that, we'll touch on a little bit here today.

[00:12:34] Harri: That sounds good. So why don't we hand the mic over to somebody who has answered, we're actively working on developer productivity or developer experience at work? I'd love to hear from you. So if you answered blue, can you please put your hand up and we'll get the microphone over to you. It's a fun microphone, it's a huge soft cube, if you haven't seen it. They say it's throwable. I'm keen to throw it, let's share it around a little bit if we can. Anybody else said we're actively working on developer productivity? Surely. Too many shy people.

[00:13:03] Max: It was phantom buttons, maybe.

[00:13:04] Harri: Maybe it's bugs in the code. No, all right, we can move on, that's all right.

An audience view: where the real priorities lie

[00:13:09] Harri: Okay, so what about somebody in the yellow category, so we talk about developer productivity but aren't working on it? I'd love to better understand how you facilitate those conversations internally and what precipitated the initiation of those conversations.

[00:13:22] Max: This one's interesting to me, because this is the conversation I have with customers. Anyone would like to talk?

[00:13:31] Harri: You'll get a warm round of applause regardless of what you say afterwards, so it'll be good.

[00:13:35] Max: All right, nobody? Nobody? Come on guys, it's meant to be interactive. I'm getting some sheepish smiles. I know that there are some yellows out there. Thank you, sir.

[00:13:46] Audience: Yeah, don't look at me. Thank you. Yeah, I'm a bit in between the working on it and not working on it. We actually have a team in the company that's called a developer experience team, which are supposed to do things that make the developer productivity higher. But I think it's sometimes still a little bit questionable about where the real priorities lie, and whether we're really working on improving the developer productivity, because it starts with tackling tech debt and the tech structures around it, and we're kind of trying to do that a bit in between the things, but it's not really a focus. And I feel like that kind of ultimately slows or decreases development productivity again, because you're trying to set up new tech systems while at the same time trying to build new features, and that combination is sometimes quite mismatching, I would say, to really make progress.

[00:14:32] Max: So if you could have a vote or some authority or power into changing this, do you have ideas how you would go about doing that? Is there a way that you would prefer to see this work?

[00:14:46] Audience: I'd sometimes just love to, so I'm a product owner, and I would just sometimes love to listen to the developers and just give them like a month to clean up the biggest technical blockers that they have, without any kind of other products or anything high prio coming up at the same time. And I think that's a difficult one to get done, or pretty much impossible, I'd say.

[00:15:05] Max: And yeah, how large is your organization, where do you work?

[00:15:07] Audience: It's like 250 people, it's really purely digital, so almost everybody is in tech or related.

[00:15:13] Harri: It's an interesting point, isn't it, because there's this real urgency and speed required in the development processes, and it can sometimes feel like taking time out to operationalize and improve developer experience can reduce speed. But I think what Max would argue, and don't let me put words in your mouth, is that taking some time away from writing code to work on those operations would actually speed it up in the long term. But it's about making sure that you get alignment with executives so that you can find time in what is otherwise a busy road map to do some of that.

[00:15:44] Max: I think, as developers and engineers, there is a tendency to jump into technology. And I think this area in tech around engineering tools and practices, I think it's quite prolific, so it's very easy and attractive to say, oh, we're not productive because we don't have that tool, or I want a new IDE, or I want to be able to use this CI/CD tool, or whatever, I want to move to this language. And I'm always tempted to focus more on the people and process first, and then find some of the evidence that would suggest moving to a tool or improving some of these technology aspects is actually going to have the benefit to the metric that I wanted to improve in the first place. So I always think, and this is where we come into the research side, doing some lightweight research initially can really help. But I don't want to give too much away in the next one, so let's understand.

Finding the waste

[00:16:40] Harri: I don't think that this one's for us, so let's do this one. So, what are the different types of waste you see and feel in your teams? Let's focus on engineering teams, but I know that we've got a good spread of folks as well, so product teams in general. I'd love to get your input on this one.

[00:16:55] Max: So for those of you that know anything about lean manufacturing or lean process, there's a lot of waste analysis that goes on, this idea of Kaizen, and it all came back from the Toyota production systems back in the 50s. But very fascinating stuff about value stream mapping, if anybody's familiar with that, where you can really map out your value stream in a flow, whether that's how your team works or how the organization makes business. But these are just a small selection that I think are quite relevant to most teams, and there's a lot more you can do on this. It's kind of fascinating stuff and I love digging into this in organizations, because it quite often shows a lot of low-hanging fruit that you can go after quite cheaply.

[00:17:38] Harri: And it's a change of perspective as well, because we think about developer productivity as output, but this is almost the yin to the yang, and we're thinking about, okay, where are the areas of waste? And so you can come at this from two different angles, and I think this particular perspective is quite interesting. And so we see complex dependencies in there. I can see a lot of crying emojis as well, I hope that's because of the waste and not because of our presentation style so far. Okay, discovery of information is interesting, waiting time.

[00:18:03] Harri: But I guess one question that I'm keen to dig into is, how do you go about discovering where these different types of waste are within your organization? So everybody has a sense of where they are, but how do you go about qualifying and quantifying some of these issues? Because inevitably, to the point that you made earlier, you'll often need to make a case to present internally, or funnel up the chain, in order to get the resources and allocation of time in order to go about fixing them. So how do you understand these issues, what's the process for doing that?

[00:18:33] Max: Yeah, I mean, this is where we borrow a lot from user research and the UX side, so looking at qualitative, quantitative research. There are companies now that are spinning up that are solely focused on this, companies like DX, or getdx.com. So how can you, as a product owner, start the revolution on this, or start the initiative, by thinking about some key questions that you could ask your engineering team or a set of engineering teams, trying to get that heartbeat or that pulse quickly on what are the problems?

[00:19:04] Max: Again, with engineers it's easy for them to focus on solutions and dive into, this is the thing that I want. But trying to push back on them and actually characterize the problem. I think a lot of the talks here over the last couple of days, when we start looking at Double Diamond and that discovery process, really trying to anchor people in the problem space. Find quick tools, I'm talking about some workshops that might take an hour or two, getting people in a room, finding out where some of those problem spaces might exist, and then finding those areas of commonality between teams. Obviously if you've got one special team that has one special problem, we all know that's probably not the first place you want to start. So looking at what are some of those common problems with common systems that are spanning across multiple teams, because that's where the margins are going to be made.

[00:19:45] Harri: And who's doing those investigations? Is it other engineers, or is it researchers or designers who are in a parallel problem space and maybe are better equipped to conduct some of those interviews?

[00:19:58] Max: I don't think we involve designers and researchers enough in this DX space. We do do it, Thoughtworks does it in all of our developer experience projects, whether we're building developer platforms or developer portals, we try to introduce designers that are designing good interfaces, doing good developer research. But it's not common, and I think we need a lot more of this. There's a lot we can learn from UX.

[00:20:21] Harri: Well, I think what we're seeing at Great Question certainly is that there's broadly speaking the democratization of research within organizations. So research is moving away from being a core role to being a strategic advisor or enabler to other parts of the organization. And as part of that they're looking to apply those research skills, which oftentimes come with PhDs and years of experience, and applying them in different areas of the business. So they'll go into a sales organization and understand, okay, what are our customers saying and how can we improve the sales process from that perspective using research principles. And I'm sure that a similar approach could be taken here, although I haven't heard of it happening. So maybe this is the final frontier for research.

[00:21:02] Max: The final boss.

Fewer meetings, and starting lightweight

[00:21:04] Harri: Yeah. So moving over onto ways to reduce waste in engineering teams. I know that the speaker earlier was talking about collaboration opportunities, and I thought that that was particularly interesting. Now I think that one thing here, fewer meetings, is actually interesting, because you would often think that meetings are held up as an opportunity to collaborate and that might be a shortcut to reducing waste. But Max created the survey, not me, so tell us a little bit about these fewer meetings.

[00:21:32] Max: It's just something that comes through quite a lot from engineers, around getting dragged into meetings. But you could reword this, fewer meetings, I would say fewer ineffective meetings. So meetings that don't have agendas, meetings that are just wholesale pulling people in. Obviously there's a huge amount of remote work happening now, so it's just very easy to click, add people in, no one has to actually walk up to a meeting room, meeting rooms don't tend to get full, so you can have a cheap way of just inviting everybody. And so from a productive meeting perspective, that's really important.

[00:22:06] Max: There were a couple of options on here as well that have been trimmed off. But yeah, there's no right or wrong answer to this, but I am really happy to see a swing towards the research to understand the baseline. Yes, well done everybody, full marks.

[00:22:21] Harri: I did kind of tee that one up for you.

[00:22:24] Max: 10 points. But it's about how can you do this in an efficient, lightweight canvassing mode. And so I've helped product owners who know this but they don't know how to get started. My answer is always just try to get started, find other people in the organization that are feeling the same thing, start to build a group of people that can build some critical mass around this, because ultimately you might need to build a business case, this might need some investment. I think the gentleman earlier was talking about wanting to pause feature development so you can go and work for a month on this tech debt. That's very unlikely going to happen, it's very rare that that happens. So it's how can we claw back 10%, 20% from our day-to-day, and that's where you start looking for this waste, so that we can now work on some of these tech debt paydowns and things like that.

[00:23:11] Harri: And I think that that's a really good point. I don't think that it needs to be a wholesale change, it doesn't need to be an all-in-one type solution, it doesn't need to be a cultural revolution. It can be something that teams can start and thread in through general processes and systems.

[00:23:24] Max: Yeah, fantastic. Let's do another question.

Research inside engineering teams

[00:23:29] Harri: Let's do another question. Not that one again, not that one. Okay. What can engineering learn from research, how should research be applied in engineering teams?

[00:23:39] Max: Interesting. Yeah, I'm keen to hear from the audience on this one as well. So whilst you're answering, have a think if you've done any kind of research in an engineering team, whether that's looking at an internal developer product. I think Daria earlier was talking, who now works for Miro, was talking about moving AWS. If anyone's used AWS in their console, it was pretty horrific, now it's a lot less horrific. So it's that kind of research and thinking that I'm keen to hear about, if there's anyone in the audience. I know it's after lunch, we've got the difficult slot to get good answers[?]. But if anyone's willing to speak up, any research going on in engineering teams?

[00:24:23] Harri: Do we have any researchers in the audience who have done research with engineering teams? Yes, fantastic, up front, I'd love to hear from you. What was the project, what was the objective and how did it come about, what was the initiator of the project?

[00:24:39] Audience: Yeah, so about four years ago we were hit with a pretty bad cyber attack that took the whole company offline for a while. In coming back from that, understandably our security team were petrified and locked everything down. A lot of the developers were complaining about this, they couldn't work the way they used to, they used to have full autonomy on everything, which wasn't necessarily the thing to go back to. But we did a research piece to really put that down on paper, got some actual numbers behind that to say, okay, we understand we can't go back to the old way of working, but right now we are wasting someone. We did a user journey map of, right, they do this, they've got to apply for this, this person's in Australia, they need to then wait a day for someone in security to approve it, to add them to this, to do this. We are losing X many hours, we have 500 developers, that is 35 grand a day we are losing.

[00:25:35] Audience: And then, I suppose, money talks. It's going back to what's been spoken about, and Kelly spoke about it, a lot of, put it in money and that will get exec talking. So yeah, that's the way, I suppose, we managed to get around that situation with the devs.

[00:25:48] Max: How long did it take for that research to happen, was it a big initiative?

[00:25:56] Audience: It was probably about six weeks. So yeah, six weeks, probably about 50% of my time over six weeks.

[00:26:05] Max: Okay, so in the grand scheme of things, nothing. So one person, one or two people kind of activity. Yeah. And you're right to mention on that, developers cost a lot of money, and so you only need to find these five, 10, 15% inefficiencies, and if you're a large enough organization that can claw back quite a lot of money quite quickly.

[00:26:25] Max: But the security discussion is super interesting, because once you start putting the dollar amounts in there, you start looking at the probability of a cyber attack and how much that costs. It stops being about this fear driven behavior and it starts being like almost a balance sheet conversation. It's like, we understand the probability of this happening is, I know, one in a 100, one in a thousand, but when it happens it costs us this amount and it takes us out for this amount of business. But then by limiting developers and what they can do, that's actually a slow burn, boiling a frog by a degree at a time.

[00:26:57] Harri: Yeah, absolutely. So we have somebody here who thinks that research shouldn't be applied in engineering teams. Do we have anybody who wants to put their hand up on that one? Contrarian in the audience?

[00:27:10] Audience: No, but it's my colleague sitting over there.

[00:27:12] Harri: Yes, it was you. Yeah, Pedro over at the birth[?]. Okay, not a problem.

Signals that it's time to do research

[00:27:17] Harri: And so I think that, unfortunately, that was an unfortunate incident with that cyber attack, but cyber attacks are becoming a lot more common these days, aren't they, with lots more opportunities for people to do nefarious things with your software which can shut businesses down. And it took quite a drastic moment for your company to come to that realization. So I think it sounds like, what are some of the signals that engineering teams can be looking for that in fact we should start doing research with them? Is it just a general sense, or are there some flares that are put up in the air that, hey, it's time to do research? Or are we going to have to wait until the next cyber attack hits to think about implementing and improving the way in which developers produce their work?

[00:27:57] Max: Yeah, good question, it's a broad one as well. The thing that came to my mind is around how different functions within organizations are trying, I think, a lot harder now. It doesn't matter whether it's security or product or PMO or testing, people are thinking about these functions and about how they can reduce the friction that happens between different parts of the organization. So, like I said, I'm kind of from a platform engineering background, from a cloud engineering background, there's a lot of thought that's going into how can these things be more self-serve from a security perspective. Some of the people that are doing this very well, the security team, they don't sit in the middle of the whole process, they describe exactly what good looks like and then they mandate that developers or product teams meet those requirements, and then they have a way of testing that those requirements are being met. It's very different to the old school ways.

[00:28:51] Max: But to the question about how do you know when to do research: I think you've got smart people in your organization, most people know that things need to change, even if they can't articulate it that well. So again it comes back to that lightweight research. Who are the people that we need to talk to, how can we characterize this information, how can we build the emerging business case, like the gentleman said, about just getting some rough numbers together about what this would look like. And then I think the important people in the business that are making those business decisions, they have to listen.

[00:29:19] Harri: Well, I think part of the friction that I certainly see is, as a non-developer it's hard for me to go in and to critique the work of an engineer, because I don't know what productivity should look like, or I don't have a benchmark for that. So in my mind the way that I think about solving this problem is to get engineers into the mode of conducting research on processes and systems, but obviously that becomes an expensive proposition because you're taking them away from being productive to study productivity. And so how do you see the development of this as a specialism within organizations, and where should these roles be coming from? Is it switching a UX researcher who's dealing with customers over to doing research, or should it be driven from within engineering teams?

[00:30:01] Max: Yeah, I mean it's hard to say one or the other, because I think it depends a lot on the size of the organization, the types of skills that you've got available. If you've got UX researchers who are interested in getting into this, it's quite hard, actually. Getting product owners and product managers into internal developer platforms has been quite difficult, because there's this affinity with the end customer and understanding the business. So it's only a small bunch of people who are technical product managers, who are interested in this and understand it and understand the tools and systems and processes and understand what good looks like, really. So I think it can come from both ways. I think if you have UX people getting a little bit more technical and leaning into that, developers getting a little bit more familiar with, what are some of the things that I care about, how can we canvas these things and build feedback loops quite quickly and throughout the process, so it's not just a once-and-done thing.

[00:30:49] Harri: Yeah, absolutely. And so we've got one minute left, Max. And so if you could leave everybody today with a big takeaway, what's one big idea that you can leave them with that will impact their organizations, that they can take back and say, hey manager, boss, CEO, whoever it is that they're reporting into, we should do this?

[00:31:06] Max: Okay, no pressure. I think regardless of where you are in your business, there's probably some information, metric, data around how you're working as a team, how you're seeing value. There's probably some that you already know about that you're not sharing with other teams enough, or there's probably some very small amount of work that you could do to start getting some of that data. And I think organizations that have some of this data at their fingertips can make better data informed decisions, whether it's reorganizing, whether it's spending some investment on optimizing workflows, things like that. So I'd be thinking about where you sit in your overall value stream, what are some of the metrics or things, data, information that you could be sharing to your neighboring teams.

[00:31:46] Harri: Great answer, thank you everybody. And that brings us up to time. My name is Harri, I'll be over at the booth over there if you'd like to come say hi. Thanks Max for joining us. Cheers everybody.