Incremental Metrics: Getting Your Kpi's Right

Natalia Kaijalainen, Paula Hellemaa at UXDX Europe. Video: https://youtu.be/8hPi3q2j5w4

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Trimble, Tekla and DevOps as a culture

[00:00:01] Natalia: We are both coming from Finland, from the Trimble company. Trimble is a global company which helps and tries to transform the way the world works. I like that slogan very much, because it tells quite a lot: we try to digitalize the world of our customers and users. We work on agriculture, on GPS and satellites, and Paula and I come from the construction side, from buildings, and our product, the Tekla family, which is about construction and construction engineering. It's used for planning huge skyscrapers and stadiums, Olympic stadiums, and I think we're doing amazing work with that, and we are working on that product to make it possible. I'm an engineering manager. I was a software engineer for eight years, working on steel structures, and for the last five years I have been leading the DevOps infrastructure team, which has been making possible our ability to release more frequently internally and to automate tests and installations.

[00:01:12] Paula: Yes. My name is Paula Hellemaa, and I take care of the user feedback program in Trimble Solutions, which means that it's my job to engage users in the development of our products.

[00:01:25] Host: Natalia, you mentioned that you manage the DevOps team. What would you say DevOps means to you, and how do you employ DevOps in your team?

[00:01:38] Natalia: This is a very interesting topic, because I totally think DevOps is not the team itself. It's the process where everybody's working together on making things happen, in multifunctional teams. But the reason why I'm saying I have been leading the ops team is that our team has been making that possible. We have built an infrastructure in our company where we have automation and the ability to deploy things fast, for our development teams, which have every specialist, like QA and UX, to be able to do the features. If the framework is there and it's available, and that DevOps culture is in the company, it's much easier for every development team to be independent and do features like they want.

[00:02:25] Host: You mentioned that you're deploying quickly. How frequently are you deploying to production?

[00:02:34] Natalia: This is an interesting question, in the sense that I would like to separate commercial releases from the ability to deploy fast for our customers. As we are in the construction industry, it has quite traditional processes. They have a very long history and long expertise on these topics. They build big buildings, big stadiums; one project takes a couple of years. So usually they don't want to change software very frequently. And we have desktop software, on Windows, which means that they might even keep the same version for a couple of years.

[00:03:10] But for us it's very important that we, as a software company, develop things faster. Our ability to deploy things and make an automatic installation is about one hour after developers make a change, and our whole infrastructure is geared towards that. So when our customers would like to change faster, we are already there. I think it's very important in that sense.

Bringing users into the development cycle

[00:03:37] Host: Great. One of the themes of UXDX is bringing UX and user feedback and user research into that development cycle. Can you tell us about how you're doing that at Trimble?

[00:03:48] Paula: Yes. We have a user feedback program where we have selected customers that are active in giving feedback through local support, and we have invited them to be part of the user feedback program. We give them our test packages, the work versions that we are developing features in, and we ask them to provide feedback on that. They do that while we are still in the development stage. So we are not giving them a beta version right before release; we are taking them in at an early stage of product development. And in the user feedback program we also give them test cases or feature descriptions, so we ask them to focus on the specific features that we are working on and provide input.

[00:04:37] Host: What sort of iterations are you planning on bringing? Have you been watching the talks today? Is there anything that you're thinking about bringing in in the future with regards to either DevOps or testing? What technologies, what tools are you excited about?

[00:05:01] Natalia: I think quite a lot of the fast prototyping that people are talking about here. Prototyping is very important for us too. We have been doing a large UI renewal, and that's the kind of project which needs a lot of prototyping and functional testing, and going to the users and talking to them. When that project is finished, I think that's a very good way to keep going forward.

Cycle time, incremental delivery and monitoring

[00:05:24] Host: Great. What else have we got here? How are you tracking your cycle time, from idea to production, and what are your goals for cycle time?

[00:05:39] Natalia: It's important for us, for some users, customers, or maybe users who know what they need next, to track the cycle time, because we need, for our maintenance releases, to be able to deliver things in two weeks. So even though I said earlier that they have two-year big maintenance releases, we also have a very fast way of releasing things if there is a need for that for users, and keeping that fast is very important. The automated system we have behind it helps with that. It also helps with understanding how much time we use for testing, how much time for automated testing, and how much time it takes from an idea to UX to prototype, going through paper prototypes, or going with a PowerPoint slide to the user, or functional prototypes. So yes, we do follow up how much time it takes.

[00:06:36] Host: Okay. And can you quantify the increase in production that this really fast release speed has made for you?

[00:06:46] Paula: Well, the faster release cycles give us the possibility to get feedback at an earlier stage and get this incremental way of developing. We can deliver the same feature in short cycles, developing it incrementally and collecting feedback on it. With that, we can focus on doing the right things that bring value to our customers, and drop development of those features that they don't really need and where they don't see any additional value.

[00:07:19] Host: Great. And are you doing any real-time monitoring on one key KPI?

[00:07:27] Natalia: For us, yes. It's important, for example, to look after performance, because we have quite a long history in our software. It's over 20 years of development, and it's quite complex software, so when we do new features we need to be sure that we don't make it slower or inefficient. So we follow that up all the time. And of course we also follow what users do, automatically, with their permission. We follow up what kinds of things they're doing, and work on topics based on that.

[00:08:00] Paula: Yeah, we have automatic user feedback collection. Our users give us permission to follow their clicks on the user interface, and then we get a log. We see which workflows they use and how much time they spend on each command, and based on that we can identify new workflows that we weren't aware of before. Then we make those workflows easier for our customers: make them more visible in the UI, maybe remove extra steps, and make them more efficient.

[00:08:33] Host: Is that tooling that you've built yourselves, or is it a specific piece of software that you're using?

[00:08:38] Paula: It's built into the software. It has been there for about 10 years already, so we're starting to have quite a lot of data there.

[00:08:44] Host: That's amazing. And how are you gaining useful insights from your data? Are you doing visualizations? How are you gaining insight?

[00:08:57] Paula: Well, data mining. And with customer interviews, we sometimes spot that they have some difficulties with our software, and then we search their data, find the functionality, and investigate what they are doing and how they are using the software. It might be that we need to improve the documentation sometimes, because we have given too vague instructions on how the feature is supposed to be used, and it might even be misunderstood sometimes.

[00:09:33] Host: First of all, do you also do in-person testing, where you watch the user physically, as well as the click tracking?

[00:09:44] Paula: Yeah, on customer visits and user days where we have users present, we do that kind of testing. We give them the features that we are developing and ask them to play around with them, and we monitor how they are trying to use them and make notes on that.

[00:10:06] Host: Okay. And are there any super surprising insights that you've had from the data that you've collected?

[00:10:14] Paula: Well, we have found these workflows that we weren't aware of, based on the automatic user feedback data that we collect. After finding those, we have been really surprised how common they are. We never thought that anyone would use the software that way, and we never would have found that with customer interviews or paper prototyping. It's only in real production use that we can get this kind of information.

[00:10:37] Host: It's definitely something that I've noticed: you build something, and you will always find a user who uses it in a way that you never imagined possible. We've got maybe 10 minutes left, so why don't we open up questions to the floor, if we've got any audience members?

Q&A

[00:10:56] Audience: How's it going? My question is: you've defined your KPIs, so how do you tie them to your team and departmental OKRs, and what's the relationship between them?

[00:11:08] Natalia: For us, we collect, for example, test coverage. I'll talk about the DevOps part, maybe. We have quite heavy checks every day on how much coverage we have at the moment. At the moment we try to reach 80% average unit test automation coverage on our code, so we follow that up, and if that's not happening, we get Jira tickets and we give feedback to developers that they need to reach it. Nowadays I don't see that it even needs very heavy follow-up anymore, because people think that it's important to have unit testing or test automation there. We also have code reviews and all these processes, and performance. Because of the follow-up, it has changed the culture, in the sense that people try to achieve that.

[00:12:05] Audience: All right. So you never tie them to product-based KPIs or key results?

[00:12:13] Natalia: I'm probably not the best one to answer that question, because I'm not aware of that side.

[00:12:19] Audience: Hello. My question is: you said you have software where companies can be using versions for a long time, like three years old. How do you try to keep users on those versions happy, when potentially you could have solved their problem today, but they're still using an older version?

[00:12:34] Paula: Well, we follow their incoming support cases, and we track the version the support case is coming from. So we can also see if one of the older versions has deteriorated with a service pack or another maintenance release. And we categorize the support cases, so we can see if there is an increase in a specific topic.

[00:13:03] Audience: And would you ever, let's say, I'm trying to think of an example, but let's say you have an older version. Would you ever do a smaller fix for that three-year-old version that might fix a very small problem, instead of forcing them to upgrade to something that might not be compatible with their use case?

[00:13:19] Natalia: We have a two-year maintenance period, so we try to push our users to take new versions and maintenance versions into use, because it's quite costly to update our old versions. In that sense, of course, if we have big customers who work on an older version while building, I don't know, some big stadium, then we do think about crucial fixes and deliver special packages very fast for them. But yeah, we do try to get our users to use the newest version, because they are missing great features which have been developed for better use. So we don't want our users to stay on old versions.

[00:14:03] Audience: Thanks very much. My question is, product teams can be very focused on new features and new functionality and building those. In what ways, or through what channels, do you highlight the older problems that users are having, versus the newer things that people want to build going forward, since the new things sound more attractive to tackle? What kind of channels or inroads do you have for getting things into the backlog?

[00:14:35] Natalia: Sure, I got the question. We do deliver maintenance, which is very important for, for example, our steel customers. They have been using our software for steel structures for a long time, so they value those maintenance features, and we bring them in service packs and maintenance releases for them.

[00:14:59] Audience: Oh, okay. So many of these might get resolved in maintenance releases. But I suppose what I mean is, do you need to have a voice or a seat at the table to get things into the backlog, or is it just driven by the size of the customer and the necessity of the contract that you have with them? So it's all based on the maintenance contract in this situation?

[00:15:19] Natalia: Yeah. We have a very large user base. We have offices in 30 different countries. I don't remember exactly the number of languages, but it's a huge amount of localization, which means that for key customers, on very important cases, of course we do process differently. It's important for us to keep good connections. But then, thinking about the large user base, we have people going to users to find which issues are most important, and that's quite difficult with that large user base and so many markets, because legislation might be completely different when we go to the Japanese market, and the construction industry is very heavy on legislation. So there are challenges in prioritizing these kinds of things.

[00:16:09] Host: Could either or both of you give me the thing that you would really like the audience to take away from UXDX? What do you think is the most important thing from each of your jobs?

[00:16:23] Natalia: For me, it's multifunctional teams, where people work together closely. I liked very much the picture at the beginning about the boundaries, which we try to crush, and have one team working on the topic, where the different expertise is fulfilled together: developers and user experience people and product owners who have all the knowledge in one team. I would like to see that happen more and more in the industry.

[00:16:51] Paula: Great. Well, for me it's easy. I think the user is the reason why we exist, so we need to remember the user in all phases of product development, from planning to development and then also in the deployment phase.

[00:17:08] Host: Thank you so much. Everybody give a round of applause to Natalia and to Paula. Thank you so much for coming in.