Being Devops In Desktop Software Development

Natalia Kaijalainen at UXDX Community: Helsinki. Video: https://youtu.be/MxxwssLX-k0

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Eight years of legacy code, and a DevOps team

[00:00:11] I've been doing eight years software engineering with software where you have over seven million lines of code, so it's quite a legacy code. After that I have been working with leading the team, the DevOps team. First I wanted to say that I'm talking about a DevOps team even though I know that DevOps is obviously more a kind of culture, the way of doing things together to get the end result out. But still, I have been leading a DevOps team for five, already six years, and I'm going to talk to you about how to do that in this kind of legacy software, legacy environment.

[00:00:47] At the beginning I'm going to show a little about my company and the products we do, just to make the base about the environments that we work in. So I'm from Trimble, but I don't know how many of you have heard about the company. Trimble is trying to transform the way the world works, so we do quite much business-to-business software, hardware, things which improve our customers', the users', way of working in future. For example we do things like GPS, satellites, some optimization to agriculture. Me myself, I'm working with the buildings division, so we do this 3D software, construction industry, and trying to improve that to be more agile and more transparent. I'll show you about my division, a short video.

The Trimble video

[00:01:50] How would you create a world with no boundaries? Would you design structures that defy gravity? Would you design software that lets you visualize the future? Would you create systems that could turn night into day, and winter into summer? Would you create a way to make space feel infinite? Would you build buildings with the touch of a button? At Trimble we believe you can do it, by uniting hardware, software and services, converging people and projects, so you can build for everything except boundaries.

Tekla and the construction industry

[00:03:00] So that video was talking about no boundaries, and now we come to construction industry, where my product is, where I'm working. We try to make our customers and our users feel the no boundaries in their projects. The Tekla product family is exactly what I'm working with. This is the model. I don't know how many of you have seen that. In Southeast Asia there is this kind of customer model done by our software, so it has all information, from the smallest bolts, if you zoom in very closely, so you can see how different beams and columns are connected, to this big view of huge buildings. So you can count how much steel you need, how much concrete you need to that. And these buildings are huge in reality. If you go to Southeast Asia, go to check them out, quite cool.

[00:03:53] Then we come to construction industry. Construction industry as such is quite a legacy one, so it means it has, I don't know, since people have been working there already building the houses and building a very different way of buildings. We have our own projects and processes, so we are quite far away from IT projects. They have quite heavy national exceptions[?] and national regulations, meaning that different countries have different rules how you can produce your models, your drawings. You need to have, okay, metrics, and this in cheese[?] is very easy, but then you need to have different dimensions, so you need to have dimensions that if one line or two lines, number up or down. So our software product provides that for every different country, so that we can produce construction drawings automatically. So there is quite much still 2D construction drawing, such different governments' demands.

[00:04:59] We have a 3D model which has all information, but still people need it with 2D drawings. There are still some customers which use pen and kind of paper to do the construction drawings. And now we're talking about continuous release and UX and all technology stuff which we are not used to.

Why our customers don't take updates

[00:05:17] They usually have many years long projects. So with our software usually are modeled skyscrapers or Olympic stadiums, of course smaller buildings too, but this is where our software is used most[?]. Then you build some big building like an Olympic stadium, it will take years, it will take years, and it will produce a lot of documents, so you can do that automatically with our software. But then our users don't want to change from their version to a newer one. We don't take updates, we're scared of that, because, you know, that's probably the finish line symmetry[?] already. But they have building projects which are getting over budget, they are getting over timeline, they are very scared to update a software which they have there on their desk, and we have always information which we provide to their kind of customers, to government.

[00:06:06] So it's not agile how we understand it in IT, it's not lean at all. But we need to be, as software engineers we are proud of what we do, as a company, as product development we need to be agile anyway. So how do you then balance these two things, non-agile kind of customers, old-fashioned partly, and then us, who would like to do things and release things fast?

Starting the BAT team in 2012

[00:06:33] So 2012 I was appointed to start a BAT team. The name is an interesting one, so we ended up having a team with naming: building, automation, analysis, testing, whatever you can just fit to these three letters, which kind of comes from CI. At that point we didn't have any DevOps word even, we didn't know even about that, so it came later, when we realized, okay, DevOps practices are quite about what we do.

[00:07:03] At that point we already had some tools in place, so we didn't start from the beginning. Our idea was to fasten our release. We had Surround SCM as version control management, which meant that all code changes were done to a master line, so to the main line, which you had to production. We had CruiseControl, so we ran our continuous integration builds every two hours, I guess. Usually when people did the code changes and it broke the code, for, let's just say, 40 developers, it was faster to get feedback from your teammate or somebody sitting on the same floor, that hey, you broke our code, you broke our software, than waiting for continuous integration to come out.

[00:07:44] We had already at that point some test automation. So when I started in 2005 we had test automation in place already on some level. We would have wished for much more test automation, but I'm still saying some, because we had it running once a day to the master, the same branch which we had all the coding in, which meant that good morning, if something broke up some test, then all our people were wondering, who from you, who have put their changes in, was breaking it. So it was quite chaotic in that sense.

Transforming the tools

[00:08:16] We needed to transform to DevOps, and that is what my team was helping to transform. Transformation is not easy, you need to change quite many things: tools, processes, workflows, I would say, and mindsets, that's the most important things. So changing the tools is not enough. We started with transforming the tools. Like I said, we had at that point a client-server version control system, so code was put directly to everybody. We changed it to Git, a distributed system.

[00:08:49] We had at that point code reviews from documents. So it's 2012, people collected together and then we checked the code from that, and we did give valuable feedback, but now we had Bitbucket in place. We transformed — now I'm talking about 100 people, transforming the tools for 100 people, not one team or two teams, but quite a big amount of people. We changed to the pull request process. We had at the very beginning this kind of code reviews, we had very much comments about style. Nowadays tools help us probably recognize that, oh, you have a big letter there, you should have a small one, you don't follow our style. It's very annoying when you have very highly skilled people commenting about very unnecessary things, but they should be correct.

[00:09:37] Now we set up the tool for this kind of feedback, then we could talk about the really real code architecture, how, which solution, all the features are done. So we had SonarQube now for checking our style issues. And then we had continuous integration in place at that time, but we produced binaries. I'm talking about a product which people are using on Windows. They run the installer and then they get the product to use. We produced at that point, in our continuous integration, binaries, so it wasn't the same what our users did get. So we needed to change our continuous integration system, we changed to TeamCity at that point, and we changed our process with TeamCity to produce installers, real installers, after when you do the code change. So transforming all these tools, it does take time.

New processes: feature branches, tests and pull requests

[00:10:29] Besides that, with all these new tools it comes new processes. So it's not just changing the tool, you need to teach people how to use the tools. At the beginning we had these direct commits with version control system. Now we changed to these feature branches, and when I talk about feature branches I mean that the change is happening, in our case, in one to two days, not one week. So a feature branch in our case is not long. Our development teams are doing changes to the same codebase, to the same installer. They usually have their own workflow for a couple of days, and then they merge it for everybody to be able to test.

[00:11:04] Test automation: we had that in our main branch, on the main product we did tests nightly. Nowadays we have test automation everywhere, so we have of course the nightly one, and then we do test our product also in feature branches, on every kind of way where development teams are working on. So we try to get the feedback very fast to developers. We had code reviews after merge before, because we had that one codebase where people do the changes and then they change the code. Nowadays we have pull request and code reviews approved beforehand. It changes, it's small when you read from here, but it does take time for developers to understand why we are doing this this way.

[00:11:55] We had previously this nightly test automation, and nowadays we have tests with pull requests, so we need to get quality code beforehand. You do the changes, you need to check that the fix works beforehand. Why I'm doing it: it's most important to fail fast. The faster we fail, because if our customer takes our installation into use in a years-long project, after two years it's quite a slow feedback loop to development. Developers would like to know much faster when we fail, so we help them to get the feedback faster. Even if our customer maybe does not take into use the installer yet, everybody else in the company can take it, our offices can take it and get that feedback for us.

Changing mindsets: the comments I heard

[00:12:43] For us the most important, like I said, is transforming the mindset. Now we were changing quite many people's way of working. The comments which I got — I got a lot of different comments, but I think these are the most often heard ones. If you wait for code review, our release can be delayed. The question as itself is quite a very interesting one. So if you try to do quality software, you are not afraid of putting your changes or new features, which nobody tests or checks the code before, to be released to a customer? But you could just push the release out and don't think about that. I think it's safer to do a code review or testing beforehand.

[00:13:28] Next one: I know my changes will not break anything, QA folks will test anyway. So somebody else will check if my code is ok and give me feedback when needed. Our developers need to understand that hey, we are proud of our work, we are very proud of what we do, so you need to do quality software. So it's not QA or somebody else to say to you, hey, you need to be better. You need to understand that you are proud of what you do, and I think feeling that proud of my work is very important, which comes through this mindset change.

[00:14:04] And then the last one, we need to find someone to take the responsibility. I heard that from one of the previous speakers, so others are hearing that too, probably some here too. But is it so, you need to find somebody guilty after we fail? I think we need to be doing bad things[?] beforehand, we need to be proud of what we do, like the previous one. If we are proud, we don't need to find somebody who is responsible, we need to fix things when they come up. The responsibility is quite — I kind of combined this question with, we need to have somebody guilty at work, which is not kind of productive.

Continuous deployment when customers update once in three years

[00:14:46] Continuous deployment. Now we come to this question: our customers are taking installers into use once in three years, some, the biggest ones who do this kind of plants and metro projects. They don't want updates all the time, they are afraid their product gets more delayed for one small fix which might break one of their thousands of drawings. But what we do, we need to deploy all the time. So are we DevOps, is what we do correct? My team have ended up with results that we have 25 installations per version today. I'm saying per version, which means we have three, four current versions which we maintain, which means 100 installations, installers on the table of anybody to test.

[00:15:29] Every change where people do, they go automatically to the installer phase, and because we are kind of an international company, any office in the US can check if that would work. So in pre-release[?] and even show to a customer if that works, very model of a new feature. We can do that in Japan too, in one hour after somebody did the change. We have user feedback programs. We have UX people on our teams and they have also user feedback profiles. We ask about new features from our users. Even if they don't take newer versions maybe into use, they do comment, and some of them get so excited. For 12 years we check when to use updates[?]. We demand quality, we don't want our product to be out of date, even if customers are slower. And we have a fast feedback loop, we get the feedback very fast, and I guess we have 1,000 builds[?]. So it's because of all these feature branches, they do get tested, and we have creation of installations, all quite much work, and a VM farm[?].

Why we still have a DevOps team

[00:16:35] And I started with the DevOps team, but I have a DevOps team, and I understand that DevOps is important as a culture, and why we have a DevOps team at the end. So this huge transformation which I was talking about, it needs support inside. We support people, of course. There is this tool administration, it needs to, for 400 people, when you have tools you need to help them with these tools. And at the end, my team is not the team which you can give a package and say, please deploy it. We don't do that, our development teams do that themselves. My team is just helping them. So if there is something in our process which does not work, we try to improve that, but we don't do anything for them.

[00:17:20] And then for continuous improvement: as we would like having continuous releases, we need to be ready for our customers, so we need to improve our processes towards that, that we will be even more ready in future. I would say that yes, you can be DevOps in legacy surroundings, and I think you should be, even if the outer world is not there yet still. But one day we will need that anyway, and we will be ready at that point. Thank you. Do you have any questions?