How to Approach a Product Overhaul

Dan Harper, Kristine Joy Paas, Linmic Lin at UXDX APAC. Video: https://youtu.be/P0S8bH2Atxc

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introductions and experience with legacy code

[00:00:00] Host: First off, I'd like to get everybody just to give a quick introduction to the audience about who you are, and then also your experience with legacy code, because what we're going to be talking about today is product overhauls. So can I start with Dan?

[00:00:15] Dan: Yeah, sure. So yes, I've been at Xero for a little while now, and Xero has a history of about 10-ish years or more of building software very quickly, so scaling very fast. And that's meant that as the company scales we have a lot of challenges with code and platforms that have been built over time. And so if you think about some of the ways that things were done 10 years ago, we have every representation of different ways to do things, which makes things very complex.

[00:00:46] Host: Love that spaghetti code. And Joy, can you introduce yourself and also just add in a bit of your background with dealing with legacy code?

[00:00:59] Joy: Yeah. So I'm Joy and I'm an engineering manager at Quipper. We are an EdTech company and I've been with the company since the early days, from seven years ago. So we started from a small startup to now one of the leading EdTech providers in the Philippines and Indonesia. And because of the demand in our services due to the COVID-19 last year, we experienced some issues dealing with these, specifically with technical debt.

[00:01:39] Host: Great, I'll dig into that in a bit more detail in a minute. And Linmic, can you introduce yourself please?

[00:01:46] Linmic: Hey, of course. Everyone, hi, I'm Linmic. Actually, I currently work for Paidy, a fintech company in Japan. So we are basically a buy now pay later company, and I'm currently supporting and building projects for the web and mobile. And we definitely have a lot of sessions in revisiting our products in terms of UX issues and performance and continuing to improve it. It was really good to help you all in the panel here.

[00:02:16] Host: Great, yeah, thank you, and welcome everybody. I'm looking forward to this discussion because it's a very engineering focused discussion that we're going to have today, and one of the things that you often hear from engineers is they always want to rewrite and they're always unhappy with the current state of code. So we're going to dig into a few examples where this is happening, just trying to understand why was there a need to do the product overhaul and how was it approached to manage it and make sure to minimize risk.

Paidy: rewriting the checkout

[00:02:46] Host: So Linmic, while you're here I'll start with you. Can you give us an introduction to the product that you overhauled, and then afterwards I guess we'll touch on how you managed to control the risk?

[00:03:00] Linmic: Sure, of course. So the product we overhauled was actually the checkout. The checkout was actually one of the most important driving streams of the company. So basically it is how you make the purchase, how you actually buy a product at the end of the purchase flow. So now, even in Japan, you can even use Paidy at Amazon and many other merchants as well. I think that we actually had a lot of concerns when we actually tried to overhaul the product, because it was too important to have anything go wrong. But we finally made it, you know, had a good result.

[00:03:43] Host: So why did you have to overhaul it? What was wrong with the product in the first place?

[00:03:48] Linmic: Yeah, of course. So at first, when I joined Paidy, after doing Paidy for a couple of months, I started playing with this product over and over, trying to make sure I understand the ins and outs of this product. But the more I played with it, the more I felt like there were many, many UX issues in the product, and also I felt like the performance could be better from a technical perspective. So I tried to initiate the rewrite. As you mentioned, it's always not good enough for us, the code, we always want to improve everything. But there's a limit, and there's a plan we need to actually build up to convince people, let's do it.

[00:04:32] Linmic: And also I think another big issue was that the checkout was actually written in Scala by the backend engineers, whereas as we grow the team we have more frontend engineers, and we are not able to actually maintain the product effectively. So whenever we want to make changes it usually takes like weeks to do so, and it's just not efficient enough. So that's basically why we wanted to overhaul the product.

[00:05:02] Host: Great. Yeah, it sounds like for a buy now pay later, the checkout is such a fundamental element. How did you get stakeholder buy-in to do this, or how did you manage, sorry, if I rephrase it slightly: one of the biggest risks with a rewrite is that scope will just keep exploding. So how did you keep control of the scope and keep your stakeholders happy while doing this rewrite?

[00:05:31] Linmic: Okay, right. Yeah, I mean that was definitely one of the concerns. When we try to redo everything there are a lot of inputs from different stakeholders, they want to improve everything here and there. But I feel like we were actually fortunate enough that when we explained to the stakeholders where we wanted to minimize the risk as best as we can, they actually understood. And also I explained to them that after the rewriting, all the new changes will be more effectively done, and they understood, and then they agreed that they would only add to the scope when the rewrite was done.

[00:06:14] Linmic: And also we actually had a lot of plans. For example, we wanted to make sure that we can roll back as easily as possible. We also made sure that we test enough, we also made sure that we have enough end-to-end tests for all the devices we want to support, and so forth. And for all those I actually got the buy-in from the stakeholders, and then finally we could do it.

Quipper: performance under COVID load

[00:06:38] Host: Great. I think a lot of people will be hoping they had the stakeholders actually pushing back on scope because of the risk. That's good. Joy, can I ask you then about your product? What was it that you overhauled and why did it need that?

[00:06:55] Joy: So during my time with Quipper we have overhauled the product many times, but the latest one was driven by COVID. Basically, if in Paidy the core feature is the checkout, in an EdTech company like ours it's the activity feature. So when there are classes, many, many teachers are sending activities to their students, and that's where we had some performance issues, and it was really affecting the flow of the learning.

[00:07:39] Joy: And so in this case it was easy to get buy-in because there was a lot of pressure from the customers. So the first thing that we had to do is to stabilize it first. So we did the quick fixes that could give a big impact on the value already, like we improved some caching, set up some CDNs and such. And there are other things that we needed to dive deeper into. Like I said, we accumulated some technical debt, and unfortunately that cannot be fixed quickly. So basically we prioritized based on the complexity and the value that it can give.

[00:08:33] Host: Okay. So I guess the same question to you, Joy: there is a big risk when you're doing a rewrite that the scope will just balloon out. I've heard examples, and I'm sure a lot of the audience have examples, of something that was supposed to take six months taking two or three years. How did you control that scope and deliver efficiently in this rewrite?

[00:08:55] Joy: Yeah, so in our case we had less than a month to fix the problem. The thing was, we were expecting the public schools to start in the following month and we were expecting like triple the traffic. So although there are many things that we needed to do, we had to focus on the things that they commonly use, so the features that they commonly use, and we focused our efforts on that.

[00:09:33] Joy: And instead of getting a lot of buy-ins from the stakeholders, the engineering team said, we have to do this whether you agree or not. We believe that we have to do this. So we didn't waste time on cascading the decisions. Of course we also communicated with them, but we said we have to really start this otherwise we will not make it. But the good thing is that we made it. So there were, as I said, there are still many issues that we're still fixing right now, but the important ones we were able to deal with despite the short timeline.

[00:10:15] Host: So you didn't do a large scale rewrite. I guess you were able to pull out different pieces to rewrite, would that be right?

[00:10:23] Joy: Yes. So of course there are long term things that we need to do, and we are doing it now in phases.

Xero: breaking the three-plan assumption

[00:10:31] Host: Great, now that's an interesting approach. I think Linmic's was more of a wholesale rewrite from Scala to JavaScript, but interesting to see a different approach. And Dan, can I ask the same question to you? What was a product that you had to overhaul and why?

[00:10:48] Dan: Yes. So if you look at xero.com and you go to the pricing page and you want to sign up, Xero has always mostly been sold in like three different plans, like a basic, a standard and a premium offer. And there's been this concept for a long time at Xero, and so a lot of the code, legacy code in particular, kind of has this assumption of a three plan structure, with a particular way of looking at the world. But what we wanted to do at Xero was to say, well, can we mix and match things? Can we add a fourth plan? Can we tweak this for a different type of customer and offer more value for them?

[00:11:27] Dan: So looking at that, we had to change all of the assumptions across the legacy platform and build some new platforms to kind of understand the world in a different way. And the question was based around, when a customer buys something from Xero, how do we know that they've been offered, or they've got access to, the right things? So answering that question was actually quite difficult, because we had this kind of three plan structure world and we had to break that whole world and say, well, now we're doing it in this new way which has a lot more flexibility. When you ask the question in legacy code of what someone has access to, we're going to change basically the path of where the answer comes from, from these new greenfields platforms.

[00:12:10] Dan: And it was very complicated because we had the scope of that, and how many parts of Xero assumed these types of authorization models, if you like. There were a lot of them, and we didn't know all of them either. So that was a huge challenge, to kind of say, well, we're stepping into a world where we don't know, and we can go and find as much information as we can but we still think we're going to miss things. So we really had to plan our work so that we could do things in very small chunks and validate things in production as early as possible, just so we could find these edge cases, and also put some things in place to mitigate impact if we got it wrong. So, to have very low risk approaches to making these kinds of changes.

[00:12:54] Host: So let's just dig into that a little bit. You're saying though it was a broad thing, that I guess it was embedded into the code. How could you do little changes? Would you not have to do one big change to redirect it? How were you able to chunk this up and manage the scope?

[00:13:11] Dan: Yeah, so from the very early stages we took one approach which ended up being quite good, which was to say that the new platform also had to be invisible, basically, to existing code. So we designed a kind of method of change in the code so that we could say, okay, even if you're calling from a legacy platform or a legacy system, we're going to support the same kind of logic flow in the new platform. So the new platform kind of had these branching characteristics, if you like, to serve both types of clients.

[00:13:46] Dan: So we had some teams that were able to move with us at the same time, which was great, so they were able to refactor their code to talk to the new world and they got some new fancy features because of that. But we also made sure that we still maintained contracts, if you like, and assumptions with the way the old code worked. So that was extra work on our part, we had to carefully design that, but the outcome ended up being successful, which is great. There was just no way we could change every single part of the code within a reasonable level of scope to be able to understand the new world, so we kind of had to make sure we had backwards compatibility.

[00:14:23] Host: Okay. And it sounds like that one, I guess if it's so entangled and touching everything, how did you even start with figuring out who you needed to talk to, what other teams are impacted?

[00:14:36] Dan: Yeah, sure. So we definitely had to pull a lot of strings and favors around Xero to say, hey, can you help us figure this out? So we had a few people across different domains and areas within Xero. We had some principals who had been around Xero for a long time and had a lot of depth of knowledge about how Xero worked. We also had people in a particular domain area which was greatly related to and impacted by the change.

[00:15:02] Dan: So one thing we did was get everyone in a room. We kind of flew people from New Zealand into Melbourne, we flew over to New Zealand one time to have one of these sessions, and we would just all be in a room and try to work through some of these challenges and hash things out. And part of that was around looking at what is our scope going to be, can we simplify things and make it easier, can we share understanding of what we're actually talking about. Because across different domains you might use the same terms but they might mean different things. So that was definitely a challenge, to try to make sure, all in the same room, we had these experts agreeing on what the approaches could be and validating some of the approaches we were going to take.

Why greenfield rather than improving the legacy code

[00:15:47] Host: Great. So the fact that you chose to do greenfields rather than modify, is that falling into the devs always want to rewrite? Why didn't you just improve the existing code? What drove the decision to actually say, no, look, we have to give up on the existing, start again?

[00:16:09] Dan: Sure. We, I guess, didn't totally abandon the improvement in the legacy world. Even though we were building greenfields, when we made changes to legacy, which we had to do as a part of the change, we did take with us an approach to say, well, we're going to make the world better, because we know generally at Xero we have a strategy to move a lot of our legacy code bases into new architectures like microservices and things of that nature. So that was already kind of a strategic driver, from CTO level all through engineering at Xero.

[00:16:40] Dan: And so we also actually had some issues with, you know, Xero, because it's grown quite quickly there are scaling challenges to this as well. And because we were changing authorization patterns, if we're going to do anything centralized we have quite a big scaling challenge, because every single click within Xero's application is going to cause an authorization call, potentially maybe more than one.

[00:17:07] Dan: So in that sense we kind of had to really, I mean, didn't have to, but we think the best engineering approach was to actually build some new greenfields platform so that we could scale it independently of the monolith. And we could just kind of have some clear boundaries around, say, an authorization domain and just solve that a little bit simpler. Because the legacy platform, of course, some of the code structures were fairly good, but you do have a problem with concepts bleeding into other areas of the application.

[00:17:37] Dan: So building it outside made sure that we could reason about the domain better, we could understand some of these edge cases and represent them in this standalone code base. And that kind of helped us to make sure that it was isolated from other things and make sure that we could handle edge cases better. So it actually ended up being lower risk, I think, because we could reason about it easier.

[00:18:00] Host: Great. And I'm going to throw a question at you here. You mentioned when you first came on, you just said Xero's been around 10 years, it's got every architectural pattern that's been popular over those years. Have you just introduced a new one?

[00:18:12] Dan: Yes. It's interesting when you look at authorization in particular. You know, there were many ways to do things even within a lot of the code that we have. The legacy platform is based on .NET Framework, and over that time, just within that one technology, there's many different ways to do authorization over that time. And so yes, we did introduce a new one.

[00:18:37] Dan: I guess the advantage with this one maybe is that we've created our own and kind of said, at Xero this is our authorization API that we're going to provide to teams. And so it's centrally managed, in a way, so it's a team that actually looks after this platform now. And so even though we've created it, we've actually created something that can help teams leverage off of that and save them some time if they need to build new features. And even for product teams, if they're building a strategy which requires a new go to market strategy, they can use this authorization platform which gives a lot more flexibility, so it unlocks a bunch of opportunity for that.

Agreeing the technical scope at Paidy

[00:19:16] Host: Great, thanks for digging deep there. And Linmic, I want to just ask you a bit more detail, particularly around, so we asked earlier and you mentioned about stakeholders, that they were on board with the scope, but to Dan's point about the technology side of things, how did you as a team agree on the scope and keep to a strict scope during the rewrite for what you were going to do from a technology perspective?

[00:19:44] Linmic: Yeah, that's a really good question. I think actually to us at the time it was fairly, really easy I would say, because what we wanted to do was rewriting what we already have. So basically we actually wanted to implement everything they already have but in a better way, in a better way from the UX perspective, from the frontend perspective. And we were only focusing on the frontend work and the design work, because we didn't really have any resources on the backend side. But that also effectively minimized the risk, because there were no changes on the backend side. So we were able to actually try as good as we want on the design perspective.

[00:20:31] Linmic: And along the process we actually talked to many stakeholders and we explained to them and kept them posted on what we are doing and where we are and where we're heading, and they were pretty much aligned all the time. So that's pretty much how we went for the rewrite.

[00:20:52] Host: And how did you, so I guess the risk is, because it was written in Scala and now it's been rewritten in JavaScript, in a year, two years' time are you going to have to write it again? How did you make decisions on your technology architecture to try as much as possible to future proof it?

[00:21:10] Linmic: Oh, sure, of course. So I think at the time the technical stack that we actually employed was React, the one actually created by Facebook. Actually we're doing it on React right now. But back then, before we rewrote it, we were using, I think we were using Scala.js, so basically it was mainly Scala. And also because the frontend engineers were not able to actually take on the tasks requiring Scala, we decided that we actually went ahead with JavaScript. And we also use Flow type for typing, and made sure that this is something really, really fundamental and won't be outdated along the way.

[00:21:54] Host: Okay, great. So no plans to upgrade to TypeScript and Vue then?

[00:21:58] Linmic: Uh, yeah, actually, we recently migrated to TypeScript, incrementally. But the reason why we migrated to TypeScript was that at the time when we actually did it, TypeScript didn't really support the React ecosystem really well. But after a couple of years the support has become much, much better, and now we have really good support, so that's why we wanted to migrate from Flow type to TypeScript at the end of the day.

Quipper: taking inventory of an old API

[00:22:29] Host: Great. And Joy, I'll ask you the same thing. So you were explaining the latency, the problems the end users were experiencing, and that got the stakeholders on board. How did you decide technically what you would use for your rewrite, and to make sure that you don't have the same problems again?

[00:22:50] Joy: Yeah, so basically for that less than a month period we didn't do big rewrites, and that's like on this phase now that we are doing some optimizations on the database. And actually we have rewritten one of the APIs that we were using. So what we did there is that we did an inventory of the endpoints that we've been using, because that API has been around since the start of Quipper.

[00:23:25] Joy: So, like Dan said, it used the patterns that were popular. So it started from a normal Rails app with the HTML and JS view, and then we used Backbone.js, then we used React. So the thing is, it's not designed to be a standalone API, and so it wasn't optimized for that. And during the inventory we found out that we are using only one third of the available endpoints now. And the second thing is that many of the libraries were outdated, and upgrading some of those libraries would mean rewriting some of the code. So we decided, let's just rewrite things, and whatever we did wrong before, let's do it right this time.

[00:24:25] Joy: So the thing that we did wrong before was keeping the libraries updated. So this time we are seriously monitoring the library upgrades, so we are using this tool called Dependabot on GitHub, so it actually notifies us if there's a new version of a library that we are using so we can upgrade more easily. And the other thing is that we're taking the documentation more seriously, because in the lifespan of Quipper many developers have left, and sometimes when people ask about this feature no one knows, so it was also hard to study.

[00:25:13] Joy: So basically that's what drove us, because the cost of rewriting the other app that's so outdated, we think that it's much greater than just rewriting things. As for the scope, we also prioritized it, we are aligning it to our roadmap. Because we started with, okay, the new features, let's develop it on this new API, and then that's how we start, and then we will slowly migrate the other things onto the new API as we go along.

Tips for anyone facing a rewrite

[00:25:58] Host: Okay, great. So thanks for digging into both the stakeholder side and the technical side of the rewrites. Everything sounds perfect from our discussion, but from my experience, whenever I've been involved in rewrites they have never gone perfectly smoothly and there have been delays. So are there any tips that we can finish up on, and I'll start with you Joy, that people can use when they have to face a rewrite, to try to keep things under control?

[00:26:28] Joy: So, like I said earlier, we have rewritten a lot of things in Quipper before. Usually we go about it with starting with the new features on the new platform, and then we're slowly migrating the existing endpoints there. As far as for the buy-ins, the situation that we had before, it was easy because it was sort of an emergency, so it was easy to convince them.

[00:27:05] Joy: But as for a typical rewrite, basically, if we observe that the needs of the users are different now, so the way they use it now differs from the way they used it before, that means we also need to consider the UX and the UIs. So usually with that, yeah, we have a user first value. So if we feel that it will benefit the user, that's our greater motivation for that. So our dev team, if we want to push for something, we must really show empathy to the users, how can this improve their experience, and not just because the devs want the latest tech or latest craze.

[00:28:06] Host: Right, so we have like the strangler pattern that you were describing there, where you don't try to rewrite everything, you just take the little bits off it. And Dan, do you have any advice that you would give to somebody who is embarking on a rewrite, to make sure it stayed under control?

[00:28:24] Dan: I'd say first thing, not to rely on a rewrite to come down the track at some point in time. I'm a big fan of incremental change, I guess. So even when working within an existing legacy platform, I like to encourage practices where we leave it better than we found it, and we add tests over time, and we do things that we think are good practices now that maybe we didn't at the time. I think that's a good thing. But I think that rule actually applies for when you're tackling a rewrite.

[00:28:51] Dan: So I think similar advice to Joy: you kind of want to take things bit by bit, and if you can somehow isolate components and do them separately without impacting other parts, I think that's a really good way to roll out change over time. I think it's pretty rare, I mean it happens, but it's a rare moment where you have a company, or a part of a company, in commercial realities where you can just stop work on what you're doing and just rewrite everything. It's not really realistic. So I think taking that approach, and trying to think about when you're going to take the approach, like how to do it well and how to always improve at it, I think it's always something we can get better at.

[00:29:32] Host: Great, and thanks. And Linmic, do you have any advice for people if they're tackling an overhaul?

[00:29:40] Linmic: Sure. I think Dan and Joy mentioned enough about how to do so, but I think another one I feel is valuable to share is that for many stakeholders they tend to be easily satisfied with the current stage. They feel like this product is good enough, stable, let's not touch it, as long as it's working just don't do anything. But I think it's always good, from a consumer perspective, to always revisit the flows, to revisit the user behaviors, because they can always change. And we should adapt to the change to actually make sure that the product can be even better. And there's always room for improvement, and I definitely encourage everyone to try to do that for their company, for their product, at all times.

[00:30:28] Host: Great. Well, thank you very much to Joy, Dan and Linmic. I really enjoyed the conversation, lots of great advice, so I hope people enjoyed it at home.