Paying off Tech Debt - Scaling Up To Grow

Catherine Cornell at UXDX USA. Video: https://youtu.be/8m6GglQTDhY

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Stash and the stashers who wanted more

[00:00:00] All right. Hello everyone. My name is Catherine Cornell. I've been a product manager for about seven years. I'm currently a Director of Product at Stash, focusing on Stash's consumer investing business. For the record, all the information in this discussion is for educational and information purposes only. Today I'm here to talk to you about tech debt, and how we as product people can be more engaged and a better partner to engineering in this vital, vital, vital thing that is often overlooked.

[00:00:29] First, a little background about Stash and what I do here. Stash is the first subscription platform empowering middle-class Americans to invest and build wealth. In addition to our brokerage products, we also have a banking debit card and a stock reward program on that debit card. Our market is middle-class Americans. The median household income of a stasher is around 50 thousand dollars a year. Our over five million customers are diverse, coming from all over the US with unique financial backgrounds and situations. The common thread between all of them is that they're first-time investors. So over at the consumer investing business, we really focus on how to make first-time investors feel confident and empowered when they're getting started. It can be really scary to start investing.

[00:01:19] So I'm going to take you back to 2019, which was also a very scary time. I cringe looking at this purple screen. I also cringe thinking about all of the crowded subways, the crowded Times Squares, where this purple screen has been pulled up on customers' phones, with germy hands touching it all over without sanitizing. But this is what our app looked like in November of 2019. As you can see, we have groupings based on "I believe," "I want," "I like." We had spent the four years leading up to this time on Invest really focused on expanding the number of products we offered, different types of brokerage products, and iterating to find the right fit, so that when new investors came onto the platform, they could find the right brokerage product for them and get started.

[00:02:13] We had around 250 investments offered, a mix of ETFs and single stocks, but mostly ETFs at the time. We executed orders twice a day, we went to the market, and we added investments every so often, whenever customers requested [?] them. Adding an investment took around three weeks and required coordination between four teams. The reason I bring that up is that during this time we started to hear from stashers who had been with Stash for a while that they felt like they had grown up with Stash. They had really taken to the education, they had really absorbed the advice, and they were wanting more. This was a great sign for us. It meant that what we were doing was working. We had found the product-market fit that we're all searching for. But how did we help these newly savvy stashers grow even further and stay with the platform?

[00:03:02] Well, they told us. These are two five-star reviews from the Apple App Store and the Google Play Store. This iOS user, sandwiched in between their lovely comments and their five-star review, tells us, "Unfortunately, you only have the ability to purchase whatever is listed on the app. They might want to figure out how to broaden their investment options." This Android user, after telling us that it's great, it's simple, all the stuff you want to hear from a user, ends with a dramatic ellipsis: "But does it offer the full range of companies to invest in?"

A clear mandate and ten pages of blockers

[00:03:46] We really heard that, and we heard it many different times: our stashers were ready for more. And if we were going to retain all of these users that we had worked so hard to acquire, we needed to have more investment options on the platform. So we aligned on our mission. We've all been here, right? You have a pretty clear mandate. You feel really confident going forward. You've sized your opportunity. We were really excited about the increase in potential AUM, that's assets under management. That's one of our key metrics we track. Not only do we make revenue off it, it's also a great sign of our customers' health, and of whether the customer is getting value out of the Stash platform.

[00:04:23] We knew that whenever we added a new investment, we would see a jump in AUM, as well as a jump in our second key metric, which is buys and deposits. So doing some forecasting, we were really excited about the opportunity to grow our AUM and grow our deposits and buy rates. We aligned on the mission, everyone was excited, and we had a pretty clear mandate: scale from 250 to the over three thousand domestically traded stocks, domestically traded investments over the counter.

[00:04:58] So you have a mission and you're ready to kick off. You start writing your spec. It's an exciting time, and you identify blockers, which is part of the process. That's what we go through. That's why product managers actually have a career: because blockers are a thing. What I didn't expect, and none of my stakeholders really expected, was that when we started to actually get stuff onto paper and think things through, we had 10 pages, single-spaced, of known blockers.

[00:05:26] Everything, ranging from the mobile layout, which was designed with three segment controls that were scroll-less, and you can't really scroll through a list of a thousand things that way. The iOS app couldn't even handle loading pricing for that many investments. The way we did pricing altogether wasn't scalable: we were updating a single table in our model every five minutes. And when we started to think about how we would actually trade, the operations tools that we were using tapped out at five hundred. That's way less than three thousand. And for all of those forecast gains that we had been so jazzed about, our trading infrastructure couldn't even begin to handle the volume that we were anticipating growing to.

Tech debt as a graduation

[00:06:12] So we had to take a step back and also think about the infinite unknown blockers that we all know are going to come up. It was really worrisome. It was a questioning time: how did we make so many mistakes? How did we back ourselves into 10 pages of mistakes? And we all know the answer: it's tech debt. Tech debt is the implied cost of additional work in the future caused by choosing an easy solution today. You're essentially borrowing labor and capital from future you. By taking a shortcut today, you might need to double back around in the future.

[00:06:49] And it doesn't mean you made a bad choice. It's the cost of building. I think sometimes we tend to think of tech debt as bad engineering or laziness, but it's often the most practical thing to do. I see it as a graduation. Your system has grown and scaled to the point where it's outgrown its little shoes and it needs new ones. So when you encounter it, I want to challenge you to reframe it as a graduation. Get your little product a cap and gown, pop some Vitamin C, and celebrate before you get to work.

[00:07:27] I also want to encourage product to think of itself as the co-signer of tech debt. We're just as responsible as engineering for the choices we've made in the past, and we can't put it all on engineering to fix it. We need to be engaged with the process. At Stash we have a working agreement: engineering is responsible for identifying what tech debt there is and explaining it to product. Product is responsible for truly understanding what the tech debt is, what the trade-offs are of addressing it now versus addressing it tomorrow, and prioritizing it as appropriate. Product needs to create an environment where it's easier for the two parties to have a consistent and constant dialogue.

Checkpoints for identifying tech debt

[00:08:09] I want to talk to you about some of the ways that we identify tech debt. These are checkpoints that we discovered along this multi-month journey that we use today. So again, these are checkpoints, or moments where it's appropriate, and needed, to check in on your tech debt balance, as we like to say. They're in order from most ideal to least ideal, so if you want to exclude the last two, that's fine by me.

[00:08:37] The first is roundups. Roundups are a ceremony that the Invest engineering team has introduced into our agile ceremonies. It happens maybe biweekly or monthly, depending on the team. It's led by an engineering manager or a tech lead, and it's attended only by the engineers in a chapter. For instance, we have a backend roundup biweekly with just the Invest backend engineers. It's a space for them to think through and opine on what tech debt there is and what risks they could see coming up. The most important part of the ceremony is to have documentation. Meeting over meeting, they're writing down their concerns and collecting a backlog that can then be discussed with product. In our case, the engineering manager runs this meeting and is responsible for bringing that document back to product and having the discussion: "These are the trade-offs, these are the risks we're seeing. Help us get these prioritized so that we're on the same page."

[00:09:39] The next is OKR planning. I know we all love OKRs. They're very trendy; they've been trending for a minute. I know we're all excited, in a week or two, to kick off Q3 with our new OKRs. A lot of times I've fallen into this trap: you make an OKR of "I want to increase the top of my funnel by 10%," but you don't think about whether the rest of your funnel can handle the 10% increase in traffic that's going to come if it does. So whenever you're planning OKRs for growth, it's a great time to check in and make sure all of your downstream systems can handle the growth that you're planning for or aspiring to.

[00:10:19] The third is new feature exploration. If you're planning a new feature inside of an existing ecosystem, you need to check in and make sure that the rest of the ecosystem can handle whatever the new feature is bringing, whether it's more traffic or more complexity. The last is a test win. This is really a last-ditch resort, and I would put a marketing launch in the same boat. If you're shipping a test that saw a 15% lift in metric Y, you need to make sure that the rest of your system is ready for the entire population to see maybe a 15% lift. Same with marketing launches. If you have a marketing launch that you anticipate flooding traffic to your app or your system, you need to make sure that you're sending it out at a rate your system can handle.

Four sizes of debt repayment

[00:11:10] Now that we've identified the debt, I want to talk you through some of the approaches we use for actually tackling it. What this really did for us at Stash was create a shared language between product and engineering to understand the commitment needed to tackle debt. They're essentially fancy t-shirt sizes with some implications. Let's go through them.

[00:11:30] The first is the good citizen rule. I've also heard it called the Girl Scout rule. It comes from the mantra that when you stumble upon a campsite, you need to leave it better than how you found it. In this case, whenever developers come across a code base, they need to leave it better than how they found it. This is opportunistic refactoring while coding features, and it's included in existing story points. They go in to work on a distinct ticket, and they might be removing verboseness they come across, removing dead or unused code, or adding more test coverage when they see problematic code.

[00:12:11] Next is small bites. These are small improvements baked into the sprint. They're pointed on the smaller side; think ones or threes. The key to a small bite is that it's a standalone piece of work. It's not dependent on any work being done in front of it, and it's not blocking any work behind it. Examples are query optimizations, bug fixing, or creating tools for things that suck up developer time. I'll talk about this a little more later, but we found that in the existing investment management, we had so many problems that we were just dealing with day in and day out. It sucked up so much developer time. Taking a day to bug fix something, or to make a tool so that an operations person could address a problem rather than a developer, freed up so much time for us to work not only on tech debt, but also on new features.

[00:13:02] The third is design it in. These are design changes to a larger platform when implementing new features. I spoke before about how we had so many issues with pricing. We knew pricing was not going to work in our monolith, and one of our wider engineering goals at Stash was to break up our Rails monolith. So we took this opportunity to break pricing out of our Rails monolith into a microservice. Some more global examples of design it in would be moving to shared components in iOS when you're designing a new feature in your iOS app, or introducing new technologies into your stack that can make developing easier overall. Design it in items are usually multi-ticket initiatives, but maybe they don't span more than a couple of sprints.

[00:13:49] The last is demolition. These are major changes to a larger foundation, and they often span multiple sprints. An example we faced at Stash was upgrading from Rails 5, or maybe rewriting a service in a new language. A more global example would be moving to a streaming event architecture, or a migration into AWS or out of AWS. Either way, these are big commitments that often span multiple sprints, and to me they also imply that there are going to be multiple teams collaborating on this. So it's a big commitment, not just from your team, but from other teams as well.

The snowball method and the rollout

[00:14:27] So those are the four types. We used all of them, and we had to create a debt repayment plan. I like to joke that we used the snowball method. The snowball method is a debt repayment strategy in which a person pays off their smallest debts first while only making minimum monthly payments on their other debts. It's sort of like an MVP, but there are multiple work streams. So that's what we did. We had other lanes of debt we had to be paying off at the same time, specifically small bites and our good citizens. We continued to work on those while we calculated the maximum number of investments we could add to the platform without toppling the system.

[00:15:08] The back-of-the-napkin math came out to around 250. We decided to be conservative and only add two hundred. So the day after Thanksgiving, we added two hundred investments to the platform. We took note of the largest pain points in the actual process of adding the investments. That helped guide what automations we needed, and also what tools we needed to empower operations to own this process going forward, rather than developers. We uncovered some of the unknown blockers. And the most important thing was that it validated the hypothesis that if we added new investments to the platform, we would see big movements in the key metrics we were trying to shift.

[00:15:51] Now I'm going to show you the rate at which we added investments to the Stash platform. This is all 2020. Throughout 2020, we added 3,463 investments. We add investments pretty much on a weekly basis today at Stash, between IPOs and corporate actions. The market's always in flux, and we need to keep up with that. You can see our little jump at the end of 2019, but most of our investments were added in late February or March, specifically with 1,555 being added in March.

[00:16:32] I just want to call out, on a personal note, that we added roughly 850 of those the first week that Stash was remote in March. We were forced out of our office very last minute, a couple of days before our entire city went into lockdown. It was a really scary time to be in our home city of New York. Some of us were sick, others of us were very scared and worried about our families, some of whom were sick. And I'm so proud that we were still able to come together, support each other and get this done during that really intense time in our city.

Paying off the tech debt paid off

[00:17:08] So you can see the shape of this graph. Now let's look at the rate of buys and deposits in 2020 on Stash. You can see a slight uptick in December into January and February, when we added those two hundred. But when we really started to step on the gas in March, we saw a significant and sustained increase in our buys and deposits. So paying off our tech debt really paid off. We started 2020 with 900 million in AUM, and we ended it with 2.2 billion, a 142% increase. Our customers earned almost four million dollars in dividends. Similar to AUM, we love to track dividends for our customers because, again, it's a measure of their financial health, of how much value they're getting out of the Stash platform, and of how well we're meeting our core mission of increasing the middle class's wealth and financial stability. We also saw a 37% year-over-year increase in purchases.

[00:18:15] What I'm most proud of is that we didn't just get a lot of investments onto the platform in March and then let it go. We continued to commit to our tech debt identification ceremonies and kept slowly working through the backlog, little by little, always making our trading system and our investment platform better. And doing that actually paid dividends. What you're seeing here is Google searches for the word "stocks" in the United States. In January of 2021, there was a short squeeze on the GameStop stock and some other securities, now known as the YOLO stocks. The media coverage given to retail investors, specifically retail investors in the subreddit WallStreetBets, created a retail investing frenzy in the United States. Many Americans were learning through this media coverage that they could start investing without needing to front hundreds or thousands of dollars, and a lot of those first-time investors found their way to Stash.

[00:19:15] This is our Stash subscriber creation during that same time period. You can see that in a single day in January, we captured almost 30,000 new stashers. During that week, we were able not only to capture these new subscribers, but to fill all of their orders without incident. There was no concern about our ability to serve up investment prices and investment pages, and to facilitate all of the record-breaking buys that were happening on our platform. And that's only because we had been so committed to our tech debt management and our repayment plan throughout all of 2020. We didn't just stop. A lot of these stashers are still on the platform because of the advice and guidance we have, and because we were able to keep chugging when some competitors maybe went down [?] that day.

[00:20:11] So just to recap: there will always be tech debt. In that fantastic week I just told you about, we had other issues with our app. We had issues with our sign-up, with loading the home screen, because you can never anticipate when you're going to go viral. So my advice to you, if you're trying to grow your business or you're trying to go viral: there will always be tech debt. You need to be proud of your tech debt, because it means that you're growing, and you need to have a process and a feedback loop between engineering and product to ensure that you're always paying it down little bit by little bit. So when you do go viral, you're ready to capture all of that traffic. Thank you.