Handling Sudden Growth

Kristine Joy Paas at UXDX APAC. Video: https://www.youtube.com/watch?v=b_FbuoPnLjU

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Quipper, and where we were in early 2020

[00:00:01] Hi everyone. Good day. I hope you're enjoying the talks and workshops here at UXDX so far. Today I'm going to share about our experience at Quipper about handling sudden growth. I am Kristine Joy Paas and you can reach me by this handle on GitHub and on social media.

[00:00:24] I'm currently an engineering manager at Quipper. I mostly work with Ruby, Ruby on Rails, JavaScript, Unix systems and Kubernetes. My hobbies during the pre-COVID era were traveling and swimming, and they helped me relax from the very stressful rigors of the job. Unfortunately we cannot do those now because of the restrictions. So recently I've been reading a lot and watching series and movies via Netflix basically. This photo was taken in Busan in South Korea a couple of years back with my friends. Cats love Busan, and I also love cats by the way, and I hope to come back here soon.

[00:01:17] My company is Quipper. We are an ed-tech company. One of our services that we offer is a learning management system. This is used by teachers, students and even parents for education. That's our company, and our agenda for today is first our company state in early 2020, and the challenges we had last year, our response, the results of our efforts there, the future, and some of the takeaways from the experience.

[00:02:05] In early 2020, the first couple of months there, we just finished our restructuring. This was a very big decision for us. We decided to split the Japanese business from the rest of the global businesses. The implication of this in the product is we had to do a code split, so that means we made a copy of the code base and basically went our own way from there. As part of the global product team our goal was to develop features for the global markets, and we had the product roadmap ready for this.

[00:02:51] And why did we have to make this big decision? Mainly it's because of the difference in academic calendars. What makes it complicated when you're in the ed-tech business, especially if it's used by schools, is that you cannot release very, very big changes on the platforms while the school year is ongoing. Because of the different academic calendars in different countries, that makes it a lot more complicated, and because of that it affects our business sustainability. That's one of the reasons we had to do this.

March 2020: how usage changed

[00:03:40] But then in March 2020, and I think this affects every one of us in the world now, the COVID situation happened, and many countries had to go on lockdown, including the Philippines and Indonesia, where we have our operations. So we had to switch to our work from home set-up, and everything had to be done online. This was not a big change for the product team, as we work from home once in a while as well, but for the other departments and our clients it's a very big change.

[00:04:25] And in effect, our users also had to adapt to the situation. If before our services were mostly used as a supplemental service, so things like assignments to give the students for them to answer after class, now they use our services for synchronous education. And if before the teachers only sent Quipper contents to their students, now they also create their own. For example, the exams: for the departmental exams, they have to actually align the contents of the exam based on what they teach their students. That's why they have to customize their content, so they create their own content on our platforms.

[00:05:20] To illustrate this change in behavior, let's see the usage pattern in 2019 and 2020. For the teachers, there's really not much changed. There was a bit of spikes here and there, and here in 2020 there's a drop here, which I assume is their lunchtime, but basically the trend is similar. However, if you see the students' pattern, it's a really big difference. Students were more active in 2019 in the evening, so that supports the behavior that they were only answering their assignments in Quipper after their classes. And come 2020, this usage pattern of the students closely mirrors the usage pattern of the teachers. So they are using it at the same time, synchronously.

[00:06:32] We also observed the change in the number of classes. The top one is the graph for teachers and the bottom one is for the students. We had our lockdown around the middle of March, and as you can see, it's like overnight we already have a drastic increase in our usage. It dwindled down around May, but the Philippines school calendar ends around April or May, so this change was expected. For the students it was the same pattern, but it looks very small compared to the teachers, and that's because at this time only the existing clients were using it and finishing their contract for the year. But around August and September new clients came in and they started using more of our services.

The challenges this created

[00:07:43] These changes brought about a lot of challenges and they had effects on our product. The first is that there was a higher demand for features to facilitate online education. As I said, at the start of the year after the code split we had a product roadmap ready, but because of this change we had to revisit that roadmap and reprioritize, and check which ones should be prioritized to match with the demand of the teachers and students.

[00:08:27] The second is that there is higher traffic, as explained before, and it's not just because of the increase in the number of users, but they are spending a lot more time on our platforms. For example, if teachers were just sending assignments with contents prepared by Quipper, now they also had to author their own contents there, so in turn they had to use our systems for a longer time.

[00:09:06] And the third one is that more bugs were discovered with increased usage, because they were covering a lot more use cases. And frankly, some of these are things that we didn't design for. We had to fix those because they were adapting, and we had to adapt to how they use our systems too. And since the usage is synchronous, we also had to fix and address these issues with a higher urgency, because if they experience an issue while they are doing class, it just disrupts the entire flow of the class.

[00:09:58] And at the same time, around August, the new school year started. Some of the private schools started in August, but the public schools, which have a much higher population, were scheduled to start in October. However, in September we already had incidents in the week where it was hard to use our platforms. So the challenge for us was to address this issue. From September until October we had less than a month to make the system stable.

[00:10:46] Some of the things that were experienced during those incidents, some of the main symptoms, were slow access on some pages, which eventually led to errors. The second, slow delivery of activities to students. For the second one, imagine the situation where you have a class from 9:00 AM to 11:00 AM and the teacher is sending a quiz to students. When the teacher sends the quiz, the students are expecting to receive it right away. But what happened was it was taking a long time. There were some cases in which they were not able to receive it within the scheduled class, so it was a really big disruption in their class.

[00:11:41] What we found that caused this: first, there was slow performance on the DB, so we had to optimize this further. There was no CDN, or content delivery network, set up on some platforms, so we had to set that up. Next is that the auto scaling set-up didn't work because it was based on assumptions that were not true any more. So we improved the scaling based on the observed patterns, based on the data and considering the peak times.

Our response: platform optimization

[00:12:22] Some more details on our response. This was our reaction: everyone was really panicking because we were getting a lot of feedback from the users. We felt we were not ready for this and we didn't expect to grow as fast. If your business grows too fast, the problems grow too fast as well. So we had to really do something about this.

[00:13:00] On the platform side we had these efforts that we call platform optimization efforts. The reasons why we had to do this is that we realized that by focusing on developing features we had less attention on scalability, and because we focused on that we also accumulated technical debt, and now it started to be that bad. And lastly, new features are useless, no matter how flashy they are, if users cannot use our core features. Which led to our decision to put a freeze on developing new features and focus on improving our systems for the meantime.

[00:13:50] I was chosen to lead the efforts, and we broke it down to three main focus points. As an engineering manager, I said that my main focus is on the technical part, but we also have to maintain communications and the operations.

[00:14:10] For the technical part of the efforts, our purpose was to prevent these incidents from happening again. We had to do a lot of things, like clean up code, remove unnecessary requests, because we observed our web clients and some mobile clients were sending unnecessary requests to the API, which just adds load to our systems, but eventually the client apps don't use that data. So we removed those. Third is we improved performance by various techniques like caching, and N+1 fixes, algorithm improvements and others. And next is we studied our data and we set up auto-scaling and scheduled scaling based on those. And we also ensured that the apps had enough resources so that we don't crash. And lastly, we had to improve the database for higher capacity. So, everything we had to do within a month, and to be honest it wasn't finished and it's still an ongoing effort, but we had to do what we had to do with the time given to us.

Communication and operations

[00:15:44] The second is communication. The goal for the communication part is to keep everyone on the same page. First, before we proceeded with the technical things that we had to do, we needed to agree on the timeline with everyone, from the product, the business, the support. And at the same time we also communicated to the users about the things that we are doing, so that they are assured that we are doing things to improve their experience.

[00:16:17] And third is we set up our weekly check-up with the platform teams so that we can maintain cross-platform communication while we are changing a lot of code, because what had been happening before is that the platform teams were doing their own thing, and most of the time communication is forgotten.

[00:16:46] And the fourth one is that we compiled common issues encountered so that the support officers can respond to users faster. And also, these issues take a lot of time to investigate. Some of the issues we had to investigate for an entire day, or sometimes even multiple days. And we had to focus on improving our system, so this is time better spent on that. We did this also so that the devs can focus on enhancing our systems.

[00:17:20] Lastly, we created an incident management flow so that everyone is up to date on the status of incidents that may happen. So basically we had a single source of truth when there's an incident happening, so that it's not chaotic, where people are communicating on different channels.

[00:17:51] And for the operations, our goal was to recover from the incidents as quickly as possible. The main thing that we had to do for this is to have developers on standby on a rotation basis from 6:30 AM, because that's when the teachers start to get active. This was a big sacrifice for the devs, especially because we usually start our work at 9:00 AM or 10:00 AM in Philippine time, so starting at 6:30 is really, really early for most of us.

Results

[00:18:34] So what are the results of this? First is that there were much less crashes on our applications. The red bars here are the error occurrences. When we had incidents in September, as you can see, it was crashing a lot, but before October came we were able to stabilize the system. And when the big schools came in in October, it was already stable.

[00:19:09] Second is we improved the overall response time. This is a result of all the things that we did to optimize the system, and it's been on a downtrend since we started it, and we are pushing to maintain this trend. So it's a continuous improvement.

[00:19:36] And the next one for me is that there were no major incidents despite increasing usage. We were actually crossing our fingers when October came, that we hope this thing doesn't happen, and good thing it didn't, not to the extent that we had in September. Our scaling system worked perfectly with the new configurations because it was based on the data of the current situation.

[00:20:08] And the third is the cross-platform communications. The platform teams liked the weekly updates, because aside from giving other teams a heads up of what they are doing, they also learn from each other what the other teams are doing, and they also adapt some processes to their own teams. And fourth is that we raised awareness on how developments in different departments affect each other. It's not just the product team educating the business and support on how we do things, but we also learn from them how they respond to a crisis.

What's next

[00:21:05] So it was really a big win for us, and we managed the traffic at peak times, but the war is not yet over. It's an ongoing improvement. And actually in October there was a small thing that was still happening. There were degradations at some points of the day at one time in October, and we found that we hit our limit of concurrent usage. So many students and teachers were using it at that exact time, and we found that our system's capacity was hit. So we have to do something to increase that capacity. That was already increased, but we are still ongoing and further improving that, because if we want to grow our business we have to allow more people to access our systems at the same time.

[00:22:14] Other things that we need to do are to establish workflows and processes. Some of this we had to think on our feet to solve the problems, but moving forward it's not enough. We have to establish workflows and processes for that.

[00:22:35] And the second is to improve our incident response process. The first time we used the incident management flow that we developed, it was a bit chaotic and people were still confused. We found some points of improvement on that, and we hope that the next time we use this, although it's not good if we get to use it, I hope it's more organized.

[00:23:06] And the next would be to switch from being reactive to proactive. We survived the crisis because of the efforts that we made, but it was a very reactive response, because we waited for the things to happen before thinking of ways to solve it. In the future we would like to be more proactive, in the sense that we will anticipate things that can go wrong and already have some plans in place to resolve them when they happen.

[00:23:41] And fourth is to have changes on an organizational level from the lessons learned. As I said earlier, we learned from different teams how we handle a crisis. Because of that, it opened the communication line not just within the product team but with other departments. And so we have to make some improvements in our systems from the things that we've learned from each other.

Takeaways

[00:24:14] To summarize everything, here are some big takeaways from the experience. First is that data is your friend. Always analyze the data to make sound decisions, and especially when in unfamiliar territory like these things that we experienced. No one had really the exact experience in that. It's like we were blind, but checking the data and analyzing it made us see what's really happening, and that helped us prioritize and decide on things more confidently.

[00:24:58] And second is to not procrastinate on technical issues. They will come back to bite you when you are less prepared. So don't wait for them to snowball. We're a young company, so growth is very important, and many startups also focus on growth, but we shouldn't forget this, because later on, when we get the big users that we want, it's going to bite us back. So don't procrastinate, and at least allocate some resources to pay back the technical debt.

[00:25:51] And third is, we can prepare all we want. We can have roadmaps, we can have different workflows, processes and plans, but there will always be situations which we are not prepared for and we'll be caught off guard. So we must learn to think on our feet and learn to adapt, basically.

[00:26:11] And fourth is, be decisive for the users. In Quipper one of our values is user first, so we think about what is good for the users. And well, in the company we like order, but sometimes cascading decisions and discussions through that process can be costly. In our situation we had less than a month to solve the problems. If we decide in one week, two weeks, that's time that we could already use to solve a lot of issues. So we had to be decisive on this, and we just made the call and owned whatever the result of that is. So this is not always applicable, but in our experience this is one of the situations in which you need to decide quickly.

[00:27:18] I hope you have learned a lot from the experiences that I've shared, and this is not the only challenge that we face, and as I said earlier, we are continuously learning and continuously improving. So if you want to improve and grow with us, be a distributor of wisdom and join us. We are looking for some members of our product team in Japan, Indonesia and the Philippines. And if you are interested, please visit our career page at career.quipper.com. Thank you everyone, and I hope you've learned a lot.