War Room Warrior: How To Manage War Room Situations

Rashi Khurana at UXDX Europe. Video: https://youtu.be/SEAtYv4BE64

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

A manhole fire at 1:30 in the morning

[00:00:01] It was a sunny Saturday. Breakneck Ridge[?] in upstate New York, Cold Spring, was much more than a hike, it was more like a rock climbing experience, but it was a lot of fun. After 10 hours in the sun and multiple delayed trains I get home, head swinging, extremely tired. I get ready to get to my bed and it's like 10:30 in the night, and my phone rings obnoxiously. It's 1:30 a.m. and I realized that there's a work incident that's happened that's causing me a PagerDuty alert.

[00:00:37] I muster the strength to get out of my bed and find my laptop, log into it, to understand that there's a manhole fire that's cut all access to a data center. I'm in my warrior mode. My talk here will be a short presentation about a real-life incident of a manhole fire that caused a company outage for us. The talk would actually help you prepare for such situations, and then be able to manage that from the center of the tornado when the situation is happening, and then understand what you can take away from it and learn from those situations.

[00:01:14] Hi, I'm Rashi Khurana. I'm vice president of engineering at Shutterstock. I lead the e-commerce front-end teams and the platform on which the front end is running. The front end is basically where customers are downloading their content, searching for their content. Shutterstock, if you don't know, is a global technology company. It offers a creative platform, tools and services to manage all your high-quality assets. You'll understand from my deck that I'm from a stock photography company, you'll see a lot of pictures here.

[00:01:46] For those of you who do not know what the war room is: war room is a word used in technical organizations to refer to a place where you can organize yourselves when such situations happen. Some companies call it the service operating center, some call it the network operating center. We call it the war room.

Why an outage is so critical

[00:02:05] Let's talk about why the situation of a company outage is so critical. First of all, stress. We've all been through some stressful situations in our life, and we know that stress is very subjective. If a bear is running towards you, you could get stressed out, but if a spider is crawling towards you, you can still get stressed out. It's subjective, and we are not at our best behavior when we are stressed out.

[00:02:32] Secondly, let's talk about the revenue and the customer impact. In 2016 there was research done that showed that the B2C business was reaching a three trillion marketplace, and it also showed that 38% of the customers, when they go to a mobile device to go to a site, they find that the site is down. And you know what, 100% of them turn around and go find another competitor. It's like you're going to a store and you find the store is locked down. So you're not just losing your customers and the bad experience that they're having, but it's also the revenue that you're losing to your competitors in those situations.

[00:03:12] And it's not just about the experience, it's also about, what if they are in the middle of a critical bank transaction? They want to transfer money and that's critical. Or if they're working on a creative project with a site like Shutterstock which has a deadline. So that's also very important, that it can actually be critical work.

[00:03:36] Let's go back to the version of me at 1:30 a.m. in my apartment scratching my head. The next two to three to five minutes are so important to lay the ground for what would happen in the night. And multiple hours and hours and days of preparation goes into making those two, three, five minutes the least stressful and the most effective.

[00:03:58] So how do you know if your company or your team or your organization are ready for these situations? What questions do you ask to know you're prepared for it? The first one is mostly, what's your uptime? How much does it take for you to recover a site from an outage? Does it take a long time, or are you able to recover smoothly, easily? The other is, in your engineering teams, are your engineers feeling a lot of PagerDuty fatigue, which means that they are spending more time just maintaining the running of your website and less time actually innovating and building products? Third is, like the one we saw, are you getting bad media, bad press because of the instability of a website? And fourth, of course, when you know that you're losing your customers, are you losing your employees because of all of this?

Preparing to reduce mean time to detect

[00:04:45] The two key metrics when you talk about uptime, the two key metrics that you're trying to drive to keep very low, are mean time to detect and mean time to resolve. Mean time to detect is the time it takes to detect from when the outage started, and mean time to resolve is from when the outage started and you got a detected page, to when it is resolved actually.

[00:05:11] So we will see what kind of preparations we can do to minimize the mean time to detect. First of all, it's definitely logging and monitoring. Like Chris was saying, if you don't monitor your systems and you don't get alerted, you would not even know that there's an outage that's happening or there's an issue that's happening. And logging comes in very handy, because if you have the right level of logging you can aggregate those logs to put the thresholds for your error rates and stuff, so that you can actually get alerted, so that you know something is going wrong and you can act and fix that.

[00:05:46] Then there's service ownership. In the middle of the night you're not at your best wits to know who's owning what parts of the service, or what application is owned by whom. All of that needs to be predefined and pre-understood between the organizations and the teams, so you're ready for the midnight crisis. Then there's PagerDuty, like rotation. What is your SLA if the first person doesn't respond? What is the SLA for the next person to respond? All of that needs to be pre-set up so that your mean time to detect is minimum and you're not again finding it out in the middle of the night.

[00:06:21] You need a place to organize yourself. Like I said, in our company we have a war room which is actually a physical room, so if we're in the office and something is happening we can go to the physical war room. We have a Slack channel that's called the war room where we collaborate, and then we all have a Hangout where we can all get online to collaborate if it's the middle of the night. You need to know what is the right crew that's coming to help you at the time of the crisis. And then you obviously need some training for traceroute and other tools so that you can troubleshoot what's happening.

[00:06:53] "We built, launched this thing. There was no monitoring, there was nothing, there was no way to know. There's literally no dashboard, there was nothing, there was no place to look to find out whether the site was up today or not except CNN, which was literally how we found out about problems a good part of the time in the beginning." So this was a video from when in the United States of America the healthcare.gov website was launched. Millions of US citizens went online to buy health insurance but the site was down, and what Mikey Dickerson is saying, who was pulled in to bring the site back to stability, is that they had no idea that the site was down except from CNN. That was the only place of truth, because there was no monitoring that they would find out that there's something actually happening. Live TV.

[00:07:48] Here is a sample of the monitoring and alerting that is set up. There are multiple tools that do that for one of us. The PagerDuty tool is on the right-hand side, where you can see that when something goes beyond a threshold you can set up an alert and then you get alerted and an incident is open. This is again in the PagerDuty UI. And then on the left-hand side you see that there's an integration with New Relic to Slack to actually also notify that something is wrong, and that we've got a severity critical incident going on right now. So you need to set up all of that in advance.

Preparing to reduce mean time to resolve

[00:08:28] Let's now talk about mean time to resolution, mean time to resolve, and the things that you have to make sure that you're ready for. These are things like, do you have the right setups, login credentials? If you're dialing in from home, do you have VPN set up? Do you have all the SSO set up to log into your logs? Do you have the right run books? Run books are very important, because again in the middle of the night you don't know where to even get started to see what is happening, why is the site down. You need some hints for you to get started on, all right, this is where I can start looking. So those run books are very important.

[00:09:05] And then you need some kind of training for your teams. A version of this deck I had presented to my team, telling them what is the expectation from them during an outage, so that they know, all right, if they cannot respond, what are the next steps they need to take, so that we are ready for that. And then there's also the follow-the-sun approach. Since the manhole fire — we didn't have this before, but since the manhole fire we've set up a tier one triage layer at a global location, and that helps our engineers to be freed from every incident that happens, because they are the first line of defense. They put the eyes on the glass 24 by 7 and they respond first, and then they page the teams when something else happens. Now this is a very expensive thing to do so it may not apply to everyone, but this is what we learned we needed, because of the PagerDuty fatigue we were having in the company because of all the war room stuff.

[00:10:07] Here is a sample of the run book, the run book that I use for my front-end teams. But what is most important about these run books is not just about your application, that the run book has information about it. It also has information about the ecosystem in which your application is running: your network configurations, your upstream services, where the data is coming in, what the dependencies are. Because when things are going wrong you need to understand all of that, and not just what is happening to your application. The run books also come in very handy, by the way, when you have new onboarding engineers, so they can read this and get to know, okay, what is the ecosystem of your application.

Change management and change control

[00:10:53] Change management. More often than not we try to keep changes from happening, but we spend less time in making sure that when they happen — not if, but when they happen — we can actually manage them effectively. Because in agile we need to welcome these changes. So change management is the process that we set up so that we can repeat these low-risk changes with a lot of confidence. What comes in handy to do that is our agile process. We're all designers, product and engineers, and we follow agile. That helps us to do collaboration and communication so that we have a better change management system. This is through our sprint planning, our estimations, our prioritization, our testing, our documentation, that makes sure that low-risk changes can be repeated in production. So we need that process set up, prepared in advance, for things like these.

[00:11:50] And then change control is a subset of change management, but what change control does is it evaluates if the right thing is actually happening. So it tells you there's actually authority with certain people to make decisions for very high risk profile changes. We have a CAB meeting, and the CAB meeting allows us to all come together in a room and see what are the most high-profile changes that are going out in the next 24 hours. This is the change advisory board, and that helps us organize ourselves so that we know that we're not stepping on each other when high-risk profile changes are happening.

[00:12:30] Some questions we ask in these meetings are: do we have rollback procedures for the high risk profile changes? Should those changes go out at the high peak time or should they be rescheduled for when the traffic is low? Because a high risk profile change, for instance, is when you're switching over a firewall or you're switching over the whole DNS lookup system. Those are high risk and we have to mitigate that, and this helps us coordinate that. What services or products can be impacted, so you can notify those? Sometimes there are side effects and you want all your stakeholders to be aware of the changes that are coming. And then obviously simple things like, was this even tested in pre-production, and what were the procedures that were used to test it?

[00:13:17] And when you're preparing — these are very simple things — when things are not working in your company, everybody is stressing out, everybody wants to know what's happening, but they don't know who to go to. Because the right people to go to to ask what is happening may be the real task force that's working on bringing things back to stability, and they don't want to distract that. So it is very important to set up in advance a communication strategy for your internal employees. An email that is easily digestible and easily drafted also, so that when it comes, this can be put together.

[00:13:52] You'll see a sample. This is a sample from the manhole fire that I was referring to, and you see how on the left-hand side the first email that went out is all red, so you know that there's something drastically wrong happening in the company. It's a sev zero. And then you will see later how we mitigated that to an orange, which is a lower medium priority, and what did we do to get there, and we'll talk about that.

[00:14:19] And then the whole reason we are here awake in the middle of the night is for our customers. So it is very important to have an external communication strategy. Figure out whether it is through our social media and comms, whether it is actually reaching out to our high revenue customers through our sales team on phone calls, or it's actually putting up a maintenance page for our customers letting them know that we are aware of an issue, we're working on it, stay put and something is going to be resolved very soon. So we need that kind of communication channel for our customers.

In the war room: assessing the situation

[00:14:52] All right, so that was preparation. Now it's the real deal. I'm in the war room, I am the liaison between multiple parts of my organization, and I need to quickly figure out what to do. The three things I need to figure out is assess the situation, see if the right people are online, and understand what has already happened. So now we'll see how all the preparation comes in handy, because there's a manhole fire and it's very late in the night.

[00:15:22] How many people here have been in situations where they felt that there's a crisis and they need to act very quickly? And then how many of you actually thought that you wish you had better resources available or something available to you to handle the situation? And then how many of you actually went back to fix that, so you had those ready for the next time? That's awesome, good crew.

[00:15:51] Let's get some definitions out of the way. I don't know if this is standard between multiple organizations, but we use severities to define the customer impact. Sev zero is super critical, everything is down, customers cannot reach your site. Sev one is loss of critical functionality, but not the whole site is down, it's just parts of the site are not functioning. Sev two is loss of critical functionality but there are workarounds around that system, and sev three we can ignore.

[00:16:24] So using the grid above, we know that the manhole fire has cut access from the data center to AWS. This was a 10 gig connection, and the whole site is unresponsive. So we know it's a sev zero. My incident manager is on call, so there's an incident manager who's drafting the email that we just saw to let the company know that the site is down and it's unresponsive, so that people can get aware. So we have her online. I'm the technical recovery manager, I got paged, I'm online. A lot of engineering teams were paged as well, they're also online, and our CTO actually also got to hear about it and he was also online at 2 a.m., because sometimes you need that kind of authority to make very critical decisions when the site is down and you need to act, do something.

[00:17:17] So no further action has happened, and the first five minutes — this is all the first five minutes of getting on the Hangout — they are over.

The vicious circle of retries

[00:17:26] Going further, we know that the site is unresponsive, and based on what we've seen before, I go to New Relic, our monitoring tool, to see what's going on. Is it just my application that's wrong, or there's other things that are also impacted? I saw that it was not just my application, it was the internal admin site, all our brands, everything was down. So that gives me an instant hunch that this is not an application issue, it's probably a networking issue, it's more widespread than just me and my team. I also checked the CAB to see if any changes went out during that time that would have caused this, but none of them were, no changes were sent through our site, so that was good.

[00:18:10] So our instant reaction, when your 10 gig connection is broken, is to fall back. We had a one gig backup, so we said, all right, let's fall back to the one gig connection. And we did fall back, but the site did not come back in its entirety. It came back in very slow, responsive parts, and we will see what happened next.

[00:18:35] We saw again, application is out of the way, there's a relief to all the application owners, but we saw that on the infrastructure, on the networking side, there was a surge of traffic to our firewalls. Because what had happened is, as we fell back from the 10 gig connection to the one gig, we started having a lot of slowness, because there were multiple retries that were set up in the front ends, and because the site was so slow because of the one gig bandwidth, the timeouts were causing more retries. So there was a vicious circle, there was a circle of retries surging the firewalls, and the whole ripple effect on our infrastructure. So even as we were trying to recover, it just became hell. It wasn't easy. And at least thanks to the documentation we knew where the retries were put in.

[00:19:31] So this is also the third part of it. There's infrastructure, there's application code, but there's also infrastructure as code, like we know today, the Helm charts and all. And it's very important to define those boundaries of ownership, because I've more often than not seen infrastructure teams get paged for scaling issues in application clusters which are actually configurations owned by application teams. So it's also very important to identify boundaries for the ownership of such configurations. But at least in this case we were clear that this was a networking issue, the fix for the fall back was out of scope for the application teams, and it was the infrastructure team that had to jump in.

[00:20:12] So this was not my team's issue, can I go back to sleep? No, I'm still in my warrior mode, because the site is still down. So while the manhole was not in the application, there was stuff that was required from the application teams. When the retries were happening, we as application team owners started changing our code to remove those retries, to take some of the traffic away from the firewalls. And then sometimes there are multiple brands running, sometimes you need those application engineers online still to give you feedback that yes, everything is actually recovered successfully. And sometimes there are very lazy old legacy systems that will just not auto connect or auto recover and they need a retake, they need to be like, all right, let's restart this application because it's not auto connecting. So we still need the application engineers even though we've identified that it's an infrastructure issue. And the Hangout sometimes can get very overwhelmed with a lot of people, and with a lot of stress it can be a little funny.

[00:21:24] It's also important to remember the slip-ups. In this case one of the biggest things we realized was, did we move too quick to fall back to the one gig without thinking the repercussions of actually doing that fall back? I'm running out of time, go faster.

Post-mortem and root cause analysis

[00:21:41] Anyway, so the situation is resolved, and I can go back to sleep, because after removing the retries we started giving some breather to the firewalls. But I know the next day is going to be a good day for a retrospective, because we'll have to begin to understand what was the root cause of the issue, the ripple effect of it into the infrastructure, and then learn and document from it. The next day would be a time for post-mortem and root cause analysis.

[00:22:09] And this is the best part of it, because the storm is over. There's calm, everybody is collected, they are thoughtful, and you can actually go through the step-by-step of what did we do and what can we learn from that. It's the calm part of the whole episode. And the most important part is, there's rules of engagement here. There's no punishment for services that did not work as expected, or there's no punishment for decisions that were made during that time that did not work in our favor. So it is very important to set that safe culture, so that we can understand, all right, what were the vulnerabilities in a process that led to that thing to happen, or what was the information that was available or not available that made that bad decision. So it has to be a very, very safe place to actually be able to get to the root of this.

[00:23:01] Here is the sample post-mortem from the manhole fire, and we'll go through some parts of it. The first one is, you have an overview of what happened. We fell back and then there was slowness. And then we followed the five whys strategy, so we asked ourselves five times, why did this happen, why did that happen. So we come to a conclusion of a process problem and not a person problem when we go through such thinking.

[00:23:34] And then we also document the root cause. In this case what we identified as the root cause was that we had ordered a 10 gig backup but that had not arrived on time. So had we had the 10 gig backup we would not have had this issue, we would have just done a fall back. Action items: we had to audit all of our retries to see, all right, where else do we have these hidden retries that we are not aware of, and how can we actually get ahead of it for the next time. We need to understand the impact.

[00:24:02] Lessons learned: well, the lesson here was that something like a manhole fire is external to our control, but we can definitely build the resiliency into our systems to make sure that when they happen, we're ready for it. And then we realized that in this situation we did all we could with the best of resources we had available.

[00:24:24] So that does conclude my talk. I know that the next time we're awakened after midnight to awaken my warrior, I know I'm better prepared for it. I'm not going to stress about it, I know how to deal with it, and then I also know what my team and I can take from it for later, so it doesn't happen again. That's it, thank you.