Startup experience at Microsoft

Amanda Goncalves Dias at UXDX Community: USA Online Series. Video: https://youtu.be/uU9AUFWQlY8

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

A startup inside Microsoft

[00:00:00] Hey everyone, again, thank you for joining. I would like to share with you today my experience working on a project from the beginning to now at Microsoft. It was really like a startup experience inside a big tech company, so that was really interesting, and it had a lot of learnings that I can share with you. I was one of the founding members [?] of the shopping product that I'll explain in detail, which has already saved more than two billion dollars in savings. So let's get into it.

[00:00:32] A little bit about myself first. I am from Brazil, and today I work in Seattle, Washington. I have a master's in computer science, more specifically computer graphics, and exactly today I'm completing three years as a software engineer at Microsoft. I'm a full-stack engineer, but I've specialized a lot in UX since I started this project. My goal for this talk is to give you a set of action steps and some learnings that you can use to solidify your startup products.

[00:01:08] A little bit more about the product. It's a shopping assistant, default on the Edge browser, so if you have the Edge browser you've probably seen this before. You don't have to install anything, and it finds deals and makes sure you have the best price when you shop online. It goes from coupons to cashback to comparing with other retailers and so on. Let me just show you a quick commercial on what the product is, to give a bit more context.

[00:01:36] [Commercial:] Meet the Turner twins, Big Chill [?]. They compete. They crew. They do everything together, except... "Wait, this isn't mine." "You don't use Edge to shop?" Microsoft Edge is the best browser for shopping. Download it yourself.

From browser extension to default in Edge

[00:02:07] Cool. Our journey so far really started as an idea. Maybe you're familiar with Honey, an extension that you have to install; it has the coupons and it auto-applies them for you. That's kind of where we started. Our goal was to be default on the Edge browser, because that way we could get a million users overnight. But first we had to prove the idea and prove it would work. So we started with a simple browser extension and made sure that we could do the technical part of it, which is getting the coupons and how to apply coupons. That was the first main feature that we had, and eventually we were default on Edge. And with that, a lot of other things and changes came up, so I will go into details.

[00:03:05] As you can see in these two images, our visual identity also changed a lot. On the left, we could try whatever we wanted to try, experiment with different designs and animations. On the right side, we had to follow more of the Edge identity, to add visual identity so people wouldn't think we were a third-party extension but part of the browser.

[00:03:36] We began in 2020. We had four people in the team, including my manager. We created that browser extension, and we had just coupons and price comparison. Today we have more than 10 devs, we are default on the Edge browser, and we have more than 20 features. So I will show a bit of the challenges of that growth and also what we learned. Here's a list of action steps, and let's go into details.

A/B experiments and user research

[00:04:05] The first one is A/B experiments. If you're not familiar with A/B experiments, it's basically a controlled experiment where some of the changes that you want to make go to a group of users, and the rest of the users continue getting the same experience as before. That is good because then you can compare both groups and see if your change improved the product, or maybe made it worse, or even flat. That is really good, especially when starting a product. It was one of the first setups that we had in our project, because when you are new and creating a product from scratch, there are a lot of ideas, a lot of hypotheses, people want to try new things. A/B experiments were a good way for us to safely try new ideas without being afraid of causing a big DSAT on our product.

[00:05:01] It was also good to support hypotheses, if you have something you want to try and want to do. For example, even to become default on Edge, we had to prove with data that our feature was beneficial and not making the browser worse. That can also justify business decisions and so on. Everything we do today is behind an A/B experiment, so we can back it up afterwards. I think now we have more than 60 experiments, and they run in parallel. We have a lot.

[00:05:36] But they don't tell the full story, so it's also important to do user research and really talk to users, because A/B experiments are just numbers. With user research you can get really into the details of what you can do better. It can be a long user research, where you have multiple participants and long hours of interviews, or you can have short ones that happen more frequently. You might be familiar with continuous discovery, so that can also be an option. We did both approaches.

[00:06:14] This first one was a shorter user research, but we found out some interesting things. For example, we used to have those images on the left in our UI after we were default on Edge, and we found out that users didn't like those images. That was really interesting. That is something that an A/B experiment would hardly tell us, but by talking to users we were able to identify this issue and remove those images to save a bit more space, and also reset the UX to be really part of the browser.

[00:06:50] But we didn't want to end at that. We thought that UX was a bit flat, so we did a bigger user research, with multiple interviews and long hours, to uplift our designs a little bit. With that, we added a lot more colors and some modern elements like emojis and this kind of thing, and still tried to make the feature appear trustworthy, because that's one of our main goals.

[00:07:23] For this user research, I actually participated as a listener. I'm a developer; I'm not qualified to do the user research, but it was really helpful for me to be there and participate, because that helped me understand the path that the designers were taking. I don't think that happens very often, where a developer participates in the process. I'll talk a little bit more about it in the learnings, because it really helps the teams to work together and unify, so everyone is following the same direction. Another little thing that we added is this feedback feature, so people could also add feedback in our UI directly, and you can always get that and act upon it.

[00:08:10] Afterwards, this was August 2021, we did this full refresh of the UX, but then we had to step back a little bit, in my opinion, in terms of designs, because of a business decision. So you always have to be prepared for these kinds of situations as well, and at the same time, don't stop experimenting and proving your hypotheses through A/B experiments. Basically, Edge had this new feature where they added a vertical bar on the right side, with a lot of features that you can have right there in the browser, and one of them was our feature, Microsoft Shopping. We couldn't just do whatever we wanted in terms of designs, because now all the features needed to talk to each other. That's why we had to go back to following the colors of the browser and be a bit more serious, let's say, in terms of our designs.

Logs, metrics and the tools behind them

[00:09:15] The next action step is to add logs to your features. That will help you get the quantitative (I have a very hard time with this word) data about your feature. You can collect many things. Some straightforward things are daily active users: how many users are using your feature every day. Also, this is really important, errors. Make sure you have a system where you can see those errors in real time, so you can act upon them really fast. Also user behavior: have logs for user behavior, like are users clicking on that feature, are they interacting with your feature.

[00:09:56] Especially if you're growing your product from zero and it's not established yet, you have to be prepared to move and adapt. Just because a feature showed an improvement to your product before doesn't mean that feature continues to do so. Always keep an eye on it. You might have to keep a feature, and that's fine, but it's important to always be updated. Something else that is really important to measure, and that's not given a lot of care, especially at the beginning, is performance, because performance can really impact your product. If your product seems to be slow, people might just leave it and not come back. And performance is important to measure on machines that are not as fast as your machine. I usually look at the 75th percentile of all the measures, so I have an idea of how long my product is taking to load on users' machines.

[00:10:56] The products we use to get all of these logs, make sense of our data and make sense of how users are interacting with our system are Azure Data Explorer, which is just a database, Power BI, and then Clarity. The first one, I think, is well known. Azure Data Explorer is an analytics platform for big data, and we can consider data in real time. That's how we get the errors fast and can fix them right away. Then, with all the data we get in Azure, we create dashboards and reports in Power BI, so people outside of our team can get metrics in a way that is easier to ingest. It also keeps them up to date. They don't have to ask us for new data; we can automatically refresh the data in Power BI, and everything can be tracked easily.

[00:11:53] The third one is Clarity. This one's really cool because it's a newish feature and it's free, and it helps you to understand and learn how users are using your feature or your website, for example. On the left side you can see one of our UIs. You can see the UI is a bit weird, because Clarity makes some changes in the UI to protect user privacy. But you can see the mouse action, where the user went on that particular UI. So it's good to understand if users are having any blockers with your feature and how you can help them.

[00:12:35] They also have a lot of other features. Another cool one is a heat map. On the right, the parts where it's really red are the parts users are clicking the most. You can see that they are trying to click on the image and on this title here, which weren't doing anything; we didn't add click functionality to them, but people were clicking on them to get more information. Based on this, we were able to identify this issue that users had and update the UI to do something about it. That is something that you can't really get from logs, so that's why it's really nice to have Clarity.

Unite devs, PMs and designers

[00:13:17] The next action step is to unite devs, PMs and designers. I feel like usually those areas stay within their realm and just output things to the other realms. As I was saying about the user research, it's so nice when you can collaborate in a way that everyone understands each other, and it's not just action steps that you keep sending to the other team. We got better when we unified this process more, because communication is very complex. If you have just these separate realms... This is just to illustrate how communication can get very complex. For every person, there are dots, and we have communication lines. You can see that with 14 people, we already have 91 lines of communication. So you have to have a process.

[00:14:13] There are ways to reduce those communication issues, and I'll just share three that I think work very well. One is to keep people in the loop, and don't assume that just because someone has a technical role, they don't have an interest in understanding the user or understanding design directions. I'm not saying that everyone should have an opinion on design or user research, because everyone has their own experience and specializes in something, and you have to respect that. But at the same time, people can understand points of view.

[00:14:56] That is the same for the opposite. I feel like there's always a conflict point where designers create these really nice things, and they share them with developers, and developers are like, "We cannot do this." That is very frustrating, and I think a lot of developers miss communicating the right technical limitations that the engineers have in some areas. By communicating those limitations... because everything can be done, right? Everything can be done; it just depends on how much time you have and how many resources you have. So if you have a limitation that is limiting a really cool design, maybe both the designers and the engineering team can communicate that to the leadership, and then the leadership might consider prioritizing fixing the limitations.

[00:15:45] That actually really helped, because after we communicated the correct limitations, what the designers would usually do is share the design as if there were no limitation, and an alternative design. So the leadership can say, "Okay, if you really like this design on the left, you need to allow the engineering team to take the time to fix the limitations." Does that make sense?

[00:16:09] And third, use tools to keep everyone on the same page. Figma is really well known, and we've also recently added Storybook. Storybook, for those that don't know, basically creates stories of different elements that you have in your product. It can be as specific as you want, or you can have a more complex story. That has been really helpful, especially for designers to evaluate design regressions and the fit and finish that they want to do, and also to access certain scenarios that are harder to trigger in real time. By having a story there, they can just go to a link really easily and communicate with the developer what needs to be changed.

Tests, staging and failing gracefully

[00:16:59] Next is tests. We want to prevent errors from reaching users, and we have many different types of tests. One is unit tests, very well known. We use Jest for our product, but to be honest, we don't have as many unit tests as we should, because in a startup everything seems good in theory: we have to add tests for every feature, for every function we add. Realistically, fast development had the priority over those kinds of safeguards.

[00:17:34] Instead, we added a more general safeguard at the beginning, which was having a staging phase. Basically, when we make a change in the code, instead of sending it to everyone at once, we have this staging phase where we send the update to a group of users, and then after a few hours we verify the logs, and if they are okay, we continue pushing out the changes.

[00:18:01] Another test that is actually really cool [?] is the visual parity test. We've been using TestCafe, because it's free and we didn't have to set up anything very complex to start using it. It's really simple. I know Storybook has a really nice one, Chromatic. We also explored that path, but since TestCafe is simpler to set up, we went for that. Just a tip here: both Storybook and TestCafe were new things that I was trying to create a culture around in our team, to keep adding stories and keep adding visual parity tests. So my idea here was to combine those two, so the TestCafe tests run on Storybook stories instead of the actual environment.

[00:18:55] One of the reasons is that it's harder since our feature is in the browser; it's not like a link that you can go to. But it also helped, because now if people want the benefit of locking in what they develop, so there is no visual regression, they have to add it to Storybook. So one pushes the other, and I can keep both updated.

[00:19:21] Just to demonstrate how it works: the image on the left is the expected image that the system is expecting, and that is retrieved directly from Figma, from what the designers did. I just use that to create my visual parity test, so that I make sure that my UX is polished. Then, if there is a change that regresses it, it will show in the image in the middle. You can see that the image was zoomed in because of a change. And the third one outputs the difference in pixels. It basically compares pixel by pixel, and things need to be matching.

[00:20:04] But even if you add a lot of tests and everything, it's possible that your system will fail from some error or something. So it's also important to fail gracefully as much as you can, so you don't compromise your reputation, especially as a new product. One way is to organize your code so it fails per feature and not the entire thing. Maybe users are seeing just a small part of what your product is, because the rest had an error or something, but at least they are seeing something, and most of the time they won't know there is something broken. And then of course, I'm sure you're logging and tracking all of this, so you can fix it right away.

Documentation and learning sessions

[00:20:54] Next (oh no, we are almost there) is documentation. When you're new, you kind of push back these kinds of things to ship faster, but it's really important to keep track of the documentation so people that are new can help right away. Your expectation is to grow and grow and grow, and that includes hiring more people, so it's good to have people able to ramp up really fast.

[00:21:23] One of the nice things that one of our colleagues in our team did was create these learning sessions every other week. People that had been in the team for longer would talk about a specific part of the product. It could be the code architecture, it could be how we log things, it could be how to analyze our metrics and all of that. All of those learning sessions were recorded, so new people could also access the previous ones. That was pretty cool. It would require having everyone in a meeting for 30 minutes or one hour, but that really helps people that are joining.

Learnings

[00:22:04] To finalize, let me share the learnings that we have from the six action steps I shared. The first one is that people need to be aligned on priorities and goals, and that should include everyone. Don't assume that because people are from the design team they don't have an interest in knowing your technical limitations, and the other way around.

[00:22:25] Also, data is a friend. Always lead with data. As I said, it's really easy on a new product: everyone has ideas, everyone has something to contribute. But you also want to make sure you are still going in the right direction, and for that, A/B tests can really be your friend. Lead with data.

[00:22:45] The third one: create best practices and safeguards that make sense for your team and product. I know that what works on paper doesn't always transfer to the real world, so make adaptations to your own environment. For example, we had a limitation that we couldn't use React. When you talk about UX, you immediately think of React, but we couldn't use it, so we had to adapt, and that's fine.

[00:23:12] Next, don't try to convince or change a culture by decree. I wanted to introduce Storybook and visual parity tests, but I cannot just say, "Okay, now for every feature you have to add a test and add Storybook," because people just think of that as another to-do list. Instead, you should show an example, make it easy and promote the benefits. For example, every time the visual parity test catches something, I'm promoting it. I'm saying, "Okay, this could have gone out, but the visual parity test was there." In the same way, if something goes wrong, I'm like, "If we had visual parity tests for that, that shouldn't have happened." With that, people start seeing the benefits, and the change happens more organically.

[00:23:55] Next, your product will never be perfect, so don't be afraid to experiment. As I said, we have a lot of experiments going on in parallel. And lastly, don't forget about the power of shipping. Don't wait for a product to be perfect before you start sending it to users, because you might get things that you couldn't have gotten in the development phase. Also, someone else might put the product out there first, and even though your product was better when it was released afterwards, you might have lost time. So don't underestimate the power of shipping. And mostly, with big growth comes big responsibility. I'd like to thank everyone again, and I'll be here for questions afterwards.