UX, DX, DSX: Developers And Data Scientists As Users

Dot Gong at UXDX Europe. Video: https://youtu.be/L8xa1fvTumE

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Data in the BuzzFeed DNA

[00:00:00] Hello everyone, hi. It's great to be here in Dublin. It's my first time, and it's a beautiful city. My name is Dot Gong, I work at BuzzFeed, and the title of my talk is UX, DX, DSX. The first time I heard about the conference, I looked at the conference name and I was like, "Where are the data scientists?" So I added them to my talk. I'm going to talk about my personal experience building tools and infrastructure for developers and data scientists.

[00:00:30] First of all, I want to talk a little bit about myself. I started at BuzzFeed over 5 years ago. When I started at BuzzFeed, I applied to the job because I saw an interesting article online about BuzzFeed's data operations, and I was very impressed by it. Data is very much in the BuzzFeed DNA. We use and collect data in pretty much every aspect of our work, and we invest a lot in building out our data capabilities. We do everything using data, from automating laborious tasks to using algorithms to publish and curate our content. Data provides information to our managers so they can make smarter decisions, data provides efficiencies to our internal processes, and data makes the experience better for our audience. As we see it internally, data gives us superpowers.

[00:01:27] When I started at BuzzFeed, everyone in the product team worked on buzzfeed.com, because that was our only destination. Five years later, we have so many media brands across our owned and operated platforms and various distributed platforms; BuzzFeed has grown into a content network. On BuzzFeed News we write important articles about what's happening around the world. On BuzzFeed we make heartwarming posts and quizzes that put a smile on your face on a bad day. We make videos that cover everything from how you make a grilled cheese sandwich to sharing more important perspectives and stories with the rest of the world. We even have a morning show that we stream on Twitter called AM to DM.

[00:02:25] As the complexity of this company grew, my job also shifted. At first I was building consumer-facing products for our advertising team, and then I switched to building tools and infrastructure for our developers and data scientists. This was a unique, interesting assignment for me at first. On the one hand, I was very happy. I had developed great rapport with my colleagues, and my users sit next to me, so I enjoy great proximity to all my users. But on the other hand, because I'm so close to them, if they don't like the services and tools they're using, you bet they're going to let me know.

[00:03:06] Building infrastructure for a company that's famous for its data usage also comes with its pros and cons. On one hand, this company has a great internal data culture: everyone really appreciates and values the type of work we do. But on the other hand, the types of data usage at BuzzFeed are so diverse that sometimes it becomes really hard to decide what we are going to build. So today I'm going to share with you all a few things that I have learned in the past few years.

Engineers as users, and the data quality problem

[00:03:36] First of all, when I think about building data infrastructure, my biggest user group is engineers. Data infrastructure is very diverse; in a way, every person's interaction with data is a use case. But engineers are definitely my biggest user group. I love engineers. I've worked with engineers my entire career, I've written a few lines of code, my husband is an engineer, and, coming from the consumer side, I've already used the data infrastructure products we offer, so I feel like I've eaten my own dog food. My complacency was shown on this slide: they're engineers, they need carrots.

[00:04:23] But what I assumed at first would be an easy walk in the park turned out to be a very different experience. We have close to a hundred engineers at BuzzFeed, and they work across ten different product areas. Their skills vary from iOS developers to site reliability engineers. They work on our owned and operated platforms, on internal tools for our staff, and on things that are infrastructure problems. Very soon we realized that our users, our engineers, were not doing the things that we expected them to be doing, and that they had very different perspectives and experience from us.

[00:05:05] Even though we saw that everyone was using our data pipeline and collecting data, the data they collected was not always usable or correct at first. Even though we have many talented engineers across different teams, with academic computer science backgrounds, the data was not always modeled with the downstream consumers in mind. And even though we have great transparency in the company and a very democratic attitude towards data sharing, people didn't always benefit from the data transparency and open access to data the way we expected them to. We realized we didn't really know our users.

[00:05:54] This is another problem that I observed. How many of you here are feature engineers? Anyone? Yes, a few of you. Feature engineers, when you're trying to ship a feature, the number one most important thing is racing against the clock and getting it out the door. That's why invisible tasks such as instrumentation or monitoring usually take a back seat to getting the feature out the door. Nobody would deny that data is important to every tech company, but a lot of the time data collection and instrumentation is a second-class citizen when it comes to feature development.

[00:06:38] The other thing that you see on this slide: at a lot of companies like BuzzFeed, we have two different types of groups. We have the feature groups and we have data consumers, and usually these are separate teams of people. That creates misaligned incentives and misaligned understanding. The purpose for which you collect the data is not always the use case that people have downstream when they're consuming the same data. This exacerbates the problem we have with data quality.

[00:07:14] This is not only a BuzzFeed problem. Here is a talk from former data scientist Peter Skomoroch, on product management for AI. In this talk he cited the VP of engineering of Salesforce. This is a quote: "Every single company I've worked at and talked to has the same problem, without a single exception so far: poor data quality, especially tracking data." On the right I have this tweet from Monika Alexander [?] that we laughed about in the office a while ago: you think data collection is great, and you end up with rows and columns of nulls. So this problem is pretty pervasive in our industry, and it also applies to us.

[00:08:02] How did we solve it? At BuzzFeed we realized that, as data infrastructure builders, infrastructure is a vehicle for our data-driven decisions. We want to make sure that data infrastructure is there to really benefit our end users. We cannot just think about building the most state-of-the-art and the most scalable infrastructure without thinking about downstream use cases. So the first thing we did is, in addition to thinking about scalability, capacity planning and these infrastructure technical challenges, we started treating developer experience and user experience as one of our top priorities.

[00:08:46] We went to talk to our front-end developers, and we started to think about what type of tools we could build them to make data collection tasks error-proof and intuitive. With that, not only did we improve our data quality, we also won the hearts of our front-end developers, because we made their lives easier. We provide not only UIs for them to do data validation, we also have command-line tools that we give them. And this kind of conversation organically got everyone mingling. We have data producers talking to consumers; we have data scientists, data engineers and front-end engineers all talking and caring about each other's jobs. With that, our producer and consumer incentives are aligned.

Data scientists as users

[00:09:33] Now I'm going to talk about my second group of users, which is very dear to my heart: our data scientist group. With the data scientists, I was very cautious, because I'd learned my lessons with engineers, and I really wanted to learn more about them. Engineers and data scientists have a lot of similarities. Data scientists are very technical; a lot of the time they use the same Python, the same language or the same stack as engineers. But the more you learn about them, the more you see the differences between the two groups.

[00:10:06] With engineers, when they start a project, they usually start with something like a product requirements doc, and they make this type of architecture diagram. Before our engineers start building things, they have a fairly good idea about the final product that they're going to have. Data scientists, however, even though they also use Python, work very differently. This is a screenshot of a Jupyter notebook. Our data scientists at BuzzFeed mostly use notebooks on the cloud or local Jupyter notebooks to do their work, and they iterate on the strategy and approach of their work. When they start on a project, they usually start with exploration of the data set, and they have no idea where they are going to end up.

[00:10:59] This is a screenshot I took, and the funny part is that, where the algorithm should be running, the data scientist wrote "TK," which is really the purpose and the whole point of this notebook. So this is my Irish joke here. To me, our engineers are like Oscar Wilde. They're the Oscar Wildes of the world: they plot incredibly elaborate schemes and plots before they start working, and they work towards that. On the other hand, our data scientist group are like the James Joyces of the world. They go on a long stream of consciousness: they start on something and they don't always end up where they thought they were going to end up. Both of them are great writers with a great body of work, but very different.

[00:11:46] Nowadays, in 2019, companies really care about the type of data investment they make, and this is how we get competitive advantage in the marketplace. You need both types of talent to do the type of work you want them to do. You need data scientists to come up with interesting optimization algorithms, and you need engineers who can collaborate well with the data scientists to make it production ready, to make it efficient, and to make it into the reliable and scalable production services you have that are user facing. To me personally, this relationship between engineers and data scientists, their collaboration and their workflow, is one of the most interesting user experience problems I face.

Research techniques for technical users

[00:12:34] With user experience problems, I'm sure every one of you is familiar with a lot of UX research techniques. In the next section I'm going to talk a little bit about the techniques that we have used at BuzzFeed and a few things that we have discovered for technical users specifically. My go-to technique is a very common one: I love to do UX interviews. As I mentioned, my users sit next to me on the same floor, so UX interviews are really easy to schedule and conduct with my users. I find this type of interview incredibly useful, because it gives you a state of the union on the landscape of the problem, and it also helps you identify the common pain points and common patterns that people are experiencing.

[00:13:23] However, just like traditional UX research, UX interviews come with their downsides. User recall is one problem; confirmation bias is another. The other thing is, when your users are friends who grab a beer with you after work, social desirability is very much an issue, because they want to say the things you want to hear. So we need other techniques to complement this type of interview research.

[00:13:49] One thing I personally like to use is GitHub. A lot of technical users use GitHub for their pull requests and their branches, and it's great for me to see the entire development history. On GitHub you see the conversation between the reviewers and the author. On GitHub you see the entire commit history and the logs, so you see where their hurdles are and where they are having misconceptions. Another thing is you can see the number of files changed. If you're making technical infrastructure products, you want to see how many things people have to go through and learn in order to complete their tasks.

[00:14:29] The other thing I really like is using Slack search. Similar to GitHub, Slack is often used in modern technology companies as an internal communication tool, and some companies may use other chat messenger apps. A lot of companies set up support channels in Slack to help streamline the support process, and this is also great when you search for people's questions about a specific topic. If you have a GitHub branch and you cross-reference it against Slack, you get the full picture of what was happening when the developer was doing a task.

[00:15:13] The other thing that I've found is that, because technical product development is very abstract, it usually spans a long period of time, and people use many different types of services. This type of user journey mapping is really useful to us. It gives a 30,000-foot view of the entire product development. Because it visually represents all of the pain points in the product development cycle, it's easy to get executive buy-in, and also, when I show these things to my users, they feel very heard and appreciated.

[00:15:53] The other great benefit of having visual artifacts like this is that it establishes a common language for everyone on the team, for your stakeholders and for your users to talk about it. Here we have four stages of machine learning product development at BuzzFeed, and now we have the common language to talk about which phase we're talking about. One thing that I find really interesting about this type of research is that it shows you things you didn't know. When I was doing this UX research on machine learning product development at BuzzFeed, very early on I identified some technical challenges that we had assumed. For example, we had really bad task coordination in the prototyping stage, or data scientists didn't have enough computing power. But some things I discovered really surprised me. For example, we had organizational and structural challenges for data scientists and developers.

Organization as part of the user experience

[00:16:51] This leads me to my last point, which is that we need to think about how to adapt our organization. Organization is very much a part of the user experience of the developers' and data scientists' work, and sometimes the solution is not technical but organizational. Organization is very much part of the work experience for everyone. There are a lot of discussions here, and I think this is sometimes one of the most interesting and useful solutions to our problems. Not only can improving the organization improve the dynamic and the productivity of a team, improving your organization can also restructure your teams so that they can realign on the most important priorities and focus resources on the most important and impactful areas to invest in.

[00:17:48] When I talk about data science team structure, it's currently talked about a lot in the industry, and there's a lot of debate about it. Mostly people fall into two camps: the centralized camp and the embedded camp. I think there is no correct or right answer here. You can be either centralized or embedded, based on your company's maturity and your data literacy. However, I think it's very much possible to have a hybrid model where you land somewhere in the middle of the axis.

[00:18:21] At BuzzFeed we recently did this type of restructuring. For four years we were an embedded data scientist team; however, we recently switched to a hybrid model. What we discovered is that in our embedded team, data scientists developed really great product understanding; however, they were very much working in silos. Usually we had one or two data scientists working with one product manager, and sometimes our junior product managers didn't really know the best and most efficient way to deploy their data science resource. So we moved to a centralized team where we centralized the project management. In addition to that, we also centralized data engineering, analytics and data scientists under the same umbrella, all in the big data team.

[00:19:13] We're very greedy: we want both the benefits of the centralized model and the benefits of the embedded model. So we created two new data roles to complement our centralized project management. The first is called a strategic data partner. Every data scientist at BuzzFeed is assigned a product area that they are the expert on, and they are what we internally call the product manager's data best friend. Throughout that partnership, they build a relationship with the product manager and the team, and they also grow in that product data area. They consult on things like what the KPI is going to be for our next quarter, and on things like whether we should fundamentally change how we design experiments in this product. Quarterly, they conduct data reviews with leadership to say, "Hey, here are some areas I see that are worth investing in," or "Here are the areas that are slowing down."

[00:20:15] The second thing is that, now that we don't have data scientists embedded full-time on teams, we still want teams to really care about their data usage. So we borrowed a term, from Google I think: data steward. Usually that's a PM or an engineer, but we also have designers as data stewards at BuzzFeed. What they do is come together and form a centralized data council that establishes the cross-organization standards for things like data governance, instrumentation and data namespaces. Then they take those standards and best practices back to their respective products, to lead and guide them on their product data responsibilities.

Building bigger pyramids from smaller ones

[00:21:09] Also, no data science talk is complete without mentioning Monica Rogati's data science hierarchy of needs. This is very commonly talked about. The big thing is that this is a very daunting diagram. It says that if you want to get to the tip of the pyramid, doing really advanced data science, there's a lot of instrumentation and a lot of infrastructure investment you have to do up front. A lot of the time people talk about the first two levels, the collection and storage levels, as at least two years of investment. And even at companies like BuzzFeed, who have built all our data infrastructure and invested years in building out data capabilities, sometimes when you get to the top and you're working on a machine learning project, you realize, "Oh, I don't have the right data." This is very frustrating, as it feels like the Sisyphean task of getting to the top and having to come back to the first level and rethink again.

[00:22:15] At BuzzFeed we looked at this, and with our restructure putting data engineers and data scientists together, we started to rethink this pyramid. You don't necessarily have to build a pyramid from the ground up; it is possible to build a bigger pyramid from a smaller pyramid. So we started looking at projects as slivers that we can add to pyramids. This is the thing about data science work: you don't know that your prototype is always going to perform better than your production model. That is scary, because you need 2 years of infrastructure investment, and at the end of the day you might not get anything.

[00:22:57] However, if you look at projects with both engineering and data scientists in mind, the type of research and development that data scientists do can become much cheaper. In a project, you can think about it as: we're working on a new algorithm, but maybe the big goal is to get a feature store, and with a feature store any future iterations become much cheaper. Or maybe in another project your goal for the engineers is to think about new ways to test and evaluate in production with a much simpler process, and that lowers future investment in data products.

Recap

[00:23:40] I want to recap my talk. If you take away anything from my talk today, I hope it's that you think technical users are as fascinating as I do. I'm incredibly passionate about working with technical users, and I think they're incredibly interesting as a UX problem. The first thing I've learned myself is that I need to remove my subjectivity when doing technical user research. The second thing is to think outside the box: really think about new ways that we can learn more about our users, and think outside the box about solutions too, because the solution is not always a new product, a new workflow or a technical solution; it might be an organizational solution. Also, I want to mention that we are hiring in New York City, LA and London, in all disciplines: developers, data scientists, product analysts, you name it. Thank you.