Data Science at The New York Times

Chris Wiggins at UXDX USA. Video: https://youtu.be/J3rfvItzeqo

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

From dead trees to a digital bundle

[00:00:10] Thank you very much, and thank you for that kind introduction. It's been wonderful meeting the UXDX community over the last few days. It's clearly a very multidisciplinary community. Just so I know my audience, how many of you self-identify as designers? Okay. And how many of you self-identify as data scientists? Okay, good. How many of you are designers at a company that has a data scientist? Okay, good. And how many of you are at companies that don't have a data scientist, but you wish you did? Okay. My goal is to make it easy for those of you who are at companies that have a data scientist to interact with them and to understand how they're trying to add value. And hopefully by the end of the talk, all of you will raise your hands to say you wish you had a data scientist at your company.

[00:00:57] As advertised, I'm Chris Wiggins, and I'm chief data scientist at the New York Times. I thought I'd tell you a little bit about that role and how it relates to the goals of UXDX. Part of the history of the New York Times is this kind of UX on the left, which is a dead tree that people interact with. They spread it out on their kitchen table and interact with it for a couple of hours. When they have a particular job to be done, they get that job done by switching from section A to section C, or by flipping to a particular place where they know, in terms of the layout of the information, how that location relates to their user need. It could be crosswords, it could be sports, it could be, I don't know, what time the ships were showing up in New York City in 1851. All that changed on January 22nd, 1996, when the New York Times introduced a website, which they reported on with the quaint title "The New York Times Introduces a Website."

[00:01:55] Of course, all of us live in a digital world. Particularly those of you who are employed at any company with a website know that it means you're getting abundant data about the way users and customers feel about your product. Right now, let's say up to 2022, the way that you interact with the New York Times is probably not in the form of a dead tree. Thank goodness some of you still actually subscribe to the New York Times in hard copy; it's wonderful. But you're probably experiencing it digitally, and that's the environment in which you may want to have a data scientist in your company to try to make sense of, and also optimize, the digital experience. You may be experiencing the core news product on a desktop, or you might be interacting with Wirecutter or other non-newsy parts of the bundle, including Cooking or Games, and as of 2022 Wordle, and also audio content. Interacting with the New York Times now is a wide variety of user experiences, not just a dead tree.

Church and state, and where data science sits

[00:02:50] Let me explain where you might put a data science team in that organization. To explain that, I need to explain to you the org chart of the New York Times. Here's the org chart of the New York Times at its most coarse level: church and state. The things I'm going to tell you about today are not going to be about the journalism parts. They're going to be about the ways that data scientists try to make numbers go up. I'm going to tell you about a couple of problems in which the job of the data science team is to use data in order to develop an optimized experience that drives some sort of enterprise KPIs. It's mostly going to be about things that involve not generating individual stories, but rather not going out of business: making the New York Times economically strong.

[00:03:37] I'll divide that into a couple of stories. The data science team develops and deploys machine learning, so we'll be talking about machine learning today. How many people use machine learning in their day-to-day life? Okay, like four. Great. I'm going to tell you more about why you might want to partner with a data science team to use machine learning, and why you might want to try to get your company to employ some people doing machine learning. The stories I'll tell you today are about algorithmic targeting, which is in the lower right-hand side, and then I'll tell you two stories about algorithmic recommendations, and then I'll try to close with some lessons learned.

Algorithmic targeting and the paywall

[00:04:13] As I said, algorithmic targeting is the first team. Algorithmic targeting is a team that we christened in 2019 to think about, among other things, the paywall. If you'd like to know more, there's a great blog post by Dr. Rohit Supekar [?], who wrote it back in 2022, about the way we use machine learning as part of the paywall experience. How many people have ever experienced a paywall? Good. I'm really just trying to get you to raise your hands there. The paywall has been around for a while, since 2011. It was an experiment. There's an opinion guy on the right-hand side, so you can tell this is an old website. People thought it was a stupid idea when the New York Times announced that they were going to have a website in 2011. Really smart people said things like "it is destined for failure," or, as we used to write listicles in journalism, "five reasons why the New York Times paywall will fail and why it's really dumb," and that was written by somebody who knew what they were talking about.

[00:05:14] In fact, it was not doomed. Things have worked out pretty well, actually. By having a differentiated free/paid experience, the New York Times has really defied this old adage that information wants to be free. If you know that old adage, if you read the original, it's a full paragraph that at the end also says information wants to be expensive. So the New York Times has had a pretty successful digital subscription project. Around 2013 is when I showed up, so New York Times profit is doing great, and it's good times, good times.

[00:05:45] What is a paywall? If you think about the paywall experience, one way to model it is as a funnel of people. It's broad in the sense that many people have heard of the New York Times. A few people show up and register, and that's one of the experiences that you might want a machine learning team to optimize: when do you ask them to register? Then eventually they move to a higher level of commitment in their relationship with the New York Times, that is, money, and that's the paywall. That's what the "pay" means in paywall. You might want to use a machine learning team to try to optimize that experience, and you need to think very carefully about what you're optimizing. As was said in the introduction, you need to be really careful about what questions you're asking. Optimizing for the right thing is a good idea. Optimizing for the wrong thing is a bad idea. That's another good reason why you want designers, UX researchers and machine learning people all to work together.

Prediction versus prescription

[00:06:38] We try to frame it for our friends in terms of the difference between prediction and prescription. Predictive machine learning has been in a lot of the papers. When you have an algorithm that can tell if a picture has a hot dog or not a hot dog in it, that is a prediction. You give it an image and it predicts what's in the image. When you have a chatbot that's effectively predicting the next word based on the words it's been trained on, that is a prediction problem. But often when you hire a machine learning team, you don't actually want to know what's going to happen in the absence of treatment. You actually want to know what treatment is going to drive the particular outcome you want. So we try to help everybody understand that what we're trying to do as a machine learning team is prescription.

[00:07:19] Predictive ML is what will happen without some sort of intervention. For example, I can look at people who use the website and predict who's going to cancel their subscription. Having done that back in 2013, the immediate response I got from my friends in product and marketing was, "Okay, that's great, but I don't really care who the at-risk individuals are. I want to know the risky behaviors, and more importantly, I want to know what things I can do to make those people not cancel." So we tend to think about this in terms of the modeling question: what is the best action in order to maximize some outcome? We look at people's previous activity, which includes some intervention, some type of action, and then based on the action we try to predict the outcome, for example whether the person subscribed, in the case of the paywall. We typically don't know what would have happened to this person if we had not given them a paywall, for example, or if we had given them more access to the content.

[00:08:12] I mentioned earlier that it's important to ask the right question; in the introduction we talked about knowing the right question. You also have to know what you want to maximize. You might want to get as many new subscribers as possible, in which case you should have a very hard paywall: everybody should have to become a subscriber right now. However, then people don't experience the breadth and depth of the content, so you may lose out on the possibility of having more subscribers in the future.

Randomized trials and tuning the knob

[00:08:36] How do we do this? We do this leveraging A/B tests. You will experience A/B tests at some point today as part of the content of this conference. We do them, and we often refer to them as RCTs, which is an older term: randomized controlled trials. Often we're not just doing two variants. We're doing lots of random things to random people in order to learn what works best. In the case of the paywall, it could be something like this. I'll describe it in the original language of the paywall. Instead of saying you get 10 articles a month and then you pay $10, imagine that you say to people, "Okay, you get three articles a month," or two articles a month. By doing it at random, you get an awesome gold data set for seeing what the trade-off is between getting as much engagement as possible and getting as many new starts as possible. Those are examples of two different business goals that you might have. One is a statement about how they use the product, and one is a statement that will make your chief financial officer happy.

[00:09:31] How do we turn that into machine learning modeling? We do some randomized controlled trials. Some people get lots of paywalls, some people get fewer paywalls. We can trace out and learn from the data what the trade-off is between eating your cake and having it too, that is, people exploring the breadth of content and people converting. Then we can show that if we have a targeted model, meaning something that uses information about people in order to give different amounts of friction, or different amounts of paywall, to different people, we can defy gravity and move off that line. This model is called Raptors [?], which is a real-time model, and it works.

[00:10:05] How many people here love math? Awesome. That's about five more hands than I expected. How do we think about this? One way to do it is to leverage two predictive models: one model that predicts how much engagement you're going to get as a function of the things you were allowed to know before you decide how many paywalls you're going to give them, and the treatment, and then to balance between the two. The utility of doing that once and then balancing it like a knob is that your product friends can say, "This month we really want more starts," or "This month we really want more engagement," and you really can tune that knob between two different things you're trying to optimize at once. We actually used to have meetings called knob meetings, where we retuned that knob.

[00:10:46] A lot of this is based on back testing. You do some random things to random people, and then you can learn what a new policy would do, and by "do" I mean how it would lift KPIs or not. Even though five of you said math, you probably don't want to hear about Hájek estimation and self-normalized weighting. Just trust me, it works. It works in the sense that we can go and optimize some balance between clicks and engagement; sorry, between starts and engagement.

Algorithmic content recommendations and bandits

[00:11:13] That's a story about algorithmic targeting, which is paywall specific. If your company doesn't have a paywall, you may not have exactly that problem, but you probably have content, and you might want to optimize which people see which content. So I'll tell you a little bit about the way we do algorithmic content recommendations. Content recommendations also have an old history at the New York Times. In 2011, the New York Times launched a "recommended for you" widget in the lower right-hand side and had a quaint little press release about it called "The New York Times Adds a Recommendation Feature." When it first appeared, it was a structuralist feed in the lower right-hand side, where you would see anything, opinion content, hard news, recipes, all in a structuralist feed of stuff. These days, and by these days I mean this blog post from Dr. Anna Coenen [?] from 2019, we think about it as an optimization problem. Instead of thinking "let's show people stuff," we think about what is the KPI we want to go up, the key performance indicator, the number we want to drive, and how do we design an algorithm that is an optimization algorithm in order to make that number go up.

[00:12:18] That crucially involves experimenting. I mentioned the RCT, the randomized controlled trial, for thinking about the paywall. Similarly, here you have to actually do something. This is a painful transition for a company. Often you want to hire a data scientist, and you say to them, "Here's a bunch of data, make magic happen." It is very difficult to know what will happen if you have never done any intervention. So for the sake of your future data scientist friends, please consider at least doing an A/B test or some sort of experiment, because that is a gold data set for learning what the optimal treatment to deliver is.

[00:12:59] As Dr. Coenen writes in this blog post, we do this using a particular approach called contextual multi-armed bandits. Just so I know my audience, how many people have ever heard of bandits? Two, four people. Okay, good. I'm going to tell you about bandits. All y'all are going to become bandit fans in the next two minutes; it's going to be great. We do this using a technique which, since 2007 or so, has been called contextual multi-armed bandits. Bandits is actually a very old word. This is the first paper where anybody used the word, from 1955. They had this experiment with Harvard undergraduates where they made them press two buttons, and if they pressed the right button, it delivered a coin. The thing is, both of the buttons were like a slot machine, and if you've ever been to a casino, slot machines are sometimes called one-armed bandits. So they called this a two-armed bandit. The tantalizing part of this old paper from 1955 is the little footnote that says the optimum rule for playing the two-armed bandit seems not to be known. Which is true. There's a whole field now of figuring out the right way to decide in real time: should I show people the blue button or the black button?

[00:14:06] The way you do that, of course, is you deliver both of them at random to random people. You have probably heard of A/B tests. How many people have heard of A/B tests? Okay, good. If you've ever been through a pandemic and heard about people giving vaccines at random to random people, the reason you do that is so you can learn from the data what the causal impact is of doing different things to different people. A/B testing is the term for this, although the idea in statistics goes back about a hundred years. What you're used to is: I do random things to random people for six weeks; I show them A, B and C. The idea of an adaptive A/B test is that as you learn from the data that option A seems to be better, rather than waiting until week five and throwing money out the door, you gradually upweight the probability that you show people the thing that actually works. This is from a very old blog post by my friend Matt Gershoff, illustrating it as a visual to compare and contrast A/B testing and bandit selection.

[00:15:09] That's a mathematical observation. The other thing is that you don't have to have a meeting. When you have an A/B test, generally right around yonder you have a meeting, and one of the SVPs says, "My KPI is really important," and then another SVP says, "My KPI is really important," and then you have to duke it out, and the highest-paid person's opinion wins, which makes everybody sad. It's better if you just decide in advance what number you are trying to make go up, and then you let go and let math. A bandit will make that number go up, and everybody's happy.

Recommendations that defend and advance news judgment

[00:15:44] We've been bringing the gospel of bandits to a lot of different problems at the New York Times. For example, there's this great blog post by Dr. Jean Yang [?] where she writes about how we do algorithmic recommendations in a way that defends and advances news judgment. One of the things about the New York Times is that it is a brand-conservative, publicly traded company from 1851. It's not the kind of place where the editors are going to say, "Whatever, optimize whatever." It's actually something where we worked very closely, hand in glove, with the editors to develop algorithms that are optimizing the right thing. We worked with them quite a bit to think about what we are actually trying to optimize. What does success look like when we're recommending content to different people?

[00:16:26] One of the things we did was move very much away from the idea of a little structuralist feed in the lower right-hand side, which was the 2011 design. Instead, we started thinking about different jobs to be done, and how there's really a different part of the homepage or the article site for each of those different jobs to be done for the user. Does the user want to experience a set of opinion content, or lifestyle content, or Wirecutter content, or maybe Athletic content, or beautiful news at the top? All those different modules have separate algorithms optimizing different things, and we work with different editors who say, "Here's a set of articles that we think are appropriate to go into that particular widget": news, features, repromotions and sub-brands, sub-brands here meaning Cooking, Wirecutter, The Athletic.

[00:17:13] The other step is to work closely with editors to say, okay, we can think about this in three steps. The following is not a trade secret; almost every company that does recommendations will tell you that the process is, first, thinking carefully about what the action space is. That's the technical term; it should be the list of articles that's going to go into a particular widget. Then doing the algorithmic ranking, which means you've worked with editors to figure out what the success score is, what number we're trying to make go up. Then applying any business rules. For example, they may not want two stories of the same ilk to appear next to each other. Various finishing functions are applied.

[00:17:48] Now we have a process in our relationship with editors where we've made things very self-serve. I was talking on Monday about provocations, prototypes, products and platforms. Once we had prototyped this a bit, we got to the part where it was really a product: the data science team was building a product for editors, and the editors could do self-service. Self-service means that the editors can say, "Here are a couple of articles that we think are appropriate for this," and then the algorithm can optimize an agreed-upon score, and they don't even talk to the data scientists. When there's breaking news, and the Queen dying was a big one for us, there were a lot of articles, and it was unclear how we should go about promoting them all as the story developed. That was an example where the editors had a self-service tool, a product that we provided them, to do that optimization without even talking to us.

Lessons learned: prescription, data engineering and the hype cycle

[00:18:47] That was a couple of stories: optimizing the paywall, optimizing content, and optimizing content in partnership with editors. Editors are not data scientists, so a lot of that involved finding common ground, finding a common language and iterating with them until we were literally looking at things. The thing could be content or it could be a score. What are some of the lessons that we've learned in the data science team as an internal product team at the New York Times? One is a lesson that I really stole from Gartner. Gartner had this blog post in 2012 or so about the difference between prediction, description and prescription. We try to emphasize to our friends that you can take a bunch of data where people do what they do and you don't intervene, and you can learn a model that says, for example, which users are going to cancel their subscription. But it's a bit like having a hospital that never gives anybody any medicine, and then you hire statisticians and ask them to predict who's going to get better and who's going to get worse. That matters; we want people to get better. But if you're the doctor, you don't necessarily need somebody to predict what's going to happen in the absence of treatment. You really want the statistician to tell you what treatment is going to drive the outcome you want.

[00:19:54] That requires you to get to a point where the editors, product people or marketing people are willing to let the data scientists do some experimentation and possibly drive part of their process. So we've tried to emphasize the value of prescriptive analytics rather than predictive analytics. That is more difficult. You have to think not only about what signals you think matter, but what number you're trying to make go up and what levers you're willing to pull in order to make that number go up. If you're willing to think through all three of those things, then you have the opportunity to turn the product experience into an optimized product experience.

[00:20:29] Another lesson learned, and this is from my friend Dr. Monica Rogati, is from a blog post she wrote, I believe, in the summer of 2019. She was working as a consultant, having already worked at LinkedIn and a bunch of other places, and she got tired of saying the same thing to people over and over again when they said, "Okay, we're just going to hire an AI person and sprinkle AI on the whole experience, and it's going to get better." She said, no, you're doing it all wrong. That's not the way this works, that's not the way any of this works. In particular, you need to have a good data engineering team and have the tooling correct, and that's not the thing people usually blog about and get excited about, tooling. But the truth is you're only doing a provocation if you just hire an AI person. You're not actually building a prototype, let alone a product, let alone a platform you can stand on.

[00:21:18] You need first to make sure that you have really good data engineering. Then you can start making sure that your data pipelines are working. Then you can start making sure that the data don't have huge pathologies, a break between your mental model of how the data were created and how they actually were created. Then you can start thinking about what is. Then you can start thinking about what could be: A/B testing is the basics of doing some experimentation and intervention with the world. Then you get to think about the fancy AI. If you bring in somebody who's doing machine learning and you haven't really built out the data engineering function, it might be fun, and they can do something provocative, but it's not really going to be the basis for the product unless you have the fundamentals correct.

[00:21:58] The other thing to learn, and this is another thing we were talking about on Monday, or maybe it was Tuesday this week, is the hype cycle. The idea of the hype cycle, also stolen from Gartner, from 2003, is that when a new innovation is introduced, there's a period when people are very excited about it. Often there's a good reason for them to be excited, because the investors and the people selling the product are telling you that this thing is the best thing since sliced bread. Then when you start using the thing for a while, you may realize that it doesn't actually leap over buildings in a single bound, etc., etc., and you experience the trough of disillusionment. There are tears. If you drop it and never use it again, you will prevent yourself from getting to the efficient part of the hype cycle, which is the plateau of productivity. Or, as I sometimes say, you move from irrational exuberance to rational exuberance.

[00:22:46] This is happening right now with AI. AI as a term has been around since 1955, but around November of 2022 a lot of people got very excited about AI. That's great for me as a machine learning person; I'm glad that people are excited about AI. Once I get them in with the gateway drug of whatever the latest AI is, I can tell them about statistics and predicting things. There are lots of things you can do with data; let's do that stuff. The inefficient thing for us all is that the technology is changing very quickly. If you put tens of billions of dollars of capital into a technology, it'll change real quick. The other inefficient thing for us is that everybody is at a different point on this hype cycle at the same time, so it's a very difficult time to form alignment between different teams and different companies.

[00:23:29] Another lesson learned is varied voices. We try to hire people from a variety of fields: neuroscience, computer science, mechanical engineering, biology, political science, chemical engineering, physics, I'm just naming them off the top of my head here, applied math, cognitive science, geophysics. We try to hire people from a variety of fields, and they really bring a variety of perspectives to this. Those are some of the lessons learned. If you'd like to know more of the lessons learned, there were two books that I wrote about data during the plague. The one on the left is with a history professor, so it's more of a book for normal people. The one on the right is with some very august and accomplished computer scientists, and it's a little bit more technical. Both of them, I think, will give you a useful view on data and how to put data to work. The history and ethics of data is a real focus of the book on the left, and how to think about putting data to work in a product is really the focus of the one on the right. With that, I think we've got a little bit of time for questions. Thank you very much, and I look forward to your questions.

Q&A

[00:24:32] Host: Thank you, Chris. Wonderful. I'm forever now going to think about physical newspapers as dead trees on kitchen tables. Thank you for that phrase. As we get some questions in from all of you, I'm curious, Chris: at the beginning of your talk you mentioned you want to optimize for the right things, and you spoke about jobs to be done and understanding where people are. Tell us a little bit more about how those conversations work in the room when you're talking with cross-functional partners, or even with your team from all over. How are you deciding what to optimize?

[00:25:05] Chris: Those conversations can be very nuanced for different parts of the org chart. With our friends in business, particularly with a subscription product, as opposed to maybe an advertising product where the revenue is aligned with something that happens in a few milliseconds, it can be hard to think through what we are really trying to get people to experience with our product. The thing you want to optimize for is a long-term relationship with a subscriber, and that's slow. You have to wait six months to figure out if somebody's going to stay a subscriber for six months. That's one of the reasons why we think about how we want to eat our cake and have it too. The other thing about waiting six months is that, effectively, the laws of physics change. Here it's not the laws of physics; it's the way users behave, it's the state of the digital product. Who knows, we might acquire Wordle or something, and the way people interact with the product totally changes. So the model that you build predicting what's going to happen in six or 12 months is immaterial, because effectively all the laws have changed. That's another reason why, when working with business partners, you really need to think about what you are trying to optimize, particularly for a subscription product where there are slow metrics.

[00:26:13] Chris: For the editors, we worked really hard to think about clicks. You have a variety of signals that you get. If somebody clicks, it could mean that they read it and then realized it wasn't for them. So you might want to think about clicks and actually engaging with the content. Of course, there's an infinite number of ways to measure engagement, and not everything that can be counted counts, and not everything that counts can be counted. So we worked hard with them to think about not only digital engagement but also editorial importance. One of the things that's described in that blog post about working with the editors is how we would put a score on editorial importance. For that, the behavior that we tracked was not the behavior of our users; it was our editors. We looked at the way editors promoted different content and learned from the data: based on what this digital asset is, here's how we think an editor is going to promote this piece of content. That was just the beginning. Of course, we had to work with editors to actually look at the ranking and see if editors liked it. That is not easy to quantify. As you know from doing UX research, a lot of the time it's really sitting with editors, asking them how they feel about things. The hard work for a machine learner is how you are going to take somebody's qualitative feedback and implement it as a change in the model. That's not easy. That's why we hire people who I think are good communicators, not just good at computers.

[00:27:32] Host: Yeah, that communication is essential as well. Thank you. We've got some great questions. What's your perspective on ethics? I know this is part of your specialty. There's a question here on the ethics of using algorithms and changing how people read news for KPIs.

[00:27:44] Chris: That's a real tough question, and in a way I threw a grenade at my own foot by mentioning that the book on the left is about the history and ethics of data. My other hat, which I didn't talk about today, is from the academy, from being a professor. The way that people think about ethics in the academy has been really stress tested for about 50 years under a particular framework around principles. Operationalizing ethics in academic research is often framed in terms of a set of principles, which I like to cartoon as rights, harms and justice. Those three principles give people a shared vocabulary for having at least difficult conversations about those trade-offs. For example, you might develop a product that's so good that you just want to roll it out for everybody and not tell them. That's an example of leaning into harms but not leaning into rights: you're leaning into what you do that's a benefit for people, but you're not leaning into giving people informed consent. For journalism, there already is a very long conception of journalistic ethics that has been hammered out by centuries of journalism. So one of the things we've been doing is working with the newsroom to think about how we reconcile ethics in the form of a long checklist, and ethics in the form of a small set of journalistic principles like trust and transparency, with the way that data scientists and academics have thought about ethics in terms of giving people informed consent, and also monitoring things so you actually know, long term, what the impact is of the algorithms that you roll out.

[00:29:13] Host: Thank you. There's a question from Kate about data visualizations in the New York Times. They're phenomenal, I completely agree. How do journalists work with data scientists to create these visualizations and determine what to prioritize?

[00:29:25] Chris: That is a great team, and I don't want to cause any confusion with people at the New York Times. In terms of church and state, there's a great team on the church side that does data journalism, and another team that does great data visualization, another team of data graphics people. We talk regularly, but in many ways the skills that my team has focused on are not the skills that they need. For example, part of working in data journalism is making visualizations and explanations that are clear and easy to understand, where you don't need to unpack what the machine learning algorithm behind it is. There are a couple of rare exceptions. Years ago ProPublica had this great story where they reverse engineered a decision tree for Obama fundraising emails. That's when you have to explain, as part of the story, how a decision tree works, in addition to what it means in terms of politics. Generally, we try to focus on things where we're optimizing a KPI and developing something that's in production, actually rolling out in prod and a reliable piece of infrastructure. So we tend to work on different stuff, although we talk regularly. I've definitely benefited from them thinking about how to visualize stuff in a way that's interpretable. Occasionally I hope they've benefited from talking to me about modeling things, but I have to say they're very good at their job. The job that they're doing is sufficiently different that I would say we're really solving different problems, and they're quite good at the problem they solve.

[00:30:46] Host: Absolutely. And speaking of people solving different problems, there's a question about UX researchers and data scientists working together. It reminds me of the fabulous conversation we heard yesterday about peanut butter and chocolate, for those that were there: analytics and UX research. I'm curious how that transpires at the Times.

[00:31:02] Chris: That is one of the reasons I was so excited to accept the invitation to come here. I feel like I, as a data science lead, have not done enough to talk to designers. I do occasionally. When we're trying to brainstorm on a new product, we're always interested in, for example, what UXR has been done. In a recent one, a data scientist working for me was thinking, "Maybe people have the following behavior, and we could model that." I said, that's great; before we start writing code or looking at data, we should talk to the UX research team and see if they've already done research on that. So I definitely try to work with them. The other, different way that the field, but not the group, of UXR impacts us is when we're building internal products, where we actually know our internal market. When we build out editor-facing tooling, which I didn't talk about today, I can remember in 2015 saying to one of the machine learning engineers, "You are now the ambassador for machine learning to the newsroom. Go shadow them. Go read this book on analytics for success and how to think about using analytics to drive products. Go understand their pain and empathize with them." That was not a natural place for that particular software engineer, but he did a bang-up job.

[00:32:12] Chris: So we definitely use those principles, and really think about, when we're developing an internal product, how we go and meet the people who are allegedly going to use it. How do we make sure that we're doing iterative and incremental product development, so that we meet with them regularly? We ask them, "Do we have your permission to continue?" We try to build our features in slices rather than layers: build out one little MVP solution and try to get the reaction. Building in slices rather than layers is really important, because we will often build a very simple data science solution and then do the organizational integration, meaning, have we built something that people can actually integrate into their process or not? Once or twice in the early days, we built out something really complicated, and then we showed it to the user, here meaning the editor, and they were like, "Well, that's interesting." Which is really sad, when a technologist spends six months building a thing and then the user says, "It's interesting; I can't really integrate it into my life." So that sort of organizational integration is definitely something we focus on in data science for building internal products that win.

[00:33:09] Host: Amazing to hear. You definitely want to have those learnings sooner in the funnel than after you've spent all of that time.

[00:33:16] Chris: It's very sad for the technologists if they build something that people don't want.

[00:33:19] Host: Yes. Okay, well, that's all our time for questions. Thank you all for your fabulous questions, and thank you, Chris.

[00:33:24] Chris: Thank you.