From Research to Reinvention: Prayag Narula on Startups, Setbacks & AI Innovation

Ryan Leffel, Prayag Narula at UXDX USA. Video: https://youtu.be/sF7Grw6XKQU

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introductions and getting into research

[00:00:07] Ryan: All right, I'm really excited about this one. I'm Ryan. As Amanda mentioned, I oversee design and research at Priceline and host a podcast. So I'm super excited to be able to do this today and get to talk to Prayag. The podcast is really drilling into not so much what you do, but how you got to where you are, challenges you faced along the way, and setbacks that became opportunities. It's been really fun having these conversations, and I'm excited about this one. Do you want to start by just talking about what you're up to today?

[00:00:37] Prayag: Sure, absolutely. Before I get into that, I want to answer the burning question for all of those who were at the party last night: I am not wearing the same clothes. I have two of the same t-shirts. My name is Prayag. I'm the co-founder and CEO of Marvin, or HeyMarvin. We are a research repository. We are a research co-pilot. You can check out more about us at heymarvin.com. We have a booth right across from the coffee place, so it's easy to get there. Check us out, and we'll talk in detail about what we do and what it means to be a repository and research co-pilot, hopefully in the session today.

[00:01:24] Ryan: Definitely. Let's start with your background and how you got into research.

[00:01:30] Prayag: How I got into research? I think a better question is how I got out of research, because there was a very brief period of my life where I was actually not doing research. My background is in research. I went to school for research. In fact, I came to the US to study research at Berkeley. I taught design research at Berkeley. I was supposed to get a PhD in research, and I was going to become a professor. That was going to be my career journey. And I do think part of my dad expects me to go back and finish and become a professor. I don't think that's happening, but don't tell him.

[00:02:08] How I got into research: I had a computer science background, and I was very interested in this idea of HCI, human-computer interaction. We're building all these products, but how are people using them? Back in the day we didn't have HCI as a field that you could go study, so I moved to Helsinki, Finland, and did some research there. Back in the day, for the younger people in the audience, if you wanted to learn HCI you went to Scandinavia, and especially Helsinki, because that's where Nokia was. Back in the day they had 70% market share, and they did amazing HCI research. The trouble with them was they would do this research, they would have amazing findings, they would put those findings in a closet, lock the closet and never visit it again. There's a lesson there somewhere.

[00:03:05] When I was doing that research, I noticed that around me everybody had a master's degree or a PhD, so I thought I'd better get an advanced degree. And that's how I got to the US, and the rest is history. Hopefully history in the making.

From research to a demand generation startup

[00:03:25] Ryan: But you became an entrepreneur, which is really interesting. One of the things that I wanted to ask you about is that your first company was in demand generation and marketing. What made a researcher open up a company focused on demand generation?

[00:03:38] Prayag: That's an interesting story. I would not recommend researchers go into marketing and sales unless you have to. In fact, though, I will say a lot of selling is probing and asking questions. I actually get a bunch of people who have a design background saying, "I don't like selling." You do; you just don't know it yet.

[00:04:07] How did we get into marketing? When we first launched, it wasn't actually an HCI problem that we were trying to solve. We were actually doing AI. What we were trying to do was build data sets for AI to get trained on. This is 2011, 2012, back when hardly anybody had any faith in AI, and nobody was doing AI, honestly. We had one customer that was doing real AI, and nobody else was interested in that. Which is funny, because three or four years later Scale AI launched. They were doing the exact same thing, and they're a $20 billion company now. That just goes to tell you how important timing is when you're launching a product. There are a lot of things that you can control, but a lot of things you can't control, like where the market is and what the timing is.

[00:04:56] So we thought, well, we have this ability to build really interesting, really clean data sets; where can we apply it? We went and launched four different products, and the product that won out was, "Hey, we need a ton of sales and marketing data," which is as true today as it was 10 or 12 years ago. So we said, all right, we've got to do something, we have this technology, and here's the market. We basically kept the same technology and pivoted into that market, and that was my life for 10 years. That was crazy. I was teaching people how to do sales when I had never done sales before.

Leadership is mostly doing stuff

[00:05:36] Ryan: What would you say was the biggest lesson you learned during your time starting a business for the first time?

[00:05:44] Prayag: That's a great question. A lot of lessons. I would say there are a couple of lessons that I think are relevant for today. Yesterday somebody was asking about leadership: what does it mean, how do you become a leader? I've been thinking about that, and that was one of the big lessons I took. I became the CEO of my company at 27. I knew nothing, and I'm not being humble; I literally knew nothing. I had never had a real job. I'd worked in a research institute, and I had worked as a low-level coder in a services company, but I had no product development experience.

[00:06:36] Then we hired a lot of leaders, and one of the things that kept surprising me was how much of leadership was just doing stuff, rather than having a title, you know what I mean? Part of it was we built this product and we just showed up, and I said, "Hey, I'm the CEO." And they didn't ask, "Oh, you're CEO of a four-person company?" They said, "Oh, it's the CEO. You are the leader." I guess I'm a leader of a group of three people, none of whom report to me. But I basically started talking about what we had built, and we just kept building stuff, and we got a ton of press, and people said, "These people are leaders in the space." We actually had a social enterprise, so, "They are leaders in social enterprise."

[00:07:40] Most of it was doing stuff and showing up, and that's what a lot of people in my companies have done. It's less about having a title. It's more about just showing up and doing stuff that nobody is doing. And that is, I would say, 80% of leadership.

[00:08:03] Ryan: I love that. One of my favorite quotes is "Leadership isn't a position, it's a choice," by Stephen Covey. It just makes so much sense. So how did starting that first company, and what you learned there, make you want to do it again? And how did that lead into where you are now with Marvin?

Leaving LeadGenius and starting Marvin

[00:08:23] Prayag: I left leading my first company. "More pushed out" is probably a better word for it. But let's say I left my company. You know how, when you get dumped, you say, "Oh, it was consensual"? This is basically the same thing. That's one way to put it. So I got dumped. It was a 10-year-long journey, and I took a little bit of a break, and I told my wife, "Hey, I want to do something new." The first question she asked was, "Okay, for how long are you going to try to do this before you actually get a job?" She didn't ask me what I was trying to do or what the goal was. No: what is the deadline after which, if you don't get a salary by that point, you're going to go and find a job? December 31st. This was January or February.

[00:09:30] I was exploring a bunch of ideas. There was a challenge I was facing at LeadGenius, where I was trying to ask my product managers what our customers had said. Everybody was asking the same question: "Hey, what have our customers said about this?" And other than going and tapping people, "Did you talk to the customer? What did they say about this?", there was literally no way to have that access to customer feedback, because we had multiple systems. We had a support system. We ended up getting Gong for the product team, because they wanted to record some of those conversations, so we had that Gong system being less used by sales and more used by product. Then we had our Salesforce, then we had our NPS. And I thought, this is a mess.

[00:10:22] I was very motivated to build a centralized location where I and everybody else could go and see what the market feedback was, and it was just so hard. So I filed that problem away in the back of my head: if I ever want to do something, this is a worthy problem to solve. But I was exploring a ton of problems. I think the biggest reason I decided to solve this problem was that I wanted to work with my tribe. I wanted to get back into research. I'd done sales and marketing, and I have some of my sales and marketing people here; no offense, we love you guys. But I really, really love the design, the research, the builder community. And I thought, if I'm going to spend another 10, 12, 15 years doing something, I would rather do it with people in a community that I really love and appreciate.

[00:11:16] I'm not just saying that to butter you guys up. Genuinely, if you've got to do it for 12 or 15 years, which is what every great product and great company takes, you've got to do it for the right reasons.

[00:11:29] Ryan: That may have answered what I wanted to ask you next, because we all have tough days, and sometimes just getting up, it's, "Oh man, I have to go through this again today." When you're running a business, I imagine there are some very difficult, trying times where you have to ask yourself, why am I doing this? So what keeps you going during those times when you're questioning what you're doing?

[00:11:48] Prayag: That's a great question. Sometimes just freaking grit. I'm going to grind my teeth, I'm going to try to get to 9:00 PM, I'm going to go to bed, and tomorrow is future Prayag's problem. I'm just going to get through today. That happens from time to time. So honestly, a lot of the time it's freaking get to the end of the day and go to bed; that's what you can do. But over a long period you can't do this. There were times at my previous company, especially at the end, where it was, "Dude, I don't know if I can do this for too long." With Marvin especially, it keeps coming back to: we are working with people that genuinely... I would have gotten a job as a researcher, or a designer, or a design leader, or whatever. And if this is going to be my life's work, then once you know this is it, that's what gets you through the day.

Researchers as early hires, and democratizing research

[00:12:54] Ryan: Cool. All right, let's get into some of the spicy stuff here. From what I understand, at startups a researcher is usually one of the later hires. Could you talk about why you think that's the case, or disagree?

[00:13:12] Prayag: That's a great question. At LeadGenius, our first product hire was a researcher. Because I'd come from a research background, I thought we should build out a good research practice before we build out even a good product practice. But ultimately what ended up happening was that researcher became responsible for not just understanding customer problems, but also trying to figure out how to solve those problems, working with the rest of the organization to figure out, engineering-wise and product design-wise, what do you do? His mandate got so broad that ultimately we said, well, you're a product manager at this point.

[00:14:01] So I think what ends up happening is, even if you build out a research function in a small company, you end up doing so much that it goes beyond just research, and that's just the nature of the beast. The short of it is, research becomes a responsibility that goes across the organization in a startup. It's the CEO's role, it's the product head's role, it's a product manager's role, it's the salespeople's role. That's why I think researchers are a great hire, and we have hired a ton of researchers at Marvin, but they're doing product management.

[00:14:46] Ryan: Okay. One thing that comes up a lot is democratizing research. I think one of the challenges with that is quality. So my question, and I realize this is maybe a broad question: is bad research better than no research at all?

[00:15:01] Prayag: That's a great question, because what I'm talking about is, "Oh, everybody should do research," but can everybody do good research? Can everybody do thorough research? Back in grad school, I had a qualitative research professor, and we were talking about doing research, and she was saying you need to go in depth. And I said, "You know what, I have this project that I'm doing for my design class," it was tangible user interfaces, "and I'm a student. I can't even pay people 20 bucks; if I pay 20 bucks, I'm not eating tonight. So how do I go and do the research?"

[00:15:53] She said, "Well, just go find the next person that's sitting next to you, show them the design and see what they say." And I asked, "Is that real research?" And she said, "Yeah, it's called guerrilla research." And I said, "Guerrilla? This is just freaking tapping somebody on the shoulder: give me feedback." "No, no, there are studies about guerrilla research, and it actually works." A ton of studies about guerrilla research.

[00:16:22] So, is bad research... The definition of bad research varies. If you're a designer and you go to a customer or a user with your puppy-dog eyes, "Please, do you like my product? Do you like my design?", and they say, "Yes, of course, of course we love you," that's probably not super helpful. But if there's a baseline met, in my opinion that research has value. Where researchers come in is they can help you curate: "Actually, maybe less emphasis on this, more emphasis on this." So the job of a researcher, in my opinion, is changing too.

AI as a research coach, and bias

[00:17:01] Ryan: Do you think there's an opportunity there for AI to help correct bad questions, or to help make sure that the problem you're trying to solve is the correct problem, and that you're tackling it the right way with the customers you're talking to?

[00:17:16] Prayag: 100%. That's partly getting into what Marvin does. I promise this is not a pitch, but one of the things that Marvin does is help create these guardrails for our users, to make them better researchers. One of our features, and this was a throwaway feature that we launched, was that we'll give you feedback on your interview. The AI would analyze the interview and say, "Maybe ask this question this way," or, "This is a leading question," or, "Maybe don't spend too much time blabbing on about the product."

[00:17:57] And what we found was people were way better at taking feedback from AI than they were from their peers in the research team. They were like, "Oh, it's just AI telling me. I'm not taking it personally. It's just an AI answer." So there are all these tools that we have built, and the industry has built, that help build these guardrails. That's what I mean when I say the role of a researcher is changing. I think the role of the researcher today is as much truth seeker as it is helping others seek truth: be the guru, be the mentor, be a comrade, review people, be a teacher. AI can do some of that, but I think the researcher's role is probably also turning into that.

[00:18:50] Ryan: One thing that I think is really interesting, and we've talked about this before: how about bias? Do you think there's an opportunity for AI to remove bias from research?

[00:19:03] Prayag: I have a spicy take on bias.

[00:19:05] Ryan: Let's hear it.

[00:19:06] Prayag: Okay. I think trying to completely remove bias is futile. I don't think you can completely remove bias. And that's as true for qual research as it is for quant research. The way you phrase questions, where you are, how you find people: we live in a real world, and you cannot completely remove bias from your research. What you can do is address bias. If you're a designer and you're testing a couple of designs, and you have a preference for one over the other, address it. Say, "Hey, I like this design. I tried to minimize it, but it's definitely possible that I might have introduced a slight preference for this design." So I think removal of bias should not be the goal. Maybe try to minimize it, but definitely address the biases.

[00:20:19] Ryan: So it's transparency.

[00:20:22] Prayag: Yes, absolutely. It's a transparency issue rather than a methodology issue.

Synthetic users and whether AI is coming for your job

[00:20:28] Ryan: Okay. A few more, and I know we're coming up against time here. Another thing we're hearing a lot about these days is this idea of synthetic users: essentially LLMs that are trained to come in, mimic a population and give feedback. You're hearing about it in both quant and qual. What do you think about synthetic users?

[00:20:50] Prayag: It's a great question. My undergrad is in computer science, and one of the things that has been proven time and again in AI is that synthetic training data actually works very nicely, and a lot of what LLMs today get trained on is synthetic data. But it is not a replacement for real data; you can't just train AI purely on synthetic data. Where I've seen a couple of very compelling use cases for synthetic users is taking quant research that is not statistically significant and adding synthetic data on top of it to turn it into a statistically significant data set. That sounds weird, but statistically it works. So that's one application that I've seen that really works.

[00:21:49] The other application is trying to figure out if you have the right discussion guide, or the right methodology. You basically simulate your research on the first five synthetic users and see where it goes. There are other use cases that we can talk about, and there are ways in which you can do this, but those are the two main use cases I see.

[00:22:12] Ryan: Okay. I think we might be just about at time here, but I have one more question, and I feel like if I don't ask this question, I'm doing the wrong thing, because this is the question that everyone always asks. Is AI coming for your job?

[00:22:26] Prayag: No. But it doesn't hurt to look over your shoulder.

[00:22:34] Ryan: Seriously, explain that one.

[00:22:36] Prayag: I think the people in the room are all going to be fine. AI can do a lot of things. What it doesn't have is judgment and taste. As long as you're tastemakers, you're going to be fine, and I think most people in this room, all of us, are tastemakers. What I feel is the challenge is how you give people enough experience for them to become tastemakers. When you're fresh out of school... I didn't have any taste. I was a negative tastemaker; don't ask me. So how do you go from there to, "Hey, I have judgment"?

[00:23:17] Because AI is going to do junior jobs so well, we won't be able to train fresh college grads on the stuff that helps them get to that tastemaker level. That worries me. I have some kind of answer to that, at least some theory on how we can help with that. But no, as long as you're making judgments and building taste, I think you should be fine.

Q&A

[00:23:47] Ryan: It's a great, optimistic answer. I'm sure everyone is happy to hear that one. I see the clock is at zero, and Amanda's standing. Do we have time for questions, or should we...

[00:23:58] Prayag: It's fine. I'm going to park myself at the booth, so if you have any questions, come see us. Get a coloring book, get some swag, talk to us.

[00:24:06] Ryan: The top-rated question. Okay. What advice do you have for a scrappy UX generalist at a startup or small company? And how do you learn quickly and effectively? Those are two questions. Good job, they snuck it in.

[00:24:24] Prayag: If you're a UX generalist, I think your primary job, especially in a small startup, is not talking to your users. It is to help everybody in your organization have better conversations with your users. Become that trainer, become that guru, become that annoying person: "Hey, what did you learn? Can we put this somewhere?" Help people become better researchers. I think that's the job of the future, especially in a startup. So that's one big thing I'll say.

[00:25:01] And how do you learn quickly and effectively? Conferences like this. Read research papers. Go to your favorite scholar site, Google Scholar. A lot of them are actually written to be very accessible, and I don't see enough people doing that, so that's why I'm going to put that out there.

[00:25:21] Ryan: I would also add to that, for anyone who's starting off in UX: be a generalist. I think it's the best way to start off, because you can learn a lot of different things and really figure out what you are most interested in, and that becomes what you really want to look at, pursue or go after next.

[00:25:39] Prayag: Absolutely.

[00:25:40] Ryan: Prayag, thank you so much. I'm glad we got to do this. Thank you, everybody.

[00:25:44] Prayag: Thank you, Ryan.