How to Use Generative AI to Find Research Insights Faster

Xuan Zheng at UXDX Community: AI for UX Research & Digital Design's Foundational Roots. Video: https://youtu.be/qXePi4dH3ss

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

The problem: nobody could find research insights

[00:00:00] I am Xuan, and today I'm going to talk about how to use AI to improve the discoverability of research insights. Before going into that, I also want to have a disclaimer: I am by no means trying to say that this is the only way to do it. I'd rather use this to kick off a discussion about how to leverage AI in research processes in general. I'm also going to talk just a little bit more about myself. Again, I am Xuan, and I am based out of the Seattle area in the US. Before Instacart I worked at TikTok as well as yo [?].

[00:00:35] Some really quick questions for you all. I know I cannot see you, but I would still encourage you to actually raise your hand if you have ever struggled to find research insights. The meme here, I know it's paperwork, and we don't really read any physical documents anymore; it's just an analogy to use. All right, and for others, raise your hand if you have ever been asked to share the same research insights about that one specific topic over and over and over and over again. I'm sure you've had those moments saying, "Oh, not again, not this topic again."

[00:01:12] I just want to mention that in this process of discovering research insights, as well as delivering research insights, no one was quite happy, at least in my experience. For non-researchers, there can be a lot of blockers, where they may not even know whom to reach out to, or they may not know where to find research insights, and it is possible that there's not a centralized place for all the research insights. Even when they do know whom to reach out to, they may not get timely responses from researchers, because researchers have other priorities.

[00:01:43] On the other side of this relationship, researchers feel frustrated as well. A lot of times, consolidating research insights feels like a repetitive task. It feels like a chore, and it also takes a lot of time, especially when there are times where it involves multiple researchers.

[00:02:07] At Instacart we have tried a lot of different methods. We have built out a consolidated research repository using some of these tools that are indicated here. We also tried to increase the awareness and usefulness of the repository. We tried to rebrand it, we tried to standardize how we present research reports in those portals, in the research repository. However, that still led to low engagement. One of our hypotheses is that all these tools are outside of the everyday, naturalistic workflow of the stakeholders that we have. It is just another app for them to sign up for.

[00:02:52] In addition to that, at Instacart we use Slack for daily communications, and we also created a Slack channel called research-insights, where we would proactively post research insights. It is also a place where our cross-functional team members can reach out to us for any user insight related questions. However, the engagement was quite low in this scenario as well, as in there were few reactions to the posts that we posted as researchers, and also there were few requests from non-researchers. Sometimes there's also notification fatigue. It is just another channel that people don't really check that much.

Assembling a team of enthusiasts

[00:03:31] Then GPT came out, and some of us thought, "Hey, what if we just used generative AI to deliver on-demand research insights?" We quickly acted on that. We gathered a mighty team, where there were multiple functions on the team. We had a product manager, some engineers, and also some researchers, and each of us had different responsibilities. The product manager acted more like an advertiser, sorry, an advisor role. He wasn't involved a lot throughout the process, but he was very involved in the beginning. The engineers were the behind-the-scenes heroes; they handled all the technical stuff. And researchers played a bunch of different roles. I was acting like the product manager or the program manager; I did a little bit of herding the cats. Some other researchers split some responsibilities, where we provided feature input, we gathered feedback from different groups of people, we reviewed AI-generated responses, and we handled all the communications.

[00:04:36] Before moving on, I do want to pause a little bit and talk about one of the lessons learned. While I was talking about this effort that we had at Instacart with some of my other researcher friends, they had a question: "Hey, how did you get buy-in from leadership? How did you get the resources to actually build a tool like this?" I just want to say, you have to unite enthusiasts and gather people who are already passionate about it. I think we got a little bit lucky in this case. All the people that were involved in this project were already working on something AI related, and they were all super passionate about it. For example, one of the engineers was building internal tools using AI, and the other one was already thinking about how to use AI to improve research processes. So it was just very natural for us, and we learned about their passion through just building personal connections through work.

[00:05:32] On top of that, we also had research leadership support. At the time we had a 20% carve-out in research OKRs, where basically we were encouraged to use 20% of our time to improve research processes. And on top of that, our company culture also supports using AI to boost productivity. So I think the lesson learned here is to learn more about your co-workers' enthusiasm, what they're passionate about, and then really find people who resonate with your passion, specifically improving research processes, maybe leveraging AI as well, and then gather those people together. That'll just make things a lot faster.

Moving fast with an agile approach

[00:06:19] I also want to talk a little bit more about the timeline. We kicked this project off in September last year, and we launched the alpha version of the tool in October, which was a really speedy process. I also want to highlight that this was not an all-hands-on-deck situation. Researchers were only using 20% of our time to work on it, and some of the engineers were even working extra hours for it. So this was, in my opinion, a really speedy launch. After the launch we kept getting feedback and we kept making improvements, and until today we're still improving the tool that we have built.

[00:07:04] I think that's another lesson learned, from my perspective at least, which is to move fast. I want to just go back a little bit. At the very beginning, I kicked off the project and I proposed leveraging a more traditional waterfall method to approach the project. What I proposed was that we would gather a round of user feedback first, then we would propose some features, and then we would start designing and developing the tool. However, the product manager came in and he was like, "Why don't we just adopt a more agile approach?"

[00:07:43] I think that made a lot of sense, because first of all, we were building a product that is AI powered, and because it was so new, what we could predict in terms of how people would interact with it was very limited. So what we could do was just build something and throw it out there, and then see how people would interact with it. Especially since this was an internal project, an internal tool, the stakes were pretty low, so we could afford making some mistakes and learning quickly, and improving the product over time. So I think that was super valuable advice that we got from the product manager: move fast, test often, and iterate quickly. And we benefited a lot from this.

How Genie works inside Slack

[00:08:32] I know I talked a lot about the team and the timeline; I haven't talked about what we built yet, and I want to show you behind the scenes a little bit. This is what we built behind the scenes. What we did was connect GPT with our research repository that we built through Sharper [?], which is one of the tools that I mentioned earlier. Whenever a question comes in, GPT would reorganize the question a little bit. For example, if there are multiple parts of the question, it would try to consolidate the multiple parts together. Then it would go into our research repository and find the most relevant research reports, and from there it would try to summarize the research insights and then provide the sources, the links to the sources.

[00:09:24] In the spotlight, what the users would experience is just another contact in your Slack. We actually built that into our Instacart internal AI agent, which is called Genie. There is a Slackbot of Genie, and we just leveraged that. We invited that chatbot into our research insights channel, where users can just tag Genie and ask a user-related question, and from there Genie would go into our research repository and bring up some summarized AI insights.

[00:10:02] This is a video of Genie answering questions in action. As you can see, it is thinking for a bit, and then it would just summarize the most relevant insights and then provide the source for it. Sometimes, when the information is available, it would also tag the author, the researcher who conducted the research, so they can also provide additional help when necessary.

[00:10:39] That led us to another lesson learned. If you remember, earlier I mentioned one of the hypotheses that we had about why some of the other tools that we had tried, to build the research repository, weren't so successful. It was because it was just another tool; the friction was pretty high; it was another tool to sign up for. So I think one of the reasons why this project has had some success is that we actually built it into the naturalistic workflow. People use Slack every day, and we just built another contact into their Slack, instead of being maybe one of a dozen new tools that they have in their Okta dashboard. I will also talk a little bit more about the engagement that we have had with this tool later on.

Evaluating AI-summarized insights

[00:11:32] So far it sounds great, but I know many of you, especially researchers, may have some concerns, specifically related to AI-summarized research insights. We also took that into consideration. We know AI-generated insights might not be perfect, so we created a heuristic evaluation rubric. It is a template that includes multiple criteria that we would use to evaluate AI-generated, or AI-summarized, research insights. Some of the criteria include accuracy, timeliness, comprehensiveness, consistency, sources, and also readability. For each of them we have defined what it means in the rubric, and also provided some questions on how to evaluate the AI-summarized research insights.

[00:12:25] For example, for accuracy, we define it as the condition or quality of the AI-generated responses being true, correct or exact. We prompt people to think about whether the insights or recommendations presented were aligned with those in the research report, and also whether the connections or generalizations of insights made were reasonable or not.

[00:12:53] We also configured Genie, the tool, in a way to accommodate all these criteria. For example, for accuracy, we would only allow Genie to index research reports, not raw data, just so people don't take a few sentences out of the chapter. For timeliness, we only allowed Genie to index reports since 2021, to make sure that research reports are recent and relevant. In terms of comprehensiveness, we had to make sure that our researchers upload all the reports into the repository, just so they can be searched by this AI tool. If a report is not uploaded to the research repository, it wouldn't even be available for the AI to search. In terms of consistency, sources and readability, we tried to accommodate those criteria through formatting the responses. We have tried a bunch of different formats, but what we landed on was to summarize the insight first, followed by links to the sources individually.

Guidelines for using Genie

[00:14:06] When introducing this Genie tool to the rest of the company, we set clear guidelines on what Genie is for, what Genie is not for, and also how to interact with it. What Genie is for is retrieving insights from existing research, whether that is market research, marketing research, exploratory research or evaluative research. We were also very specific that it was not for external research yet. At the time, Genie was connected to our internal tools and internal resources, but not so much to the open web, so it's not the best for external research. And we did not connect Genie to any other insights outside of the research org, so it wasn't for behavioral data analysis. Also, for information that Genie could access, we couldn't guarantee that it would be publicly shareable, so if you want to share something, you still have to get legal approval or finance approval.

[00:15:14] Technically, Genie could work in direct messaging as well as in the Slack channel, but we asked people to ask Genie questions in the research insights channel, just so that as researchers we can monitor Genie's responses all the time. Whenever there are inaccurate or low-quality responses, researchers can come in and provide some correction or additional insights.

[00:15:43] We also had some very specific instructions for people in terms of how to ask Genie questions. The first one is that they need to provide context as much as possible, as if they were asking a question of a researcher. We also encourage them to get specific, and avoid using ambiguous words like "users", or words that may easily be misinterpreted. For Instacart, we have four sides of the marketplace. Sorry, I think I forgot to introduce what Instacart is. Our core business is online grocery delivery, and we have four sides of the marketplace. The first side is the consumers, the people who actually order from the Instacart platform. We also have shoppers, people who actually go into the stores and do the physical shopping. We also have retailers and advertisers. So we really encourage people to clarify the audience in this way, just so we can match them to the right research resources.

[00:16:52] We also encourage them to avoid jargon or abbreviations. We encourage them to think, "Hey, just assume Genie does not work in your field at all." And some people would also ask, "Are Genie, or AI-summarized research insights, enough? Do I still need to go into the research report and read the whole thing?" The answer is yes, please always double-check the resources, because AI oftentimes knows what to say, but doesn't really know what it's saying. So always double-check the resources.

Collecting feedback from researchers and users

[00:17:27] As I said earlier, we did a lot of iterative testing and feedback, and we collected feedback from different groups. We collected feedback from researchers, where we treated researchers as experts and their responses as the baseline, and we also collected feedback from all the other users, the general users.

[00:17:50] For researchers, we conducted an expert review session, where we asked researchers to basically use the template, the heuristic evaluation rubric that we created. We asked them to come up with some user-related questions in their own domains, and we asked them to write out their own responses to the questions, and we treated that as the baseline. Then we also asked them to ask Genie to generate responses to the same questions, and then compare and review the AI-generated research summary, and provide feedback on the specific criteria that were proposed in the evaluation template, including accuracy, timeliness, comprehensiveness, sources and readability. And then lastly, whether they would recommend the AI-summarized research insights or not.

[00:18:43] In addition to that, we also created multiple channels for general users to provide feedback. For example, in Slack we had quick reactions that people can use to provide feedback on Genie's response, whether it works or not, or if they had additional questions. We also had some automatic messages to nudge people to provide feedback in Slack. And of course, we're researchers, so we had to create some sort of survey. We created the survey to collect general feedback that was not necessarily associated with a specific response. Researchers also proactively reached out to co-workers who had used Genie for informal feedback, whether that was through informal conversations or through following up in the Slack thread.

Improvements made from feedback

[00:19:35] These are some of the feedback and improvement pairs that we acted on, so I'll just go through some of them. For example, at the very beginning we built this feature where people had to use a command. They had to tag Genie first, and then they would have to add /ux, to make sure Genie was acting as a research summary function. We noticed that people had a lot of problems with the command. It could just be formatting issues, an extra space here and there, and that created a lot of problems that just couldn't trigger the research insight search correctly. So what we did was remove the command and just ask people to tag Genie, and then in the back end we would auto-route any questions that Genie identified as a user-related question that could be answered by research insights to the Genie UX function, based on the question asked.

[00:20:45] Another example: we had a lot of engineers working on the project, and they had a lot of great ideas from the technical side. They were building some really great formatting changes to Genie, but in the VPN environment. However, most of our employees actually do not have VPN access. They didn't realize that until the researchers and the PMs on the team told them about it, and their first reaction was, "I don't know how you can function without VPN." But the fact was, a lot of people's job functions don't really rely on VPN to function. So as a result of that, we actually prioritized refining the Genie Slackbot, versus refining the VPN version of it.

[00:21:37] Another one was that at the very beginning of this project, we were only searching the first 10 pages of a research report, and that would sometimes overlook super relevant information later in the deck, especially for some of our research projects, like the customer sentiment study, which would have hundreds of pages. Not saying that pages later in the deck are not important; they're just of different importance levels for different people. So what we did later was expand that search from 10 pages to 50 pages. It's still not perfect, but it has improved a lot in terms of indexing as much information from a deck as possible, and it also prompted researchers to rethink the structure of our research decks as well.

[00:22:27] The next one would be that we realized that a natural behavior for researchers was that we had been tagging each other in the thread where a research-related question was asked, just to provide more context when needed. I think I've already shown this feature early on, but the improvement that we made based on this observation was just to ask Genie to tag the author of the report automatically, when the author is clearly noted in the report.

Where it stands and what's next

[00:23:05] All right, then I want to talk about where we are today and where we're going. The tool is still not perfect, but we have seen a lot of engagement. It has been used more than 500 times since December, and also by a variety of functions: product management, sales, marketing, design and of course research. We have some testimonials as well. From researchers: even though they think their responses might be more exactly what people need, or could be more comprehensive, they could never come up with the bullet points as quickly as Genie. So efficiency-wise, Genie definitely helped a lot. From our cross-functional teams, as you can see, they don't have a lot of words to say, but their behaviors are very encouraging, because we noticed that a lot of them were tagging their co-workers to let them know about this amazing tool, and also sharing summarized, bite-sized research insights with their co-workers.

[00:24:09] However, there are still some problems. For example, it's not perfect yet. We cannot purely rely on the AI-summarized research insights. Researchers still need to monitor the questions coming in on the research insights channel a lot, to make corrections or provide additional information. There are still a lot of technical issues. Genie at this point, or GPT, is not great at reading charts yet, and hallucination still happens. There can be inconsistency, because we sometimes use inconsistent terminology throughout different reports, so it can lead to overlooked sources. And for now our information source is also very limited. We only have research reports hooked into GPT, not any other insights yet.

[00:25:00] With that being said, in the future, what we want to explore is, as some of you may know, a few weeks ago OpenAI launched this new version of GPT, GPT-4o, so we hope to leverage some of its vision capabilities for reading charts or graphs. We will also potentially explore using a custom GPT to limit hallucination, define interchangeable terminologies, or even define the output format a little bit more. We also hope to expand to more information sources, to help with triangulating user problems, potentially including reports from data science, bizops, data ops, media analytics and even more. That is what I have to share for today. I also want to thank all of these amazing people; without their efforts this project wouldn't be possible. Next I'll just open the floor to questions.

Q&A

[00:25:59] Host: Excellent, thank you very much, Xuan. We actually have a question from Mashid [?], who's saying: how reliable do you find the quick reactions on Slack for the responses? Are you only focusing on thumbs up, thumbs down, and measuring NPS?

[00:26:14] Xuan: We're not actually measuring NPS. I don't think they are that reliable, because the engagement was not super high. Not every single time would people put a reaction to the response. But again, that's also the reason why we have multiple feedback channels, just so that we can have each of the channels complement each other.

[00:26:43] Host: Excellent. Sorry, there are questions coming in here, so I'll just fire them at you quickly, because we only have two minutes left. Sandy has asked: can you share any prompts or logic you use to sort your raw data, or maybe what helps you analyze your information to get a report or takeaways?

[00:27:04] Xuan: I'm afraid that I don't have that information. Again, I just want to emphasize that we only indexed research reports, which are already analyzed information. We didn't give access to the raw data, for example the interview scripts, because we would rather our cross-functional teams access research insights that are analyzed, so that they don't take that one extreme example and use it to inform any product decisions. But I'm afraid I can't share the specific back-end things that we use, because I am not an expert; I am not an engineer from that perspective. Sorry about that.

[00:27:52] Host: I could keep asking you lots of questions, because I've tried this myself, trying to build a custom GPT, and unfortunately the quality of the searching wasn't quite where I was hoping it would be, with a custom GPT-4o. But maybe I'm doing something wrong, so afterwards I'd love to...

[00:28:10] Xuan: Oh, sorry, just one thing, I think I forgot to mention it. The reason why we are using Sharper [?], the tool where we built our research repository, is that there is some metadata about the research reports that we could use. We had some tagging on research reports, for example, as I mentioned earlier, the four sides of the marketplace, and also the author, and some other information. I think that potentially helped a little bit to improve the accuracy of the AI-summarized research insights.