Bridging Discovery and Delivery: Automating User Feedback within the Release Lifecycle

Irad Cohen at UXDX Community: Validated Delivery: Platform Research and Automated User Feedback. Video: https://youtu.be/by6sdrj2g5s

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Intro and a testimonial

[00:00:08] Irad: Thanks, Rory. Hi everyone. Maybe I'll start with a quick intro. I'm Irad, Irad Cohen. I'm a tech lead at monday.com CRM. I used to be a co-founder and the CTO of a small startup called Renobiz[?], where I did lots of product and customer discovery. But what I'm going to talk about today and what I'm going to discuss is basically a system that we developed in Monday of how to get user feedback, which, I think everyone here knows, is a super important thing. What we did to get it, and more importantly, how we connected the engineers working on different projects with the users and with user feedback, and basically empowering them to do more than just technical work.

[00:01:06] Irad: I'll start with, nothing beats a great testimonial. So I'll start with a video of a product manager whose name is Eran, with whom I worked in a previous project where we initiated the system. You'll hear him speaking about how we took this system to his next and different projects, and I'll give the stage briefly to him.

[00:01:36] Eran: Because we saw the feature analytics and we realized that something doesn't work well. We're looking at the funnel and it doesn't convert as we expect. And we were trying to understand why it happens. So getting this feedback directly from the users was super valuable for us, but sadly, we weren't able to reach out to the relevant customers and get this feedback. And the results were amazing, because out of 700 users that were part of this early beta phase, we got more than 40 email responses that brought lots of insights to us, to understand what is currently lacking, what works well, and how we can improve the feature much faster than before.

The validation gap

[00:02:30] Irad: So I think there's nothing new here, but it's always happy to see someone enthusiastic about how he is getting user feedback, and lots of it. I think we all know this, right? The feature or a new project is shipped, the dashboards are green, but are our users really happy? And that's a difficult question to answer, especially when you're just deploying stuff.

[00:03:02] Irad: Writing code, I would say it's easy. Some people may argue it's different types of easy, and maybe nowadays it's even a lot easier, like AI can generate tons of code, and I think building the technical aspects of the product became more and more easy. And I think that's the difference. So even if writing code is easy, building products is still a really, really hard thing to do.

[00:03:35] Irad: And I think we've all seen this. There's a PM, a really talented PM, building a brilliant spec for an amazing product that's going to change the world and move all the needles that the business needs to move. And engineers, as we said, can do their work and ship these features. But then what happens? And usually there is silence.

[00:04:01] Irad: And I call this silence the validation gap. It's basically the time that passes between, it's the dead zone between when you're deploying something and knowing if it's actually working, because at the beginning there's a few initial users. Maybe you select a couple of your users, maybe you gradually open it to some percentage of them, but it's really hard to make sense of the dashboards when you don't have enough data. And it's really difficult for us as builders.

[00:04:35] Irad: Just FYI, at Monday when we say builders, we mean everyone who makes products happen. So it's UX researchers, it's product managers, and designers, and it's engineers. All of them are considered builders. And it's really difficult for us builders to answer these questions: are we working on the right things, or are we working on the wrong things? And I think these two questions are questions that we should be able to answer as fast as possible.

[00:05:11] Irad: Because in our world, building bad software is inevitable, and working on the wrong things is inevitable. And the question is how long does it take us to understand it? Because I think if it takes you a whole quarter to understand that you're building the wrong thing, then I guess you're going to be unhappy, and your leaders are going to be unhappy. If it takes you a month to understand that you're building the wrong thing, I think that's feasible or understandable in a way. But if it takes you a week to understand you're building the wrong things, that's amazing, and I think that's somewhere we should all strive for.

[00:05:55] Irad: Metrics, they tell us what is happening, but they don't tell us why. And only users can tell us why. We have observability tools. We use Datadog, and there's tons of different tools out there that tell you what is the latency, how long does it take for you to answer questions, or what are the error rates of your product and what is the throughput. But it's not enough. And then on top of it come different analytics tools, if it's Mixpanel or Amplitude, and they allow us to see what's going on, see different funnels and conversion rates and how are users behaving with our product. But it still doesn't tell us why.

[00:06:42] Irad: And if we want to debug these funnel drops or user frustrations, it makes no sense to look at a trace. We really have to hear their story. And it doesn't have to be, but I think the best feedback that we can get is face-to-face.

Three moves: recruit, automate, connect

[00:07:06] Irad: So what I'm going to tell you about is the system we've built in order to really understand this why as fast as possible. And some would call this a nice bonus, some would say this is the most important aspect of the system, but we also got the whole team on board. And we're also going to discuss why it matters so much.

[00:07:32] Irad: It boils down to three different moves: recruit, automate and connect. I'm going to dive deeply into each one of these steps, but if you want to take a screenshot, I think this is the most important slide to take a screenshot for, because these are the three, quote unquote, moves you can steal right away.

[00:08:02] Irad: You have to recruit the right users, the ones who will be the best candidates for, first of all, trying your new product, and the most eager ones to actually give you feedback. You have to automate some kind of a reach-out for them, and there's quite a bit to it. And then you have to connect with them.

[00:08:27] Irad: I think the majority of the crowd here are UX researchers and, I would say, more product people, and connection for them is the default. It's what you guys do every day. But for most engineers, this is something that they don't even reckon that they're able to do, or they should do, and they don't see the benefits of it. And this is something that really makes good teams better, or it makes teams work.

[00:09:00] Irad: And I think another important aspect of this, and you're going to see it soon, is that you don't need a fancy stack. You can start off with a simple Google form and some manual emails, sending them out, and it can get you quite far.

The case study: quotes and invoices

[00:09:17] Irad: I'm going to present a specific case study of where we initiated the system. monday.com has a feature called quotes and invoices, and I don't know how many of the people listening here ever created or generated a quote or invoice in their life, but it's a cumbersome, tiring, mundane process.

[00:09:46] Irad: And at monday.com we started out with this feature, I think, pretty straightforward, like having a form, there's some fields to fill out, you scroll and you scroll and you scroll and you scroll, and eventually you finish filling it out and you have a quote generated. But what we saw is that this thing doesn't reach its full potential. We knew from onboarding sign-up questionnaires that about 17% of our user base needed this feature, but only 8% of them used it.

[00:10:28] Irad: And then we figured out, okay, we need to create something different. And of course our researchers and our product managers spoke with these users and understood their needs and wants, and what holds them from using it, what's wrong with the product. And we came up with a really nice WYSIWYG, or what you see is what you get, experience. It's a complete change of this workflow.

[00:10:57] Irad: And our non-engineering builders, they really had high hopes and high dreams for this change, and they baked everything they could dream of into it. And you want to move fast, and when you want to move fast you have to cut an MVP. Cutting an MVP is not an easy task. It's a hard thing to do, but it's manageable. The more important question is maybe what to build next. After this MVP, what should you do? And we were really thirsty to get users on board, and we were really thirsty to get them to help us focus, because we didn't really know how to do it.

Recruit: a banner where the pain is

[00:11:45] Irad: So the first move is understanding who these users are and how should we recruit them. And to be honest, we don't know. We don't know exactly, okay, so I want to take this guy and this guy and that guy and take them into the alpha. We have to segment them. And in our case, we wanted to segment them by the users that have the most pain with the existing product.

[00:12:15] Irad: So it made all the sense to put a banner where they have the most pain. They go into this, they have trouble, we're asking them, hey, do you want to be the first to experience the updated quotes and invoices feature? And we tell them, hey, you can join the waiting list. And I'm going to put a marker around this, so remember the waiting list wording, and remember the "want to be the first". It's going to come back in a future slide.

[00:12:49] Irad: So basically I think that when the pain is high, the intent is higher, and we try to create a psychological effect or a framing where they feel a part of something. They feel like they're exclusive. We wanted them to feel like they belong, not like we're begging for their feedback.

[00:13:15] Irad: But plans and reality aside, doing our initial plan that I just showed, of a surgical in-feature manner, would take us two days to build. And then, like every engineering or building team ever, we went to the easier solution, and we figured out that we already have a Monday-wide framework to show, I would say, product-wide banners. And we said, okay, intent is nice, pain is nice, but let's just spend two hours and ship this banner to start to get users into the pipeline.

[00:14:06] Irad: And this actually turned out to be a good idea, because what we missed originally was the fact that if we show this banner only in the product, we're going to miss all the churned users, all the users who used the product and churned away, or tried it and it didn't fit their needs. And putting this banner outside the specific feature helped us reach out to those users as well.

[00:14:34] Irad: We started with releasing this banner to just 3% of the accounts, and the conversion rates were incredible. On the first day we got more than 200 opt-ins. We actually had to kill the Slack notification. We had a Slack notification for every new opt-in and we had to kill it after, I think, after one hour. So this was really fun to see.

Automate: the loop and the heartbeat channel

[00:15:02] Irad: So I'll move on, and maybe the next part is the automation. We created a loop. You have to create a loop around it, and put all of the results of this loop into one channel, wherever you're working, if it's Slack or something else. And this channel acts as the product's heartbeat.

[00:15:28] Irad: And what this means. The loop was pretty simple. Users would click the banner, we would save their details, anything we need, and we would choose specifically which of those accounts and which of these users fit our needs. For these specific users, we opened the feature flag to see the new experience. And once we opened the feature flag, we sent them a personal email telling them, hey Rebecca, thanks for opting in, or some better wording for it. Thanks for opting in to try out the new quotes and invoices feature. We just opened it up for you. Feel free to use it and send us any feedback that you have, either by replying to this email directly or by using the in-product feedback form that we have.

[00:16:20] Irad: These emails were personal. They were not like Monday team selling this, but like, hi Rebecca, I'm Irad, I'm a tech lead at monday.com CRM team working on quotes and invoices. We started making it relatable. We created a real connection there.

[00:16:38] Irad: And we had an automation that after three days we would send a follow-up email. And this follow-up email would say something along the lines of, hey, I hope you enjoyed the feature, got the chance to check it out, blah blah blah. Would you like to meet us for a chat and give us your feedback? We want to better understand your needs.

[00:17:04] Irad: And what we saw, and this is what Eran from the video at the beginning of the talk was so enthusiastic about, the conversion rate for users actually wanting to speak with us, without a hundred dollar Amazon gift card or all of these things, they were actually eager to speak with us because they felt a part of something. So this was really, really nice.

[00:17:33] Irad: Every reply to this email, and every in-product feedback that we got, we sent into a dedicated Slack channel. And the whole team was a member of this channel. It was the devs, it was the PM, it was the designer, it was the team lead, it was the UX researcher. Everyone saw it. And I think that's a mechanism that really made everyone wholesome, and helped the whole team look at the target and really understand the users.

[00:18:05] Irad: So I think it's super important that not only the PMs and UX researchers understand users, but we wanted to get engineers into that mindset as well. And this channel really helped it. And we got everything there. We got bug reports, we got feature requests, we even got praises. And to be honest, for someone building a CRM, having a user compare us with Salesforce and telling us our experience is a field of dreams, this is beyond any expectation that I ever had.

Connect: engineers in the room with users

[00:18:53] Irad: And the last part is the connection. And it's not only the, I would say, passive intake of feedback in this Slack channel, which also was a place where we could actually discuss this feedback, but we also wanted to put the builders, or specifically the engineers, which are not used to doing it, and our users in the same room.

[00:19:23] Irad: And because we got so many users wanting to actually speak with us, this is also, to be honest, as a user of other products who participated in user interviews, having a technical person in the call really makes you feel important in a way. And I think this is the feeling we wanted to give our users. We wanted to give them the feeling that we want their voice to be heard. We want to build something with them. They're not only using our products, but they are helping to shape it.

[00:20:05] Irad: And as a user, this is I think one of the greatest experiences ever. Seeing something you ask for, and then first of all getting acknowledged for it, and then two days, one week, one month afterwards, seeing it happening in the product and getting a personal email from the dev you talked with last month saying, hey Rebecca, this thing that you wanted, it's live now, let us know what you think. That's priceless. Both for the dev, who now has a sense of meaning and understanding that they're not building things in a void, but there's actually people using their products, and for the users.

[00:20:56] Irad: And what this creates is what I think is really the difference between a great coder and a great builder. It's empathy. And I think this empathy is one of the most important overlooked engineering skills that there are. It creates opinionated engineers, and it creates proactive engineers, and it creates scenarios where a PM comes up with a spec and an engineer says, hey, I understand where you're coming from, I understand what you're trying to achieve, I think that if we change it a little bit we can make it happen in a shorter time. Or even give ideas for features and products that could be quick wins. And that's something that I personally, and I guess everyone here, we want engineers to be empowered that way.

Results, and what to ask when planning

[00:21:55] Irad: So how did it come out? We got dozens of quality insights in just under a month. And these insights actually helped us pivot our roadmap. So after the MVP, we understood what should we do now. And I think that's the thing. It was not just feel-good feedback. It was feedback that actually helped us change what we're building. We pivoted off planned features. We doubled down on ideas that we deprioritized before, because we now understood the importance.

[00:22:37] Irad: And I think that's one of the main things to ask when planning your next feature. It's not just asking, what's the MVP, but there's two important questions to also understand, that we have the answer for. It's: if this MVP fails, then how do we know why? Do we have the tools in place to understand? And if this MVP succeeds, how do we know what we should build next?

[00:23:10] Irad: As I said, you don't need any fancy stack. You can start off with a simple Google form, with a link to it in your product, sending manual emails to the people who filled it out, and just one Slack channel that acts as the heartbeat. You can start doing this tomorrow, and really this infrastructure is irrelevant. What matters is the commitment for the whole team to listen. And this is really low effort, high impact. So I urge you all to just go do it. And you should just get your whole team closer to your users. Both will thank you.

Q&A

[00:24:05] Rory: Excellent. Thank you very much, Irad. That was a great talk. It really sings to what we're trying to achieve at UXDX, which is break down those silos and get people working closer together. And if anybody has any questions, please do reach out. But in the meantime, just to start off: how did it come about? You said at the start that researchers went and did some work before the MVP, but how did it happen at Monday that you got them involved? Sorry, that instead of it just staying with the researchers and then handing over to engineers, that they got more involved. What was the impetus for that?

[00:24:47] Irad: Cool. So I'll give you maybe an understanding of where I'm operating at. The CRM group in Monday currently has, I would say, about 100 people. About 50 of them are builders. And there's one UX researcher for this whole group. And usually the responsibilities for this one UX researcher are more strategic. What happens is that mostly the designers and the product managers for the specific teams are the ones conducting research, speaking with users, interviewing users, et cetera. And they gain tons of material, which they crunch up, they kind of summarize everything or send it to their cloud, whatever. And if they present, I would say, a chewed-up presentation in a meeting, or even send a summary in Slack, it's a bit detached.

[00:25:58] Irad: And this detachment is hurting me, as someone who wants to work in better organizations. What I try to do is understand how we break this detachment. Because recently, people read nothing. Whatever you send on Slack, no one will read it. And I think the only way to break this detachment and these silos is to get the whole team understanding users. And it's really easy for people building dev tools because they're their own users, but I'm building a CRM, and I'm not a CRM user. And there's a big gap here to fill. And the only way I can fill it is conversing with our users, understanding who they are generally, things that are usually only for UX researchers, like what's your daily routine? What do you do when you start working in the morning? It's the kind of questions that, if you can't tap into them, it's really difficult to build a good product, in my opinion.

[00:27:11] Irad: I can't hear you, Rory. I think you're muted.

[00:27:14] Rory: Apologies. Was there openness on both sides for that, or was that just you being the driving force?

[00:27:18] Irad: So for me, my experience was, I'm an engineer that loves product, so it's easy. And in my experience, I was a CTO and I had to do it, I had no other option. And kind of like my personal agenda, which is arguable, my personal agenda is if you get engineers with users, you'll build better products. And I'm trying to prove that this agenda is correct, even though deep down inside I know it. And I see it. Unfortunately I can't really measure it, but I feel that it's a bit of a quality issue, and I see the engineers around me growing, and I see engineers who were used to getting a spec and just blindly implementing it starting to ask questions. And that's a huge success, in my opinion.

[00:28:21] Rory: Brilliant. So I know you're trying to measure it, but excellent. Thank you very much, Irad. That actually brings us to time, so thank you very much for sharing, and I hope you enjoyed it out there as much as I did.

[00:28:31] Irad: Definitely. Thanks for listening.