Securing success of an open beta through mixed methods research

Jaclyn Touchstone at UXDX Community: Maximizing Outcomes & Securing Beta Success. Video: https://youtu.be/kiHLy2qq9hQ

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

What this talk covers, and what a beta is

[00:00:00] I wanted to talk about what startups or smaller businesses often run up against, which is running an open beta, or testing out your product, and how you as a researcher, a product designer, can work with other teams and contribute and help ensure that the success of that beta is met, and all of your stakeholders are happy and you can exit that beta very comfortably.

[00:00:26] I'm going to go through a little bit of an overview. I'm going to do an intro, we're going to define beta, and then work through four different phases of how you can break down your processes within the time period of the beta that will be set by you and your team, and then a little bit of a wrap up at the end.

[00:00:43] About me: I'm a senior product designer at Bitwarden. I've worked on B2C apps, I've worked on B2B apps and SaaS apps, and I also have some experience working with agencies, startups, as well as larger corporations. So I've kind of been all over the place in terms of business size and scale.

[00:01:03] Definition of beta. What is a beta? Typically when you're talking about a beta, the key goals for a company are to resolve bugs that are detected, but it can also be an opportunity for you to deliver continuous insights through user research. It gives you the opportunity to uncover actionable insights by talking to your beta testers that can have an impact on future product improvements, help you identify areas for improvement in usability, as well as aid sales and product marketing teams with insights to craft engaging stories for prospective users and customers.

[00:01:38] There's two main ways you can think about a beta: an open beta or a closed beta. For the purpose of this talk I'll be focusing on an open beta, which means it's a little bit more public and anyone can participate, depending on what product you're trying to test out. As well as your business, and maybe the privacy or security of it, your organization may choose to go with an open or a closed beta.

[00:02:07] Regardless, this information should be helpful for you if you are going to work at all with a beta program. I also want to mention, the type of research methods might change depending on whether you're in an open or closed beta, but again for the purposes of this talk I've kept it pretty general, so you can apply anything I discuss in an open or closed beta.

Prerequisites and the discovery phase

[00:02:32] To begin, there are some prerequisites that you're going to need, and hopefully you have, if you are involved in the beta discussions with your teammates. You're going to have to have a hypothesis about the product concept, and that includes a value proposition and a general understanding of who your target users are. You really can't do much without these baseline prerequisites.

[00:02:58] Once you have those things you can start in the discovery phase. This sort of research can begin and include any activities that help you and your stakeholders become better acquainted with the underlying problem that your product idea is trying to resolve. It's a really great time for you to get to know your primary stakeholders and their motivations, so that together you can work as a team and uncover insights that will inform the actual beta release. Once key stakeholders are aligned and the product idea has been validated, you can start planning for that launch of the beta together and set timelines and milestones for the course of the beta.

[00:03:36] Reaching out to your stakeholders and getting to know them is super important. I would say that I would not start any research without getting to know the key stakeholders involved and including them early on in the process. I like to think of user research as something that is done in service of others and toward a greater goal, which is the success of a product.

[00:04:00] Your key stakeholders need to be involved early in any research effort you're doing, and their input will often help formulate early timelines and focus areas. So it may make sense to form a core team of stakeholders involved in all the research that you're planning for the beta, and this might allow you to share some duties of the research among a multidisciplinary group. Having key stakeholders from different departments can help disseminate results further and increase the teams that benefit from those insights. I like to think a bit about it like the fellowship in Lord of the Rings: Frodo didn't make it to Mount Doom on his own, he had a fellowship of elves, hobbits and men.

[00:04:41] I also want to bring up an example. If research is driven by a UX designer who is mainly focused on the product's usability, you might miss insights that you would have uncovered had you included a product marketer or a salesperson, or someone that's thinking more about things that happen further down the line.

[00:05:05] I'm going to glaze over this slide a little bit, but I do want to just briefly touch on organization maturity, and how that can increase or decrease the complexity of how many stakeholders you have to have and would want to include in different parts of your process. If you're at a higher complexity organization with a lot of different stakeholders who have an interest in the success of the beta that you're running, not all of them need to be involved in every aspect of research planning that you're doing, but they may come in at certain points during the process. For example, it may not make sense to fully involve QA and support teams in product discovery, but once you hit the phase of usability testing further down in the process, they're definitely going to be more interested in those research results and they will help them achieve their goals as their own teams.

Validating the value proposition and competitive analysis

[00:05:57] To validate the product's value proposition: again, you've had that hypothesis about the value that your product can bring to the market and your users, and you want to start to validate that and make sure, okay, actually we want to spend engineering resources and build this product. That should be a key part of your discovery process. Your team should have a solid hypothesis, and then it's time to get that in front of users and get feedback.

[00:06:30] At this point you don't need prototypes, you can just do user interviews, focus groups. You really just want to see how people are thinking about the problem that you are going to solve, and where your product can fit in and get an edge. Competitive analysis is a really great tool to use. I like to start most product projects I work on with a bit of competitive analysis.

[00:06:55] It's one of the best tools in the user research toolkit, because it doesn't require a lot of time in recruiting participants, and it's really just something you can do as desk research that's pretty solid and has actionable results, in my experience basically all the time. You can focus primarily on a UX competitive analysis, where you're looking at task flows, features and usability. Or you could approach it with your stakeholders as a blended UX and marketing competitive analysis, where you're working more with the marketing stakeholders and you take a wider, holistic view of the customer experience and include things like how the competitor products are being positioned and what their marketing and sales strategies are.

Building and keeping a panel of beta participants

[00:07:47] Again, based on that hypothesis you formulated, you can start to get a general understanding of who your target user is and how you're going to recruit them for the purposes of the beta. With an open or a closed beta you're still going to need to find people and draw them in to want to use it.

[00:08:10] This step is mainly about building a large core group of beta participants who are going to get in your product. It's really good to do that early on, because you can use that sort of participant panel list to do discovery research as well as have them get in the product. That list is going to aid a couple of different departments, like marketing, as well as product, in doing things like general communications, and eventually down to sales once you actually launch the post-beta version of the product.

[00:08:49] Building a set of beta testers may end up being trial and error, and so you might want to use a paid service or a recruiting firm, and that's always an option. But another way you could go is to build your own list, and you could do that through finding out where your target user congregates, online or in person. Again, getting to know your target users, you can do a little bit of secondary desk research and add that into how you're building out that list and that base understanding of your users.

[00:09:26] I might go a little bit quick over these slides as I pay attention to time, but since we're talking about that core base of beta participants, I wanted to bring up how you need to keep in mind that you may need to keep them engaged. If you have a longer beta program that lasts like six months or something, you might see a drop off in engagement of your participants just because of the time allowed. Say you generate a lot of excitement with your beta testers at the beginning and they're excited to get in the product, that excitement is going to wane, most likely.

[00:09:59] So you need to keep in mind and work with your stakeholders here on how you can continue to have them be engaged with your product as you continue to get findings and resolve bugs and everything you need to do with the beta. Milestone compensation is a really great way to use a carrot to entice people to continue using the product if you're noticing a drop off.

[00:10:23] Based on the results from the initial research you can also start to define feature concepts and terminology, and bring in practices like prototyping, journey mapping and then surveys. Say you have a question about what you want to call a specific feature. The discovery phase is a great time to run a quick survey with the beta participants that you have contacted and built a panel for, to determine, when we launch the beta, what word is going to make the most sense for this abstract feature that we've got.

Timelines, exit criteria and navigating change

[00:10:59] Now that you're getting into the groundwork discovery research and you're able to validate the product idea with your stakeholders, you can start to begin thinking through timelines. With your key stakeholders, every team is going to have their own timeline. Marketing is going to have their specific timeline and their general communications timeline and their own priorities, product management is going to have their own, engineering is going to have its own, and then product design, as a product designer you would have your own. So you need to take that in mind and get to know their timelines and milestones as much as possible, because it should inform yours as a product designer and a researcher.

[00:11:41] At this point stakeholders may or may not have fully developed timelines and milestones. In an ideal world, as a product designer you're brought in on the process extremely early so you can formulate timelines together with your stakeholders, but that might not always happen. If it doesn't, just reach out to them, get to know what they have going on, what their priorities are. And I think Stephen mentioned this earlier: check in with them regularly, check in with them often, take a pulse on the situation. As you continue your beta research and uncover findings, with each finding you should be meeting with your stakeholders, assessing your timelines and adjusting milestones if you need to.

[00:12:26] Skipping that slide. One thing that's important to talk about is beta exit criteria. When you're running a beta there's going to be an end of that beta. It's something that's probably going to be determined by leadership and your stakeholders, and isn't necessarily going to be a product designer's decision, what the exit criteria is. But hopefully, again, you will ideally be brought in and are able to influence what you want to find out as a product designer, researcher, that would be included in the beta exit criteria.

[00:13:04] Ultimately beta exit criteria should be the definition of success. A successful beta is meeting all that exit criteria that is set before you launch the beta. And it's also, if you have a time, say that exit criteria is we want people to be using the product for three months, that would determine your timeline of all the research activities you need to do. So it would influence how many studies you could run, and really when you needed to get findings by to stakeholders.

[00:13:37] Another point about this that's important is navigating change. In the startup world, or in software in general, things change quickly. You need to adapt to changing circumstances, and often as you're uncovering continuous insights during a beta program things will change, things could change dramatically. So you need to be as adaptable as possible. It's important to continue to report on research results as you get them, regularly, in order to keep your stakeholders informed in case you may need to pivot your research plan or change things around.

Phase two: product definition, terminology and pricing

[00:14:13] When you complete the groundwork and you've formulated a research plan, you can start to move into phase two, or what you could term product definition. At this point the beta is still not in the hands of your beta testers, but you have some room to do a little bit more validation on your flows and feature concepts, and then you can start to run research about pricing, and determine how you're going to collect quantitative data and qualitative data via unmoderated methods.

[00:14:43] Once you have some initial definitions and ideas for feature concepts you can move into the validation stage. You can use things like surveys to get feedback on terminology, and that is a great way to get quick insights in an unmoderated way that allow you to move forward, particularly if parts of your product require terminology that isn't universal or needs to be specific or proprietary to your product. Validating decisions about terminology at this point is going to help stakeholders formulate talking points about the product that are accurate to the mental models of the target user base.

[00:15:19] For pricing: pricing is an interesting thing, and I would say doing pricing research early has the potential to have a great impact on how the product is positioned, as well as how sales and marketing teams are going to approach where the product sits within the marketplace, if there is one. If you're able to do this research early on, it can allow teams within your organization to make more accurate financial predictions, more confidently forecast projected ARR or MRR. So it can enable even teams that you wouldn't regularly work with, like finance, things like that.

[00:16:03] As a researcher your goal is to determine how much users in general are willing to pay with a feature set that you launch with. There are a couple of different methods you could choose depending on exactly what your product offers or what the early concepts for the plan structure are like. Two really great ones that you could use are a conjoint analysis survey, or a Van Westendorp, a pricing sensitivity survey. If you want to see what the optimal price point is for a product and you care less about the feature set, then a pricing sensitivity survey may be more beneficial for you. But if you want to see how much users are willing to pay for one set of features compared to a different set of features, you may want to use something like conjoint analysis.

Collecting quantitative and qualitative data

[00:16:49] For quantitative data collection, you're going to have some time before the launch of the beta where you can work with your stakeholders to determine what quantitative metrics you want to start collecting, what you want to focus on, to see how people are starting to use the beta product when you first launch it. Analytics data is fantastic at telling you what is happening and what users are or are not doing with a product, but it won't ever tell you why. So be careful to not rely too much on quantitative data, and follow up on any observations in that data with qualitative research before making any recommendations about changes made to the product.

[00:17:24] You might also, as a product designer, want to educate stakeholders on the pitfalls of relying on quantitative data to make product decisions, because sometimes teams can jump to conclusions about why they're seeing a pattern in the data, and those things aren't backed up by qualitative or more contextual information.

[00:17:41] Depending on the industry or technical limitations of your product, you may not be able to use a third-party analytics tool like Google Analytics or Pendo. But consult with your stakeholders and your team and determine what methods are available to you. There may be a way for you to collect data that you hadn't considered. You just need to be reaching out to your stakeholders and getting as informed as possible about what your limitations are.

[00:18:10] For qualitative, I'm going to skip over this slide, because it's really just bread and butter for user research: user interviews, online forums for beta testers, and feedback forms. You might also have access to support tickets if you're working with a support team, so mining qualitative data from those channels are really great ways to take a pulse on the product's usability. Also, if you want to target feedback about a specific aspect of a product or beta, this is where you could do an unmoderated survey or just a general feedback form and mine that source of data for qualitative themes that you could then explore further.

Phase three: launch and monitoring

[00:18:54] Phase three. Now that your initial discovery and definition are complete, say your designs are handoff-ready for engineering, you can move into the beta launch and monitoring phase. At this phase your participants are going to get access to the product, and you can start to dig in on that usage data and build on that initial value proposition that your beta is attempting to serve.

[00:19:18] Monitoring, like I just mentioned, monitoring that quantitative data collection if you have it, is a really great way to see what is going on. Based on what you and your stakeholders observe, you can determine focus for any check-in interviews that you may want to run with your beta participants. By this point you should have multiple data streams from which you can pick out themes about how users are interacting with the product.

[00:19:46] Whether you're doing user interviews or just reviewing support cases and online forums, themes will emerge that will show you and stakeholders different focus areas or gaps in the product offering. Say for example, if a product is wanting to support new parents, a gap that multiple beta users could have reported is that it takes a lot of time to get in touch with their doctor if they have a health concern about their new baby. This taking too long to contact the doctor is an emerging theme that you could then dig into in order to develop a feature to solve that problem for users.

Phase four: post beta and usability benchmarking

[00:20:20] For phase four, post beta. The beta will have progressed, your key stakeholders will be able to see when you were near meeting all of that beta exit criteria that was set in the initial phases. At this phase you can begin focusing on how to make improvements to the product after the beta ends. You're planning for the future, you're planning for the product to go into general availability launch. You're ready for the rubber to meet the road.

[00:20:45] If the data shows a certain feature or aspect of the beta product is underused, you and your key stakeholders can discuss the priority of those issues. If you're lucky enough that all the critical bugs have been resolved, then it's a really great time to do a usability benchmark with your beta participants. Depending on your product and circumstances you could consider either moderated or unmoderated usability benchmarking. It's going to depend on timelines and level of access that you have to beta participants.

[00:21:14] Once you've identified those priority areas, you and your stakeholders have been able to uncover themes that you can address as future product enhancements. If you find that a feature is underutilized and you've validated that what that feature offers is valuable, you can begin to work on ways to encourage adoption of that feature as a future product and design initiative. If you've uncovered ongoing themes with qualitative usability feedback that you've been collecting, you can use that to inform usability enhancements for future releases of the product. At this point your stakeholders are product, marketing, engineering and support.

[00:21:55] Again, going back into usability benchmarking: the best time to run a usability benchmark of a live product is when you know that the release is stable. At this point in the beta all your bugs have been solved, it's as stable as that product has been in real life up to that point. So run it as soon as you know that all the bugs, or most of the bugs, are resolved. Take every effort you can to run a usability benchmark at that point. It's a really great opportunity to set yourself up for future success.

[00:22:26] I'm a huge proponent of usability benchmarking because it sets the groundwork for you to be able to compare anything you do in the future to a pre-existing metric. So it shows that yes, you've been improving the product over time, every new design that you're launching has been building on the initial value of the product. And it really helps you as a design team show, yes, we are improving the product, we're having an impact on usability, we're moving the needle. I would say it's also important to do this as early as it makes sense to, because it will also build back into that generating value: design improvements, product improvements, generating value for your company.

Prioritizing features and the Kano model

[00:23:11] Since you now have a laundry list of features in your backlog, you can take some effort to assess the priority level of those features. You might have a grab bag of features that you've uncovered from themes that you may want to introduce to the post-beta product, but you may not know the value level of each in relation to each other. So it might be a little bit difficult for your team to assess which thing we're going to work on.

[00:23:42] At this stage you can find out how proposed features fit within defined value categories, on a scale from least valuable to users to most valuable to users. You're going to want to understand which features are critical for users, but you may also want to understand the engineering effort required to build each of those features. Things you could do in order to determine how valuable a feature is in relation to others for users is to use something like the Kano model, which I'm going to go into detail on in a minute. But internally among your stakeholders you could run workshops like an impact effort matrix, feasibility, desirability and viability studies, and then something called the MoSCoW analysis.

[00:24:28] I absolutely love the Kano model. It has been extremely helpful for me and my teams. You can do this unmoderated or moderated. You could build it into, say you have a user interview that you want to run and you want to assess value, you could just run a Kano model questionnaire at the end of that. Or you could do it unmoderated in the form of a survey. There are a lot of great tools out there, Qualtrics is obviously the best. So I would highly recommend, if you're ever in a situation where you need to prioritize features among users, you should definitely do a Kano model study and add it to your research toolkit for determining which features are most important.

Key takeaways

[00:25:12] I think I have made it another 30 minutes. I just wanted to wrap up with some key takeaways. Your stakeholders: relationships with your stakeholders, and working in service of your stakeholders, is one of the best things you can do for yourself, and for the success of the beta, and for the success of you individually. Your stakeholders are your key collaborators and they're super important to the success of anything you're ever working on as a researcher.

[00:25:46] Use stakeholder timelines and constraints to get creative with planning. Make a plan to keep participants engaged, whether that's milestone compensation, or maybe you want to send them merch if your product is super hip like TikTok or something. Report your insights frequently. You should be checking in with stakeholders, but depending on your specific circumstances you may want to be disseminating that research result information more widely, so just keep that in mind. And keep in mind that it benefits everyone for you to report on insights as quickly as you get them in.

[00:26:21] Usability benchmarks are going to serve you well in the long run, don't miss the opportunity to run it during the beta. And then balance qualitative and quantitative data when you can. And ultimately, have fun. Work with your stakeholders, build those relationships. I've had a lot of fun doing research during beta programs, so I think it's always important to think about it as a fun activity. That's about it for my presentation today, thanks everybody.

[00:26:49] Host: Thank you very much, that was great, I really loved it. It's kind of like a full product lifecycle, you're starting from zero the whole way through to successful products. So thanks for that run-through. Unfortunately we are over time, but again, if you have any questions, please do post them in whichever platform you're watching these on and we'll invite Jaclyn to come into the Slack channel and answer those questions for you. But I just want to say thank you once again to Jaclyn for sharing her knowledge. Thanks so much everybody.