Are your words working? Creating and sustaining a content-focused research practice

Erica Jorgensen at UXDX USA. Video: https://youtu.be/OyuBuYKDH74

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Why content-focused research

[00:00:00] Hello and welcome to "Are your words working? Creating and sustaining a content research practice." I'm Erica Jorgensen and I'm a Senior Content Designer and Manager at Microsoft. I've been there about four years, the last year or so on the UX team. I have also worked at startups like Amazon and Rover, and I've taught at the University of Washington's Digital Communications program, specifically on web analytics.

[00:00:20] Here's what we're going to cover today: why and how to do content-focused research. How can I help you identify the exact words that are clearest and resonate the most with your audience? You'll learn how to create business impact with content, maybe even drum up some additional headcount for your team because you're driving so much impact with content. And how UX writers and content designers can be inspired by content research, and also how it will build respect, more respect for your content writers because it is so strong.

[00:00:50] So what am I talking about when I talk about content-focused research? Basically it's a way to evaluate your content and make sure that it's helping your audience make decisions and take action, and if it's not helping them do that, why not? It's the why not, I think, that is really where the golden nuggets can be found. It helps you find out what's clear and what's not, which exact phrases to use. And sometimes these words and phrases will come out of the very mouths of your customers, and that's awesome. They can help you do your work.

[00:01:24] What I find super powerful about content research is that it helps you optimize content before you launch it. Now, it's great to optimize your content after it's published, but why not know when you're publishing what's going to work? What would it feel like to feel super confident that you're using the right words? Maybe I need a Doug Henning magic slide here, but this is exactly the power of content research.

[00:01:51] It's fairly quick. You can often get results within an hour, depending on which audience you're using for your participants. You can do research with current, potential or past customers. And one thing that I think is super important is it helps you check your bias. We're all biased, and when we think, oh, this is just right, I know it's going to work, how do you know that? You don't know that. Content research will help you check yourself and make sure that you're being as customer centric as possible.

[00:02:20] And there's gold here. There are just so many golden nuggets that come out of content research. You'll get insight after insight, you'll learn things that you didn't even think about, and I'll provide an example of that in just a minute. You'll learn how to speak your customer's language and that builds trust and loyalty, and I think that's invaluable. And you'll drive business impact, because you're using the words that are clear to your audience. They're more likely to do what they need to do with your website, with your app.

What to test

[00:02:44] Content research can be done on all sorts of types of content, but mostly I think messaging frameworks, that's a great thing to test. Calls to action, product and feature names. Those don't get tested nearly enough, and definitely at Microsoft, I'll speak for myself, I think there are some feature names that could be improved, could have been user tested before they went out the door. And any of the most important words, the words you use all the time, those go-to words that are in the style guide. That's a good word? I would say, hold it, hold it right there. You don't know. The words in your style guide might not have been validated. That's what I'm trying to say. And so you can use content research to make sure that those words in your style guide are spot on.

[00:03:37] This is a little hierarchy of what to test. Like I said, the frequently used words, those things that you are assuming are clear, test them. The words on your home page, on the landing page of your app, test those, test those, those are important. You shouldn't have any jargon on your site, but this is here because we all have jargon no matter what industry we work in. Jargon seems to be inescapable, as much as we try to eradicate it. Test all these things.

[00:04:03] This is Microsoft's writing style guide, which is available online if you want to check it out. This A-to-Z word list over here, I asked myself, how do these words get there? A lot of my smart coworkers in marketing and branding develop some of the words in the word list that we use, but I can tell you that some of them haven't been validated. We try our best, our brand voice is to be warm and clear and ready to lend a helping hand. I think the style guide is awesome, but we can go further. We can become more customer centric by validating the words that are in our style guide. And if you don't have a style guide, you can create one with all validated content, because you can use user testing to make sure that the words in your style guide are perfect.

What content research is not

[00:04:47] Quickly, what is content research not? It's not a couple of things and I want to clarify these. It's not regular UX research. You might have done prototype research with design and visual design and content design together. That research is great, I'm not putting it down, but if you can decouple the words, if you can separate the words from the design and focus on the words, this is what I'm talking about. Sometimes you'll need to include a screen grab, or maybe even a prototype, in content research. But the more you can just use the words to ask people, is this word clear to you, what do you think of this, tell me more about this, that's where you'll really get a lot of value.

[00:05:24] So it's not the same as A/B experimentation. A/B experimentation is good up to a point. I think it's time consuming, it's resource intensive. And I think sometimes UX teams, I'll get on a soapbox here, they get a little bit bent out of shape about statistical significance. It's great if you run an A/B experiment and you reach statistical significance, but do we really have to? Do we really have to be that precise? We're not creating drugs, we're not creating a new COVID vaccine. I think people tend to get overly worked up about A/B experimentation and it slows down organizations.

[00:06:05] I think you can move faster, be more nimble with content testing instead. If you want to do content testing prior to A/B experiments, that's a great way to go. I say, if you have a version A as your control and version B is your test version, make sure that version B is something that from content research is resonating with your audience. If you're doing multivariate experimentation, make sure those variations that you're testing have been validated by content research.

What to know before you start

[00:06:33] So, what to know before you start. You can use any number of online tools to accomplish online content testing: UserTesting, UserZoom, even SurveyMonkey. You don't need them, though. They're great, they're fairly affordable, you can also try the trials if you're interested, but you can get super scrappy.

[00:06:50] When I worked at Rover it was a startup and we were strapped for cash, we were bootstrapped. And when I wanted to know what words were going to work, we needed to know what worked with our audience because the competition was so stiff. Rover has since bought their competition, which is great. But I didn't have a staff, I was a content team of one, and basically ran around downtown Seattle with a clipboard and a bunch of Starbucks gift cards for five bucks each and just ran up to people and said, do you have a dog? If you do, can I talk to you about some of our website content? And people would say, oh, free latte, sure, I'll talk to you. So you don't need a fancy tool, is what I'm saying, to do content research. You can do it on a bootstrap.

[00:07:42] I like to plan. This is a sample test plan. I like to loop in my stakeholders and just kind of get out of my brain what am I trying to learn, what do I want to accomplish with my user tests that focus on content. And I loop in my product manager, my visual designer, even my software team. I want them to know what I'm doing, so I share my test plan with them. Sometimes I collaborate with user research on these, if I think they need to be moderated. But most of the time, if I'm doing an unmoderated content test, I set it up myself in UserTesting and just let my team know what's up.

[00:08:18] One thing to be careful about when you're doing content research, as you should with all usability research, is to be sensitive. If there's content that's already live, someone went through, your content designers and UX writers did their best creating the content that's live. Don't be critical of that content. Look at user research as a way to help the team grow, to get more customer centric. It's not being critical. You shouldn't be seeing it as a critical thing. It's a way to grow and learn, so approach it that way.

[00:08:49] And on the same page, don't think of words as winners or losers. If you run a content test and one word is much preferred over another, don't call that the winner word and the word that was less resonant the loser word. I think that's a dangerous way to think about it, because it kind of puts down content and we don't need that. I'd say look at it from a growth mindset perspective and you'll be all the better for it.

[00:09:14] And keep your tests short. Usually I aim for about 10 or 15 minutes maximum for the time it should take a participant to finish a test. Longer than that and they'll lose interest, and with the questions that come at the end of your test, from the answers you can tell that their energy will be sapped by the end. It's just best to keep them short and sweet.

An example: seat versus license

[00:09:43] So let's get to an example. This is a marketing page for Microsoft 365, formerly known as Office. You might hear me use those terms interchangeably. It's been over a year since we rebranded it, but most people are still more familiar with Office than Microsoft 365. Anyway, this is a landing page that people come to when they're buying, and when they buy they need to buy licenses for their users. Usually people are signing up their whole organization at the same time, so they need to get one license per user.

[00:10:11] And when I started on the team, I thought, licensed, that's kind of a heavy word. It reminded me of the DMV. It reminded me of waiting in line on a Saturday morning to get my license renewed. That's just the connotation I had in my brain about it. And I thought, what if we tried to swap the word seat in there? Why not seat instead of license? And I think license is pretty industry standard, I figured a lot of people were familiar with it, but could we lighten it a little bit? It's all over the place on our site, it's all over the place when you're managing Office, a license here, your license there. Could we use a different word that's not only shorter but maybe a little bit more clear?

[00:10:51] So I set up a test in usertesting.com to see if seat was clear or if license was clear. My goal was to just find out which one works better for our audience, and that's the important part. This is Microsoft shoppers. They are different than your customers. But anyway, you can target your test to the participants you need. And I focused on 20 small business owners. Small business owner means 10 or fewer employees in Microsoft land, so keeping in mind that this is our target market. We have other customers, of course, but for the sake of this particular UX work I was focusing on the very small business, or VSB, audience.

[00:11:36] So you don't need 20. You can test with as few as five people, and Nielsen Norman Group have got a great article on this. A lot of people are like, five, that's too few. It's not, it's not. I like 20 just because it gives me a whole lot, when I ask people open-ended questions I get a whole lot of answers in there that I find super enlightening, but five might be the sweet spot for you. It's also easier to analyze only five user testing participants' feedback.

[00:12:09] So, testing seat versus license. I set up, the first question was multiple choice, a quantitative question: which word would you use to describe the thing that gives you permission to use software? Now, if my eighth grade English teacher read that sentence she would say, your syntax is off, this is a clunky sentence. It is clunky deliberately. I don't want to use the words I'm trying to evaluate in the test question. I am bending over backwards to avoid saying it in that sentence, to avoid biasing my participants' responses. So this is a clunky question, but it's useful.

[00:12:43] And whenever I do a multiple-choice question, I want to leave room for none of the above, for something else. I don't want to bias my participants into feeling forced into answering one way or another. I always want to give them the out of none of the above, no, that doesn't work for me, none of them. And then ask them, what word would you use if none of these are working for you?

[00:13:06] The next question, the follow-up question to the multiple-choice question, is an open-ended or qualitative question. Now, qualitative, a lot of people get confused by that. It has nothing to do with quality, just don't think of it like that. Your qualitative questions ask the why, why people are thinking the way they are. And I like to ask, tell me a bit more, tell me a bit more about why you answered the way you did. You can also think of these open-ended questions as the Olivia Newton-John questions: tell me more, tell me more.

[00:13:41] I like to combine this quantitative and qualitative approach. I think it's super powerful to ask a what question with a why question. It's really a great way to go.

What the open-ended answers revealed

[00:13:54] So this is what we learned from testing seat and license. License was preferable to 17 in 20 people. A couple of coworkers told me, I told you so. I said, good for you. I had no idea, I really didn't think that license was going to be this preferable. It could be because it's industry standard, or people weren't as bothered by license as I was. And that's great to know, this is super powerful information. This is a bit of an aha moment for me. Checked my bias, certainly.

[00:14:29] And it was the open-ended question, that tell me more, tell me why, this is where the nuggets of gold come from. So we ask people, tell me why you responded the way you did, and people provide additional information that pulls the curtain back. It showed me basically that people are comfortable to a certain point with the word license, but they could not explain it in their own words very well. They stumbled around it, and they provided some feedback that made it super clear that they don't know, not all of them but enough of them, didn't know how many licenses you need.

[00:14:58] And this was super important to our customer experience. You don't need to get a license for each device. If you have a computer and a phone and a tablet, you need only one license. You need one license per user, basically. People were stumbling. People said, oh, you need one license for your whole organization. No, that would be a subscription. So you need a license for each user, but only one license. So this was an epiphany: people don't know how many licenses are needed.

[00:15:31] And we were assuming the UX was clear. When people are buying a hundred licenses for their organization, that's expensive. We don't want them to buy 200 when they should only have a hundred, that's ridiculous. Then that will also make them think that Microsoft products are more expensive than they really are. We don't want people to over-purchase. We want them to be happy, we don't want to undermine their confidence, and we don't want them to get super confused when they go to set up the product and get their company going with it.

[00:15:57] We don't want any confusion. We don't want them to have to return to the website to buy more. We don't want them to get their invoice and go, wait, wait. We don't want them to over-purchase. Worst of all, what I'm trying to say is many customers didn't realize they were over-purchasing, which is awful, awful, awful.

[00:16:18] So, long story short, the UX had to be clarified to make sure that people were buying one license per user. You need one license for each user. Guess what? The designer pushed back on this. She didn't want to add it, and we showed her, we brought her along for the ride. We showed her the results from the user testing study that showed people did not know what to do, that they didn't know how many licenses to buy. This copy is necessary. This content has to go here to prevent a customer experience snafu.

[00:16:56] And it's going to reduce calls and emails and texts to the help desk. It's super, super important. We're looking at quantifying exactly what the business impact is because that just recently launched, but we know it's going to help our customers. We know it's going to help customers be more loyal and more likely to renew, because they're not over-purchasing, they're not having trouble when they're doing the setup, when they initially buy the product. We know this is a huge, really important collection of six words that's making a world of difference.

[00:17:23] And we published the study. We have a SharePoint site where we publish all of our user research, and there's a special category for content research now, because our team is doing so much of it we can filter on content research. This is a report I published in collaboration with my user researcher, Veronica, who is amazing. I think it's important to collaborate really closely with user research and not step on their toes when doing content research.

[00:17:51] We did test another batch of words in the same study, and this report has been read by over 250 people across Microsoft, not just from my organization but people around the company, which is a huge company, a hundred thousand people or so. Everyone has access to the study and I'm thrilled. So now marketing and branding and other departments outside of UX are referencing this terminology study, and people keep bringing it up in meetings. Well, let's look at that terminology study. Remember we looked at those words, these are the right words to use, we know because we tested them.

Setting up a study

[00:18:26] So how do you go about setting something up in UserTesting? You can use UserZoom, don't get me wrong, but UserTesting is what we use at Microsoft, and so I'm just going to walk you through a sample setup in UserTesting.

[00:18:42] Choose your number of participants. You can do five, you can do 20, you can do a thousand. I think you can do a thousand, I think it actually goes up to 999 if I'm not mistaken. But for each test you run you delineate how many participants you need and what device type you want them to use, whether smartphone, tablet or computer. You can test all over the world. There are people participating in user testing all over the world, which is great, and you can even specify which state if you need to target your research accordingly. For that reason you can use user testing to validate the localization of your content, to make sure that the language is clear and that culturally it's landing well with your audiences that are in different countries.

[00:19:24] This is a sample of a screener question setup, so you can see I want to make sure that I'm getting feedback from people who work at small companies of 10 or fewer people. Those people could get welcomed into the study, and then people who work at bigger companies do not get to take the study. And people are compensated by UserTesting for their participation, by the way. And there are a whole bunch of other filters, by industry, by job role, by seniority, all sorts of things, so you can get really, really granular really quickly.

[00:19:57] The more granular your test is, the longer it might take to run, but generally I get results within three days if I've got a basic type of audience. If you're looking for something super specific, a really specific industry, like security professionals with 10 or more years' experience, it's going to take longer to run. But I usually allot a week for my user testing studies, just to give myself a little bit of a cushion.

[00:20:22] And create your test questions. This is really drag and drop, it's super fast. You can do a rating scale in addition to multiple choice, for a different quantitative question type. And that's sort of like, say, on a scale of one to nine, where one is not at all well and nine is extremely well, how do you understand the word blah? These are different scale question types that you can use, and you can also customize the end points. It can be one to five, one to seven, one to nine, so you know which end of the scale people are falling on.

[00:20:54] And this is a sample output from UserTesting. I call these donut graphs. I don't know if that's accurate but I think of them as donuts. Maybe I'm hungry, maybe I really like donuts, but I call them donut graphs. And this is not from the seat versus license study, this is one that a coworker ran because she wanted to know what to name a dashboard that was going to be super important to our business. We didn't want to have to go back and rename it, we wanted to make sure that we landed the name of the dashboard super well. And it showed that simplified was the word to use, by both people who had a lot of experience and people who did not have a lot of experience. Both audiences preferred that word, which was really interesting feedback.

[00:21:37] If you ask a scale question, this is the sort of bar graph that you'd get as an output. I love putting these graphics in PowerPoint presentations for my leadership team so they can understand, this is what we know, this is how we know it. But I don't want to leave out the qualitative responses, because they add so much color, so much information.

[00:21:57] This is a sample output from a qualitative question. There's a lot of content here. You can export it to Excel and then go through and highlight where the nuggets of gold are. This is where I was uncovering, teasing out the fact that people were comfortable with the word license but they couldn't explain it in their own words. They sure couldn't understand how many they need for the organization, and that was a big light bulb moment for our organization.

Building a content research program

[00:22:24] So if you want to build a content research program, go for it. I say do it now. You can get your whole team comfortable doing it. I started with a group training program and that didn't land very well. I think maybe because our content team is spread like peanut butter a little bit, we're spread thin, and people just didn't have the bandwidth initially to wrap their heads around adding another thing to their plate, adding content research to their very full plates. So I think an each-one-teach-one approach is maybe more helpful, at least for our team. Think about how your coworkers learn. Are they visual learners? Do they like to learn in groups or on their own? Something to consider.

[00:23:03] Creating templates was really key for our teams so we could plug and play, so we could really get crisp on our thinking about what do we want to learn from this research, and make sure that they were focused and we kept them in scope, so we could run more short tests quickly. Like I said, short and sweet for the win.

[00:23:20] And this is again the report that I shared. Share your results widely. Make sure that people are aware of the testing that's going on. Share it with your feature team, share it more broadly than your feature team. This is a newsletter that we send out, so at the end of the month we put in all the research, including content research, into a newsletter that goes out to the organization. This is the one email that everyone is excited to get every month. People can't wait to open this and dig in and read about it. It's fantastic.

[00:23:52] And make it a habit. If you can add content research to your sprints, add it in for every sprint. Make a backlog of what you want to test and then start chipping away at it. We've also added content research to our team's goals. We have annual reviews, and if people work this into their goals it's one way to make sure that people are participating and making it part of their day-to-day work. And then things take off, people will talk about it. People will talk about it widely and you'll soon be up to your ears in requests for content research, which is pretty cool. At least that's what happened at our organization, and it's fantastic. We're driving a lot of business impact with it and I wish there were more hours in the day to do more of them. All right, well, that's my presentation. Thank you for coming.