Utilising Guilds To Develop & Support A Culture Of Research

David Sheridan at UXDX Community: Ireland. Video: https://youtu.be/74HioGSBe68

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Storyful, scrum and scaling

[00:00:05] I'm going to be talking to you today about utilizing guilds to develop and support a culture of research in your company. A little bit about me: I'm a senior digital product designer here at Storyful, and my work involves leading the user research initiatives. I'm also a member of the steering committee of our design system. I joined the company in October 2017, so I've been here nearly three years.

[00:00:30] I'm going to go into the background story for this talk to give a bit more context. We're a social media intelligence and news agency, and we use unparalleled access to data, proprietary technology and sophisticated analysis to deliver actionable insights to our partners. What that means is our journalists and analysts license and sell social media videos, and they also perform social media analysis on behalf of news organizations and our commercial partners. On the product team, we develop web-based applications to support these tasks.

[00:01:05] Within Storyful we use agile as our delivery method; in particular, we use scrum. We're set up in multidisciplinary teams, and these teams comprise one designer, one product owner, one QA engineer and three to five developers. We work in two-week cycles called sprints, and at the end of these sprints we have demos where we show stakeholders the work that we've done. At the start, we were using demos as a way to gather user feedback. We were small, about one or two scrum teams at the time, and essentially we showed the stakeholders the work that we'd done and asked them: do you like it? Do you see value in it? Will you use it? What do you want us to work on next? Because we were small, the risk of this was relatively low, and if we went down the wrong path we could stop, change and go back. So it worked for us; it was fine.

[00:02:02] Well, in 2013 we were acquired by News Corp, and we scaled rapidly. We went to over six product teams, each team with many more products, more stakeholders; there was a lot going on. So we decided to adopt the Spotify model and the SAFe [?] model, just to help agile scale more easily across the company and prevent people from feeling isolated. We added chapters as a way for people of the same discipline to share knowledge, come together and chat about the work they do. And we also added guilds, so that people across multiple disciplines could come together, talk about specific topics and share knowledge.

[00:02:47] But as we were scaling, we started seeing a lot more issues. What people were asking for versus what they needed wasn't really lining up. We would hear a lot of complaints, such as, well, we built the feature but nobody's using it, or we put it in but it's not what they really needed. So we were building a lot of unnecessary features, and we were wasting a lot of time doing the wrong thing. Agile methods are focused on developers; they came out of programmers' attempts to solve common pain points experienced during big software development projects. Agile was really good at helping us move quickly and ship features really quickly, but it wasn't really helping us understand what we should be shipping and where we should be going with the products.

Early research wins within one scrum team

[00:03:33] When I first joined Storyful, I was finishing up my master's in user experience at IADT [?], and we were doing a lot of user research. I kept thinking that maybe this is something we could really use in the company, and it might mitigate a lot of the issues that we were having. But as I was new to the company, and this was a completely different way of working that they'd never experienced before, we had to test the waters first and see whether it would be the right fit.

[00:04:04] Initially I joined a team that was working on the beta release of a new product, and from the start it was pretty clear that we needed to do some work on customer and user research. I had a chat with the product owner at the time and discussed the financial issues of releasing a product that isn't fit for market: releasing something people can't or won't use is essentially wasting your time. He agreed, and he let me go out for a few days and do some research. We ran some usability studies with SMEs within the company, just to understand some of the pain points around using the product, and then we did some secondary customer research to understand what our customers really needed from this product. From it we were able to develop personas and understand a lot of the issues that still existed within the product, and thankfully we managed to fix these before the beta release.

[00:05:01] A couple of months went by, and my team was later refocused to work on a new search tool in the company, to replace some of its predecessors. As a team, we had a chat with the product owner, and we were quite worried about repeating the past mistakes of the previous tools. Again, thankfully, he agreed, and he let us kick off some field research. Myself and one of the developers went off and shadowed stakeholders from across the company, to understand what their goals were, what they were trying to do, what issues they encountered, not just around our tools but external tools they were using and their workarounds, and what opportunities there were for us in developing this new product. We created an affinity diagram at the end and uncovered a number of issues, a number of different directions and opportunities. We had a meeting with the product owner and were able to help him develop that initial roadmap to get the product off the ground.

Field research across the product team

[00:05:55] With the success of these early studies within my specific scrum team, I was looking for opportunities to scale research across the product team on a wider basis. At Storyful, we have a product off-site every six months or so. At this off-site, we meet with key stakeholders from each pillar group to understand their business needs, so that we can align our product development to help them meet them. At this particular off-site, we found there was a considerable issue with stakeholder capacity across the whole company, but we had absolutely no idea why this was happening. So the head of product [?] asked the design team to look into this, do some research and see what we could do about it.

[00:06:38] We recruited 12 people from across the product team, developers and QA engineers, to help us with this, and we decided to do some field research. Before doing this, just to make sure everyone was comfortable and up to speed on the fundamentals of what's expected, we ran a half-day workshop. In this workshop we went over the format of a field study, the different things you should be looking for, and general considerations in running this kind of study, and we left some time at the end to discuss any questions or concerns. We then created a note-taking template based on the AEIOU framework, again just to help people understand what they should be looking for while they're doing this.

[00:07:20] Once we had this out of the way, we created a timeline for who we were going to be researching with and when. We paired one designer with another person from the product team, and they went off and shadowed multiple users. At the end of this two-week period we set up a war room. We put together a skeleton service blueprint map, and we chose this technique because we wanted to see the issues from start to finish across the entire pipeline, from sales all the way to customer delivery. When we ran the session, we brought back in all the people who helped us gather this research, as well as the product owners. We wanted the product owners there so they could see the value and the impact of what we were trying to do. As a group we worked through all the findings, we uncovered a number of issues and we agreed next steps. At the end of this we put together a report summarizing it all, which we shared with the exec team.

Starting a research guild

[00:08:22] At this point there was a lot of excitement about research, and we were looking for ways to really help support and develop this culture more. I remembered that we had quite a few guilds in the company, which were a great way for people across disciplines to come together and discuss this kind of thing. So I reached out to my manager, and we both agreed this would be a good idea, and its sole purpose should be to develop and support this culture of research we'd been working on.

[00:08:47] We started the sessions last summer, and we left the agenda quite open. We wanted people to just bring the topics and ideas they were interested in, and the group would talk about them. Well, we had a lot more people coming than we expected who actually wanted to learn some fundamentals and some techniques that they could bring into their work, and with this open format they weren't really getting much value from it. So we noticed numbers started to drop pretty quickly. We stopped and changed tack. We made a list of things that we thought would be really good for people to learn, things that they could apply straightaway. Now we run the meetings as workshops. The first half hour is an introduction to the technique we're covering that day, and in the second half people are paired up in groups and go off and practice it, and they can ask us for advice on better ways to perform it.

Minimum viable research and a small set of methods

[00:09:41] At this point people were starting to bring research into their teams, and it was starting to scale up quite quickly. We realized we needed some principles around this to help guide and structure the approach. We didn't want to just bring in any generic principles; we wanted something that would match the agile way that we'd been working. We work in MVPs, and I was reading Just Enough Research by Erika Hall. The book covers the idea of minimum viable research, MVR, and the whole idea is that you do just enough research to answer a well-defined research question, the exact same way we develop our features. We realized this was perfect, as research could be done in tandem with product development, so one wasn't slowing down the other.

[00:10:31] Once we'd done this, we started looking at our methodologies. We had a number of people who were quite new to research, really excited by it but really intimidated by how vast the landscape is, so we decided to hone in on what we thought would be the most effective methodologies that people could use. We got it down to five, based on six key use cases that we have. We recommend using focus groups to uncover stakeholder needs. We use interviews to define potential future development opportunities, and we also interview our sales teams to understand our customers' goals and pain points. We run usability studies to uncover issues before and during development. We run field studies to understand higher-level issues across stakeholder workflows. And we run remote surveys to measure the impact and usability of recently released features.

Product release cycles

[00:11:32] Again, to reduce the complexity of figuring out when to do research or what methodology should be used, we looked at the product life cycle that we have. We still use two-week sprints, but we've added another layer of organization on top, which I call product release cycles. These cycles comprise three sprints, and the goal of the product release cycle is that the team is working towards this one release, this one feature, this one overarching goal, using the three sprints to make it.

[00:12:04] In sprint one we're now recommending that people do field studies or focus groups or interviews to understand the problem landscape and the opportunities within it. Then, once those are defined and the team starts working on a possible solution, they start running usability studies, just to understand: is this the right direction? Is there anything major I'm forgetting here? Am I doing the right thing, or should I stop and start again? Then the same thing in sprint two, as we've nailed down the direction we're going to go and the way we're going to solve this problem; we start working through a lot of the bigger issues with it. And sprint three is all about refining and making sure it's ready for release, and that we're quite confident it's going to solve the problem that we defined earlier.

[00:12:47] We also recommend doing your remote survey at this point in time, and that's for the feature released in the previous cycle. It's been out in the world now for six weeks, so you're going to get a sense of realistic usage from it, and you'll be able to measure these things quite effectively. And you can do this on an ongoing basis.

Templates and shared research

[00:13:03] In another attempt to standardize our principles, our methodologies and our timelines, we started looking at the studies themselves and whether there was a way we could standardize the approach. Our goal was to reduce the time spent rewriting usability studies and surveys and everything else over and over again, even though they follow the same format pretty much every time, and to reduce the cognitive load of this as well, because it takes quite a lot out of you to create these studies sometimes. Essentially, we put together full usability studies and forms for everything that we covered in our list of methodologies, but we left placeholders so that people can add content that's relevant for their study. That's the only little bit that they actually need to worry about, and the hope was that it would make doing research more efficient.

[00:13:59] We created a shared Google Drive to house all this, and we shared it across the whole team. Over the coming months we found people were starting to share their research in this folder, not just the designers or people working on the product direction side, but developers and QA engineers as well. Seeing this, we created a dedicated area for people to share this stuff, and we created discipline-based folders as well, so people from front end, back end, QA and DevOps can all share their material with each other. Because we're a small company, we share the same stakeholders, and a lot of this research is relevant to different people at different times, so it's good to always have that there.

Key learnings

[00:14:44] There are three key learnings we picked up along the way. If you try to do too big an initiative too quickly, you might find you get a lot of pushback, and there are multiple reasons for this. Sometimes it's financial; sometimes it could just be quite a big risk. But often, from my experience, if people are very used to working in a very defined way and you try to change that overnight, you're going to make people extremely uncomfortable, and they're naturally going to fight back. So start small. The smaller you go, the less risk and money involved, and the easier it is to get that opportunity and that space to try something out. And the more of these small initiatives you do, the more people you get on your side, because they'll start to see the value of what you're doing. It's a bit like a snowball: the more of it you do, the more people you get on your side, the more momentum you get, and the easier it is for that culture to start developing almost on its own.

[00:15:42] Then, when you've got to the point where you have people interested, you have a culture and you have people involved, you'll look for best practices to frame this and structure it a little better. There are a lot of these best practices floating around, but every company is different: different shapes, different sizes, different budgets, et cetera. For us, we're a small product team and we don't have a budget for dedicated researchers, so using MVR was the most appropriate approach for us. Just ensure whatever you do matches your culture and matches your scale, because if it's seamless, it'll help the culture to sustain itself.

[00:16:24] These are some of the resources that I used along the way. If you're trying to do something similar, hopefully these can help. I've also attached my social media details, so if you want to reach out to me, I'm more than happy to talk about this.