How BMW Motorrad Integrates Fans into their Digital Product Development

Jan Bode, Jann Kirchhoff at UXDX Europe. Video: https://youtu.be/TyBJepnuCds

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introductions and why testing matters for BMW Motorrad

[00:00:00] Jan Bode: Hi, I'm Jan from the Testbirds community team, and I'm here with our partner, Jann Kirchhoff from BMW Motorrad. We want to talk a bit about our collaboration and how we at Testbirds help BMW Motorrad to test their digital products with real riders of their motorcycles on our platform. We will talk a bit about the why, why there is a need for it, and also how it works. I will also give you some insights on how we built up a dedicated community of BMW motorcycle riders who frequently test the BMW Motorrad digital products. I will hand it over to Jann who can take it away from here, and I'll talk about our community a bit later on in the presentation.

[00:00:48] Jann Kirchhoff: Thank you very much, Jan. Hi to the audience. My name is Jann Kirchhoff. As Jan already said, I'm working with BMW Motorrad in the area of digital apps and I'm employed as a product success manager. What this means to my business is basically keeping the customer happy, giving the customer a seat at the desk when it comes to decisions about our apps, and taking care that our digital services actually offer what the customers need. Testing is a huge part of that, and therefore, for proper testing, for proper customer opinions, we are working together with Testbirds.

[00:01:29] But first let me take you a little bit into the world of what it is like riding a motorcycle. If we talk about motorcycles, we are not necessarily talking about digital stuff. When you are riding outside with the motorcycle, it normally gets dirty, you are pretty close to nature, you're on the road, you're on the track, you're together with people, sometimes you're off-road in the mountains, and it's pretty intense. So necessarily, you don't need any app. But we want to provide a digital product that makes riding better than before and that helps you do better rides.

[00:02:07] We have a small agenda to walk through today. First, I want to introduce the product to you, what we actually do, what we offer. Then what quality means for us, which challenges we are facing, how our crowd testing is actually working, how we actually get in touch with the customer, and what Testbirds actually means to us if we talk about the tooling landscape.

The BMW Motorrad Connected app

[00:02:33] First of all, BMW Motorrad is offering a product called the BMW Motorrad Connected app. This is available in the App Store for free, and you can install it on your smartphone. Your smartphone can then be connected to your motorcycle that features a 6.5 inch TFT display, and it's connected via Bluetooth, and you can also connect your helmet. With this combination, you can take phone calls, you can listen to media, and in combination with the BMW Motorrad Connected app, you can also use navigation from your smartphone, you can record your rides and the data you create, and you can also see valuable vehicle information in the app.

[00:03:11] You can control all of this with the so-called MMC, the multimedia controller that is mounted to your left side handlebar. So you can control multimedia functions without taking your hands off the handlebar, and this is one of the most important things while riding, because you want to keep riding very secure and our riders healthy.

[00:03:35] This video just gives you a small impression of how it works. On the left-hand side you'll see how the MMC is manipulated, and in the display you can see the navigation instruction changing. Again, this all comes out of the app and it's transmitted via Bluetooth to the display. That's why our BMW Motorrad Connected app is also pretty important for the customer, since it delivers a great navigation functionality out of your smartphone.

What quality means, and the three puzzle pieces

[00:04:03] Talking about quality, the actual challenge has changed. We are also producing motorcycles, of course, but if you talk about digital products we have three puzzle pieces that need to go together. Of course we have the vehicles, the software versions and the vehicle models that we have that feature the TFT display. But we also have the BMW Motorrad Connected app that is featuring updates even every month. And on the left-hand side, we have user devices. As you can see, there are two smartphone platforms. There are more than eight smartphone OS versions featured. There are more than 24 countries that need to be supported. And 97% of our users have more than 30 smartphone models between them.

[00:04:48] Now you need to make sure that all those puzzle pieces work perfectly together to create this high quality user experience we want to deliver and we want to see for our users, since this is the reason why we get up every morning and get into the office. We really want to deliver a high quality user experience, and Testbirds is a pretty valuable part of making sure this high quality user experience is created.

[00:05:08] Of course, we have our profound testing cycles from vehicle development. We do internal testing. We have other testing partners that ride our motorcycles for thousands of kilometers to make sure the vehicle is properly working. But if we talk about digital products, the game changes a little bit and we need other tools to cope with that challenge.

Zero suitable testers, and building our own crowd

[00:05:40] When we started first with crowd testing and we had the first appointments with Testbirds, of course Testbirds told us, yeah, we have this great Testbirds test crowd. At that point in time there were about 300,000 testers that we potentially could involve in our testing. And we pretty fast came to the point where we said, okay, how many of those 300,000 testers have a motorcycle? It's like this. And how many of those 300,000 testers have a BMW motorbike? It's like this. And how many of them actually had BMW motorbikes with exactly this equipment, the TFT display? It was zero.

[00:06:28] This was a big problem, because there was just no one who could test our app. Of course, to test that app, they need a motorcycle. What we've been working on is the bring your own crowd concept together with Testbirds, where we created our own crowd. Together with Testbirds we recruited testers, customers of BMW Motorrad, that actually own a motorcycle that is equipped with the TFT display. So we built up a crowd that today features more than 250 testers in more than 20 countries, and it's growing.

[00:07:05] This crowd is so valuable to us, as we can directly address those customers, ask them for their opinion, ask them to try our app, the newest beta version of our app, and give us direct feedback. Of course, Testbirds did the heavy lifting in recruiting those users. We went through Facebook groups, we went through influencers, but also our email support, our affairs at BMW Motorrad, helped to get users into that crowd and to build it up. Jan is also facing some challenges, I guess, and he will now tell you about what he actually experienced during our time with Testbirds.

Recruiting riders worldwide, and why they join

[00:07:48] Jan Bode: Exactly, thank you. To give you a little bit more background, we started working with BMW Motorrad in the fall of 2018. Jann has already mentioned that we had precisely zero suitable testers in our community of way over 300,000 people back then. So everything that we built here in terms of this testing community for BMW Motorrad, we built from the ground up.

[00:08:17] The first riders that we recruited for this community were all mostly from Europe, mostly Germany and the UK, with some testers from the US. But then very soon winter came, and when we sent our test invitations where we need testers to take their BMW motorcycles on a ride to test the new beta version of the app, some of the testers' responses were just a photo of their street covered in snow. Not the ideal riding conditions. From that point on we realized that this community needs to be a worldwide one, where riders coming from everywhere can participate, and somewhere on the planet there's always decent weather for riding a motorcycle.

[00:09:04] What we basically did is we also approached motorcycle clubs everywhere in the world. There's literally a motorcycle club for BMW motorcycle riders in Namibia or in Peru. That's what we did in the first place. We approached these BMW clubs and we laid out the benefits for riders if they participate here. For those riders the benefits aren't just monetary. The bigger incentive is actually being part of this development, to help BMW test their products, to give feedback, to participate in interviews, and make their voices and opinions heard. Because everything that we do here is going straight into the product team at BMW, and to a lot of people being able to do this counts even more than the monetary aspect that we also provide to everybody who participates in our tests.

Authenticity, trust and guiding a non-digital-native crowd

[00:10:12] The next thing is authenticity. Talking about authenticity, it's a challenge that we have been facing all the way through the project. We are still Testbirds. We're a small company based in Munich and not many people have heard of us, especially not motorcycle riders in South Africa. So naturally people ask themselves why they would put their trust in us, or if we're even working with a company like BMW, or why they should provide us with their personal data.

[00:10:48] Our task therefore is to create a level of trust. Fortunately, Jann from BMW is very cooperative in that manner. We shot a welcome video where Jann explains the reasons for the testing community even existing. We also sometimes attach videos to tests, or we give an update on the development status, for example during the times of Corona where testing has slowed down a little bit, just to bridge the gap a little bit. All of this direct communication helps us tremendously to create the level of trust that we need to keep our testers active and motivated and engaged.

[00:11:31] However, having gained people's trust and having them sign up with us is only really the first step to a successful testing experience. We have to remember that we are dealing with a target group that is a little bit older, and testing is maybe not really familiar to them. Just for comparison, our average tester at Testbirds is around 30 years old. We also test with the car side of BMW, and those testers are on average 37 years old. And our motorcycle riders are 48 years old. So we're not really talking about digital natives all the way through.

[00:12:10] We need to provide a certain level of guidance and assistance to them all the way through, from explaining how the testing works, to troubleshooting within tests, and also making them understand how the development process works. But fortunately these testers are some of the most enthusiastic and passionate, and also most relaxed, testers that I have come across, and that's just great to see.

[00:12:44] To bring my part to an end, this is the current status of the testing crowd that we've built up for BMW Motorrad at its current stage. We have testers everywhere, in over 30 countries and more than 250 testers in total. So again, somebody always has good weather to take their motorcycle on a ride and test the new beta versions. Back to you, Jann.

Getting through to the user in a multinational corporate

[00:13:12] Jann Kirchhoff: This is something we are also very proud of, that we have such enthusiastic, not only users and testers, but also fans, that are supporting us in the development process, that are really passionately testing our app, getting us feedback, sharing their opinion. Testbirds in that game is really the one who catches them all and who consolidates this feedback into an understandable piece of information that we can use in our development process. This is also the big value of having a partner like Testbirds.

[00:13:49] If we are talking about challenges, especially if you are a multinational corporate, then it's pretty tough to get through to the actual user. As Jan already mentioned, it's pretty easy to say, yeah, let's build up a beta user group. But if you are a multinational corporate, it's not that easy. You have to get through to the actual user, and if you are in the headquarters, basically a product team, you have to go through your sales department, you have to also go through your national sales corporation, and then you need to approach a dealer, in our case a dealer, and then you have to go through to the user.

[00:14:31] Ideally, it works like this. Everyone is bored and of course has time to do the process, to sign up, the data privacy statement and stuff like that. But let's be honest, this is not reality. Reality more or less looks like this. It would take us ages to get through this hierarchy, this organization, and in the end all those people need to do their jobs, they need to serve the customers, and they are not a beta testing department.

[00:14:59] What we did, and for me as an employee of a big corporation this is one of the most important benefits of Testbirds, is that we installed a bypass. Testbirds is doing all the heavy lifting for us, gets through to the user, works with them and builds up this beta user group without bothering our whole organization. And we can work really agile and really, really fast with the users and get direct customer feedback.

A rider who recorded 30 minutes of his own screen

[00:15:28] We just want to share a short video that shows how close we are with the user, and how passionately they work for us, and how they want to share their insight, they want to share what they found, bugs and so on. This video was taken by one of our riders, and he's basically recording the TFT screen for, Jan, I don't know how long the video was, like 30 minutes? And he was editing it after that, and then he was also commenting on what actually went wrong. We were totally stunned by the way he leaned in.

[00:16:05] What we did was we decided to send him a small care package from BMW Motorrad, together with Testbirds. And something that was absolutely unforeseen was that he basically posted this on Instagram. He was really thankful for what we sent to him, and he was really like, wow, he didn't expect that. This is the kind of personal relation we want to build up with our fans, our testers. This is what I really love about working with Testbirds and our customers, that you can get directly in touch with the actual user that is using the product.

Testbirds in the tooling landscape, and the benefits

[00:16:44] Talking about product development, of course we are using a lot of tools, and this is maybe the last point I want to come to. I think a lot of those tools are well known, if you talk about Jira, Confluence, Bitbucket, Jenkins and so on. Testbirds is one of our tools to make that certain product happen.

[00:17:02] In a nutshell, if someone would meet me on the street and ask me, what are the benefits of Testbirds, why do you need Testbirds to do your job, I would say, first of all, it's testing done by real authentic users, so they are not biased, they really paid the money for the product and they want to get the value. Also, Testbirds is a good instrument to motivate our community and to really maintain our fanbase in that small beta user group. People really feel special about being part of the product development. We can quickly generate new ideas and also validate our ideas, whether we are on the wrong track or on the right track. And there is a huge geographic coverage, so we can easily do a test drive in Spain if we need it tomorrow.

[00:17:51] One thing that is overlooked by a lot of product development teams is localization coverage. We can check for localization coverage, and we can check if our translations are actually right or if we have bad translation quality. And of course, in the end, there are a thousand different models of smartphones out there, and we are not alone on those smartphones. We are next to other apps. There are system settings, talking about energy saving options with Android. Testbirds and the test crowd and our testers are a great opportunity to test our app in the real world that is not isolated in the laboratory, before actually releasing it. This makes Testbirds a great tool for us. I think I've come to an end. Famous last words, Jan.

[00:18:44] Jan Bode: Well, I think you made it pretty obvious how Testbirds is helping you in product development. One point that I would love to add is that you're also doing a lot of interviews with our testers. When you're developing a new product, it doesn't even have to be for the app, it can just be anything, and you're early in the development stage, you can basically reach out to our testers directly, have one of our project managers on it, and interview the testers to see and validate if you guys are on the right track, if the product or the prototype makes a lot of sense. I think that is also something that helps you a lot on your way to developing the final product for the customer, who is in the end the one who you want to buy the product. That's something that I wanted to add at the end. There's not much more for me to say, besides thanks everybody for listening, and hopefully see you soon. Bye.