AI and ML in Drone Navigation: Insights for Product Development

Alan Hicks at UXDX EMEA. Video: https://youtu.be/g77N0BPC-UU

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

About Manna and how drone delivery works

[00:00:08] Hi everyone, hope everybody's refreshed. When I found out I was doing this talk, because Manna tend to go out and do a lot of talks, our marketing team told us it was about AI and ML, and that they tried to put as many buzzwords into a talk as possible. So I apologize that this talk is actually just about data, which, if we all know, is really what AI and machine learning is.

[00:00:39] What I'm going to do is first of all tell you a little bit about Manna, just in case you don't know who we are, so you have some context. Then I'm going to show you what we're trying to do with data and AI, and then some practical things that we've done immediately and what we're looking to do in the future.

[00:00:53] For those of you who haven't heard about Manna, we are on a mission to improve the world by making lightning fast suburban deliveries green and affordable. What does that mean? That means that when you typically get your takeaway by a road-based vehicle, using up carbon emissions, we will now do that delivery by drone, which means we don't have any traffic, we're fully battery powered, and we're very green and society friendly.

[00:01:23] How does it work? It works the same as your normal delivery. You go onto your aggregator app like Just Eat or Deliveroo, you shop, you decide what you want, you put in your address, you pay for the goods. The only difference is that instead of it being late and cold, it's fast, it's hot and it's quick. That's really it. So we have all the normal user flows that other businesses would have, replicated some of the same technology, just to prove out the market fit for drone technology.

[00:02:01] Why drone technology will win in this space: it's just faster and a much, much better quality product. Our typical delivery will be about 8 minutes 43 seconds from when the customer actually places their order. The actual delivery drone part of that is about 2 minutes 48 seconds average. It's gone up a little bit since this slide because we're delivering a further radius.

[00:02:27] So that's what Manna do. We're live in Dublin at the moment. It is the largest drone delivery operation in all of Europe, probably in the United States as well. Australia is the only country in the world with more operations at this moment in time.

[00:02:48] Rory really wanted us to have a drone here today to show you, and we just didn't have any spare. But when people think about Manna, the first thing they say is "oh, the drone." That's the entire business, that's what everybody is focused on. And it is, it's the cool part of our business. If there's any software people in the room, which I think UXDX is a lot of software, it's very hard to get excited about software because it's intangible. As soon as somebody puts in a fingerprint scanner or some physical bit of hardware, as simple as it may be, everybody gets really excited, and we're no different. So we always try and say to people, look, the drone is a tiny part of our business. It's actually much more about the applications, the system and the data that we actually work on.

The drone and the Blanchardstown operation

[00:03:33] But to cover the drone. Our drone is reasonably large. Most people typically think of drones in and around this size. Our drone is about one and a half meters all the way around, and it's built for redundancy. It's not the prettiest drone in the world, it's a workhorse. It flies 12 hours a day doing multiple deliveries an hour, and it's designed with redundancy in mind.

[00:04:00] It allows us to do a sub-five minute delivery. We cruise at about 85 km an hour, so super quick. We can go as the crow flies, so we don't get stuck in traffic. As I mentioned, we're green, and most of all we're very, very safe. You'll see, and this will become important later on, on the drone we've got propellers on every arm, we've got two engines, and we can afford to lose any engine on any arm and we can still fly. That's part of our safety case. Internally we've got lots of triplex systems to keep us safe. But that is not what today is about.

[00:04:41] This is again the largest drone delivery operation in Europe. It's down the road in Blanchardstown in D15. There we are operating deliveries to 45,000 households, Tuesday to Sunday, 9:00 in the morning till 9:00 at night. We've got about 25 restaurants there. We've just opened a second base on the outskirts of D15 to expand our delivery radius. So if you go down there, have a look in the sky and you should see us flying.

[00:05:12] This is the area, and this is what it looks like if we overlay all of the drone routes that we've ever flown. Each orange marker there, each orange line, is a flight that we've actually taken and done a real delivery. To date I think we're about 11 and a half thousand deliveries this year alone in Blanchardstown. So again, customers love it, really, really popular service.

How we use data: Metabase and flight logs

[00:05:41] Now, data. How do we use data? You might recognize that's actually an operation we have in the US, I should say. It's not a very Irish-looking home, that's not in Blanchardstown.

[00:05:56] Data. I think if anybody leaves here today with one thing to look at, it's a tool that we use for data called Metabase, and it is very, very, very useful. Very simple, very useful. If you take one thing away from today, it's Google Metabase and see how it can help your business. If anybody in your business says that they need to build a custom dashboard, ask that person to take lunch, go Google Metabase, come back and have it running.

[00:06:24] With all of our flights, we've done about 200,000 flights, with every flight we get three log files, because we've got three flight controllers on board. That's about 50 megs of data, automatically uploaded to the cloud, and we process it. Normally we dump it into a database and then we run all of these fancy SQL queries on it to see, are we within tolerance for what we wanted to be achieving in that mission? Our viewpoint into that is Metabase.

[00:06:54] All of these are custom SQL queries. This is one particular flight. In the top right-hand side for you guys, left-hand side for you guys, you can see some metrics which I won't bore you with, but it essentially tells us that the flight is healthy, what altitude it flew at, how much energy it used and things like that. This is something that we've been doing for five years. We've been using these tools and we've built upon them, and nobody in this room, I think, would call that machine learning or AI. That's just basic stuff that we should all be doing with our data.

First learning: lots of small specialized models

[00:07:26] What we did about a year ago is we said, right, we are going to be scaling out our fleet. Just to give you an example, if we wanted to cover the UK in drones, we would need 49,000 drones. So quite a lot of drones, and this type of data won't scale well for that problem, for a few reasons. One, it's very complex data, and there's nuances in what is an anomaly and what is not an anomaly. So about 12 months ago we hired one guy, Tung, to come in and play with our data and see what he could find.

[00:08:01] We had great plans. We said, yes, we're going to use machine learning for all of our data, and we're automatically going to anomaly-detect every problem we ever thought we could have. What we first quickly realized, as soon as Tung started looking at it, was we hadn't labeled any of our data, which is a big problem. The number one problem is you don't record the data in the first place. The second problem, very closely followed, is you don't label the data, which means your data scientists have a very, very hard job of trying to figure out what the data does. So that was the first problem.

[00:08:36] I'll jump back and forth. What Tung did is he ran us through unsupervised learning, and he figured out some plots and we grouped our flights. We went to our engineers and we said, take this group and this group, and tell us is it a good group or a bad group. We identified these purple dots here, and this particular plot is around vibration, which is an important factor for us. We said, okay, that's great, we can detect the flights and everything like that. I'm losing my screen here guys, sorry, I don't know if that's affecting that.

[00:09:15] So we discovered that this can detect some anomalies, but it was giving us a lot of false positives, and what that meant was that it was only so useful to us. What we ended up doing was figuring out that actually one large model is not a good use case for us, and that's because wind has a big impact on vibration, if we take that example, or voltage and things like that. So what we've settled on is a lot of small specialized models, which means that we don't have to have as much trained data or labeled data, because each model is focused on an individual part of the problem. So that's our first learning.

Second learning: replacing the pre-flight check

[00:10:03] Our second learning is that we can actually use these models to great effect in our operation. If you can imagine our drone operation, we have four drones on four pads. An order comes in, the restaurant bring the food to us, we're co-located with the restaurants. One of our operators, the loader, takes the bag and puts it into the aircraft. Now, before every flight, until we studied this data and actually understood the problem, this is what the loader had to do.

[00:10:45] What's happening here is they're doing a pre-flight inspection. They slowed this down a little for the video today. What they're doing is they're checking bolts underneath on the motors, and they're rubbing their hands along the props to look for cracks or looseness in the props. This is something that we've committed to doing, again, to have a safe operation. Our props are folding props, and what can happen is the washers in there can get worn out and the props begin to shake, and they're not as efficient as they otherwise should be.

[00:11:21] So this takes, let's say, 30 to 40 seconds every flight. I promised the guy doing this I would show it all. So think about this time for every order we get, this is the process we're doing.

[00:11:46] Okay, so we went to the data and we said, what can we do to solve this? And this is still one guy, Tung, working away looking at our data to understand. What he figured out is if we look at the two seconds before takeoff, just as the propellers are starting to spin up, they've got a different vibration profile if they're ready to go, if they're tight, or if they're a bit loose. They've got a different vibration profile. So we plotted that data on this bell curve, and what you can see is the blue here indicates that we've got an issue, the orange indicates we don't have an issue.

[00:12:27] Now, a bit clearer in this violin distribution. If we look at it by day, the larger the base and the higher the spike, the more vibration, which means we need to change the props, and we've got tolerances. So this was great data, but we couldn't corroborate it until we started bringing in our labeled data, which is our maintenance records. This was only something that we started to really digitize properly in a programmatic way, again, about eight months ago.

[00:12:57] These numbers here are different work orders that were carried out on the craft. Every time an aircraft is touched, we actually have an entry that we fill out and we explain what parts were affected, who did the work, etc. What Tung was able to do is he was able to correlate motor changes with decreases in vibration. You can see on the second we changed one motor and it went down, fixed another motor, it went down, and then we came all the way down into our tolerance.

[00:13:30] What this meant is that we have a proof point now for understanding what vibration is good and bad. We now do not need a pre-flight check, which means we've saved, based on our current flight numbers, 41 hours a month of a human's time, which makes our operation more efficient. The biggest challenge with road-based or last mile delivery in general is the human cost that applies. So that's one example where a data model, an AI model, has really helped us in a practical way, where it hasn't just given us results that we can't really action. So that's what we like.

Drone routing and the 4D model

[00:14:14] The next problem we had that we solved with some data is actually our drone routing. What we used to do until very recently with our drone routing is, when you fly you need to reserve airspace, so if you've got two drones in the sky they're not going to collide. We have a virtual grid that we make of the sky based on aerial surveys and population densities, and that we figure out where to fly with different algorithms.

[00:14:45] What we have here, in the old picture, we have how we used to do it, which was, if you see these yellow boxes or hexagons, we used to actually reserve the entire route. Which means that when we were flying other drones, we would have that in our grid and we could route around them. Very, very simple, very, very effective. What that meant, though, was that we weren't clearing that airspace until the drone came home. We go out and back the same route at the moment. So that limited our capacity in the air, which really meant we can only fit about five drones in the sky at one time, which on a Saturday night when everybody wants their takeaway is not very efficient.

[00:15:25] So we have recently moved to this new system, which we call 4D, four-dimensional. What you can see here is the green is a sliding window in time, space and distance. The white dots are the drone, and they need to be there. What we do is we guarantee where the drone will be inside that green worm, or our sliding buffer, and that allows us to have crisscrossing routes. We're not constrained by time, which gives us basically infinite capacity in the sky.

Weather and the forecasting model

[00:16:02] That's all very easy to say. It's harder to do, and the main reason it's harder to do is what Irish people love to talk about all the time, which is weather. Wind has a big, big, big impact on drones and anything in the sky, really. I personally didn't appreciate how difficult a job the weather forecasters have until we actually hit this problem. What happens is, if we're flying into a headwind, it will really affect the performance of the craft. Likewise, if we've got a tailwind, it'll make it go faster, which in our 4D model will push us outside containment.

[00:16:45] So we said, okay, simple, we'll put up a weather station, which we have already, because we use that to determine if we can fly or not fly. But it turns out our local weather station wasn't accurate enough, because it was being shielded by buildings and things like that. What we later found out is actually no weather station is accurate enough, because they'll always skew their data.

[00:17:09] So what we did is we took three different types of weather sources. One of them, the external one, which we validated, is a 10-minute weather source. Our current on-base weather source was, I think, what do I have here, one minute exact by five seconds. What we needed was at least a one minute, but we couldn't get that from an external reliable data source. So what we did is we plugged all of our data into a model and we formed a forecasting model.

[00:17:45] What you can see here is the blue dots are the external source of data, which we get every 10 minutes, which is why we've got flat lines, and then the other ones are more real-time data. You can see they match sometimes, depending on wind direction and things like that. So it's not an exact science, which is where the model has really helped us.

[00:18:06] What you can see here is our predicted weather value, which is the orange, based on the actual different weather sources that we're coming from. What this does is this allows us to maintain our ability in that column, which means we can guarantee that we can stay in tolerance of our 4D window, which again, as I said, means that we can have unlimited craft in the sky at any one time. I'll just go back because we're quite proud of this video. Okay, so you can imagine what would happen if one of these aircraft came outside of containment.

What's next for Manna's data journey

[00:19:05] I think what I wanted to show today was just a bit about how we're using data practically. We've got big plans for how we want to use these AI models, and we're getting better and better at labeling data. I think it's a slow process. Everything's a slow process when you're doing it for the first time. Something that we intend to have in place by this time next year is an aggregated model of all of these smaller models, so that when we do scale our drone fleets to thousands, we've actually got a single health check, a fleet check, to be able to verify what we're doing.

[00:19:41] So what's next for Manna in terms of our data journey? We're going to develop more and more smaller models to pick up things that otherwise we would miss, keep us safe. We're going to refine our models to enable preventative maintenance. The MTBF, minimum breakage time for a part or for a fault, is a key stat for us, and we believe AI can help us do that, as long as we're maintaining our work order records correctly so we can programmatically use those in our models.

[00:20:13] And then, as I said there, we want to scale these models. So hopefully, if I come back next year, oh, you're not in Dublin next year, we have to worry about that, or get a trip to New York. The next time we give this talk, we intend to have predictive maintenance. So not only will we detect that our props are loose and stop flying, we will actually be able to tell our maintenance team two weeks in advance so they can actually schedule maintenance and not react to maintenance. I think that's the power of data for us.

[00:20:45] A great thing one of our advisers always reminds us of is "all models are wrong, but some are useful." I was able to find on Google this was attributed to George Box, and it's one of the great things, if you're working with data scientists that are looking for perfect: there is no perfect, perfect doesn't exist. So I would always remind them that all models are wrong, but then some become useful. Thank you.

Q&A

[00:21:22] Host: How do you manage route planning if competitors enter the market, and are there regulations on sharing flight routes?

[00:21:31] Alan: Great question, somebody's been thinking. Yeah, it's one of the big challenges that the drone industry has: how do you cooperate and share airspace? One thing we recently did with one of our competitors, but also one of our friends in the industry, is Wing, which is Google's drone company. We demonstrated in the US Wing drones and Manna drones actually collaborating and swapping. It's called UTM, unmanned traffic management, which is a bit like manned aviation's air traffic control. But the expectation is that there will be 10x more drones than there are manned aircraft, so it can't be a manual process. So the industry has come together and developed UTM standards, APIs basically, and governance policies. When we go to plan a route, we can share our route with the industry, they share their routes with us, and we deconflict. That's where the 4D comes in, and that's why you need to be able to predict where you're going to be, because if you're sharing the airspace with other people, they need to know where you're going to be, and you can't just do it in real time.

[00:22:36] Host: Okay, great. Why should sky be designated to commercial deliveries, and are we looking at the end of a blue clear sky? What about the users that are not ordering stuff at the moment in time?

[00:22:48] Alan: Yeah, I think it's a fair question. I think there's loads of benefits, societal benefits, like the environment, things like that. Do people feel the same way about road-based deliveries that they're not getting? Do we like sitting behind Amazon trucks in traffic, and delivery bikes? What we do know from the data is that cyclists and road-based delivery drivers have a very, very catastrophic incident rate when they're trying to make deliveries around the world. There's some frightening stats on that.

[00:23:22] Host: Good. Okay, how much cheaper are drone deliveries compared to a person on a bicycle?

[00:23:29] Alan: Yeah, good question. Not only is our business exciting and cool and all of that stuff, it's actually very practical. If you look at all of the current delivery companies that are delivering food, they actually lose money on nearly all of their deliveries, which is why the cost of the takeaway is so high, because people are trying to make the margin somewhere else. Drone delivery, we're already lower than the cost of road-based delivery to restaurants.

[00:23:56] Host: And do you collect your data in real time?

[00:24:01] Alan: Do we collect your data? Yes, we collect real-time data from our aircraft. I suspect the question is more, do we collect environmental data, and no is the answer. We have a camera on board, but that is only used at delivery to verify the delivery zone. There's no collecting of data. That's something we're very conscious about.

[00:24:19] Host: And how did the company find product market fit?

[00:24:26] Alan: With a lot of work. Originally the business started as a B2B business, where we would power other people's deliveries, like restaurants and things like that. To actually prove market fit, we actually had to create a store and create an app and create all of these things just to prove the actual technology. So we're currently running a Manna app at the moment in Blanchardstown. We're integrating into a few other third parties to be a B2B delivery option, but we currently have all of the headache and systems that come with managing actual customer orders.

[00:25:01] Host: Nice. Okay, what percentage of time are you unable to fly due to the weather in Ireland?

[00:25:11] Alan: 3%.

[00:25:11] Host: 3%, really?

[00:25:12] Alan: Yeah, we can deal with 97% of Irish weather. It's pretty good. We wouldn't be flying in Florida right now.

[00:25:19] Host: Right. And then I think, is this the final question? What was the impact of the weather forecasting model that led to 4D versus the old route planning, and how did it impact the business? From increase in delivery, was there an increase in safety?

[00:25:35] Alan: So there was an increase in safety, because we were working on a model where we blocked the entire route, so there was no impact of the wind. We only moved to this 4D model once we were confident in the weather model that we had, so there was no impact on safety. There was a massive impact in the capacity that the sky now has, and it gives us a lot more flexibility in terms of how we route, in that we can fly out one way now and back a different direction.

[00:26:01] Host: Okay, we've got more questions, you can find out all about drones. What was the weight limit for the drone, and can I get a couch delivered?

[00:26:07] Alan: You can get a very small couch. The maximum takeoff weight for the drone is 25 kilos. At the moment we deliver two and a half kilos of payload.

[00:26:18] Host: That's pretty good. It's like a bag, like my luggage of chocolates, for a small family.

[00:26:26] Alan: Yeah, we designed it that way. The next iteration of the craft will actually have a three and a half kg payload capacity, and that's really what Tesco's called the emergency shop, where people do a single basket shop during the week. That's three and a half kilos.

[00:26:41] Host: And did you interview birds and ask for permission to fly into space?

[00:26:47] Alan: No, we nearly did. In our operation in Balbriggan, where we were for two years before Blanchardstown, we actually had a set of nesting seagulls beside us which liked to attack us every now and again. But they loved the drone, they didn't like the humans.

[00:27:01] Host: Now I remember these use cases in Dubai, I think they've been practicing this. Are you going to transport people with your drones, AKA a drone taxi?

[00:27:12] Alan: Yeah, that's a whole other ball game. Urban Air Mobility, there's a lot of companies working on that and doing a good job. It's a massive cash burn as an industry. Once you involve carrying humans it becomes very, very complex. A lot of the technologies that we're building, like the air traffic management, the UTM technology, is feeding up into that, air taxis, but it's not something Manna are looking at.

[00:27:39] Host: Okay, one more, keep going. Like a pop quiz. What does the future look like for residents, noise nuisance and privacy? And it's not talking about birds.

[00:27:50] Alan: Not birds, okay. Yeah, I think a lot of people have a bad impression of drones because they think about the hobby drones that are bought in the different electronic shops, that have quite a high-pitched, annoying buzzing sound. Our drone, again, it's about a meter and a half. The props are about 60 inch, 32 inch at either side, and it's more of a hum. So when our drone is in the sky you don't hear it. You hear it when it comes into a delivery, but it's actually quieter than a delivery van coming to your house, and it's there for far less time.

[00:28:27] Alan: So it's a new thing, and when we move into a community we do a lot of community engagement. We bring the drone to schools, we do evenings for the residents to show them the technology and how it works, and that allays a lot of fears. But we appreciate that new things can be scary. Typically, when we go into a new location, everybody looks at the sky for three weeks and then it's normal. People just love it, and they love that they can get their coffee or their ice cream or whatever they want.

[00:28:58] Host: How do you incentivize people to actually purchase the order from you and not from another kind of delivery?

[00:29:08] Alan: There's no real incentive needed. It's the same price as a road-based delivery, it's just quicker. People very quickly forget that getting it quicker is a new thing, and really they order from us on repeat for the quality. And it's in new categories as well. We actually run a coffee shop, because coffee is a huge item for delivery, and I think if you said to anybody, do you want a road-based coffee, I think they'd laugh at you. So it's opening up some new categories that we didn't realize at the start.

[00:29:47] Host: Okay, one last question. Are some people so disagreeing with drones that they shoot them? Do you have a lot of loss?

[00:30:02] Alan: We've got no loss. Our drones, I should have said earlier on, our drones don't actually land, so they don't come near humans or the customers. We typically fly at about 65 meters, and when we get to your home we descend to about 15 meters off the ground, and underneath there's two doors that open and the payload comes down on a thread. Again, coffees are one of our number one items and we don't spill a drop. We are live now in Texas, and we've had one person come to us and say they're going to shoot us down, and it turned out they were joking, which is good.

[00:30:36] Host: That's great. Thank you very much, Alan.