“You Can’t Just Turn It On”: The Hidden Work Behind Enterprise AI

Daria Tarawneh at UXDX EMEA. Video: https://youtu.be/a95CrR4_kxY

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

How I got here

[00:00:11] Daria: Hello everyone. There's a lot of light so I cannot see a lot of you, so I'm going to try to make sure that I don't get blinded. Again, my name is Daria. I actually started my career in the industry back in 2006, and as many of you here, I started in a very different field. I'm an architect. I started doing advertising games and I started doing UI, and this is how I got into the industry.

[00:00:39] Daria: From there I ended up in New Zealand and Japan, building companies there, and currently I've been in Berlin for 10 years. I worked for McKinsey. I spent seven years working for Amazon Web Services, dealing with things like compliance, administration and auditing and so on. And for the last three years I have been at Miro. Very similar space: compliance again, security again, all the fun stuff.

[00:01:06] Daria: Before I start, can I see a raise of hand? How many people here work in an enterprise product or anything related to compliance, security? I got it, cannot see. Oh, there's a lot of you here. Okay. So a lot of the information that I'm going to give here might be familiar, and some of it could be interesting, or could be like, hmm, this is too much.

Sons of Anton, and the incidents that followed

[00:01:29] Daria: Before I start: do you know this show? Another raise of hand. Who knows this show? Not a lot of you. Okay. For the ones who have not seen this show, this show is actually about a group of people who come together and they are building a startup in Silicon Valley. The interesting thing about this show is that it was released in 2014. Remember this, this is very important.

[00:01:55] Daria: And in one of the situations, one of the developers decided to build an agent. He called it Sons of Anton. He was super busy, and he realizes, oh, the agent can fix all the bugs. So he goes and says to the agent, "Fix all the bugs." What do you think happened? Yeah. The agent deleted the entire production environment. It's very logical. If there's no code, there's no bugs.

[00:02:30] Daria: The story is very funny, and as I said, the show aired on TV on HBO in 2014, and now we're laughing about it. 2025 — my mic? Yeah. Is it switched? Okay, oh, that is so much better. So we thought this was funny, and this was like, haha, this will never happen, what is this science fiction gibber jabber. Okay.

[00:03:02] Daria: 2025: an agent in a very famous software — I don't want to say the name of the software because I don't want to get in trouble — actually deletes the entire production. So that is happening. We think this is an incident, like a one-off. Happened last year, we learned so much, we're so much better.

[00:03:22] Daria: 2026: an agent at Meta gave wrong instructions, and it caused a data leak for all of your data in Meta for internal employees. So that tells you that these security incidents are not a hacker going into data. It is actually humans. And what do we do? We are listening, and we're getting so used to this habit of copying and pasting and using AI in all our daily lives.

[00:03:52] Daria: But we have other incidents that happened of this blindly copying and pasting, and this was a little bit older, but that's an interesting one, because an employee pasted company data into ChatGPT, which is insane. And that resulted in ChatGPT being banned at Samsung for quite some time, until they actually resolved it.

[00:04:15] Daria: We think we're better, right? No, we're all here doing exactly the same. In the latest statistics I found, 77% of employees are pasting data into GenAI tools. 82% of them are actively using it on their private accounts, on unmanaged accounts. And here's the scary one — when I saw this one I was completely surprised — 22% are pasting data containing PII or PCI, which means personal data, credit card numbers, addresses, social security numbers, bank details, into unmanaged accounts. And that's where things are getting scary and getting very risky.

Why this is a user experience problem

[00:05:05] Daria: So are we talking about this now? Why is it really important in a UX conference? As we have seen, some of these incidents, some of these problems that we are seeing, it's not a software problem. It is a user experience problem. And we are seeing this more and more, and the companies are actually seeing this more and more. Whether it was an enterprise or a smaller business, we're now opening our eyes not only towards the shiny object of AI, but also to how secure and how compliant it is. And that is impacting how we work with it.

[00:05:44] Daria: We are used to designing frictionless experiences. For years we thought it needs to be simple, it needs to be one button, the magic should happen in the background. It used to be amazing. But we're finding ourselves in a situation where that frictionless experience and high stakes is ending up in a disaster, and everyone started looking at that and putting speed bumps and saying, wait a minute, we need to control this beast that we have.

[00:06:15] Daria: So let's do a small quiz. It's going to be an analog quiz. In general, as a statistic, how many users are actually using AI at work, whether they're permitted or they trust it? Who here thinks 58%? Can I see a raise of hand? 58%? 80%? Oh, there's more hands. 80%? There's more 80%. Okay, 90%? 90% of users are using AI at work or are allowed to use AI at work?

[00:06:54] Daria: Here's the shocking truth: 58%. 58% of workers are allowed to use AI at work, or they trust it enough to use it at work. And there's something we're seeing from a lot of companies, whether they're big companies or smaller companies: 84% of these companies worldwide are saying that security and compliance is their number one buying criteria. And what we see sometimes is they are willing to pay extra just for that security and compliance.

What enterprises actually require

[00:07:33] Daria: And they have specific requirements. What do they need? The first requirement is data boundary. They don't want to use specific models. They want to know where the data is going to be stored — there are certain regulations for each country — and they want to make sure they are bringing their own models. They want to control access and human oversight, so they want to make sure that they know who's accessing which AI feature, in which way it's been accessed, and what content can and cannot go into any model.

[00:08:10] Daria: Every company has their own security, compliance and legal readiness checklist, and if you're building a product that is not passing these security and compliance checklists, specifically when it comes to AI, congratulations, your product is not being adopted by the companies you want to be using it.

[00:08:29] Daria: Usage and recovery. They want to know who's using AI, and they want to make sure that their customers or their employees are using AI safely, which means they want to know what happens when a specific feature is failing, or when AI is failing. Can I recover it? Can I bring it back? Or is it going to go into the vacuum or the abyss and nobody can control it?

[00:08:52] Daria: And then there's the biggest part: the ability to audit. They want to see if there's a problem, they want to be able to trace it. So they're requiring those rigorous, detailed audit trails, to be able to fix it when a problem happens.

The EU AI Act, and a five-part framework

[00:09:10] Daria: And there's the biggest beast, which is now coming like a big storm, and that is the EU AI Act. We've been talking about it since last year, and now we're seeing more and more companies adopting it. How many of you have heard about this act? It's a lot of hands. How many of you know exactly what's in the act, or have read it? Ooh, one, two. Nice.

[00:09:36] Daria: So if you look at the act, there are a lot of parts of it that are actually very much user experience. And if you look at the entirety of what it is doing and why it's there, it is actually there to give us a certain layout, for us as designers, to give us responsibility. At the core of it, it is there to help us give secure AI, compliant AI and ethical AI: what you can and what you cannot do, and how it is protecting everyone's information.

[00:10:11] Daria: So when I look at it, I can bring it into five different parts of a framework: the visibility part, control, predictability, accountability and recovery. What does that mean?

Visibility

[00:10:29] Daria: Visibility. If the users cannot see what's happening, or they cannot see that this has been generated by AI, they cannot control it, they cannot judge it. They freak out. They're immediately losing trust. They should know when AI is involved and they should know exactly what AI has done. You cannot go into a magic box and pretend, oh, this magical experience, look at it, it's bringing bubbles and champagne and it's awesome. You cannot do that any more.

[00:10:57] Daria: Sources need to be visible. I need to be able to see which source you're taking this information from. Uncertainty should be designed into the interface. AI doesn't give you 100% information. AI is a mathematical predictive probability machine, so we need to bring that into the interface itself. Complexity needs to be revealed. We always said we want our interface to be simple, simple, simple, simple. But we cannot do simple any more. We need to bring that complexity and explain that complexity for the users.

Control

[00:11:37] Daria: Control. We're talking about the human in the loop, and here I really want to make a distinction between the concept of a human in the loop and the fact that sometimes it is misused to a point where you're sitting there at the screen and you're saying, "Allow, allow, allow, allow," and then allow all. Defeats the point. So you want to bring the human in control where the human makes sense, in a way that you still have their attention and they're not just skipping next, next, next.

[00:12:12] Daria: That's where risky action comes in. If you're thinking about the risky action, that's where you should put the guardrails. Don't give AI power quietly. And then permissions. Permissions is becoming a design act, and when I say permissions, I'm saying permissions in terms of who you are, which team you are in, or which organization you are in, what AI tools you can use, what you cannot use, what the controls are. These need to be controlled from a centralized unit, to make sure that everything in the organization is going according to plan.

Predictability

[00:12:48] Daria: And then predictability. As I said, you cannot 100% control what AI is doing, but what you can show is how confident AI is with a certain solution, or with certain information that it gives you. Remember the Meta example? It would have been so much simpler if the agent had said, "I'm not very confident, this is 10, 20% confidence," and then they would not have had that security incident.

[00:13:19] Daria: And then, when you're looking at what the generated content needs to be, or what AI needs to be, it needs to be resilient — resilient to errors, failures, inconsistencies, security threats. And I'm saying resilient, I'm not saying impossible, because it's for us to put these measures and for us to put these boundaries to make sure that we are looking at the risk, we're assessing the risk, and we're helping the user make the right choices by mitigating it.

Accountability

[00:13:53] Daria: Accountability. Whose fault is it when AI gives a wrong answer or does something wrong? That's the big question, and I think this is a big question for debate. Whose fault is it? And, surprise, it is the user's fault — the user who actually ran the system without looking at what the consequences are. But then it is also the designer's fault, for not showing what the consequences and the risks are that come with making that decision.

[00:14:27] Daria: So AI needs to leave a trail, a way to trace what happened, what went wrong, what decisions have been made, by whom, in order for it to be accountable. Users should know what they're generating, who's approving it, and what they're going to do with it. And that is not only about enterprise products. This is also in consumer products, and in different consumer products it could look like a small label: why are you seeing this, and what are the risks if you take this action?

Recovery

[00:15:02] Daria: The next one is recovery. With recovery, we need to be able to give the user a way to go back, to get back into the action and see what happened and recover it. They need undos, rollbacks, reporting. They need to be able to see a failure, and to see that the failure is recoverable and not a dead end where, oopsie, things fell and I don't know what to do with it any more.

It is showing up in consumer products too

[00:15:34] Daria: So, I'm standing here and I have a lot of experience in enterprise, and you would think this is an enterprise problem, that the EU Act is going to impact the enterprise people. I like where you're smiling, I think I know where I'm going here. We're seeing it in consumer products as well, and we're seeing it very slowly seeping into our day-to-day usage of these consumer apps. A few examples that I found, and I thought that they're quite interesting.

[00:16:04] Daria: One is what's happening with Instagram. For those of you who have seen this AI info: Instagram now labels if content has been generated by AI. It gives you AI info, an AI label, in order for you to know how this content has been generated.

[00:16:28] Daria: And then what I found interesting is the example of YouTube. They have this likeness button where you can go as a user and say, hey, this content has been generated by AI, this content looks like me, I want someone to take a look at it and I want this content to be removed. So it puts you as a consumer back in control of the product.

[00:16:53] Daria: And this is from Claude. If you're using Claude web, it actually tells you what they're going to do and you can approve. You can say yes or no, I want you to do that, or they can change their actions. So the issue of trust and the issue of building products with that trust element is not an enterprise problem any more. It's not only an enterprise problem. All the solutions, all the products that you're working with, AI-native products, they will have to follow this AI Act, and now we're seeing more and more products showing examples even on the consumer side.

Ten principles

[00:17:33] Daria: So now we've gone through the wave. Now we've seen all the magical, amazing AI products and the innovation, and we're pushing the boundaries. Now we're getting to a certain situation where the winners are not the ones creating products that will feel magical. The winners will create a product that feels, for the user, understandable, controllable and accountable. And we as designers are sitting in the middle of it. We are controlling what this user experience would look like.

[00:18:07] Daria: I wanted to leave you with something. I wanted to leave you with ten principles of creating with AI when it comes to security and compliance, or bringing the trust element into how users are using your product. So the first one is design for understanding, not only use. The second one is reveal critical complexity: show it to the user, let them decide. Design for human agency — humans need to be there. We're not designing AI for AI, we're designing products for humans, actually. AI is just a cherry on top.

[00:18:46] Daria: Design also for uncertainty. And design for failures. We are used to designing the happy path, but now we need to rethink what the different failures are, and how I design for the failures. Design transparency into the experience itself. It's okay if our designs or our interfaces don't look white and shiny and very clean. They need to look understandable.

[00:19:10] Daria: Design permissions as if they are ethics, because this is how you control what is happening, who's using what, to what degree they are using it, and what they're allowed to use it for. That's how you prevent these 22% of people putting their personal data into an LLM. So design for permissions. Design for accountability: who's accountable? Is it the one who created it, the agent, or the one who ran the agent? The agent itself? Design for accountability.

[00:19:45] Daria: And design responsible friction. Not friction that is so annoying that you will hit next; friction that makes sense. And design for earning trust over time. Building a product that users trust is not a one-day thing, and it's not a one-week thing, and it's not a one-year thing. Trust is earned over time, and you can do it by building these small things into your product, so that users can trust you.

[00:20:15] Daria: At the end, designing enterprise products and designing consumer products are becoming more or less bundled together, the same, when it comes to security and compliance. Thank you so much for listening. And I'm wondering — oh, look at that, I still have time.

Q&A

[00:20:38] Host: We have lots of time for questions then.

[00:20:42] Daria: I managed to do it before the time runs out.

[00:20:46] Host: Excellent. I can see already that we're getting some questions in, and I did love that. I was surprised by the 58, and looking at most of the hands, I was in the 80% bucket as well. So it is a bit lower. If you build in these tools, it'll improve. So: did you design the graphics for this presentation by hand yourself, or with AI? If the latter, how did you make it so good?

[00:21:14] Daria: Thank you so much, I really appreciate that. I was a bit nervous with the graphics of my presentation. I was asking some of my team members, what do you think, is it too much? So thank you. It's actually a mix. I personally use a mix. I like to design some things by myself and then take some artwork designed by an LLM or by AI. But it's a mix. I'm still a designer, I still design things by myself. Yeah, it's a bit weird to say that on stage, but I still enjoy the fact that I have text — I usually work on Canva — and I have this text that I have to manually align, and I think, ooh, this is so nice when it's already aligned. That is a small little pleasure. I still do that.

[00:21:57] Host: Excellent. As a non-designer you would not have liked my answer to that. When everything is built by an agent, the content is AI-generated and AI is the scaffolding, would the content even give a human experience in this scenario?

[00:22:25] Daria: If the entire experience is AI...

[00:22:31] Host: So, if everything's being designed and built by AI, how do you get the human back in there?

[00:22:37] Daria: The thing is, if you're using an agent, the agent has been built by a human. So the human, the one who designed the agent — unless AI is designing AI agents, and that didn't happen. No AI wakes up and says, I'm designing an agent to schedule meetings. I think there's always going to be the human in the center, and the human that is creating and orchestrating these number of agents that they are creating. So yeah, I think there's always a human. I don't think there's a situation, unless there's a specific use case that I'm not understanding, I don't think there's an agent without a human, because the human created the agent. But I'm happy to discuss and to get deeper into your question after.

[00:23:25] Host: Okay, great. We have one that a lot of people are voting for. How do you see Europe lagging far behind in the AI innovation race with China and the US due to its stricter regulation framework?

[00:23:38] Daria: I don't see us lagging per se when it comes to regulations and frameworks. The US is also bringing the California AI Act, which is very, very similar to the EU AI Act. China is a different beast, but even China have a new act that is coming as well. So I think when it comes to security, or when it comes to governance, we're seeing a lot of the countries putting a little bit more control on what is happening and what is allowed.

[00:24:09] Daria: There has been a big fiasco around the deepfake images, and they're going onto Twitter and stealing people's identity and so on, and I think this is what the governments are trying to put checks and balances on. What really happens when somebody takes my picture and creates another image and puts it on YouTube and starts making money out of it? I need to have my identity protected. So that's worldwide, every country is doing that, in terms of the governance part at least.

[00:24:43] Host: Yeah, it's a difficult one. I think with Europe and America there's a big gulf in funding, because these AI models need so much funding, but Mistral in France is doing something innovative in the enterprise space.

[00:24:57] Daria: And it's great. Mistral is really awesome. The only problem with it is it's a bit more expensive for it to be mass-produced, but I think it's very, very powerful, actually. I've used it and it's really great.

[00:25:12] Host: Excellent. Looking at — we've just had somebody jump into the lead again. Is anyone thinking about the environmental impact of AI? What are your thoughts on enterprise orgs taking the lead on setting standards for environmentally responsible AI?

[00:25:28] Daria: I completely agree with you. I think we are getting to — my personal opinion, of course — I think we're going to a situation where the power that is needed to power our usage of AI is getting bigger and bigger. The need is becoming bigger, so there are more data centers opening up, they're using more resources and they're using more water for cooling and so on. So I would love to see more enterprise companies putting a brake on what can be used and how do we use it responsibly.

[00:26:05] Daria: I know that some of the enterprise companies are trying to offset their footprint. They're trying to build data warehouses that are powered by clean energy instead, or they're trying to plant trees to combat what they're doing. I think that's a good step. Is that enough? I don't think so. I think we need to do more, and we all need to think very responsibly about how much we're using it, and the environment as well. So that's a really good question.

[00:26:34] Host: I think I read somewhere that knowledge is the one infinite demand item, so I think the usage and the demand is going to keep increasing, but I think there's starting to be some innovation, as you were saying, in different clean energy and things like that. So it's a challenging one.

[00:26:52] Daria: And one thing I would share: there are a few startups and a few companies actually here in Germany that I've met that were trying to create different batteries or different ways to charge and power these data centers, in order for us not to consume so much power. So there are a few startups doing this and I think they're amazing, and I think more startups should be doing that. Not only generating AI products, but also generating the infrastructure. How do we protect our environment from what we're doing?

[00:27:26] Host: I guess the positive is that more funding has gone into fusion in the last two years, apparently, than in decades before. Do you think that the ten guidelines you've provided are inspired by the original UX heuristics, like affordance of objects, visible system status, control, etc.?

[00:27:42] Daria: Yes, they are. When I was putting them together I was thinking, I didn't want to leave empty-handed, I wanted to leave you with something that sparks a discussion. And then I thought, you know what, the heuristics, the ten-step list that we had in the early 2000s, served us so well until now, and I really wanted to do something very similar, inspired by it. If I would think about it from an AI perspective, from the security part at least, how would it look? So yeah, they are actually inspired. Good catch on this one.

[00:28:16] Host: Excellent. Can you give examples of companies designing for ethical AI?

[00:28:22] Daria: Ethical AI...

[00:28:26] Host: I'm racking my brain as well.

[00:28:28] Daria: I think the term ethical AI is very vague. What is ethical AI? And I would turn it a little bit in a different way, where ethical AI means compliant AI and secure AI, where you know that your data is protected, you know that your information is protected. So I think the act itself is giving you a really good base and a really good framework for how you can do that.

[00:28:58] Daria: And I will answer the second one because they are quite related. The second one is talking specifically about Miro. At Miro we are seeing very similar requests from enterprise in terms of how they want to manage and control their environment and their users, and the list that I put up in terms of what the requirements are, these are very similar to what we see. So we have released, in Canvas, which was this week, a new AI security and compliance release, and a lot of these things that you saw today are part of the features.

[00:29:37] Daria: For example, you can set permissions, you can set very granular settings, you can set moderation, like what can go into a model versus what cannot. For example, for companies who are doing a lot of strategy work on the boards, you want to make sure as an admin that you're not putting information into the model. But how do you decide which information can and cannot? Because the companies are different. Certain words like gun and kill would be okay for a gaming company, but for a bank they're not okay. So you need to set this moderation and so on. So we have released a number of features that are very much around the list that I mentioned in terms of enterprise requirements and what they're asking us to do.

[00:30:22] Host: So can I just clarify? You have the base models like Claude or Codex, and you're putting your own kind of layer on top that ensures that your security and compliance and stuff is being adopted?

[00:30:33] Daria: We create the options for them. We create different ways for them to be able to use which model can be used in which feature, in which operation. We're working on bringing your own model as well, for certain companies. We also allow them the ability to say that certain teams that they know work on strategy, for example, are not allowed to use AI on the board whatsoever, or for confidential boards you're not allowed to use AI, or certain words are not allowed to go into the board, into the generation itself. And then you can decide what can be generated and what cannot be generated through AI.

[00:31:11] Daria: So some companies say, you know what, we don't want any image generation on a Miro board with AI, but we're okay with content being generated. It really depends on the company, and the trick here is to create controls that are granular enough but easy to understand, because you don't want to have a lot of controls that people get lost in. So the balance here is the tricky one.

[00:31:36] Host: Okay, excellent. I think we have time for one last question. So we've answered that first one. Designing for transparency and ethics is work that's usually invisible to leadership. What skills do you think a designer needs to develop in order to become more visible and impactful, comparing it with expectations that value visual more than designing for the holistic system thinking?

[00:32:02] Daria: It depends where you work, but what I'm seeing right now is that there is a lot more talk and a lot more conversation happening around the compliance and the security of it, because they're all feeling the same thing. Every SaaS company that is creating any AI product, they're feeling the burn. They create amazing features, they want to go sell it to enterprise, the enterprise are not adopting. And then they create even more features and then the enterprise are not adopting it. So now they've started seeing why compliance and why security is really important.

[00:32:38] Daria: Unfortunately, it's not an easy path. So if you're working in this field, you're playing life on hard, unfortunately. Your life is not so much easier. But I'm seeing more and more leaders that have started to care about this, and especially with the EU act. My advice for the person who wrote this: really get into understanding what is happening, because I think my prediction is the AI act, both in the EU and the US, is going to come like a storm in the next two years, and we're going to find ourselves in another GDPR fiasco, for the ones who have witnessed the pain and experienced it. So it is coming. Get ahead of the curve, really understand what that means, because that will give you a way to have these difficult conversations with leadership when it comes to compliance and security.

[00:33:30] Host: Excellent. So the wave is coming. Get prepared.

[00:33:33] Daria: Yeah, get prepared, exactly. Get prepared.

[00:33:36] Host: Excellent. That brings us to time. So thank you very much, Daria.

[00:33:39] Daria: Thank you so much.