I was recently talking to a developer who shared this, all too common, story.

Sarah had her dream job. After having worked in small agencies creating almost repetitive, templated Wordpress sites she landed a senior frontend role in a well known, and well respected, eCommerce brand.

When she started she really dug into the code - its always easy to think that the people who wrote the code before you were either lazy or crazy because the code is a tangled mess of overloaded abstractions and weird hacks. Sarah didn't think like this though. She knew the pressure to release from her agency days and knew that corners needed to be cut from time to time to meet a deadline. Regardless of the health of the code, she made it her mission to understand how it all tied together because this was her code for the foreseeable future.

Not long after starting Sarah was assigned to a mid-flight project to introduce a new product line to the site. As she was working on a story she noticed a bug in the code. The bug hadn't been created by anyone on the project - it had been in the code for a while. While she didn't blame people for the state of the code she was going to take pride in her work and wasn't going to accept the current state of it. It wasn't a particularly difficult bug so she made the fix and then moved on. A week later she was assigned a bug from the QA team related to the feature that she was working on. The bug was that the previously broken item was now working!

Sarah went to chat to the QA team about the bug because it didn't make sense - how could fixing something be a bug? When she went to the QA team they showed her their test cases and the expected outcome didn't match now that the feature worked. Sarah explained that the expected outcome was wrong and that she had fixed a problem. Unfortunately, the team said that they can't change the scripts because that would be a change in the scope of the project. They told her that the BA owned the scope so he was the best person to talk to.

Sarah went to the BA to get him to tell the QA team to update the script and then laugh about how crazy this process was. That would have been a happy ending to this story. When chatting to the BA he told her about a previous project where they found the bug but it was "Triage Accepted" by the business so the QA team updated their regression scripts to mark the bug as the correct behaviour. Bringing the bug fix into the scope of the project would be a change now. He agreed that since the fix was already done it probably made sense to include it but the PM owned the change process so it would be best to speak with her.

You can imagine how frustrated Sarah was getting at this point. Still, she expected that the PM would realise how silly this situation was and that maybe she could sort out this process for future occasions. When Sarah raised the bug with the PM they agreed that it didn't make sense. But the PM wouldn't back her up because they were already behind schedule and she was putting pressure on the QA team to work extra to meet the deadlines. Introducing voluntary changes at this point would hurt morale. And besides that bug was assigned to the maintenance team, who ran a separate project, already - they would get around to it eventually.

Without any further options, Sarah re-introduced the bug into the code.

"A bad system beats a good person every time" - W. Edwards Deming

You might think that this is an example of a particularly dysfunctional company and that this is a pretty isolated incident. However, as part of the UXDX Community events we always ask an opening question on the biggest challenges that companies face. I have now met thousands of people from hundreds of companies and the second biggest problem that is consistently raised by people is that they are "Unable to fix problems in their product because it is not in scope".

42% of software delivery professionals highlighted being "unable to fix problems because they are not in scope" as one of their top problems.

How do situations like this arise?

To simplify a complex situation, it is down to how success is measured.

Most traditional organisations are structured into functional divisions or departments. These functions specialise in particular areas such as marketing, design, UX, frontend development, backend development, etc. If the project is expected to deliver an ROI of €5m but it only delivers €1m who is responsible? Marketing argue what they wanted would have delivered the full €5m but development didn't deliver. Development will argue that Marketing signed off on the spec and didn't provide enough input to clarify issues.

In a functional environment because there is no single owner the most important metric becomes predictability. Each function needs to say how long it will take them to complete their work and then they must stick to that at all costs.

Companies structured into functional divisions lack the accountability of delivery so they prioritise predictability over value.

Managing Predictability

To be able to predict how long something will take you need to be able to understand exactly what is involved in achieving it. This means that each function must have clarity on their work before they give an estimate or start working (when the clock starts). The business must think through all of their requirements, the designers must design every screen, the architects must create a solution that handles every requirement, the developers must code every acceptance criteria and the testers must validate every story before handing it back to the business. These handovers are cascaded from one function to another through stage gates with detailed documentation which verifies that the appropriate level of due diligence had been completed. Conway's law in action.

Trying to be Predictable in an Unpredictable World

This is the most efficient way of building software when all of the variables are knowable and known. Unfortunately, this is exceptionally rare in the real world. Leaving aside the customer and the market for a moment, which adds another layer of complexity, when a team tries to document all of the requirements for a system they will make assumptions. They are specialists in their areas - and not yours. They will not think to enter things which are basic for them and they will not even know about the questions that you need answered.

The result is like a company-wide game of telephone - the original intent of the request and the product that is returned at the end of a project differ dramatically.

If people identify a misunderstanding the most obvious approach is to try to fix it as quickly as possible, like the bug in the example above. But the stage gates of Waterfall mean that you can't go back to a previous stage without going through the Change Control process; words which can strike fear into many people's bones. But change control serves a very strong purpose - it protects the most important metric: predictability. Even at the expense of a better product.

Waterfall is a concerted effort not to learn while delivering software

The end result is that the focus on predictability harms the outcomes that we are really trying to achieve. If you are working in a system where the process acts against the outcomes you need to be prepared for when the inevitable bad results come in. This leads to CYA (cover your ass) or what I call a Teflon culture - the confrontational arguments, pointless posturing and the ridiculous number of emails and sign-offs that are imposed, which provide everyone with plausible deniability but harm the ability to actually deliver.

Senior executives will give everyone a dressing down but there are no fingers pointed too much in any direction. It's painful at the time but the teams forget about it just in time to pick up the next project and start repeating the same mistakes all over again.

What can we do?

The answer is fairly simple: make people accountable. How you do that is the hard part. You can't give someone accountability without authority (unless you want them to quit). But since projects cut across functions, how do you give authority to a single person? This leads to the opposite of Conway's Law - in order to change the process you need to change the organisation. Functional divisions make sense in low change environments. The efficiency in the functions outweigh the inefficiencies in the projects. But modern product development organisations are not low change environments. You need to shift the structure to optimise the efficiency of the cross-functional activities.

A product team structure, where there is a dedicated team of people from all necessary functional areas, is the most efficient in a high change environment. The team owns the product from cradle to grave. They do not disband after delivery but they stick around to learn about whether it worked or not and they react accordingly. The documentation handovers get replaced with conversations within the team, reducing invalid assumptions and miscommunication. Strict scope adherence is replaced by openness to change as it might improve the outcomes being targeted. And if outcomes are not achieved the team is responsible.

This is a major shift for people who are used to working in functional teams, particularly for the managers. This isn't easy but it is possible. Organisations need to unlearn the ways they worked in the past in order to create an environment that teams need to succeed. There are three core ingredients to ensure success - the motivation to change, the skills and ability of the teams on the ground and the ability of the organisation to enable the teams.

We've created the UXDX conference to help companies navigate this path from projects to cross-functional product teams who are empowered to deliver.  Our conference targets all three areas with motivating case study presentations from the leading companies, hands-on training on the latest best practices and techniques for organisations to set up the governance structures required to enable autonomous product teams.  

If you're on this journey I'd love to hear your thoughts and questions in the comments below. Every company is unique and the challenges subtly different so hopefully we can help each other. Best of luck!