How many times have you heard - or thought to yourself - things used to be much better before? We used to be able to release quickly but now there is so much overhead and things move at a snail's pace!

You're not crazy - things do slow down over time. There are many causes but I'm going to delve into three of the main causes over the next few articles.

• Predictability
• Complexity
• Scaling

Let's start with Predictability

When asking for money for a project you need to explain how much it will cost, how long it will take you to build it and how much money you will get in return. This means that you need to have estimates for work and then teams must stick to those estimates. Put another way, people want software development to be predictable, but since we never build the same thing twice we are presented with a challenge - how to be predictable when doing something novel? To solve this challenge we need to overcome three gaps:

• the knowledge gap - the information needed to solve the customers' problem
• the alignment gap - ensuring all stakeholders have the same understanding
• the effects gap - achieving the outcomes for the customer and business that you expect to achieve

Knowledge Gap

At the start of a project there is a lot of uncertainty about how you are going to solve a problem. In a traditional project you would invest time into a dedicated Analysis phase, followed by a Detailed Design phase in order to close the knowledge gap. The larger the problem the larger the upfront phases need to be. And as I highlighted in the 6 hidden costs of large waterfall projects, the way traditional projects are managed there is a bias towards larger projects.

The Alignment Gap

The Detailed Design phase is typically completed by an architect or technical lead. But the actual software creation is done by a different team of designers, developers and testers. To ensure alignment with the intended solution a detailed design document is produced which outlines how the product should be built.

Effects Gap

After your software is released a benefits-realisation phase (sometimes) takes place. And this is where people start to notice some problems. Up to 66% of features fail to deliver the value that was expected.

Given the poor results, something must be done.

The typical response

Lessons Learned workshops or Retrospectives are held to understand where things went wrong. These sessions can highlight problems from across the delivery cycle such as lack of engagement of certain stakeholders early enough, more detailed analysis required in particular areas or designs that didn't adequately solve the stated problems. Or any others of the millions of challenges that occur.

The usual response is to put in place more processes or steps into the Analysis and Design phases to ensure that the Knowledge and Alignment gaps are reduced for future projects. And these steps take more time...

If the steps actually worked and projects became more stable and predictable over time then it would be worth it. However, after 30 years of learning lessons, we still have terrible success rates. So our attempts at improving the process aren't working.

There is a great concept called the Cone of Uncertainty in Project Management. The graph shows that you only know what you need to build at the end of the project. At any point prior to that there is uncertainty. But I look at this a different way - when doing our lessons learned at the end of a project we know of all the twists and turns that we had to make. We can create a story looking back about how we could have predicted each problem we faced - essentially falling into the Hindsight Bias trap. But the items that were identified get locked into the process for every future project going forward, even if they were unique to this project.

And with that, the templates and processes bloat over time until things slow down to a crawl.

What can we do?

We need to break free from our current process that tries to impose predictability on an unpredictable problem. We need to accept the inherent complexity in software development and design our processes around this uncertainty.

When you accept that software development is complex, you shift from thinking about it as a problem that requires more analysis to thinking about it as a problem that requires small, iterative tests to validate ideas and learn more about the problem space before continuing. By breaking a large problem into small increments you can close the Knowledge gap more easily and Alignment is simpler as well. But the real benefit is in the ability to track the Effects more closely and change track if you realise that your original idea is not working.

This seemingly simple suggestion, to reduce upfront planning and replace with continuous iteration and experimentation, has far-reaching consequences. Instead of focusing on building a product you need to get people to focus on solving a customers problem in a way that delivers value for the business. Instead of preparing large documents for each stage, you work together in small teams to get the necessary alignment. And you need to track outcomes against your expected metrics continuously to ensure that you are heading in the right direction.

Simply put, you need to shift from working on projects, that deliver outputs, to working as a product team, that delivers business outcomes.

Easy to say but really hard to do because the company will fight back every step of the way because no process exists in isolation. While the iterative approach delivers better results, you can't say with certainty what the end product will be,  how much it will cost or how much time you will need? Or even how much return the company can expect? And even if you do get funding, how are people who are used to working with large, complete problems expected to break them down into small, incremental pieces, and track their success?

Arguments from management will be that you can't make any changes because the people aren't ready. Equally, outputs need to be defined because the teams don't have enough understanding of the business context to make the right decisions. Arguments from teams is that management won't give them the trust and autonomy to let them demonstrate that they can deliver. And in some cases, you will run into people who don't want to change their ways of working.

How to move forward?

You need to set up a small team to prove that the product way of working is more efficient. In a project way of working, alignment across the organisation is handled through project prioritisation conducted by senior management. These managers are using their knowledge and experience to choose projects which deliver the best returns or solve the most pressing business and customer problems. But this doesn't scale when teams are trying small experiences and moving quickly. To ensure that the teams can make the right decisions, management teams need to share the business priorities and customer problems with the teams so that they can use the same information to make the best prioritisation decisions.

In addition to the business and customer alignment, teams need to align on the processes that they use - both continuous delivery and continuous improvement. Managers are being dragged in multiple directions so they do not have enough time to focus on fixing the problems in each team. By aligning the team on the process goals that they are trying to achieve the teams can innovate more and rack up the compounding benefits of improvement. This focus on continuous improvement helps teams to identify the skills gaps that are preventing them from shifting further into a product way of working and empowers them to upskill as required.

Alignment leads to better decision making in teams and helps teams to self-manage their transition to a product way of working. But this is easy to say but hard to do. If you want to learn more about how to build the necessary alignment and execution skills check out the UXDX conference where we have stages dedicated to each of these topics. The Vision stage focuses on the business, customer, process and team alignment whereas the Execution stage focuses on the skills teams need to deliver.

But predictability isn't the only problem that causes projects to slow down over time. Subscribe to get part 2 of the series where we talk about how projects slow down as the size and complexity of the product grows.