Reg Mungomery had a problem. In 1925 he started with the Queensland Bureau of Sugar Experiment Stations  (BSES) and things were not going well. Sugar was big business in the early 1900s and Queensland looked like it hit the jackpot - it had the climate and the space to grow a lot of sugar. The problem was the greyback grub, a beetle whose larvae would eat the roots of the sugar cane and destroy the crops.

Over 10 years he had already tried fumigation and pesticides but these were expensive and the effects were only temporary. It was in 1935 that Reg was assigned a new project - to solve the cane beetle problem using a biological solution. Three years earlier a paper was published in Puerto Rico about their success solving their beetle problem by using cane toads from South America.

Reg ran the project perfectly.

• Analysis: Reg went to Hawaii to review the cane toads after they had recently been introduced there so see the benefits first hand
• Design: He identified the best release spots throughout Queensland to have the greatest impact and distribution of the cane toads.
• Build: He collected 200 cane toads from Hawaii and bred them in captivity until he had 2,000.
• Test: While breeding them he fed them cane beetles to ensure that they would eat the beetles.
• Deploy: He released the cane toads in the sites identified.

Overall the project was completed on time and with minimal cost compared to the insecticide costs. The problem was that the outcome of the project was an unmitigated disaster that has wreaked havoc on the Australian environment and is costing billions to this day to try to resolve. Not to mention that the cane toads didn't even eat the cane beetles because they resided too high up on the sugar cane compared to Puerto Rican beetles.

So would you classify it as a successful project or a failure? If Reg had discovered that the cane toads might poison native fauna should he have stopped the project?

How many IT projects are stopped when info comes to light?

What went wrong?

Reg did everything right. He followed a structured approach to the project and he adhered to the metrics he was being judged on - time and cost. But the outcomes didn't line up with what he expected. Reg assumed that past successes (Puerto Rico, Hawaii) would result in future successes in a new environment (Queensland).

Nature is complex; the ecosystem in a delicate balance. It is very difficult to cater to all of the variables when trying to figure out the impact of a change, such as adding a new species into a new environment. In the case of the cane toads, their natural predators in South America had the time to develop immunity to their poison. Australian predators did not. Unanticipated side-effects are common in complex environments.

Is your problem space complex?

When Dave Snowden and Cynthia Kurtz worked at IBM global services they created a "sense-making" framework to try to help people understand how to react to different classifications of problems. The result of their work is the Cynefin (Kuh-NEV-in) framework which identified five core problem spaces:

Simple

These are easy problems with easy answers. 1+ 1 = 2. Always has and always will.

You sense the problem (it's simple), categorise where it belongs and respond with the single best answer.

Complicated

Things are getting trickier here. What is the orbit of the Earth around the Sun? But this can be solved by getting some experts into a room, giving them the right inputs and eventually they will give you the answer.

You sense the problem (it's a tough one), analyse it to figure out the solution and then respond with your best solution.

Complex

People don't like complex problems. People like certainty and complex problems have no right answer. What do you want to eat? Well, this depends an awful lot on your mood and what has happened recently. These decisions are based more on emotion rather than logic. I could ask that same question every day and get a different response.

You can't sense complex problems initially. You need to probe - try something small - and then sense what reactions occurred. Based on the reaction then you can then respond with another probe.

Chaos

This situation is when all hell is breaking loose. You don't stop to analyse the problem or run small experiments to see how things might react - you act as quickly as possible to try to calm the situation down. Once calmer you can try to figure out the next steps - which category above does the problem fit into.

Disorder

Finally, there is disorder - when you don't know what state you are in. Chaos is pretty simple to identify but the other three can be muddled. Every problem starts in Disorder before it is classified, and this can be incredibly dangerous because you are susceptible to your biases.

• Managers want every problem to be simple with a constant process that anyone can follow.
• Professionals want every problem to be complicated - give me the time and money and I'll come up with the right answer.
• The world makes most problems complex. Anything that involves people is complex because you don't know how different people will react to the same action - or even how the same person will react on different days.

Treating a Complex problem as Complicated

Not all software is complex, but the fun stuff is! Once a problem has been worked on by many hundreds of teams over many years best practices emerge. Authentication, search, storage, image processing, messaging services and other product components have best practices. And in today's world, that means that there is a SaaS solution - Auth0, Algolia, S3, Cloudinary, Twilio, etc.. Unfortunately, these "solved" problems are not differentiators, they are the table stakes - the bits that are taken for granted by customers. The differentiators are the bits that are unique to your product. And these unique pieces do not benefit from years of experimentation. You are likely solving these problems for the first time.

For years the Waterfall process was the best practice in software delivery. It was modelled off the construction industry which is a complicated problem domain. Qualified engineers can give you precise concrete and steel combinations to ensure that the building will be able to withstand the forces exerted on it. They need time to analyse the bedrock and the weather conditions but once completed they can give detailed plans for the construction of the building. This seemed like a good analogy for software when the industry was in its infancy but unfortunately, software development is its own problem domain. A building has a fixed foundation - software continuously evolves.

When you have a waterfall process everything looks like a complicated problem. But that is dangerous.  

By treating complex problems as complicated we end up with products that don't meet the needs of our customers

Bad products leave us open to competitors as someone can solve the problem better for our customers, which is the biggest risk in product development. On top of leaving ourselves exposed, we have lost the opportunity to learn about our customers so that we can avoid similar mistakes in future.

How to manage Complex problems?

We need to shift from the "big design up front" model that we have in place for complicated problems to a more iterative sense and respond model which is more appropriate for complex problems.

Unfortunately, software delivery processes do not exist in isolation - they are supported by project initiation process, budget allocation processes, resource management, governance processes and individual social clout. In short, organisations are complex. So in order to ensure success when introducing changes in how we deliver software we can't follow the "big design up front" approach. We need to set a clear target for what we are trying to achieve and make small continuous changes towards that goal.

This is a Catch-22. If you want to introduce an iterative approach to delivery into an organisation that is optimised for Waterfall projects you need to use an iterative change process, which will be a challenge since the organisation is optimised for Waterfall.

Ultimately you need to give confidence that there are safe ways of delivering projects which contradict with the current governance processes. This is the purpose of the clear target - the high-level way of working that also covers governance, resourcing, accountability and funding.

I'll cover approaches to set these target conditions - for the product, the process and the organisation in future articles. The key thing is agreeing on a shared goal, focusing on principles rather than every detail on how they will work so that the teams can figure out the best approach through small iterative feedback loops.

If you want to learn more about how to create alignment within a team and with management so that you can start moving from "big design up front" projects to continuously improving cross-functional product teams check out the agenda for the UXDX conference.

If you're on this journey I'd love to hear your thoughts and questions in the comments below. Every company is unique and the challenges subtly different so hopefully we can help each other. Best of luck!