Why Software Projects Fail #1

Nothing is delivered.

This is article number one in a series where we muse over various reasons one of the world's most well-paid professions often turns up nothing of value.

We're starting with the most obvious reason projects fail: nothing makes it into production.

Achieving absolutely nothing is hard work

• Risk-aversion.
• Silos and Ivory Towers.
• Perfectionism, endless deliberation, design-by-committee.
• The absence of a real problem.

When it's been 6 months and nothing's in production yet, with no change on the horizon, these are some of the usual culprits.

Risky Business

Highly risk-averse teams will do anything to avoid the liability incurred by delivering value. This is often an institutional / cultural dilemma. The roots go deep.

We see it a lot in Government - projects that receive immense funding from tax-payers but never see the light of day because "what if something goes wrong?". The money slowly evaporates in meeting after dull meeting, email after vague (but ironically verbose) email, as each gatekeeper presents their own unique barrier to the delivery team. With every obstacle overcome there is another waiting, and the team ultimately fails to get adequate buy-in. The runway is never clear. They cannot take flight.

The crippling fear is thinly veiled by a seemingly well-intended and bureaucratically executed mission to avoid costly mistakes. They ironically ignore the "cost of doing nothing", either because they don't conceive of it or they are content that the status quo is not their responsibility.

When there's a chopping block, people will do whatever they must to keep their heads far away from it.

Some systems are just broken. A culture change is needed, and usually rogue internal agents to catalyse that change. Very rare. In lieu of that, external vendors can be brought in to do any real work (this happens a lot in Government). In either case, teams that can really take ownership and operate outside the usual political boundaries may succeed.

Wrong Division

Software projects often span multiple teams or departments. It's impractical for everyone involved to be fully dedicated to a single project. Conflicting business priorities and the enterprise tendency to group people by a perceived "business function" means different initiatives will compete for the "resources" (i.e. people) they depend on. Sometimes these groups are highly separated, perhaps even fully departmentalised, with their own distinct hierarchies, and no one answerable to any another among them.

Day 1 "We're waiting on Security to approve the database access policy."
Day 2 "They've received the request, but they're swamped. Give it a few days."
Day 5 "We haven't heard back yet. I'll follow up with them this week."
Day 12 "I've escalated the blockage."
Day 14 "Still nothing. Can we work around it?"

Sound familiar?

Silos

This grouping of people by perceived "business function" results in what are referred to as "silos". It's rationalised by a misunderstanding about how software is actually (or ideally would be) made, but it can work as long as their success is measured largely by how they support business initiatives (i.e. other teams who depend on them). The structures governing those groups should optimise for that. Too often, though, each department is treated as a service-desk for the others. Tickets are flooding in, they're overworked, and lacking clear direction.

Silos form within some fairly predictable (yet still arbitrary) boundaries. It may go something like this:

• Business Analysts - Gather requirements
• Designers - Turn requirements into pixels
• Developers - Turn pixels into code
• Testers - Ensure the solution meets the requirements
• Infrastructure - Provision server-side components
• Architecture - Apply the seal of approval
• Operations - Deploy the solution and provide support

They may be further subdivided. For example:

• Developers
  • Frontend - Make it pretty
  • Backend - Make it work
  • Database Admin - Make it persist

At every line of division, one can reasonably expect some overhead. If the management structures fail to optimise, or (worse) increase the overhead, then a slow-moving assembly line emerges. This is where software projects go to die.

When these silos form a hard dependency for software delivery teams, there can be almost no certainty around when (if ever) a project will be complete. Ideally, they're given the autonomy they need to prioritse the work effectively, but this is a double-edged sword - politics inevitably emerge.

Arguably the worst configuration is when silos are granted autonomy and authority but no accountability for how either are exercised...

Ivory Towers

A common example of the above in Government and enterprise is the "Architecture Review Board" (or similar). These boards are often comprised of highly disconnected Ivory Tower types, exhibiting the aforementioned anti-liability posture and/or espousing highly dogmatic principals. Imbued with authority by the business, they call the shots but are sufficiently removed from actual delivery that they face no consequences for blocking it. They will simply reject ideas that don't satisfy their opaque criteria, deciding when (and if) they will offer consultation or even clarity.

Faced with such an opponent, delivery teams will typically either:

• circumvent the rules, opting into the associated risks, or
• relinquish their aspirations to the status quo, becoming yet more cogs in the slow-moving machine.

They could mount an assault, escalate to the highest C-level, but in all likelihood they'd be staging a mutiny against the captain's guard.

The best employees will down tools and leave these environments. The worst will stay, having no practical experience to take them elsewhere, and climb the tower; the next generation of blockers.

Without the shared goal of delivering successful outcomes, these people have no incentive to take on any responsibility.

One Team, One Dream

The expectation that software development can (productively) be broken down into atomic, semi-sequential tasks and passed between separate business units with no consistent ownership is part of the reason these phenomena exist; Silos of homogenous skillsets designed to service the perceived "assembly line" of software delivery; Ivory towers of dismissive, authoritarian nay-sayers and do-nothingers who can refuse passage to anyone without consequence.

The exact opposite is the real truth. There are knowledge gaps revealed in every conversation, new requirements in every user testing session, new bugs with every feature, alternative paths at every turn, new compromises with every constraint. Alas. Reality blows.

The solution? Cross-functional teams with outcome-oriented measures for success. No more silos or ivory towers - designers, developers, testers, and architects all working together, in the trenches. One person's fight is everyone's fight. So too is their victory... or failure.

When teams are formed around a common goal, there's no room for politics; just good old fashioned gettin' it done.

Imperfection Impasse

Risk-aversion may lend itself to perfectionism, but sometimes even teams that are keen to shoulder responsibility will succumb to perfectionst stagnation. They'd prefer to design everything on paper and "get it right" before committing too much effort. Some causes include:

• Seeking to satisfy too many peoples' needs with one solution.
• A desire to avoid wasting time.
• A fear of embarrassment or appearing incompetent.

The last two are easily debunked. Nothing is more wasteful than delivering nothing, nor more embarrassing. That first one, however, is worthy of elaboration.

Just as too many cooks spoil the broth, too many diners dilute the soup. Seeking to build generic software that is "everything for everyone" will almost always fail. Some distinct use-case will invariably necessitate some idiosyncratic variation that cannot be sufficiently abstracted into the broader design. Worse, some needs may conflict with others. If one person doesn't like chilli, no one can have chilli.

Start small. Focus on one problem at a time. Don't try to please everyone all at once. Piece together the bigger picture, or perhaps the many smaller pictures, systematically and patiently. Iterate and get feedback frequently, and be prepared to throw away, redesign, and/or rebuild. To close out the tenuous analogy: deliver one delicious bowl at a time.

Understanding emerges as the solution unfolds. The unraveling of assumptions (implicit or otherwise), the subtle optimisations, the "now that I've seen it, I've realised it won't work" conundrum, and the miracle strokes of "what if we just" genius. Getting it wrong is usually the first step to getting it right.

Want perfection? Learn to master what comes before it.

The No-Problem Problem

Sometimes, nothing is delivered because nothing needs to be delivered, because there is no real problem to solve.

Unable to pin down a satisfying challenge to overcome, a team will move from idea to idea, endlessly brainstorming, prototyping, and invalidating. This isn't so much a failure in delivery as much as it is a misdirection of energy. The delivery team is doing exactly what they should: consulting, discovering, prototyping, and validating (or invalidating). The problem is that the project sponsor has incorrectly identified an area for innovation. Turns out, everything's fine. What a shame!

Just put the tools down and be happy. There are undoubtedly other problems to solve, elsewhere.

Closing

That's the end of the first (and, so far, lengthiest) article in this series on Why Software Projects Fail. There have been many failed projects out there in the wild, and we've just scratched the surface. Hopefully this provided some insight, some signals to look out for, and some ways to correct course.

Until next time, keep shipping.