This is an editorial piece that expresses my opinions about the Agile transformation journey of one Software Development team. It should be read as a story told from the perspective of someone who has experienced degrees of failure and success in other such efforts, and now hopes to share some of the common elements that consistently appear on the teams that thrive. For now, I will reveal a single thread that follows this narrative, and hopefully survives any attempt to obscure it through details.
“Culture eats Technology for lunch.”
Software development is really hard, but not in the way that you might think. Technology is getting easier to select, adopt, and implement through our products and services. The hard part is trying to reconcile this against what the industry has unfairly and clumsily labelled as its primary tool – the knowledge worker. This mindset around how to harness people is dated, and reflects the managerial ethos from a time when your placement on a GANTT chart was all that was needed. People only mattered insofar as they possessed the required skills, and precious little attention was paid to how they worked together.
This series of blogposts will weave a number of experiential concepts together to help illustrate how our initial attempts to evolve a new software development profession has been misguided through the application of older paradigms. The story is told through the lens of a team that underwent an Agile Transformation towards a more highly productive team. Along the way we’ll look at various elements of a typical R&D organisation to enrich the series with the necessary detail. The pillars I’ll focus on are Agile Management Practices, Culture and Craftsmanship, DevOps, and Test Automation. For now, let’s try and establish why software development is so unique, and why it resists the urge to standardise and institutionalise like so many professions before it.
Journey to the Center of the Earth
Sometimes helping an R&D team mature over time feels like you are boring into the center of the Earth. You toil for months and years towards a planned route, and frequently fight off the nagging sensation that you aren’t progressing. You fail to predict the occasional ore deposit that requires expensive diamond drills to tunnel through, but you’ve passed the point of no return. Worst of all, you don’t quite know where you are headed, except that you are definitely headed somewhere. There are no maps with a large red ‘X’, and you harbor a worrying suspicion that you won’t recognize it when you get there. The elusive treasure chest is something our industry has been grappling with for decades.
Welcome to Software Development – the only profession whose governing force is Chaos theory. This theory predicts behaviours and outcomes of complex systems that are highly sensitive to the most minute of changes, which leads to strikingly different outcomes. Software projects are similar in that their outcomes are extremely hard to predict, and despite our efforts to do so, we have around 50 years of project management practices and chronic failure to demonstrate this with painful clarity. Until Agile arrived, our attempts to fix this focused on becoming even more prescriptive, which manifested and peaked in the form of goliath and unwieldy frameworks like Rational Unified Process, or the dreaded Capability Maturity Model. Although they still linger in fringe regulatory and compliance driven companies, the recent Gartner attention towards Enterprise Agile adoption should help clarify the growing dominance of this approach in the wake of a wave that crashed 15 years ago.
Agile as a Philosophy – not a framework
The only consistent element in these legacy software development frameworks is not apparent from what was common, but rather, from what was missing – people.
RUP, CMM, ISO, or Waterfall do not help us to understand the importance of Individuals and Interactions from the standpoint of culture – only as it fits within an org structure. Frameworks tend not to use ambiguous terms like principles, values or tenets. The knowledge worker within IT was almost regarded casually as a plug and play resource. We then handed projects over to managers whose only role was to track progress through an ongoing and repetitive dialogue of questions “what percentage complete are you?”. We then watched the asymptotic relationship between time and work done lengthen while languishing at 99% for far too long, surrounded by growing risk and cost to the project. This is the lemming phase of our industry, because everyone knew we were walking towards the precipice, but we didn't have the tools to address what was required of us, and we lacked the empowerment to make it happen. What worked for engineering projects for thousands of years clearly did not have the same success on a Software Development project.
Enter Agile in Snowbird, Utah 2001. For the first time, the leading professionals of the day assembled to discuss this, and the outcome reflects an important pivot around people. The only constant in projects lacking causal determinism and lacking predictability, are the people who deliver them. This is what Agile represents – our initial recognition that at the core of building software are principles and values that govern how individuals interact with one another, and a far better predictor of success today. Doctors have their Hippocratic Oath, Engineers have their Ritual of the Calling of an Engineer, and we now have our Agile Manifesto. The only difference is that we aren’t required to swear by it, but hopefully this will soon change, but more on that in a future blog post.
Being Agile is a mindset towards building software partially defined by the Manifesto. Be wary and suspicious of any urge to productize it with Scrum Certification, or scaled Agile Frameworks such as DAD or SAFe. These only work under the expert guidance of skilled practitioners, and only if you guide your teams to feel stronger accountability towards those principles than the practices that derive from them. Agile is an adaptive approach, and we sometimes forget that the feedback loop also includes the frameworks themselves, but we should at least acknowledge the perceived difficulty and cost of adopting an approach that is self-optimising by definition, especially for Enterprise. Hence why we prefer to characterise Agile as a philosophy that encompasses a set of values and principles, and not a set of frameworks that sometimes incorrectly manifest as dogma no worse than their predecessors. It is a subtle distinction that may just be enough in the grind of a working day, to take a step back to review how project decisions impact those principles, and to course correct your teams away from calamity.
Principle-driven engagements are very challenging, because the only reason for writing them down is because we don't understand them at first. In order for a team to achieve a degree of success, they have to be willing to suspend disbelief for long enough to allow the principles to settle. Only then will inspiration arrive. Until that moment, we have to exercise the courage and faith required to measure every decision up against them, especially when projects come under stress. Principles aren't there for convenience, they exist to preserve the integrity of who we are as software professionals, how we build products, and the value we build for our users.
Software Team Evolution Journey Map
If you are here, we can probably agree on a few things. Developing software is really hard, predicting outcomes is inherently difficult, and we need to pay far more attention to the people who deliver software over the tools and processes we use to do the same. The team evolution map below describes one such journey that started in the spring of 2015. It shows the timeline and the relative mindshare being exercised to solve significant challenges within our environments. The discontinuation of an effort does not represent our belief that we’ve mastered it, but only that it is serviceable enough to benefit our Software Development activities.
We’ll touch on each topic over the course of the blog series, but you might be instinctually reacting with nausea at the apparent degree of change introduced. One of the hardest practices to implement is that of rapid experimentation and the courage required to leverage failure as an opportunity. Humans have an evolutionary hard-wired imperative to be risk averse, and we tend to stick to what works. Although useful while scavenging for food within an ecosystem filled with predators who see our pale skin and meagre nails as an invitation to a buffet, it doesn’t help us now once completely removed from the harsh Darwinian reality of our ancestors. Feedback is essential to adaptive systems, and the pace of maturity grows with the pace of experimentation. However, it’s important to understand that the goal is not to stop experimenting, but rather, to derive better and better results from the provided feedback.