Feed Me
My Social Links
Twitter Feed

Agile Transformation - a story told from the trenches

This is an editorial piece that expresses my opinions about the Agile transformation journey of one Software Development team. It should be read as a story told from the perspective of someone who has experienced degrees of failure and success in other such efforts, and now hopes to share some of the common elements that consistently appear on the teams that thrive. For now, I will reveal a single thread that follows this narrative, and hopefully survives any attempt to obscure it through details.

“Culture eats Technology for lunch.”


Software development is really hard, but not in the way that you might think. Technology is getting easier to select, adopt, and implement through our products and services. The hard part is trying to reconcile this against what the industry has unfairly and clumsily labelled as its primary tool – the knowledge worker. This mindset around how to harness people is dated, and reflects the managerial ethos from a time when your placement on a GANTT chart was all that was needed. People only mattered insofar as they possessed the required skills, and precious little attention was paid to how they worked together.

This series of blogposts will weave a number of experiential concepts together to help illustrate how our initial attempts to evolve a new software development profession has been misguided through the application of older paradigms. The story is told through the lens of a team that underwent an Agile Transformation towards a more highly productive team. Along the way we’ll look at various elements of a typical R&D organisation to enrich the series with the necessary detail. The pillars I’ll focus on are Agile Management Practices, Culture and Craftsmanship, DevOps, and Test Automation. For now, let’s try and establish why software development is so unique, and why it resists the urge to standardise and institutionalise like so many professions before it.

Journey to the Center of the Earth

Sometimes helping an R&D team mature over time feels like you are boring into the center of the Earth. You toil for months and years towards a planned route, and frequently fight off the nagging sensation that you aren’t progressing.  You fail to predict the occasional ore deposit that requires expensive diamond drills to tunnel through, but you’ve passed the point of no return. Worst of all, you don’t quite know where you are headed, except that you are definitely headed somewhere. There are no maps with a large red ‘X’, and you harbor a worrying suspicion that you won’t recognize it when you get there. The elusive treasure chest is something our industry has been grappling with for decades.

Chaos Theory

Welcome to Software Development – the only profession whose governing force is Chaos theory. This theory predicts behaviours and outcomes of complex systems that are highly sensitive to the most minute of changes, which leads to strikingly different outcomes. Software projects are similar in that their outcomes are extremely hard to predict, and despite our efforts to do so, we have around 50 years of project management practices and chronic failure to demonstrate this with painful clarity. Until Agile arrived, our attempts to fix this focused on becoming even more prescriptive, which manifested and peaked in the form of goliath and unwieldy frameworks like Rational Unified Process, or the dreaded Capability Maturity Model. Although they still linger in fringe regulatory and compliance driven companies, the recent Gartner attention towards Enterprise Agile adoption should help clarify the growing dominance of this approach in the wake of a wave that crashed 15 years ago.

Agile as a Philosophy – not a framework

The only consistent element in these legacy software development frameworks is not apparent from what was common, but rather, from what was missing – people.

RUP, CMM, ISO, or Waterfall do not help us to understand the importance of Individuals and Interactions from the standpoint of culture – only as it fits within an org structure. Frameworks tend not to use ambiguous terms like principles, values or tenets. The knowledge worker within IT was almost regarded casually as a plug and play resource. We then handed projects over to managers whose only role was to track progress through an ongoing and repetitive dialogue of questions “what percentage complete are you?”. We then watched the asymptotic relationship between time and work done lengthen while languishing at 99% for far too long, surrounded by growing risk and cost to the project. This is the lemming phase of our industry, because everyone knew we were walking towards the precipice, but we didn't have the tools to address what was required of us, and we lacked the empowerment to make it happen. What worked for engineering projects for thousands of years clearly did not have the same success on a Software Development project.

Enter Agile in Snowbird, Utah 2001. For the first time, the leading professionals of the day assembled to discuss this, and the outcome reflects an important pivot around people. The only constant in projects lacking causal determinism and lacking predictability, are the people who deliver them. This is what Agile represents – our initial recognition that at the core of building software are principles and values that govern how individuals interact with one another, and a far better predictor of success today. Doctors have their Hippocratic Oath, Engineers have their Ritual of the Calling of an Engineer, and we now have our Agile Manifesto. The only difference is that we aren’t required to swear by it, but hopefully this will soon change, but more on that in a future blog post.

Being Agile is a mindset towards building software partially defined by the Manifesto. Be wary and suspicious of any urge to productize it with Scrum Certification, or scaled Agile Frameworks such as DAD or SAFe. These only work under the expert guidance of skilled practitioners, and only if you guide your teams to feel stronger accountability towards those principles than the practices that derive from them. Agile is an adaptive approach, and we sometimes forget that the feedback loop also includes the frameworks themselves, but we should at least acknowledge the perceived difficulty and cost of adopting an approach that is self-optimising by definition, especially for Enterprise. Hence why we prefer to characterise Agile as a philosophy that encompasses a set of values and principles, and not a set of frameworks that sometimes incorrectly manifest as dogma no worse than their predecessors. It is a subtle distinction that may just be enough in the grind of a working day, to take a step back to review how project decisions impact those principles, and to course correct your teams away from calamity.


Principle-driven engagements are very challenging, because the only reason for writing them down is because we don't understand them at first. In order for a team to achieve a degree of success, they have to be willing to suspend disbelief for long enough to allow the principles to settle. Only then will inspiration arrive. Until that moment, we have to exercise the courage and faith required to measure every decision up against them, especially when projects come under stress. Principles aren't there for convenience, they exist to preserve the integrity of who we are as software professionals, how we build products, and the value we build for our users.

Software Team Evolution Journey Map

If you are here, we can probably agree on a few things. Developing software is really hard, predicting outcomes is inherently difficult, and we need to pay far more attention to the people who deliver software over the tools and processes we use to do the same. The team evolution map below describes one such journey that started in the spring of 2015. It shows the timeline and the relative mindshare being exercised to solve significant challenges within our environments. The discontinuation of an effort does not represent our belief that we’ve mastered it, but only that it is serviceable enough to benefit our Software Development activities.

We’ll touch on each topic over the course of the blog series, but you might be instinctually reacting with nausea at the apparent degree of change introduced. One of the hardest practices to implement is that of rapid experimentation and the courage required to leverage failure as an opportunity. Humans have an evolutionary hard-wired imperative to be risk averse, and we tend to stick to what works. Although useful while scavenging for food within an ecosystem filled with predators who see our pale skin and meagre nails as an invitation to a buffet, it doesn’t help us now once completely removed from the harsh Darwinian reality of our ancestors. Feedback is essential to adaptive systems, and the pace of maturity grows with the pace of experimentation. However, it’s important to understand that the goal is not to stop experimenting, but rather, to derive better and better results from the provided feedback.


Making a Business Case for Refactoring

Here is the business case i recently made for a necessary refactoring in our platform that emerged after the project scope was approved. This may be of additional interest to others that consider Business POs to be the sole source of prioritization in a complex matrixed organization.

The Epic Story can be described as:  “As a developer, i want the cost of adding a new <product> to the platform to be as cheap as possible, so that we can deliver to market quickly and so that we can deliver on our promise of high quality.”

Cost of refactoring = 50-75 SP

Reduced cost for adding future products= –20SP

The immediate response from Product Owners or Project Managers is to balk at the cost and to bemoan the effect it will have on approved delivery dates. Typically, after considerable humming and hawing, handshake deals are made to preserve the Iron Triangle without admitting to the quality concessions that are necessary. Where have we seen this before? It almost sounds like an endemic decision tree that perpetually afflicts our industry.

The impact this decision has on quality is worse than one might believe. The cost calculation is compounded with each successive backlog that does not implement this new software pattern. That is to say it is exponential, not linear.

Let's look 2 years into the future and assume we'll deliver 3 products / year.

Base Cost of implementing new product with older pattern = ~40SP

(+) escalating cost of low maintainability code = x^3+10

(+) escalating cost to quality and QA efforts = x^3+10

If x=1 for the first product of that year, then this project will incur an additional 22 SP in rework, for a total of 40+22 = 62 SP.

x=2 --> 36 SP

x=3 --> 74 SP

x=4 --> 148 SP

x=5 --> 270 SP

x=6 --> 452 SP

Cumulative Cost = 1000 SP (you can probably argue the math and the numbers used, but you cannot argue the exponential decay to quality)

What this leads to, of course, is another House of Cards Platform waiting to happen, and the resulting paralysis that is often faced when trying to approve a new project. This is a pattern of decay that a lot of organizations follow, and these are the types of decisions, when not acted upon, that lead  to large 6000+ SP replatforming efforts. Perhaps we finally see how the cost of Refactoring now seems small compared to the alternative.

This is why Engineering/Architecture will always have the legitimate case to push for 'Technical' POs on a given backlog. Tech debt has increasingly been associated with 'optional' engineering work to help address token quality concerns, but the truth is that we are trying to independently select work that has a definitive impact on business value - albeit through reduced cost and time to delivery, if not directly through user NPS scores. We are ensuring that the derivatives market for an organization remains healthy, if not the present stock price.

So, we have 2 choices here. We either pay for the cost of this refactoring now, or we pay for it later. The cost in choosing the latter paints an awful picture above.

1. The trade-off is between short-term business need (revenues / time to market) with intentional decrease in platform quality (must pay for it later)


2. Long-term health and stability of the product while possibly missing initial product launch dates

I'm ok with either decision, as long as we understand the ultimate cost to the future of our products, our quality, and our delivery times. Ultimately, the business must make the best decision here, and with enough trust between these groups, i am confident that it will always be the right one.


Business Value & Quality are the same: Convincing the Organization

Seeking and engendering accountability amongst an agile team is one of several difficult transformations within the maturation process. Unfortunately, some of us are still of the belief that accountability can be 'encouraged' or influenced from a motive outside the team. Although this approach may ultimately work, it's far more likely to fail, because the only lasting form of accountability, is that which the team builds within itself.

For every developer on a modern agile team, the motivation to achieve a date is still important, but what has now ascended above it, is the motivation to build a high quality product at a sustainable pace. Do no devalue the pride a developer feels in delivering high quality craftsmanship, as opposed to just simply delivering it on time. The irony here is that the business ends up with far more intrinsic value with the former. Business value and quality are one in the same, and cannot be separated. We have 40 years of software development to back up this claim.

And so, yes, i too would love it if teams could report on their status at the demo, and to explain why certain stories weren't completed, but with respect to time and maturity, we weren't able to make it work. As long as that explanation does not take the form of an apology to an external source, because if accountability is flowing upwards and outside the team, we have only paper accountability that crumbles with the slightest pressure. Bravery and not fear must be the prevailing emotion on the team.

Our current answer is to put our faith and trust in the Agile leaders of those teams, and to invest in their training. A team that is forced to adopt scrum without understanding the underlying principles and values, is a listless team with unpredictable quality and productivity. We may ultimately force them to meet dates, thinking it is the right thing to do, but at what cost? The Agile Leader is responsible for building a culture where both quality and execution are sought, and where much pride is taken from achieving both.


Manager 2.0

Managers in the IT world are slowly realizing that the previous model of command and control is a poor bedfellow to sucess. Forty years ago in a new industry that had little history and wisdom to draw upon, the tendency to resort to a more traditional, and disciplined model, was natural. Comparisons with the Engineering disciplines, and all the pressure we put on ourselves to achieve ISO certification, or CMM level 5, bestowed upon us an unhealthy expectation of certainty and predictability, where little exists, and indeed, where little is expected.

Agile forces us to focus on people and culture as the conditions for success. However, this success is often predicated on an organization's ability to transform culture throughout the older and more traditional hierarchy. The following article describes one aspect of this transition for the former Manager of the development team. How will this role change to accomodate self-organization and team accountability? Although it doesn't seem to fit into the classic Scrum model, we are increasingly finding a desperate need for leadership that applies itself differently, and towards a more noble goal - guiding and mentoring a team. If success is more often defined by the people who participate in it, rather than the technology or solutions selected, then it makes sense for a Manager to focus on this as well.

The days of General Patton like C&C are over. Wielding people like units on a battlefield has lost its appeal, and instead, our industry is transforming towards Servant Leadership and Management 2.0.





Graphical Resumes?

I just read an article in LinkedIn that really has me challenging the traditional recipe approach to creating resumes. Knowing that the 1st level of filtering is normally automated by a resume-bot that looks for keywords, we intentionally manipulate the mechanism by ensuring that these keywords appear in our own resumes.

.net / team foundation server / sharepoint / agile / tdd / scrum

What if we splashed our resume with a liberal amount of charts and graphs that portray our experience and skills in a unique and creative way? Imagine showing the breakdown of your skills in a 3D pie chart. You satisfy the word and acronym hunger of resume bots, the resume stands apart on a desk next to the .txts and .docs sitting next to it, and it also declares something about your personality for employers that aren't looking for the granola time-stamping worker. The big question for me is - would this work within the Software Development industry?