The Path to DevOps is Paved with Efficient Intentions
Achieving efficiency across a scaled engineering organization means using systems-thinking, an approach that looks at how all of the components in the delivery lifecycle work together to create value. But designing a software delivery system that produces a continuous flow of value often requires a paradigm shift in organizational thinking.
Organizations can easily fall into the trap of allowing local optimizations at the expense the system, getting caught up on resource-level problems, attacking issues that don’t actually improve productivity, thereby hurting, not helping, the overall output of the system.
So how do we even know what the right problems to solve are? The answer is to focus on Flow: i.e. how work moves through your product development system.
Leaders that embrace Flow need to transition their organizations from thinking in terms of resource efficiency (optimization of components of a system) to system efficiency (optimization of the system itself). That intellectual journey typically looks something like this:
A: The Perceived Starting Position
Point A is where organizational leaders in traditional IT organizations typically see themselves: operating with high-resource efficiency.
Resources are optimized locally, looking maximize their ability to deliver on local objectives. “My job is to create widgets. I will create as many widgets as possible!” Each unit demands more resources, more budget, more incentives, all with the goal improving the efficiency of their function.
The truth is, though, the “resource efficient” organization is in constant suffering.
Actual time spent adding value is low due to a variety of factors: Poor handoffs; Broken workflows; Unnecessary approval processes; Poor communication; Lack of knowledge integration; Gatekeeping; Misalignment.
The reality is, there is no efficiency. It is an illusion. Any success is achieved in spite of the system’s design, not because of it.
Even if a finished product does make it out of the door, it will be loaded with defects, vulnerabilities and technical debt — knock on effects from the fact that flow across the end to end delivery of value is broken.
Here we find a key lesson from Lean — functional silos form a natural impediment to any kind of product development. They results in abundance only of waste — a raft of non-value adding activities that are disguised and rationalized as necessary to compensate for the disconnection between teams.
The organization must therefore transform — transition from its myopic focus on resource-efficient functional silos and onto the concept of a flow-efficient value stream.
Unfortunately, even when the well-intentioned leader is aware that there are significant issues with functional silos, there will be a reluctance or inability to change.
Why? Consider that if the status quo is working to some degree, why would you take the risk of changing? Why risk the existing work structures when new ways of working might not even deliver on their promises? Maybe there is still some faith left in the existing structures. Why bet against the ability to just fix the resource-based approach so it works for software? In the worst case, maybe the organization has already tried to transform and failed due to lack of leadership or an inability to break down functional silos.
The transformation process therefore starts with organizational leadership realizing that they are not at Point A on the Efficiency Matrix. Rather, they are already deep into the Wasteland of inefficiency, and the only way out is through transformational change.
B: Actual Starting Position
This is the reality of where most organizations are starting at — their resources are unable to deliver without incurring extreme waste. Local optimizations are incompatible with the velocity and responsiveness needed at an organizational level.
The first step in moving forward is therefore an awakening — the realization that the status quo is inefficient in all respects. Coming to terms with the fact that products cannot be built efficiently and effectively using functional silos is fundamental Flow thinking.
It is perhaps the most difficult part of the whole process because it means changing the perception of leadership, convincing people that they are not in fact at Point A and that it is worth taking the risk to transform.
This realization is often obtained by validating the systemic benefits of Flow with a single team or project to experiment and prove out the principles. Once you can show the techniques at work in a corner of your organization, you can then start talking about rolling out the approach on a large scale.
For this, a transformational leader is needed to clearly demonstrate the benefits to key decision-makers, obtain the all-important executive buy-in, and drive major change. That leader will build the momentum needed to transition through the next part of the journey.
C: Increasing Flow Efficiency
Path C represents the crucial movement towards flow efficiency, and the goal here is clear — transform the software delivery capability into an integrated system that permits free flow of work.
There is no blueprint for transformation success in achieving this goal; however there are some significant signposts and key guidelines that create a foundation:
- Transformational leadership motivating change
- A culture of experimentation and collaboration
- Full-stack teams with shared responsibilities
- Extended reach of Agile methodologies
- Deep integration of the development toolchain
The flow-efficient organization is designed to be well integrated and connected to allow resources to be used effectively in the context of the whole system. This is the fundamental idea and hallmark of Flow thinking that we have seen drive so many successful transformations in other sectors.
Once we are effective in obtaining flow efficiency to achieve system-wide integration, then we can start pulling levers and twisting knobs on the individual resources.
D: Increasing Resource Efficiency
In path D, the organization begins optimizing resources, but, importantly, as a function of systemic flow efficiency.
A key outcome of flow efficiency is the ability to intelligently develop cross-cutting data collection and reporting capabilities to provide fast feedback on system-wide effectiveness. This provides the traceability needed to identify bottlenecks and drive continuous improvement over the long term.
Now the organization can begin benefitting from the typical DevOps techniques such as continuous integration, continuous delivery, test automation, WIP limits and so on, with a clear view for how these changes affect the overall flow of the system.
Interestingly, if the integrations for flow efficiency are executed correctly, a “collect everything” approach is not necessary. Remember what W. Edwards Deming said, “Just because you can measure everything doesn’t mean that you should.” Using the visibility gained from your integrations — monitor and collect metrics that are highly targeted and verifiably useful for decision making.
E: Final State
The final state is not a resting state — the organization will continually come up against new limitations, customer requirements, organizational bottlenecks and market forces. Having flow efficiency allows the business to calibrate resources dynamically to respond quickly to emerging issues.
The interesting thing about the final state is that resource efficiency is not maximized — capacity is always maintained. This gives the organization the responsiveness required to deal with changes in technical confidence, customer requirements, market forces, defects and vulnerabilities, technical debt and other unexpected or unplanned work.
And here we come to the question of why an organization would want to undertake a DevOps transformation in the first place: the flow-efficient system is dynamic, responsive and able to adapt to change. It is one in which the effectiveness of resources is maximized to deliver the most value to their organization.
Final Thought
At its core, a transformation means changing the way an organization delivers value — i.e. making the choice to design an organization around system-level flow efficiency. This means:
- Integrating the software development lifecycle to obtain flow efficiency
- Calibrating resources in respect of the emerging flow requirements
- Continuously monitor and rebalance as necessary
Flow efficiency is the ‘True North’ in the journey towards building and managing a lean organization as a single system, one that continuously delivers the right value to customers. With this direction organizations can forge ahead confidently with a transformation strategy for developing a first-class software delivery capability.
Don’t find yourself in a situation where you’ve got good intentions but focused on the wrong things — grab some books on Lean and DevOps and set yourself up for success.
