As many of us at Architech visit and work with a wide variety of organizations, common patterns have emerged regarding challenges with software development processes. We repeatedly find organizations with a process burdened with waste, high expenses, and software that is either buggy or misses the end user’s needs, or both. We’ve also worked with organizations that are embarking on “Agile Transformation” and need help. The driver for these initiatives is often improving software time to market.
We’re seeing the following trends in Canada today:
- Poor quality software. Many companies are struggling to release quality software.
- Expensive development. Software development is an expensive endeavour at the best of times, but in an environment of cost-cutting and increased expectations, organizations are re-evaluating the status quo.
- Release cycles are getting longer. Despite urgent demands from the business, major releases are moving in the wrong direction, going from two months to three, and from quarterly to semi-annually.
- Time to market is becoming a huge issue. This is now true for so many companies, especially in industries where a new breed of competitors like Google, Amazon, and a variety of startups are changing the rules around what customers should expect.
- Size doesn’t matter. These struggles are seen in companies regardless of size, whether small startups, mid-sized firms, or established enterprises. It’s often easier for smaller firms to change, and many are now adopting Lean Processes. The United States is generally ahead of Canada adopting Agile software development, with the US Government and a range of large US companies leading the way.
Value Stream Mapping
Value Stream Mapping (VSM) is an important Lean technique to help understand and visualize processes required to complete a product or service from beginning to end – and to see where problems exist. It’s a tool used to support the implementation of Lean strategies – including identification of waste in a process – with the goal of maximizing the overall flow. It involves drawing the current state as well as the desired future state and a plan to get there. Though VSM has its roots in Lean Manufacturing and the Toyota Production System, it has shown to be powerful in improving software development processes and often reveals hidden symptoms of larger problems.
In any process, we can think of activities as either:
- Value Add – any activity the customer cares about, changes/improves what is being built, and is done correctly the first time.
- Non Value Add (Waste) – all of the unnecessary activities that plague most organizational processes – the waiting, errors, lack of communication, extra paperwork, and so on. It notably also includes activities required for legal, governance and regulatory reasons that add no real value for the customer.
Here’s an example of a VSM for a typical waterfall software development process. Notice how the Value Add activities tend to be fairly short bursts followed by longer delays as the process winds through common governance “gates”. The overall efficiency is 30%, meaning 70% of the activities involved are wasteful, mostly in the form of delays, prioritization, reprioritization, handoffs, workarounds, duplication, documentation, etc. This diagram also shows Development & Testing as one box, implying 100% efficiency within, which is clearly not the case at any organization. This illustrates an important point about how the granularity of the diagram can expose or hide further opportunities for improvement.
Also interesting is a common situation, where the Business Unit will view the total cycle time (i.e. time to complete a single unit of production) as 315 days, whereas IT will typically view it as six months shorter since its process starts when funding is received. In fact, the overall cycle time could be substantially longer from the customers point of view, assuming what they want takes time to make it onto the Business Unit’s wish list.
How-to: creating a Lean Process
Whether an organization is looking to improve their existing (waterfall) process or move to something new, it’s important to understand what a Lean Process is, and how to derive more value from your development efforts.
A Lean Thought Process can be defined as follows:
- Define value from the customer’s perspective
- Map the value stream
- Create flow; eliminate the root causes of waste
- Create pull where flow is difficult to achieve
- Seek perfection via continuous improvement
Value Stream Mapping is therefore one of the first steps towards implementing Lean improvements to your software development life cycle. By understanding what activities deliver value to your customers and which are wasteful, you can implement processes that are more efficient, and then over time create an environment and mindset of continuous improvement to refine it even further.
There are seven common sources of waste in a system that are targeted:
- Overproduction (unused features)
- Inventory (too much partially done work)
- Waiting (bottlenecks, handoffs, approvals)
- Over-processing (too much documentation, relearning)
- Errors (defects)
- Motion (overburdened people and task switching)
- Transportation (too much email and other movement of data)
The goal is to remove all obstacles to flow – the units of production should never stop. These include rework, work in progress, bottlenecks, handoffs, setup, physical layout issues, and more.
So how does this help with time to market?
To speed up time to market, throughput usually needs to be increased and overall cycle time needs to be shortened. In Lean terminology, organizations must optimize the Process Time (the time it takes to actually perform the work) and reduce the Lead Time (the Process Time plus all the time spent on handoffs, delays and waiting for the next step).
One common finding is that the efficiency of the system when all activities are considered is less than 10%. In other words, something productive is being done to the work only a small percentage of time. Typically there are islands of activity (Process Times) within long Lead Times.
Any process is limited by exactly one constraint, which changes to another as the constraint is addressed. VSM helps to identify this constraint.
To drive faster time to market, the approach is to:
- Improve the throughput of the constraint (e.g. development), by going faster with existing capacity or expanding capacity.
- Run the whole process at the throughput of the constraint, reducing waste and work in progress.
As an example, the first constraint in organizations building software is usually development and/or testing, with a virtually unlimited supply of requirements. Employing a range of Agile techniques and adopting rigorous software discipline can help to increase development throughput. As development gets faster and faster, the constraint could then move to the Business Unit which can’t keep up with defining new (clear and thorough) requirements. Addressing that constraint could involve employing cross functional teams, reducing governance overhead, training on Agile product ownership methods, and so on. At this point, through a lot of hard work, the constraint might then shift back to development or perhaps release activities.
Shortening cycle time
VSM helps to point out the Value Add and Non Value Add activities in an organizations software development process. Shortening cycle time is about reducing the waste throughout, which often means a mix of fine-tuning and more dramatic changes. Some simple examples:
- Reducing the “batch” size of each release.
- Scheduling releases more often.
- Merging development and testing activities into one “stage” in a waterfall process, whereby testers are imbedded in the development team, and coding and testing activities run concurrently.
- Relaxing some governance steps in a waterfall process.
- Reducing the sheer number of high priorities running through the system, by getting better at prioritization.
- Admitting that not all requirements are really needed.
- Decoupling a system’s architecture and reducing dependencies in releases.
- Improving environment management to reduce the time and effort needed to spin up new environments and test effectively.
- Adopting Agile techniques (Scrum, Kanban, etc.).
- Building quality into the software to avoid defects.
- Implementing a range of software discipline techniques including test driven development, automated testing, continuous integration and continuous delivery.
Although implementing a Lean Process requires a high degree of discipline and can take time, getting started is relatively easy. Creating a VSM usually takes somewhere from a couple days to a couple weeks. When done formally, it involves:
- Quantitative analysis of the current state. Includes defining boundaries, value, tasks, flows, and resources. Delves into the as-is process metrics such as counts and rates, and often includes interviews of key staff and sourcing metrics where they might not already exist.
- Visualization of the “ideal” future state. Includes design of the future state map reconfigured to eliminate waste and add value.
- Creation of an implementation plan. Development of action plans to move from the current to future state.
It makes sense to ask for help from a firm that has created a VSM many times.
As Architech’s President, David’s focus on creative solutions is based on a strong belief that traditional software development practices are flawed, and his team routinely demonstrates a better approach with open source, Lean and Agile methods. Connect with David on Twitter or Linkedin.