Standard Agile Process

A few days ago a friend described to me how in their company they were introducing a new development process. To go with the flow they decided to introduce agile methodologies at the same time. To make sure that all worked on the same basis they are calling their new process “Standard Agile Process”.

I think that is a contradiction within itself. Either it is standard or it is agile but not both. Let me explain.

In my view at the very core of agile methodologies is the ability to adapt to the environment. Many factors can influence the decision including people, experience, customer base, technologies, product, target market, time zone differences. Since 1999 I haven’t found two teams who used the same agile approach not even within the same company.

Given sufficient autonomy and authority a team will adapt as well as they can. Is it reasonable to expect that any two teams (regardless of being in the same organization or not) start their journey at the same time, progress at the same speed, learn the same things at the same time, have the same set of experiences and end up making the same choices? Evolution is never the same if the factors influencing it are different or if the timeline is different.

It’s even more complicated - or easier depending on your viewpoint. One of the teams I’m working with is using multiple process all based on agile principles, no two are the same. Factors that have influenced their choices are type of work, product and customer. And each of the processes (at least three) is evolving at a different speed following a different set of changes. Changes are applied as the team identifies the need for them, if needed multiple times a day. Occasionally a process stays the same for a few weeks.

So when you see or hear the term “Standard Agile Process”: If you are impacted by it, keep the above in mind. Don’t switch off your brain. Think for yourself. Speak up.

Or just have a good laugh in particular if your leadership team allows you and your team to be truly agile. You may or may not believe in Darwinism. But history suggests that those who adapt faster and better not only have a better chance of survival. They also tend to have a better life, i.e. are more successful and have more fun.

Lean Startups

Agile approaches are very much about adapting and using short increments when dealing with uncertainty. Starting a new company comes with a lot of uncertainty. Agile principles can help reducing risk and improving the odds of success.

Eric Ries is the inventor/initiator of the “Lean Startup” movement. His concept is to incrementally develop your customers and your business. By running experiments, some people also call them spikes, you learn a lot about the business idea you are working on. By using small increments you avoid spending months or years building a product or offering a service that in the end is fantastic but nobody is willing to spend money on. Eric Ries helps you de-risk your business idea.

A complementary technique is customer development a term coined by Steve Blank. While product development important (your product can also be a service) it is equally important to also develop your customer base. In his book “Four Steps to the Epiphany” he describes this process in details. His book is a work book, so be prepared that you will have to do homework.

Why am I mentioning these two? The work of both of them is based on agile principles. Therefore if you are considering to test one of your business idea then the work of these two authors should be part of your preparation. And both are not limited to new companies. If you are tasked to build that brand-new product in your company then you are basically running a start-up. The only difference is that you run it in the context of an existing company, which can be beneficial (e.g. financial backup) or a hindrance (e.g. bureaucracy).

Mass Customization

If you find yourself in a project where you need to deliver and maintain a large number of customized versions of an otherwise standard product you may want to consider designing your processes in support of mass customization.
There are a number of prerequisites that you’ll need to have in place. For one you need a base product that you want to customize. That base product needs a mechanism in place that allows customizing it. For example you could design it so it supports plug-ins.
Next you need a fully automated process for building the base product plus all plug-ins. Ideally you have a continuous integration solution and an extensive automated and virtualize test environment. The latter allows automatic instantiation of different target environments and testing of various product configurations in those environments.
One option for mass customization is then to create a custom package for each individual customer, e.g. by creating an installer containing the base product and plug-ins for just that customer. While the installer might be a good option for a shrink-wrapped product, continuous deployment in a hosted environment will typically benefit from a different approach. For example instead of packaging different installers, you might have a different deployment for each customer. Only the plug-ins for that particular customer would be included to be deployed in their environment.
You can drive this one step further, for example by providing a web site where your customers can select the base product and the plug-ins they want. If they are self-hosted they would receive the custom installer. If they are hosted their deployment in the hosting environment would be maintained accordingly. This web site could include integration with payments systems or with your internal accounting system, e.g. to check whether a maintenance payment was received.
Alternatively instead of having custom installers or custom deployments, the availability of plug-ins can also be controlled through the use of licensing in the deployed product. All plug-ins are installed but only the ones that were licensed are loaded and available.
Of course there are number of other factors that must be considered, e.g. how to design the process and the product so it can be upgraded without downtime. Once you have this in place, though, you will enjoy a scalable solution that allows mass customization.

Feature Branches and Release Management

In a few previous posts (here, here and here) I described how feature branches can be used to reduce project risk. As you move along there are further ways to simplify the use of feature branches.

For example I mentioned release management branches. Release management branches can be used for the final Quality Assurance (QA) work when preparing a release. Having a separate branch for this has the benefit that trunk (or master or main) doesn’t have to be locked down until the release day. The release management branch is controlled in that all changes that are accept to it, are carefully reviewed and specific to the preparation of the release. The rest of the team – that is everybody not involved immediately in the release management – can continue to commit new work into trunk.

That comes at a price, though. While working on the release management branch, trunk still requires work, e.g. merging new features. Alternatively you can neglect trunk while working on the release management branch. Even better is reducing or even eliminating the need for a release management branch altogether.

As a first step you could reduce the length of release management branches, e.g. from a week to a couple of days. Or you go all the way and just remove the release management branch altogether. This is feasible if you release out of trunk and have tight controls over what goes into trunk (“no junk in the trunk”). For example you could give your QA person the authority to decide when a branch is merged back into trunk. The equivalent in Git could be a pull request and the QA person decides which and when to accept a pull request into master. A review on the branch could be a prerequisite, for example QA could verify whether enough automated tests where written by trying to break the system.

By giving responsibility for trunk to QA and by minimizing the required release management tasks releases can easily be delivered out of trunk (or master). With tight controls trunk/master is no longer a “moving target”. Once you have eliminated the release management branch, the overall branch management and the release management has been simplified. A new release can be made from every single successful build out of trunk/master.

Feature Branches: Making it Easy

In my previous post I wrote about the benefits of using feature branches for quality assurance. As with all tools features branches don’t come with benefits only. There are unwanted side effects. Fortunately there are ways to minimize or eliminate them.

One such area is creation of the branches. There are various factors to look at. For one your version control system (VCS) should make it easy to create branches. For some systems this is a no-brainer and for other VCS’s some more work may be required. Most of the teams I work with use Subversion or GIT, and branching is not an issue.

When creating a feature branch, the VCS is only one of the tools that is affected. You also need to consider the client side of the equation. For example, how easy is it to switch between branches? How often do you need to switch between branches? What does the support in the VCS client look like? Do you need integration of your VCS client into the integrated development environment (IDE) your team is using? By choosing an appropriate VCS introducing feature branches becomes much easier.

Apart from the VCS other systems your development team is using may be affected as well when branches are created, whether they are feature branches or others. For example you may be using a continuous integration (CI) system such as TeamCity, Jenkins or CruiseControl.NET. Once the new branch has been created you want to make sure it is picked up by the CI system as well and automatically built. Therefore each time you create a branch you typically want to set up a new branch as well.

You may have other systems that may need to reflect that a new branch has been created. For example a bug tracking system may offer the branch name as the affected version when entering a bug. Or you may use a tool to plan and track progress for a particular feature. Again this would then be set up accordingly when a new branch is created.

As these activities will have to be done each time a feature branch is created, it is an obvious candidate for automation in particular if your team works on dozens or hundreds of features each year. With appropriate APIs of affected systems automation is not an overly complex task. Instead it is just a matter of putting the time in.

In closing I’d like to give you a specific example from one of the teams I am working with. Creating feature branches is completely automated in this case and the time required is only as long as it takes to type in the name of the features branch. The remainder is automated. This includes the creation of the feature branch in the VCS, setting up the new build configuration in the CI, creating the feature as a project in the project tracking system, adding a configuration to the test bed controller and setting up automated merging. Taken together it typically takes about 10 seconds and is saving hundreds of hours of development time per year.

In one of the next posts I’ll describe techniques that help making the merge of feature branches into the main development branch easier.

Feature Branches and Quality Assurance

An almost “classic” way of managing branches is to have a main branch into which all development work is committed. When a release is prepared a release branch is created. Quality assurance (QA), including testing and bug fixing, is done on that release branch. Any code changes are usually propagated to the main branch.

When introducing feature branches the question is whether you would still do all the quality assurance work on that release branch. You can, but in my experience there are better options available to you.

With the “classic” setup you also often see teams putting a lot of quality assurance work into the release branch. This tends to increase significantly the time from when you create the branch to when you release the software. This can lead to spikes in the QA workload. For example if you have planned 4 weeks for release management (QA plus other items required for release) and you are on a quarterly release cycle, every 3rd month typically has a higher workload for the team taking care of the release.

Therefore it might make sense to look for better ways to level the release management work. Feature branches give you additional options.

Since content wise a feature branch is your main development branch plus only one feature, some quality assurance efforts can take place in the feature branch and focus on that particular feature without having to worry about changes that may be in progress in other branches. Also, all QA efforts can start as soon as the first story has been completed on the feature branch. In this case “QA efforts” doesn’t mean that quality is tested into the product. Instead it means that certain tests, e.g. platform, installation and upgrade tests can be run very early in the project. Equally you can start with usability and performance testing very early, too. Quality assurance may also include feedback sessions with customers. By using prototype version from the feature branch, those sessions gain focus as well.

The general idea is to move in as many work items as possible from the release management process and to move as many items from the release branch to the feature branches. With this approach the time between creating a release branch and shipping the software can be shortened extremely, more in the range of hours or days. Since more options exists for quality assurance work, this also reduces or avoids altogether spikes in the QA workload of the development team.

There is one other item that requires consideration. Even despite all efforts in the feature branch you cannot guarantee that the system is not broken when the branch is merged back into the main development branch. You will want to have some integration tests in place to speed up that verification process.

In summary you are distributing the release management work across three places. Firstly you put as many of these items into the feature branches as you can. Second, you need integration testing for when the feature branches are merged. And you keep only the unavoidable release management task in the release branch that you cannot reasonably do in the release management branch.

In a future post I will discuss another technique for what you can do to reduce the effort and risk for merging a feature branch back into the main development branch.

Reducing Schedule Risks: Feature Branches

Software development projects are subject to a number of risks. One of these risks is schedule risk. By this I mean that you may not be able to ship on time because of some nasty discovery while executing the project.

Of course you don’t know what you don’t know. You cannot foresee what you will discover as you work through the project plan (or backlog). No matter how much effort you put in planning your project, breaking down the stories, even running some spikes, you will find that you cannot totally eliminate surprises that add to your workload.

Although we cannot totally eliminate that risk there are techniques that help with mitigating it. Firstly you can build in an allowance for discovery into your plans. Some people would call it a buffer.

A different option is to break your deliverable into multiple features. Often you will observe that most of the features are completed within the deadline while a small number may overrun. You will run into a problem, though, if you are using a single branch in the version control system (VCS) for your work. Your only option will be to deliver late once all features are complete.

Therefore a different approach is using feature branches. For each feature that you have scheduled you create a separate branch. Once the feature is complete you merge it back into your main branch, e.g. ‘trunk’ in Subversion. As you approach the deadline, e.g. the release date, you now have options. You can either ship with just the features that are complete or you wait until some or all of the remaining features are complete as well.

Of course this changes the scope of the release. However, if the features are worked on in their priority you may be able to ship the more important features even early or at least on time. This is sometimes a viable alternative to not shipping at all.

In the teams that I have worked with this approach is working quite well. There are a few more aspects to this but I’ll cover these in a future post.