Measuring Productivity of Software Engineering

Not too long ago I had a conversation with a friend and we discussed productivity in software engineering. In particular we got hung up on the question how to measure it.

It’s not that nobody had ever tried to measure it. I have probably about 30 to 40 books just on metrics alone. So far, however, I haven’t found a method that would meet my requirements. Let me explain.

Let’s assume you have a team that is working frantically and has a high productivity. Let’s further assume that you had a metric that can measure the productivity reliably. At some point the team ships the new system, not only on time but with a record breaking productivity maybe even ahead of time.

Enter the customer or even better the actual user. They sit in front of your brand new system. The initial comment: “This is not what I wanted. I don’t need a system for making hotel reservations I need a system that helps me control my production plan.” (OK, not a real case but I have to protect the innocent and I think it makes the point.)

In this scenario, what good is it if you have the best productivity on the planet if as a result you deliver the wrong features? Wouldn’t this indicate that in the end it is the customer who judges whether they get enough value for money? And wouldn’t that be a good metric for productivity? In this case, because the wrong system was delivered the “productivity” in terms of features with business value was zero. At least from this customers perspective.

In the discussion with my friend we really go stuck at some point. We had already agreed that counting lines of code (LOC) doesn’t help, or counting the number of classes, methods, function points, statements, screens, etc. Equally counting the hours doesn’t work.

My friend and I agreed on all of this. We couldn’t find a metric that we believed would work. Then I asked him how he measures productivity and his answer was: “gut feeling”. I’m not quite sure about his answer.

As for me I have decided that if several customers tell me that they believe they get good value for their money, e.g. for their maintenance fees, then I take that as a sign for a good productivity. Productivity measured as in business value perceived by the customer. Does that mean we now lean back and have a good time? No. We still use a continuous improvement process within our team to find even better ways of working, even more waste we can eliminate. And we continue to have a good time on top of the hard work!

Some Non-Cash 'Carrots'

Leadership sometimes requires finding a 'carrot' to make things happen. Not always does the 'carrot' need to be cash or other monetary items. Let me give you two examples.

In one case I worked with development team that wasn't keen to try pair programming. However, I observed that they were using pretty old computers for development and they were not happy with the performance.

Therefore I purchased a couple of high-end workstation, shiny, brand-new and fast, along with really nice large flat panel displays. I also got a desk for each of them so it would be easy for two people to sit next to each other on those desks. And finally I announced the rule that these boxes are dedicated to pair programming hence the powerful machines.

It didn't take long for the 'carrots' to show an effect. Slowly first but then more and more people tried it. Some only for a short time, some for more. And eventually I had to get more such machines and the team was totally hooked.

In a different case I was working with a team that was sitting on very old C++ code. The available tool set was appalling, and so I gave the team the direction to get onto Microsoft .NET, in particular C# where a much better tool set would be available, including good refactoring support, unit testing, and better libraries. And all the memory management crap would disappear as well.

Again it was close to impossible to get there. And some of the reasons given were just fine. For instance there were significant technical obstacles to make that shift easy.

Therefore I asked one of the team members to find out how to move the native C++ code to Managed C++. Sure, that wasn't straight forward either but it was an enabler and good step in the right direction. As a consequence it became much easier for the team to mix and match Managed C++ and C#. And since the tool support in C# was much better, more and more code was eventually written in C#.

These are just two cases where using an appropriate 'carrot' helped the team to move faster towards a desired outcome. Sure, in particular the second example is a bit special. Yet, I am sure that you are able to find more 'carrots' which work in your particular situation. I'd be interested in what you find out!

Constraints and Bureaucracy: Do you slow down your organization?

Most people I talk to will say that they want to keep bureaucracy at the minimum. At the same time they will say that a minimum set of policies, rules, and procedures need to be in place so that an organization can exhibit discipline and consistency across the board.

Agreed.


But then think of this: Your team interacts with lots of different other teams and individuals in order to work on projects. In addition there are cross cutting concerns such as administrative things (e.g. access to the building), HR (e.g. papers you need to hand in), Accounting (e.g. your last expense report), and so forth.


From each department's perspective they impose only a small little item (or two or three). And each item is just a very small thing by itself. And from that departments perspective each of the items is required and reasonable so they can do an excellent job.

The problems start when you pile the all the items up that come from different departments. Then progress in your organization may slow down significantly and may even come to a grinding halt.

So what to do? I have friends who have simply given up on some of these items. If they are given a spreadsheet to fill in they may decide to just throw in some (almost) random data (except if it is financial data). (Sorry, but I won't reveal names here to protect the individuals.)

Let's take an example: If asked to assess your team members and fill in 50 or 60 items in a skill matrix and given a team size of 15 or 20, we are talking about between 225 and 1,200 items to fill in. You probably need to think about each item for at least 10 seconds (this is a wild guess). But maybe you want to do a good job and do justice to the people in your team. Or maybe you want a realistic and true picture so you can be better at managing your project. So you spend 30 seconds on each item (Too much? Too little?). Then just this "simple" exercise amounts to 2 to 10 solid hours of your working time spent on a single spread sheet! Doesn't sound like much? Well, don't forget those other spreadsheets, reports, presentations, etc. that are waiting in your inbox!

So whoever thinks they are at the low end of bureaucracy, please think again! From a person's perspective who is affected by your actions the world may look entirely different, and your "small" request may amount to just that tipping thing that may keep those people from being successful. You may achieve quite the opposite of what you really wanted.

My recommendation would therefore be to consider for all your actions: How could you do less and still get a good enough outcome? (Or Google this: TSTTCPW. It applies to non-software engineering items, too.)

BTW: I will do the same myself and see whether I can't find an item where I might be causing avoidable overhead myself. I'm probably as guilty of this as anyone else....

"Healthy Noise": Introducing Automated Performance Testing

Starting with the current release cycle I have introduced automated performance testing for my project team. Not that we didn't test performance in the past, not that we didn't assess performance related items during a release cycle. But the fact that it is now becoming part of the automated development environment creates a number of interesting collateral effects. I'd like to highlight a few.

Reduced Latency

First of all there is the fact of performance testing itself. I strongly recommend not to wait until you are in the last few weeks or days before a release. That may be too late. If you discover a performance related issue then you may be forced to take shortcuts so your product meets performance requirements. And indeed in the past we already did assess the performance of the product throughout a release cycle.

What is different then? In the past we had to get performance engineers to set up the test, maintain test scripts, run the tests, analyze the results, identify root causes, and suggest solutions. Now the tests are written by developers and they are integrated into the test suite and then executed immediately after integration. No need to wait until a time slot with a performance engineer becomes available. The tests are now written and maintained by the developers. The performance engineer's time is freed up and in that time they are available to consult to the developers. The feedback loop is much shorter. A few hours instead of a few days or weeks. And this is also helps reducing risk since stories that may have a performance impact can be played earlier.

General Benefits of Automation

And you also get the obvious benefits of automation such as repeatability, lower costs, higher quality. Not that the performance engineers make mistakes consciously. As humans we are not fail safe whether we like it or not. So the major drivers for the automation were: cost, time, quality.

Impact on Behavior

But then there are less obvious effects caused be the introduction of automated performance testing. For instance I am observing that the thinking of the entire team is influenced by it. The performance aspect has been promoted and is not playing a much more important role in the considerations of the cross functional teams. Performance is built in instead of bolted on. Should a particular design or implementation cause a performance issue it can be dealt with immediately. Bad practices are no longer proliferated.

With our customers I am also observing an improved understanding of considering non-functional aspects such as performance engineering when deciding about backlog priorities.

And although it is not the main driver, and I didn't think of this aspect at all, there is also the impact it has on the developer. I sense that improving the development environment with automated testing has a positive impact on the morale of the engineering team. We continue to improving the development environment providing all people an opportunity to improve velocity while improving quality at the same time.

"Healthy Noise"

Certainly this change wasn't painless. The development teams had to negotiate with their customers the right amount of performance related work that needed to be accommodated in the backlogs. The automated build environment has to be extended.

We may have to purchase additional licenses for the performance toolset. I hoped I could get away with just the floating licenses we already had but that doesn't seem to pan out! However, now the bottle neck is no longer the performance engineer. Now it looks more like the number of licenses. When I compare the "price" of an engineer with the price of an additional license, it becomes apparent that the license is definitely cheaper.

Introducing automated performance testing caused some issues. But I would call this "healthy noise". All participants - customers, user experience experts, performance engineers, developers - are working very focused to iron out these hick-ups and they have made a lot of progress.

Wrapping Up

Introducing a somewhat significant change like this requires adaptation by everybody. Processes, tools, etc. need to be adapted as well. The result, however, is that you are moving closer to a holistic approach to software engineering that also considers performance engineering and testing as an integral part of the process. In particular I am very pleased how all the people involved grew as part of this process. The team and the product will be better because of this. Well done, folks!

Mandating Velocity

Assume you are in the middle of preparing your release plan. The team - product managers, developers, performance engineers, user interface experts, and other - has come back with a plan that is based on a velocity of let's say 3.56 per week.

You are certainly interested in any improvement of speed. So you are considering to say: What if you would plan for 4 per week? Or maybe you are even considering to mandate a velocity of 4. Would that be a smart idea?

Let's have a look back to the "good old days". A manager describes a piece of work to one of his engineers and asks for the estimated effort required to do the job. The engineer comes back and says that he can build it in 28 days. The manager thinks about it and then says: I think you can build it in 25 days. You will get 25 days to build it. I expect it by ... (fill in a date).

What's wrong with this? For one, if the engineer is not stupid next time he will add what he thinks will removed from his estimate (3 days) plus some (maybe 2 days) in case the cuts are higher next time. So the initial estimate might be 32 days. The manager is not stupid either. Next time he might think that the estimates are "inflated" anyways and that "sand bagging" happens anyways and sure people are playing ping-pong or darts anyways. So it's safe to reduce the estimates by 10% or even more. Just to get to the "real" numbers.

All of this leads to a situation where nobody works with the real numbers anymore. Every single estimate becomes unreliable and as a consequence every project plan built on them is flacky and unreliable on the outset. The engineer and - using the approach on a large scale - the entire team is set up for failure. (Unless you don't have a problem with "whipping" them a little harder and expect them to work crazy hours, which may not be a problem for as maybe you don't have family and don't know how it feels when you miss important events in the lives of your kids.)

There are certainly chances that the engineer may finish the work in 25 instead of 28 days. But at what price? If it really takes that long to do the job properly and if the engineer is really working at the maximum sustainable pace then what is left? Quality. The only choice the engineer is left with is degrading quality. This could mean that code is no longer reviewed. It could mean that design is no longer reviewed. It could man that some error handling code is left out (good chances it won't get caught in system testing). It could mean that fewer tests are written or no tests at all.

How is this related to the initial story around a velocity of 3.56 versus 4? Well even if you don't challenge the estimates, mandating an increased velocity is in essence the same thing. You are saying that people are not working hard enough, are in cruise-mode, etc.

Eventually the velocity will increase. The numbers will go up, they will even achieve 5, 6, or any number that you choose. Just by osmosis the team will learn about your "technique" and it will adapt. The estimates for every single item will go up. If an item was estimated to be 2 it will then be 3 (or even 20!). Then the velocity will increase not only to 4 but they will achieve a breath taking 35! Just imagine!

But seriously. If you are interested in working with the true numbers so that you have realistic and reliable plans that you can base important decisions on then you shouldn't mandate velocity.

A different question is certainly when there is a project that is very critical to the company. If you have treated your people with openness, honesty, and fairness, I'm sure they will understand if as an exception extraordinary measures needs to be taken. However, if those are used in every other release cycle, it becomes normal mode of operation and hence unsustainable.

It all boils down to how you select your "supplier". Are you interested in getting only the cheapest one, no matter what? Or are you interested in a reliabel supplier that continuously reduces the cost / increases the output / improves the quality? -As a customer as well as a manager this is your choice. (My take on this: Sacrificing quality has never ever paid off.)

To Pair or Not To Pair

Although I have been using pair programming as a standard practice with my teams since 2001 once in a while I still run into the question - usually asked by an uninformed outsider - whether pairing really makes any sense. The naive view is that it appears that two people are doing the work of one so it doubles the cost for most of the items.

Initially I was extremely adament about pair programming. It's called EXTREME programming after all! Quite some time ago I changed my position slightly. Today, when I start working with a team I make it clear that the default for doing any software engineering related work is to pair with a second person.

Does this mean that the people in the team I work with pair all the time? No, not according to my observations. There are quite a few occasions when people work by themselves.

One such case is if people would like to explore a new technology or tool. Different people prefer different learning styles. And before they present their findings to the team for further discussions most people want to understand enough about a topic before using their colleagues time. This gathering of information could be just surfing the web or reading a book on the subject. But it could equally be running a few experiments on a development workstation.

Then there are the usual other activities that most people prefer to do alone, e.g. email, filling in time sheets (yes, there are still a lot of companies who have them), etc. Also, you might want to call your doctor or spouse or log in to your broker without a colleague sitting next to you.

Unfortunately there seems to be only few studies out there on the subject. Laurie Williams' work is available in different versions, e.g. here and here, and also as a book. But some people would like to see additional researchers/authors who come to similar conclusions. Some studies have been conducted in an academic environment, e.g. with students. These studies are easily challenged when you try to use them in an industrial context. (Please note that I'm not saying these studies are useless.)

More generally I believe that reasoning about pair programming using a linear approach doesn't help. System thinking should be employed. Pairing (or not pairing) affects too many other factors. A simple linear cause-effect thinking tends to exclude critical factors.

Based on my observations and experience I would describe pair programming as an activity that has a positive impact on quality, team learning, soft skills, communication, productivity, innovation, creativity, collaboration, training, and many more. To pick one, let's look at productivity.

While many people believe that productivity goes done (two doing the work of one?), I believe that productivity actually goes up (or at least stays the same). My reasoning for that is - again - manyfold.

First of all, some people confuse body time with brain time. Have you ever walked through a development department where people work solo? Run a little experiment. Peek at the screens of the people and take a note of the percentage of people not currently looking at your IDE (or other work related tool). Now, do the same experiment with a team that uses pair programming as a standard practice. Or even better work with a partner and try to track your stock quotes at the same time... I think, you my point. With pairing there is no way to back out or to just doze off. You're in it full time. The ratio of brain time versus body time is much higher than with solo programming.

Next, a pair can and will be much more adament about the quality of the change set they deliver if they are members of a properly trained agile team. They will merciless refactor trying to use the simplest possible design and implementation to get the job done. They will be creative and innovative in doing so. They will have automated tests in place preventing other people from accidentally breaking their code. And once they have committed their change set they usually can focus entirely on the next task without having to go back to some past work.

There are other factors that are worth exploring, but I'll save them for now. I'd like to come back to them in a future post in this blog. As always, I would be very interested in hearing about your experiences and also about your feedback to this post. Thank you!