Two More Thoughts About Selecting Tools

In a previous post I shared a thought about selecting tools, in particular whether you should decided for a tool just because you happen to be a partner of the vendor.

Here are two more thoughts about selecting tools.

Driving Towards A Decision

The first one deals with ensuring that the decision making doesn't take forever. Sometimes teams discuss about tools for weeks or months without coming to a conclusion. As a result they don't have the tool. And as a consequence they don't enjoy the benefits.

For instance you are considering adding a particular type of test tool to your tool set. Of course there is the danger that you may select the wrong tool (I'll discuss below how to mitigate that risk). On the other hand even a tool that is not a perfect fit but at least does a perfect job is better than no tool at all. No testing tool is even worse than a testing tool that may be usable only for very specific test cases.

Therefore it is important to drive to a decision. And the best technique is time boxing. Give your time a time box and insist on a decision by a specific time. Only extend the time box if an excellent reason is given. That way as a leader you can steer the team towards a decision.

And if the the team can't come to a conclusion? Well, you can always use the threat to make the decision yourself....

Homeworks

While it is important to reach a conclusion you also want to make sure that the team makes a proper assessment of the different options. I usually expect that the major player are being looked at and compared. As a tool I recommend a decision making matrix by which the columns are the different products and the rows are the selection criteria.

To get the team started I describe to them how the tool works. Don't assume they already know. Make sure they do. Next I also tell them that I won't provide them with the full list of selection criteria but that I expect them to choose those as well before they start the assessment. I guess the management speak for that would be "empowerment".

Usually I make a small number of selection criteria mandatory. For tools these are:

• Ability to be launched from a command line
• Ability to deliver the result as an XML file (or other standard format)
• Price for chosen configuration, e.g. based on number of license required

The first two items are important from my perspective since a tool needs to be easy to integrate into automated processes. By using command line and standard-format based files (XML preferred) only little 'glue code' is required if at all.

Why the decision making matrix? First, you will notice that the team will then have a recipe for their assessment. They know what questions to ask and what to find out about the tool. Second, the team will gain an overview of the major players for each tool. Often team members have different backgrounds and experience with different tools. By working together the decision will become data-driven and facts-based as opposed to 'I like tool A more than tool B'. Also the team will have a basis on which they can have a meaningful discussion. The result is much more detailed and specific.

And finally, you will observe that all the sudden all major vendors are looked at and a better decision is made. You can take the outcome including the decision making matrix to create the business case that you may need for the purchasing process anyways. And even if such a business case is not required it still is an excellent exercise to go through some number crunching to determine by when a particular tool is fully paid off and what ROI it creates.

Decision Making Styles

Making decisions is probably one of the most important activities of any leader. There are plenty of different ways to make decisions. I like the way how it has been described in Peter Senge's book "The Dance of Change":

• Telling: Make a decision and just tell the team.
• Selling: Make a decision, explain why you made that decision, then tell them to do it.
• Testing: Identify a problem, present your solution and ask for the team's comments.
• Consulting: Identify the problem, tell them you don't have a solution and ask them for their suggestions.
• Cocreating: Identify the problem, then come to a decision as a team.

While cocreating probably creates the most buy-in from your team, you may have to make decisions using one of the other styles occasionally. For instance you may have become the team leader only recently and a problem is so pressing that any further delay creates a lot of additional damage. Then it might be right to just make a decision, explain it and then have the team execute it (Selling).

One of the factors that may influence the style you want to use is maturity of the team. In very mature and homogeneous teams consulting or cocreating might be very fast approaches. On other teams different factions may simply not converge even if you give the team a generous time frame.

Again, it pays of to adapt to the situation at hand. The more of these styles you can easily work the easier it will be to select the most appropriate one, and still get decisions made both fast and properly.

A Thought About Selecting Tools

Once you are an XYZ partner (replace XYZ with your favorite partner vendor) this means that you should be using all XYZ tools. After all you get them all cheap or even for free. So choosing them must be right!

Wrong! Check what the primary reason is why your company is in business. Is it being an XYZ partner? Or is it delivering good products and services to your customers? I bet that it is the latter.

And if you deliver good products and services to your customers using XYZ technologies or tools might be the right choice. But maybe it's not. If you believe you can choose only from XYZ's product list then chances are that you miss out on the best opportunities to improve what you do and how you work.

For example, XYZ may just have large monolithic applications that require training for users and specialists for configuring and maintaining it. And maybe those large applications that try to be everything to everyone become so flexible that all that flexibility makes it hard to change and quickly adapt your processes to your environment. Do you want your tools to support your processes? Or do you want your tools to dictate your processes?

Bottom line: Don't let you limit yourself by being an XYZ partner thinking that you can use only their tools!

Getting Started

Now you are a new member of this team of smart, experienced, and very talented people. How do you get started?

Sometimes it looks to me that the most difficult part is to make sure that the team starts to shift its mindset. This does not mean at all that everything that the team did in the past or everything they are doing today is wrong. Quite the opposite.

A mindshift helps looking on everything in a new light. For instance it might help to rerun an experiment that failed a few months ago because one of the tools wasn't up to the job. If you have a new version of the tool this time the experiment might have a different outcome.

A mindshift also helps you to move to a different approach while - maybe - continue operating the same. For instance when you are used to larger and more complex projects or tools or designs then it can be quite challenging to move in the other direction. What about smaller and simpler?

A mindshift might also require challenging some of your assumptions. What if you are assuming that you get punished if an experiment fails? Maybe that assumption was correct many years ago and you internalized it so much that you are not even aware of it. What if the assumption is incorrect?

Getting started can be very difficult and sometimes it takes a lot of courage. Yes, it is good to know all those reasons why something cannot be done. In engineer's terminology this is the list of rists. But then: aren't we engineers to to make things work despite other people saying it cannot be done?

So try small experiments, maybe a couple of people for a couple of days. Maybe the experiment fails. No problem. Then you know yet another way that doesn't work and you have learned more about the challenge.

But maybe the experiment is successful. Then you have started to move. Admittedly just a tiny little step but you have moved. And you have learned as well. And maybe after you have moved you can identify the next thing already that is worth trying which was hidden or impossible before your move.

Let's look at a real life example to illustrate my point: Let's assume you have that mixture of unmanaged C++ and .NET code and you want to move all of your code to the .NET world. Then switch on that /clr flag and see what happens. Maybe it is successful and maybe then the code for communicating between the two worlds can go away, and maybe then you start seeing new options for what the next best move can be.

Product Manager as Team Members

There is that aspect of agile approaches that emphasizes the "whole team". For more details see here, here, here, here, and here. In my opinion this includes every person that works towards a deliverable of your project. Other people may call that a cross functional team since it not only includes the developers but other professionals as well.

So you may also need subject matter experts (SME's) for different areas. Maybe your application is extremely sensitive to security. Then you may want to include a security expert. Or your system might need to process a lot of data in which case you may want to include a performance expert. Or your system needs to interact with that old mainframe application in which case you want to include a person who is sufficiently familiar with that. In all these cases you may want to add the person full time or part time on an as-needed basis.

One of the most important people on such a team is the subject matter expert (SME) for the business side of the application. E.g. for a hotel reservation system you would want to have a person on the team who is familiar with that industry.

In some companies that domain expert is equal to the product manager. OK, I'm simplifying here a bit. But stay with me as the simplification is for illustration purposes only.

So with the above mentioned concept of the "whole team" you'd assume that this product manager is part of your team as well.

Well, just until a few weeks ago I would have said yes.

In the meantime I have come to the conclusion that I have to qualify this answer. And here is the reason.

To some degree, the product manager is part of the team in that she provides the input that is required from a product perspective for example the business side of the software.

But then, the product manager is also a customer. Do we treat customers the same way as we would treat our fellow colleagues?

Sure, it definitely would be nice and desirable if the relation would be just the same. We could have Friday afternoon drinks and have all the fun by telling all the war stories from the week. But wait! Is this really what you want?

Here is something to think about: Your customer is the one with the money (or budget). So here is something that is different. Your customer wants to spend money only on items that make sense to her. Your customer doesn't want to hear about that database problem that you encountered this week because it might actually mean - in her perspective - that the project is at risk, and all the sudden this anecdote, shared over a beer on Friday, takes on a life on its own.

Does this indicate a failure of the process? Does this mean you should exclude your customer from the team? I think that would go too far.

I think, based on my experience over the last 12 months, the customer (e.g. your product manager) is probably the most valuable person on the team. That means that person needs special treatment. Does that mean you should say "yes" to everything? No, it doesn't. Does it mean that the customer gets her way all the time? No, it doesn't.

So what does it mean? It means that you can still share most of the information with your product manager that you would share with your other team members as well. But it might mean that you rethink the way how you represent things.

Depending how you communicate with the product manager you craft her perception of you and your team. Say the same things, but say them in such a way that considers how you may influence perception. Don't walk on egg shells either. You still need to be self-confident, and that means for each conversation it is good if you have prepared a view. Don't go into a meeting without a view.

I have changed my model for how I look at internal customers in that I try to provide to them the same service I would provide to external customers. Although there is no written contract, a product manager (or any other business domain expert) is a customer to you, and the way you treat your customer will have a huge impact on the outcome of your projects. Work for your customer, work with your customer, come with solutions instead of problems, and ultimately make your customer happy.

Can Agile Keep You From Being Successful?

It certainly depends on what kind of career you are looking for. But sometimes if you want to become a leader you may find that being an expert on a non-leadership subject can actually keep you from being successful.

Some authors - I don't want to give reference here since I was looking at German authors - use a model that takes different thinking styles as a starting point. The knowledge focused thinker tries to become and stay an expert on a particular subject. And a person that is an expert may even have the fear that someone comes along who is an even better expert. So they frantically work on become and staying the "best". They may start to defend their beliefs and knowledge, ultimately coming across as defensive, academic, or even arrogant. Non of these perception will help if you want to become an agile leader.

As a leader you are still a subject matter expert. However, instead of being an expert on Crystal, Scrum, XP, or the like, you become an expert on agile leadership. You can delegate the agile practices to people in your team. After all: If you have coached your team over an extended period of time there should be more than enough methodology champions in your team anyways.

So you can let go without losing influence. You are no more under the pressure to create the perfect implementation of Scrum or XP regardless of what "perfect implementation" means. If XP or Scrum doesn't work perfectly it is not you who has failed. The team as such has not yet managed to adopt and adapt it sufficiently. Depersonalize the particular area from yourself.

A question that might help you to make this change towards re-inventing yourself could be: How do you define success? Is it to get pair programming rolled out through-out the organization? Or is it something different? Maybe you want to provide the best possible service to you customers. Maybe you no longer think in terms of black and white, right or wrong, works or doesn't work. Maybe you want to introduce a third category saying: it helps.

Bottom line: To make the shift towards an agile leader you might actually have to let go of trying to be an expert on an agile methodology to be successful!

Do You Need a "Quality Program"?

My answer to this question would be: It depends.

It depends on what you mean by program. If you mean an elaborate and detailed plan it will be highly likely that by the time you get to start executing it the conditions have changed and many of the details are no longer valid in the changed context.

And here is a related question: Is Quality Improvement a one-off? Again, it depends. If it means that once you have rolled out the quality improvement - you ticked off all the items on the list - you are done then in all likelihood your organization will fall back in terms of quality again even if it doesn't degrade.

But if you understand Quality Improvement as a process, if you construct it as a set of guiding principles and values, then you are on the path towards a sustainable approach.

And here is how it might look in practice.

Option 1 might be: Create a detailed program so that you cover all aspects of what is going wrong. This may take weeks or even months just to get this plan set up. And it will take some more time to roll it out. And then it will take even some more time to show results.

Option 2 could be: Create a laundry list of things that may have an immediate impact. Your people know what's broken. Poll their views. Then make small, incremental changes. Observe. If it works, do more of it. All of this can be instigated, modified, or cancelled within days or weeks. Yes, you might be wrong. But with small items all of them suggestions from your people who are as close to the problems as possible you will get the majority right. And for the few that don't work, you can react extremely fast and can cancel it.

I personally would go for option 2 since it would help showing results very fast and very early. Each small item that is cleaned up will uncover or emphasize other items that are broken. And this could be a bug that you fix, or it could be a small change in the process that you apply. With small incremental steps you achieve short term results, you don't have to flush down major changes that go wrong, and you plant the seeds for a continuous improvement and learning process.

Let me give a very concrete example. And this one amazes me again and again because it is so blatantly obvious that it is surprising that there are still companies out there who don't do this. Let's assume you have a system of a significant size. This can be an in-house solution, a one-off, or a commercial off-the-shelf (COTS) product. Assume further that you have an issue with the bug level. There are just too many of them. Then here are the rules that you can put in place to address the problem, fix it, and prevent it from reappearing:

1. Each bug must be reproduced by an automated test.
2. The automated test is to be included in the automated (regression) test suite
3. The code is modified until all tests pass.
4. A new version, e.g. a patch, can be sent out only if all tests pass.
5. If the bug is fixed on a support branch it also has to be fixed on the main development stream (potentially other streams as well but I think that's a business decision)

With this set of rules in place, meticulously followed, you can expect the bug rate to fall noticeably within a short time frame. In one case that I'm aware of the bug rate reduced by 30% within 12 months (starting with zero tests!). I'm sure that some improvement was already visible after 3 and 6 months.

Bugs are not the only type of lack of quality. You can equally find lack of quality around processes, requirements, tools, etc. The approach would still be the same. Instead of a sledgehammer or Ben Hur sized quality program, try a continuous stream of small incremental changes. It's slow at the beginning since there is so much to clean up. But then it will gain momentum, and when it has become a habit your organization has changed it's behavior. Quality has been established as a process, as a part of the culture.

So, no, I don't think you need an elaborate program. Small, incremental changes, observation, then adapt, are extremely lightweight and can be introduced today. But you need certainly guiding principles that help your team understand that you mean business when you talk quality.

Quality!

What is the important bit about quality?

Just calling for it doesn't do the job. You've got to put your money where your mouth is!

So when you ask for improving the quality it might pay also off to think about W. Edwards Deming's "Red Beads Experiment" (description for example here). Why is this relevant?

In essence the experiment describes how you limit the quality any process can achieve by not allowing the employees to improve the tools and processes to do their work. In fact allowing for continuously improving tools and processes is one of the most powerful mechanisms to improve quality. And as Deming points out, the decision to empower or prevent employees from doing this improvement comes from management. So, if you ask for quality improvements and at the same time deny your team tool or process improvements you shouldn't be surprised if the quality level doesn't go up at all or not as quickly as you'd like to.

Zero Defects

Agile concepts are not as new as you might think. In fact some concepts have been around for quite some time.

For instance look at Test-Driven Development (TDD). It is a technique that among other things uses prevention instead of inspection to achieve high quality code. And in all cases the objective must be zero known defect. It is not about "good enough" quality. What is "good enough" in this case? Is it 2 bugs, or 5, or 10? It doesn't really matter. "Good enough" is not specific enough when it comes to quality. Zero defects is the objective. That's specific. Is it achievable? Well, apparently sites like Flickr release new versions of their system up to every 30 minutes. Is this doable if you have low quality? Very unlikely. Without automated comprehensive, fast (= cheap) testing Flickr would be able to do this.

Another technique that is popular with agile approaches are reflection workshops. These are basically opportunities for taking a step back and think about improving the way we work. And if (major) issues are identified in the process then the solution is not only to fix it but to prevent it.

All of these thoughts are not new. Philipp B. Crosby coined the term Zero Defects several decades ago. His book with the title "Quality is Free" is an excellent read. I just finished the sequel "Quality Without Tears" which was published in 1984, long before "Agile" became a hype. When you read the book you will notice that although some terms are different there are a huge amount of commonalities.

Get Quality Without Tears: The Art of Hassle-Free Management

Recognition: How do you share Fame and Blame?

A while ago I wrote about Staff Turnover (here and here). In this post I'd like to look at something that you can do for the people on your team. And maybe with this you can help to provide a better environment, a better place to work at.

In some cultures not complaining is equivalent to praising someones performance. The Southwest of Germany is such a place (For those who are familiar with it: "Net g'motzt isch scho g'nug g'lobt").

In other cultures rewarding has take on almost the shape of an avalanche. New Zealand is such a place (Remember all the awards, prices, etc. that you got at school? In some schools there are more awards than students!).

In other cultures appraising each other can become a group decease. The United States appear to me sometimes like that (Ever heard of the group cheering at Wal-Mart?).

So regardless where you live there are cultural differences in terms of how recognition is used to motivate people. And as a leader you can make the difference by sharing the fame with your team, and keeping the blame away from your team (unless they really screwed up in which case you'll probably have a private session with them that you don't want to share outside of your team).

So is recognition about affection? I think these are two different things. People certainly want to be liked. And if they can choose they will work in a place where they and their work is appreciated.

Recognition, in my view, is not about expression affection. Expecting being recognized for a good performance or result is not seeking to be liked. I think people can even "survive" for a while if they are not given recognition explicitly. If that's your style, so be it. Maybe you express your recognition in other ways, e.g. by giving your people a day off, or by giving them stock options, or by asking them taking on tasks that are critical and important for the company and come with more responsibilities.

There are many different ways to express recognition without having to express affection. Some of them are even free. Or how expensive is it to say "Thank you"? How expensive is it to mention people who over years showed persistent high performance despite all obstacles, hung on to the project and the team, and in the end made it happen?

All of this depends a lot of your style. Expressing recognition is important to keep your team motivated. It is certainly not the only means and it certainly is not sufficient. And if you use low-cost tools like saying "Thank you!" you must ensure that you really mean it. If you don't have the credibility of your team, just don't.

An entirely different question is if you give recognition, e.g. by company wide announcement, but don't include the people who had critical roles for the success. If you include only the ones that you interact with most frequently, or if you reduce other people's contribution, don't be surprised if people won't be as motivated next time round. Again this has nothing to do with the need to be liked. It's about ensuring that recognition goes to the right people. And it's not about what you intended to say. It is about what actually did say.

So the questions are: How do you share fame and blame? How do you give recognition to your team and the people on your team?

Update on Bureaucracy and Constraints

This is an update to my previous post on the subject.

Indeed I was able to find an option to reduce some overhead in my area. By using shared files instead of duplicated information I was able to reduce some bureaucratic overhead with reporting.

And here is another example: The PMO (Project Mangement Office) in our organization has decided to define a new charter for themselves in order to clarify their role and responsibilities. We started with a draft that had about 6 or 8 pages. We ended up with a net amount of about 1 page when we finished refactoring. That way we reduced waste.

And the biggest fun was from my perspective that the entire group participated and contributed to simplifying the document. It's now really lean and mean!

I'm sure we are able to do the same thing for other items again. We have the opportunity to become a really lean organization. It seems important to me to alway be on the lookout for opportunities to reduce waste, to try to find simpler solutions, to remove barriers.

If you build too many fences you will be surrounded by sheep. I think that in the age of internet communities like Facebook or Twitter people are looking for flatter hierarchies, more self-organization, and more freedoms. Everything else equal, people will choose to work for a company that comes close to the ideologies that is underlying the new online communities.

Managers that continue to use a management approach that might have worked 10 years ago will eventually loose out. Instead of administrative management, we need inspiring leadership that unleashes the creativity of the people in a learning organization. And in my view one important ingredient is collaborative decision making.

Constraints and Bureaucracy: Do you slow down your organization?

Most people I talk to will say that they want to keep bureaucracy at the minimum. At the same time they will say that a minimum set of policies, rules, and procedures need to be in place so that an organization can exhibit discipline and consistency across the board.

Agreed.


But then think of this: Your team interacts with lots of different other teams and individuals in order to work on projects. In addition there are cross cutting concerns such as administrative things (e.g. access to the building), HR (e.g. papers you need to hand in), Accounting (e.g. your last expense report), and so forth.


From each department's perspective they impose only a small little item (or two or three). And each item is just a very small thing by itself. And from that departments perspective each of the items is required and reasonable so they can do an excellent job.

The problems start when you pile the all the items up that come from different departments. Then progress in your organization may slow down significantly and may even come to a grinding halt.

So what to do? I have friends who have simply given up on some of these items. If they are given a spreadsheet to fill in they may decide to just throw in some (almost) random data (except if it is financial data). (Sorry, but I won't reveal names here to protect the individuals.)

Let's take an example: If asked to assess your team members and fill in 50 or 60 items in a skill matrix and given a team size of 15 or 20, we are talking about between 225 and 1,200 items to fill in. You probably need to think about each item for at least 10 seconds (this is a wild guess). But maybe you want to do a good job and do justice to the people in your team. Or maybe you want a realistic and true picture so you can be better at managing your project. So you spend 30 seconds on each item (Too much? Too little?). Then just this "simple" exercise amounts to 2 to 10 solid hours of your working time spent on a single spread sheet! Doesn't sound like much? Well, don't forget those other spreadsheets, reports, presentations, etc. that are waiting in your inbox!

So whoever thinks they are at the low end of bureaucracy, please think again! From a person's perspective who is affected by your actions the world may look entirely different, and your "small" request may amount to just that tipping thing that may keep those people from being successful. You may achieve quite the opposite of what you really wanted.

My recommendation would therefore be to consider for all your actions: How could you do less and still get a good enough outcome? (Or Google this: TSTTCPW. It applies to non-software engineering items, too.)

BTW: I will do the same myself and see whether I can't find an item where I might be causing avoidable overhead myself. I'm probably as guilty of this as anyone else....

Hiring

Hiring people is important to the success of your team and project. I strongly believe that you cannot deliver great products if you have a low-performing team. I strongly believe that regardless what you want to achieve with your team, it heavily depends on the quality of the people. I'd like to discuss a few of the principles I use to select people for joining my team.

Elite Team

It is almost like a positive, reinforcing cycle. If you have an excellent team, people will learn about it, and then they want to join that team. This leads to a greater pool of people you can select from, which in turn provides you with high quality candidates you can select from.

So in that sense an elite team, thinking, or attitude is acceptable and even desirable I think. The challenge is certainly to avoid coming anywhere near arrogance. But if you are pushing your team all the time to become even better - to outperform themselves, and if you have hired people with good character and attitude than this is a no-brainer anyways.

Hiring Process

It is important to distinguish between trying to assess the quality of a candidate in general, and assessing how the candidate fits with your team. I think you should focus very much on whether a person fits your team. If a candidate is a perfect engineer but isn't a fit for your team, hiring that person will not do any good. Similarly rejecting a candidate doesn't mean that the person is a bad engineer. It merely says that the person wasn't a good fit for your team.

If given the choice of a) a candidate that is perfect as an engineer but has weaknesses on the interpersonal side, and b) a candidate that is a perfect fit regarding interpersonal skills but has some weaknesses on the technical side, I would always go for b).

Part of my hiring process is a programming task. This is a very simple task with about half a dozen stories. Good solutions typically fit on two letter/A4 sized pages including tests. It is amazing how good that filter works. I have seen "senior" engineers claiming to be experts in Java, C#, and C++. But when asked about the differences of generics (Java, C# - different in both cases) and templates in C++, I looked into a face that spoke a clear language: "I have no clue."

Sure that was an extreme case. But still. If a candidate cannot even do a very simple programming task it won't be much better if the task is part of a regular project.

Team Buy In

I believe this is an important aspect as well. When you hire a person always keep in mind who you expect the new person to collaborate with most of the time.

The technique I use is to basically ask two of my team members to run the actual interviews. At the end of the hiring process those two people then can make a recommendation who they believe is the best fit.

This does not take away your responsibility to make the decision. But it allows your team members to be heard and to voice their opinion about the candidate.

Once your team members have voiced their opinion and assuming you follow their advice, you will find that the buy-in of your existing team members with regards to a hiring decision is substantially higher. At the same time you ensure that you don't overlook something. When selecting new development workstations I ask my senior engineers to look at the specs as well. Why would I want to reduce the quality for the hiring process?

Hiring Managers

Admittedly I don't have a lot of experience in this field. Still I believe that the above principles apply to hiring managers as well. Even if you seek advice from your team the final decision is still with you. After all a commercial enterprise is not a democracy.

By not asking your team members for their thoughts about different candidates you basically reduce the buy-in, and potentially you might even overlook an important detail. You are missing an opportunity for building an even stronger team.

Certainly, your department is your "ship". You can run it any way you like. But I believe that with flat hierachies and with self-organizing teams, and given that we are now in the 21st century, and people are more self-directed and teams are more self-organized, it definitely is a good practice and pays off when you involve your team when hiring, including hiring managers. I don't see why the principles that work for engineers shouldn't work for managers as well.

Just my two cents. As I said I don't have a lot of experience so it might well be that I'm overlooking something here.

"Healthy Noise": Introducing Automated Performance Testing

Starting with the current release cycle I have introduced automated performance testing for my project team. Not that we didn't test performance in the past, not that we didn't assess performance related items during a release cycle. But the fact that it is now becoming part of the automated development environment creates a number of interesting collateral effects. I'd like to highlight a few.

Reduced Latency

First of all there is the fact of performance testing itself. I strongly recommend not to wait until you are in the last few weeks or days before a release. That may be too late. If you discover a performance related issue then you may be forced to take shortcuts so your product meets performance requirements. And indeed in the past we already did assess the performance of the product throughout a release cycle.

What is different then? In the past we had to get performance engineers to set up the test, maintain test scripts, run the tests, analyze the results, identify root causes, and suggest solutions. Now the tests are written by developers and they are integrated into the test suite and then executed immediately after integration. No need to wait until a time slot with a performance engineer becomes available. The tests are now written and maintained by the developers. The performance engineer's time is freed up and in that time they are available to consult to the developers. The feedback loop is much shorter. A few hours instead of a few days or weeks. And this is also helps reducing risk since stories that may have a performance impact can be played earlier.

General Benefits of Automation

And you also get the obvious benefits of automation such as repeatability, lower costs, higher quality. Not that the performance engineers make mistakes consciously. As humans we are not fail safe whether we like it or not. So the major drivers for the automation were: cost, time, quality.

Impact on Behavior

But then there are less obvious effects caused be the introduction of automated performance testing. For instance I am observing that the thinking of the entire team is influenced by it. The performance aspect has been promoted and is not playing a much more important role in the considerations of the cross functional teams. Performance is built in instead of bolted on. Should a particular design or implementation cause a performance issue it can be dealt with immediately. Bad practices are no longer proliferated.

With our customers I am also observing an improved understanding of considering non-functional aspects such as performance engineering when deciding about backlog priorities.

And although it is not the main driver, and I didn't think of this aspect at all, there is also the impact it has on the developer. I sense that improving the development environment with automated testing has a positive impact on the morale of the engineering team. We continue to improving the development environment providing all people an opportunity to improve velocity while improving quality at the same time.

"Healthy Noise"

Certainly this change wasn't painless. The development teams had to negotiate with their customers the right amount of performance related work that needed to be accommodated in the backlogs. The automated build environment has to be extended.

We may have to purchase additional licenses for the performance toolset. I hoped I could get away with just the floating licenses we already had but that doesn't seem to pan out! However, now the bottle neck is no longer the performance engineer. Now it looks more like the number of licenses. When I compare the "price" of an engineer with the price of an additional license, it becomes apparent that the license is definitely cheaper.

Introducing automated performance testing caused some issues. But I would call this "healthy noise". All participants - customers, user experience experts, performance engineers, developers - are working very focused to iron out these hick-ups and they have made a lot of progress.

Wrapping Up

Introducing a somewhat significant change like this requires adaptation by everybody. Processes, tools, etc. need to be adapted as well. The result, however, is that you are moving closer to a holistic approach to software engineering that also considers performance engineering and testing as an integral part of the process. In particular I am very pleased how all the people involved grew as part of this process. The team and the product will be better because of this. Well done, folks!

Re: Musings on Software Testing

I just stumbled over a post by Wes Dyer on software testing. The post is a very interested read.

While I share a lot of the concerns that he mentions and also have seen a few of them materialize in practice, I still get the sense that something is not quite right in Wes' post.

Reading a book on a subject that heavily depends on practical experience doesn't really give you the full experience. I'm sure he'd agree with this.

Overall the post comes across as a mostly theoretical discussion with little practical background, at least on the commercial scale or long-term application of TDD. This surprises me a bit since Wes - at least in 2005 - was a developer on Microsoft's C# compiler team.

I would love to know more about the background and context, e.g. empirical data, practical experience from commercial projects, etc.

To make a specific point: He mentions that the testing ideal is to minimize the cost of bugs. Well, that is certainly a good objective. For TDD, however, there are additional aspects that are important, e.g. trying to find a simpler implementation of the code through refactoring that becomes only available because of the comprehensive test suite that TDD creates in the first place.

I also think that the first diagram in Wes' post is not quite accurate. For instance while refactoring you also run tests to see whether or not your refactoring broke any tests. So you'd go from step 5 to step 2 or 4.

Looking at TDD in isolation doesn't make the cut either in my experience. TDD makes most sense and provides most value if it is one element in a system of interdependent and interrelated elements that comprise an agile development approach. So for instance there are interdependencies to refactoring, pair programming, and others. The techniques of XP are not just a laundry list of best practices. They support and strengthen each other.

I have been using XP (including TDD) in various projects of different sizes since 1999 when I was introduced to TDD/XP by Kent Beck. I am currently managing a 40+ people commercial software product project for an international audience. One of the key elements is TDD. The results that my teams have produced in this time have been by far superior to anything I have seen developed with a more "traditional" approach (this is certainly limited to the projects I have sufficient information about).

Bottom line: While I like Wes' post very much since it highlights a number of good points and concerns, at the same time it seems to lack quite some credibility because little empirical information is provided that support at least some of his statements. The post reads to quite some degree more like a theoretical opinion lacking sufficient practical background. Again, surprising given his (past) role at Microsoft.

One of my past managers liked to put it this way: Without numbers you are just a person with another opinion.

But, hey, maybe that's exactly what his post is: An opinion. And in that sense: Yes, I like it.

Mandating Velocity

Assume you are in the middle of preparing your release plan. The team - product managers, developers, performance engineers, user interface experts, and other - has come back with a plan that is based on a velocity of let's say 3.56 per week.

You are certainly interested in any improvement of speed. So you are considering to say: What if you would plan for 4 per week? Or maybe you are even considering to mandate a velocity of 4. Would that be a smart idea?

Let's have a look back to the "good old days". A manager describes a piece of work to one of his engineers and asks for the estimated effort required to do the job. The engineer comes back and says that he can build it in 28 days. The manager thinks about it and then says: I think you can build it in 25 days. You will get 25 days to build it. I expect it by ... (fill in a date).

What's wrong with this? For one, if the engineer is not stupid next time he will add what he thinks will removed from his estimate (3 days) plus some (maybe 2 days) in case the cuts are higher next time. So the initial estimate might be 32 days. The manager is not stupid either. Next time he might think that the estimates are "inflated" anyways and that "sand bagging" happens anyways and sure people are playing ping-pong or darts anyways. So it's safe to reduce the estimates by 10% or even more. Just to get to the "real" numbers.

All of this leads to a situation where nobody works with the real numbers anymore. Every single estimate becomes unreliable and as a consequence every project plan built on them is flacky and unreliable on the outset. The engineer and - using the approach on a large scale - the entire team is set up for failure. (Unless you don't have a problem with "whipping" them a little harder and expect them to work crazy hours, which may not be a problem for as maybe you don't have family and don't know how it feels when you miss important events in the lives of your kids.)

There are certainly chances that the engineer may finish the work in 25 instead of 28 days. But at what price? If it really takes that long to do the job properly and if the engineer is really working at the maximum sustainable pace then what is left? Quality. The only choice the engineer is left with is degrading quality. This could mean that code is no longer reviewed. It could mean that design is no longer reviewed. It could man that some error handling code is left out (good chances it won't get caught in system testing). It could mean that fewer tests are written or no tests at all.

How is this related to the initial story around a velocity of 3.56 versus 4? Well even if you don't challenge the estimates, mandating an increased velocity is in essence the same thing. You are saying that people are not working hard enough, are in cruise-mode, etc.

Eventually the velocity will increase. The numbers will go up, they will even achieve 5, 6, or any number that you choose. Just by osmosis the team will learn about your "technique" and it will adapt. The estimates for every single item will go up. If an item was estimated to be 2 it will then be 3 (or even 20!). Then the velocity will increase not only to 4 but they will achieve a breath taking 35! Just imagine!

But seriously. If you are interested in working with the true numbers so that you have realistic and reliable plans that you can base important decisions on then you shouldn't mandate velocity.

A different question is certainly when there is a project that is very critical to the company. If you have treated your people with openness, honesty, and fairness, I'm sure they will understand if as an exception extraordinary measures needs to be taken. However, if those are used in every other release cycle, it becomes normal mode of operation and hence unsustainable.

It all boils down to how you select your "supplier". Are you interested in getting only the cheapest one, no matter what? Or are you interested in a reliabel supplier that continuously reduces the cost / increases the output / improves the quality? -As a customer as well as a manager this is your choice. (My take on this: Sacrificing quality has never ever paid off.)

A reply from MindJet (makers of MindManager)

In one of my previous posts I mentioned that I wrote to MindJet and suggested a free trial with a sufficient time-out. Along with my request I sent a justification.

I already heard back from them. A person from their sales department provided me with the information the download - OK, everybody can do that - but also with details for an extended trial.

So here are the positive remarks:

1. Fast response. It was less than 24 hours after my enquiry.
2. Good enough response. I got the information that I was hoping for.

So, I guess, I'll give it a try, and then I'll take it from there.

Are your stakeholders on the same page?

Sometimes it's not good to rely on assumptions. One of those assumptions might be that you work with people from your product management team and you assume that a key stakeholder of that team is fully in line and on the same page with them.

This doesn't have to be the case as I just had to find out.

Over several months I used the assumption that the product specialist, who I have on my team as the proxy customers and backlog owners, were on the same page with their manager, the product manager. I think all of them thought, too, they were on the same page.

Only two weeks ago it turned out that the product manager had an entirely different understanding of the scope that would be available for an internal release than what at least one of the product specialists thought.

The result was a "sticker shock" for the product manager. With reason!

So what can we learn from this? There is certainly no guarantee that communication would have prevented this form happening. Remember all participants acted in good faith. Still more and more focused communication with your key stakeholders will reduce the likelihood of bad surprises.

So in my case I have arranged for weekly catch-ups with the particular product manager (Agile value: adapt). The intention is to keep each other informed as much as possible about developments that may have an impact on each others work.

Again, this highlights the superiority of direct communication (agile value!) over comprehensive documentation. Of course, regular written progress reports were provided during the same timeframe. And everybody thought everything is alright. Well, it wasn't. A key stakeholder wasn't on the same page.

Are your key stakeholders all on the same page?

More on MindManager

I just received a marketing email encouraging me to upgrade from MindManager 6 (MM6) to MindManager 7 (MM7).

Well, at least this time they don't try to sell me on all those performance improvements.

Still, why should I pay for an upgrade if I don't know whether the performance issues are fixed?

They claim that 91% of those customers who upgraded to MM7 are "satisfied" or "very satisfied" after they upgraded. What they didn't mention how many of the customers that are on version prior to MM7 are dissatisfied with the software and don't intend to upgrade any time soon.

Also, under the headline "MindManager Pro 7 by the numbers" they claim that there are over one million licenses of MindManager around the world. They didn't say whether this is just MM7 or whether this figure includes prior version. I assume the latter.

So overall, I do understand the intention. They would like to get people off old versions onto the current one, and that way they can collect the upgrade fees.

I sent an email to their sales department asking for a 90 days trial license key. I have upgraded to 5 then to 6, each time hoping that the performance issues are resolved. Each time I paid money then I got a license. This time I think it would be more than fair to do it the other way round. I get the license first - time-limited to 90 days - and if their product turns out to be as fantastic as they claim - which it is without doubt! (sarcasm!) - then I'll be happy to pay for the upgrade.

So let's see what they come back with. At the moment my confidence level regarding MindManager isn't very high.

Vodafone Vodem on Vista

It's great to have all those wireless options. It makes you much more flexible with regards to choose where you work and when you work.

Vodafone is offering their "Vodem" as one of the options for wireless data connectivity to the internet. In that sense they provide a tool for more agility. And that's the only reason I include my comment in this blog.

Officially the Vodem is supported on Vista, but in practice there are a few issues.

1. It doesn't seem to like some screen savers. I can deal with this by trying different ones.
2. It doesn't like AnyDVD, which is ok, too, as I can simply switch it off temporarily.
3. It doesn't like Hibernate. I don't like this one particularly as I don't want to reboot the machine each time I want to connect to the internet via the Vodem.
4. Sometimes it just stops working.

In any of these cases it still claims to be connected to the network but when you do an nslookup it can't find any server. Then you have to reconnect it, and restart "Vodafone Mobile Connect Lite" (Lite on what? Quality?)

Overall I have the impression that the quality of the drivers and firmware needs some serious improvement. With my previous data card I had similar issues under Windows XP and it didn't really improve in the long run.

Then I turned the Vodem around and read Huawei and "Assembled in China". I wonder whether the people at Huawei and/or Vodafone are proud of this product.

Bottom line, Vodafone's Vodem works to some degree on Vista but it is very flaky and they have still a long way to go until the reliability is where it should be. So if you are considering it for your team then you may want to give it a trial for quite some time before deciding on a full roll-out.