This time I'm writing about an item that is admittedly very specific to software development. More than once when I spoke to a members of a development team I was told "yes, we have an automated test suite". And yet, further along the conversation it turned out that despite a significant test suite the resulting quality wasn't where all the efforts put into creating those tests indicated it should be. And in all these cases when we then took a closer look at the tests themselves it turned out that at least one key element was missing.
That begs the question: What makes up a good test? What key characteristics should a good test have?
Setup, Execute, Validate
The first key ingredient is that a test consists of three parts: The first parts sets up the data that is needed for the test. This could be restoring a particular database content, it can be setting up a few objects in your programming language, it can be launching a particular user interface (e.g. browser) and many more. The second part is the actual execution of the test. This means you invoke functionality that modifies data. In the final third party you validate whether you have the expected outcome, e.g. the actual data is equal to the expected one.
Occasionally I've found, though, that people forget about the third step. I don't have data but suspect that this happens when people come from a background where executing a piece of code without crashing is almost a success. Think early/mids 90s of last century. C and C++ were still very dominant in the PC industry. Exceptions in the midst of running a program were nothing completely out of the ordinary. (Maybe you are the only one who never experienced them?) However, we can do better. Just because it doesn't crash with a nasty null pointer exception doesn't mean it performs as expected. Therefore at the end of a test always validate the outcome! The typical tool for that are the various assertions that come as part of test tools.
Repeatable
Not strictly a requirement but there are quite a few scenarios where running the same test more than once reveals and thereafter prevents certain bootstrapping type issue from happening. Assume your service implementation does some sort of housekeeping upon startup. The first time you invoke an operation on the service everything is still fine. But then perhaps as you repeat the same test (or set of tests) using operations on that service things go off track. Maybe connections are not properly closed. Maybe the service cannot handle more than 10 open connections at a time (rightly or wrongly). By repeating the same test over and over again chances increase are that discover a hidden issue and resolve it before your product is shipped.
Random Order
Tests should not depend on each other. A test should not require a different test to run first. If they do changes to one test may trigger further changes to other tests in the suite thus making changes more expensive and time consuming. You don't want to lose time. You want to be fast.
For example, lets assume you are working on a system that has Project as a concept and the name of the project becomes a unique identifier for each project. If all tests use the same project name for their tests, then each test would have to check during setup whether the project already exists. If it doesn't it would create it. The alternative would be to use a generated name in each test such as a string with the value "ProjectName" + RandomNumberAsString(). That way you make the tests independent from each other.
A corollary to this is that you can run each test in isolation, meaning you can run just that test focusing on the specific task at hand. You don't have to run other tests first and you don't have to remember the sequence for those other tests. You can - and probably want - to run the entire suite anyways once you have finished the coding part of your task.
Fast Equals Cheap
Why do tests to be fast? To an engineer time is money. The longer a test or test suite needs to execute the less likely it is that people will run the suite or portions of it. It a suite runs takes 1 minute to complete you probably run it before each commit. If a suite takes 5 hours you won't run it before each commit. So keep them fast, for example work with the smallest possible dataset, avoid or mock costly operations like talking to remote systems, databases, filesystems, anything that requires mechanical parts to move. Use in-memory databases (e.g. SQLite) instead of client-server systems such as Microsoft SQL Server or Oracle.
You may also want to continuously refactor your tests as well. Keep them lean and mean. Split the test suites into the ones you definitely want to run each time before you commit and those that are more expensive in terms of duration. Platform tests or scalability tests fall into the latter category.
Automated And Integrated
Tests have a value only when they are executed. As long as they are just sitting in your version control system they are useless. Make them work. Make them work hard. Use all those machines that sit idle while the developers using them during daytime are at home enjoying live. Integrate the execution of your automated test suites into your development processes. When tests and their execution are automated and integrated they are executed more frequently and each time a change in the code base has been committed.
Are Automated Tests Orphans In Your Team?
Automated tests are first class citizens and equally valuable as the product that you ship. Don't even think for a second that they are just an unavoidable side affect. It's not a tax report that you do because law says so. Instead fully integrated automated testing is the mechanism that allows your team to operate at full speed. Depending on your team size maintaining the testing infrastructure can well turn into a full-time job for a motivated engineer.
Treat your testing infrastructure at least as well as the other parts of your development infrastructure. Make all of it part of an end-to-end integrated development process that starts with a customer suggestion and ends with a deployed system operational for the very same customer. Automated testing is a key ingredient and the rules are simple to follow. No excuse any more for not improving software quality. Happy testing!
Showing posts with label quality. Show all posts
Showing posts with label quality. Show all posts
Saturday, January 30, 2010
Friday, June 27, 2008
Do You Need a "Quality Program"?
My answer to this question would be: It depends.
It depends on what you mean by program. If you mean an elaborate and detailed plan it will be highly likely that by the time you get to start executing it the conditions have changed and many of the details are no longer valid in the changed context.
And here is a related question: Is Quality Improvement a one-off? Again, it depends. If it means that once you have rolled out the quality improvement - you ticked off all the items on the list - you are done then in all likelihood your organization will fall back in terms of quality again even if it doesn't degrade.
But if you understand Quality Improvement as a process, if you construct it as a set of guiding principles and values, then you are on the path towards a sustainable approach.
And here is how it might look in practice.
Option 1 might be: Create a detailed program so that you cover all aspects of what is going wrong. This may take weeks or even months just to get this plan set up. And it will take some more time to roll it out. And then it will take even some more time to show results.
Option 2 could be: Create a laundry list of things that may have an immediate impact. Your people know what's broken. Poll their views. Then make small, incremental changes. Observe. If it works, do more of it. All of this can be instigated, modified, or cancelled within days or weeks. Yes, you might be wrong. But with small items all of them suggestions from your people who are as close to the problems as possible you will get the majority right. And for the few that don't work, you can react extremely fast and can cancel it.
I personally would go for option 2 since it would help showing results very fast and very early. Each small item that is cleaned up will uncover or emphasize other items that are broken. And this could be a bug that you fix, or it could be a small change in the process that you apply. With small incremental steps you achieve short term results, you don't have to flush down major changes that go wrong, and you plant the seeds for a continuous improvement and learning process.
Let me give a very concrete example. And this one amazes me again and again because it is so blatantly obvious that it is surprising that there are still companies out there who don't do this. Let's assume you have a system of a significant size. This can be an in-house solution, a one-off, or a commercial off-the-shelf (COTS) product. Assume further that you have an issue with the bug level. There are just too many of them. Then here are the rules that you can put in place to address the problem, fix it, and prevent it from reappearing:
Bugs are not the only type of lack of quality. You can equally find lack of quality around processes, requirements, tools, etc. The approach would still be the same. Instead of a sledgehammer or Ben Hur sized quality program, try a continuous stream of small incremental changes. It's slow at the beginning since there is so much to clean up. But then it will gain momentum, and when it has become a habit your organization has changed it's behavior. Quality has been established as a process, as a part of the culture.
So, no, I don't think you need an elaborate program. Small, incremental changes, observation, then adapt, are extremely lightweight and can be introduced today. But you need certainly guiding principles that help your team understand that you mean business when you talk quality.
It depends on what you mean by program. If you mean an elaborate and detailed plan it will be highly likely that by the time you get to start executing it the conditions have changed and many of the details are no longer valid in the changed context.
And here is a related question: Is Quality Improvement a one-off? Again, it depends. If it means that once you have rolled out the quality improvement - you ticked off all the items on the list - you are done then in all likelihood your organization will fall back in terms of quality again even if it doesn't degrade.
But if you understand Quality Improvement as a process, if you construct it as a set of guiding principles and values, then you are on the path towards a sustainable approach.
And here is how it might look in practice.
Option 1 might be: Create a detailed program so that you cover all aspects of what is going wrong. This may take weeks or even months just to get this plan set up. And it will take some more time to roll it out. And then it will take even some more time to show results.
Option 2 could be: Create a laundry list of things that may have an immediate impact. Your people know what's broken. Poll their views. Then make small, incremental changes. Observe. If it works, do more of it. All of this can be instigated, modified, or cancelled within days or weeks. Yes, you might be wrong. But with small items all of them suggestions from your people who are as close to the problems as possible you will get the majority right. And for the few that don't work, you can react extremely fast and can cancel it.
I personally would go for option 2 since it would help showing results very fast and very early. Each small item that is cleaned up will uncover or emphasize other items that are broken. And this could be a bug that you fix, or it could be a small change in the process that you apply. With small incremental steps you achieve short term results, you don't have to flush down major changes that go wrong, and you plant the seeds for a continuous improvement and learning process.
Let me give a very concrete example. And this one amazes me again and again because it is so blatantly obvious that it is surprising that there are still companies out there who don't do this. Let's assume you have a system of a significant size. This can be an in-house solution, a one-off, or a commercial off-the-shelf (COTS) product. Assume further that you have an issue with the bug level. There are just too many of them. Then here are the rules that you can put in place to address the problem, fix it, and prevent it from reappearing:
- Each bug must be reproduced by an automated test.
- The automated test is to be included in the automated (regression) test suite
- The code is modified until all tests pass.
- A new version, e.g. a patch, can be sent out only if all tests pass.
- If the bug is fixed on a support branch it also has to be fixed on the main development stream (potentially other streams as well but I think that's a business decision)
Bugs are not the only type of lack of quality. You can equally find lack of quality around processes, requirements, tools, etc. The approach would still be the same. Instead of a sledgehammer or Ben Hur sized quality program, try a continuous stream of small incremental changes. It's slow at the beginning since there is so much to clean up. But then it will gain momentum, and when it has become a habit your organization has changed it's behavior. Quality has been established as a process, as a part of the culture.
So, no, I don't think you need an elaborate program. Small, incremental changes, observation, then adapt, are extremely lightweight and can be introduced today. But you need certainly guiding principles that help your team understand that you mean business when you talk quality.
Thursday, June 26, 2008
Quality!
What is the important bit about quality?
Just calling for it doesn't do the job. You've got to put your money where your mouth is!
So when you ask for improving the quality it might pay also off to think about W. Edwards Deming's "Red Beads Experiment" (description for example here). Why is this relevant?
Just calling for it doesn't do the job. You've got to put your money where your mouth is!
So when you ask for improving the quality it might pay also off to think about W. Edwards Deming's "Red Beads Experiment" (description for example here). Why is this relevant?
| In essence the experiment describes how you limit the quality any process can achieve by not allowing the employees to improve the tools and processes to do their work. In fact allowing for continuously improving tools and processes is one of the most powerful mechanisms to improve quality. And as Deming points out, the decision to empower or prevent employees from doing this improvement comes from management. So, if you ask for quality improvements and at the same time deny your team tool or process improvements you shouldn't be surprised if the quality level doesn't go up at all or not as quickly as you'd like to. |
Wednesday, June 25, 2008
Zero Defects
Agile concepts are not as new as you might think. In fact some concepts have been around for quite some time.
For instance look at Test-Driven Development (TDD). It is a technique that among other things uses prevention instead of inspection to achieve high quality code. And in all cases the objective must be zero known defect. It is not about "good enough" quality. What is "good enough" in this case? Is it 2 bugs, or 5, or 10? It doesn't really matter. "Good enough" is not specific enough when it comes to quality. Zero defects is the objective. That's specific. Is it achievable? Well, apparently sites like Flickr release new versions of their system up to every 30 minutes. Is this doable if you have low quality? Very unlikely. Without automated comprehensive, fast (= cheap) testing Flickr would be able to do this.
Another technique that is popular with agile approaches are reflection workshops. These are basically opportunities for taking a step back and think about improving the way we work. And if (major) issues are identified in the process then the solution is not only to fix it but to prevent it.
All of these thoughts are not new. Philipp B. Crosby coined the term Zero Defects several decades ago. His book with the title "Quality is Free" is an excellent read. I just finished the sequel "Quality Without Tears" which was published in 1984, long before "Agile" became a hype. When you read the book you will notice that although some terms are different there are a huge amount of commonalities.
Get Quality Without Tears: The Art of Hassle-Free Management
For instance look at Test-Driven Development (TDD). It is a technique that among other things uses prevention instead of inspection to achieve high quality code. And in all cases the objective must be zero known defect. It is not about "good enough" quality. What is "good enough" in this case? Is it 2 bugs, or 5, or 10? It doesn't really matter. "Good enough" is not specific enough when it comes to quality. Zero defects is the objective. That's specific. Is it achievable? Well, apparently sites like Flickr release new versions of their system up to every 30 minutes. Is this doable if you have low quality? Very unlikely. Without automated comprehensive, fast (= cheap) testing Flickr would be able to do this.
Another technique that is popular with agile approaches are reflection workshops. These are basically opportunities for taking a step back and think about improving the way we work. And if (major) issues are identified in the process then the solution is not only to fix it but to prevent it.
All of these thoughts are not new. Philipp B. Crosby coined the term Zero Defects several decades ago. His book with the title "Quality is Free" is an excellent read. I just finished the sequel "Quality Without Tears" which was published in 1984, long before "Agile" became a hype. When you read the book you will notice that although some terms are different there are a huge amount of commonalities.
Get Quality Without Tears: The Art of Hassle-Free Management
Saturday, May 10, 2008
"Healthy Noise": Introducing Automated Performance Testing
Starting with the current release cycle I have introduced automated performance testing for my project team. Not that we didn't test performance in the past, not that we didn't assess performance related items during a release cycle. But the fact that it is now becoming part of the automated development environment creates a number of interesting collateral effects. I'd like to highlight a few.
Reduced Latency
First of all there is the fact of performance testing itself. I strongly recommend not to wait until you are in the last few weeks or days before a release. That may be too late. If you discover a performance related issue then you may be forced to take shortcuts so your product meets performance requirements. And indeed in the past we already did assess the performance of the product throughout a release cycle.
What is different then? In the past we had to get performance engineers to set up the test, maintain test scripts, run the tests, analyze the results, identify root causes, and suggest solutions. Now the tests are written by developers and they are integrated into the test suite and then executed immediately after integration. No need to wait until a time slot with a performance engineer becomes available. The tests are now written and maintained by the developers. The performance engineer's time is freed up and in that time they are available to consult to the developers. The feedback loop is much shorter. A few hours instead of a few days or weeks. And this is also helps reducing risk since stories that may have a performance impact can be played earlier.
General Benefits of Automation
And you also get the obvious benefits of automation such as repeatability, lower costs, higher quality. Not that the performance engineers make mistakes consciously. As humans we are not fail safe whether we like it or not. So the major drivers for the automation were: cost, time, quality.
Impact on Behavior
But then there are less obvious effects caused be the introduction of automated performance testing. For instance I am observing that the thinking of the entire team is influenced by it. The performance aspect has been promoted and is not playing a much more important role in the considerations of the cross functional teams. Performance is built in instead of bolted on. Should a particular design or implementation cause a performance issue it can be dealt with immediately. Bad practices are no longer proliferated.
With our customers I am also observing an improved understanding of considering non-functional aspects such as performance engineering when deciding about backlog priorities.
And although it is not the main driver, and I didn't think of this aspect at all, there is also the impact it has on the developer. I sense that improving the development environment with automated testing has a positive impact on the morale of the engineering team. We continue to improving the development environment providing all people an opportunity to improve velocity while improving quality at the same time.
"Healthy Noise"
Certainly this change wasn't painless. The development teams had to negotiate with their customers the right amount of performance related work that needed to be accommodated in the backlogs. The automated build environment has to be extended.
We may have to purchase additional licenses for the performance toolset. I hoped I could get away with just the floating licenses we already had but that doesn't seem to pan out! However, now the bottle neck is no longer the performance engineer. Now it looks more like the number of licenses. When I compare the "price" of an engineer with the price of an additional license, it becomes apparent that the license is definitely cheaper.
Introducing automated performance testing caused some issues. But I would call this "healthy noise". All participants - customers, user experience experts, performance engineers, developers - are working very focused to iron out these hick-ups and they have made a lot of progress.
Wrapping Up
Introducing a somewhat significant change like this requires adaptation by everybody. Processes, tools, etc. need to be adapted as well. The result, however, is that you are moving closer to a holistic approach to software engineering that also considers performance engineering and testing as an integral part of the process. In particular I am very pleased how all the people involved grew as part of this process. The team and the product will be better because of this. Well done, folks!
Reduced Latency
First of all there is the fact of performance testing itself. I strongly recommend not to wait until you are in the last few weeks or days before a release. That may be too late. If you discover a performance related issue then you may be forced to take shortcuts so your product meets performance requirements. And indeed in the past we already did assess the performance of the product throughout a release cycle.
What is different then? In the past we had to get performance engineers to set up the test, maintain test scripts, run the tests, analyze the results, identify root causes, and suggest solutions. Now the tests are written by developers and they are integrated into the test suite and then executed immediately after integration. No need to wait until a time slot with a performance engineer becomes available. The tests are now written and maintained by the developers. The performance engineer's time is freed up and in that time they are available to consult to the developers. The feedback loop is much shorter. A few hours instead of a few days or weeks. And this is also helps reducing risk since stories that may have a performance impact can be played earlier.
General Benefits of Automation
And you also get the obvious benefits of automation such as repeatability, lower costs, higher quality. Not that the performance engineers make mistakes consciously. As humans we are not fail safe whether we like it or not. So the major drivers for the automation were: cost, time, quality.
Impact on Behavior
But then there are less obvious effects caused be the introduction of automated performance testing. For instance I am observing that the thinking of the entire team is influenced by it. The performance aspect has been promoted and is not playing a much more important role in the considerations of the cross functional teams. Performance is built in instead of bolted on. Should a particular design or implementation cause a performance issue it can be dealt with immediately. Bad practices are no longer proliferated.
With our customers I am also observing an improved understanding of considering non-functional aspects such as performance engineering when deciding about backlog priorities.
And although it is not the main driver, and I didn't think of this aspect at all, there is also the impact it has on the developer. I sense that improving the development environment with automated testing has a positive impact on the morale of the engineering team. We continue to improving the development environment providing all people an opportunity to improve velocity while improving quality at the same time.
"Healthy Noise"
Certainly this change wasn't painless. The development teams had to negotiate with their customers the right amount of performance related work that needed to be accommodated in the backlogs. The automated build environment has to be extended.
We may have to purchase additional licenses for the performance toolset. I hoped I could get away with just the floating licenses we already had but that doesn't seem to pan out! However, now the bottle neck is no longer the performance engineer. Now it looks more like the number of licenses. When I compare the "price" of an engineer with the price of an additional license, it becomes apparent that the license is definitely cheaper.
Introducing automated performance testing caused some issues. But I would call this "healthy noise". All participants - customers, user experience experts, performance engineers, developers - are working very focused to iron out these hick-ups and they have made a lot of progress.
Wrapping Up
Introducing a somewhat significant change like this requires adaptation by everybody. Processes, tools, etc. need to be adapted as well. The result, however, is that you are moving closer to a holistic approach to software engineering that also considers performance engineering and testing as an integral part of the process. In particular I am very pleased how all the people involved grew as part of this process. The team and the product will be better because of this. Well done, folks!
Friday, April 25, 2008
Mandating Velocity
Assume you are in the middle of preparing your release plan. The team - product managers, developers, performance engineers, user interface experts, and other - has come back with a plan that is based on a velocity of let's say 3.56 per week.
You are certainly interested in any improvement of speed. So you are considering to say: What if you would plan for 4 per week? Or maybe you are even considering to mandate a velocity of 4. Would that be a smart idea?
Let's have a look back to the "good old days". A manager describes a piece of work to one of his engineers and asks for the estimated effort required to do the job. The engineer comes back and says that he can build it in 28 days. The manager thinks about it and then says: I think you can build it in 25 days. You will get 25 days to build it. I expect it by ... (fill in a date).
What's wrong with this? For one, if the engineer is not stupid next time he will add what he thinks will removed from his estimate (3 days) plus some (maybe 2 days) in case the cuts are higher next time. So the initial estimate might be 32 days. The manager is not stupid either. Next time he might think that the estimates are "inflated" anyways and that "sand bagging" happens anyways and sure people are playing ping-pong or darts anyways. So it's safe to reduce the estimates by 10% or even more. Just to get to the "real" numbers.
All of this leads to a situation where nobody works with the real numbers anymore. Every single estimate becomes unreliable and as a consequence every project plan built on them is flacky and unreliable on the outset. The engineer and - using the approach on a large scale - the entire team is set up for failure. (Unless you don't have a problem with "whipping" them a little harder and expect them to work crazy hours, which may not be a problem for as maybe you don't have family and don't know how it feels when you miss important events in the lives of your kids.)
There are certainly chances that the engineer may finish the work in 25 instead of 28 days. But at what price? If it really takes that long to do the job properly and if the engineer is really working at the maximum sustainable pace then what is left? Quality. The only choice the engineer is left with is degrading quality. This could mean that code is no longer reviewed. It could mean that design is no longer reviewed. It could man that some error handling code is left out (good chances it won't get caught in system testing). It could mean that fewer tests are written or no tests at all.
How is this related to the initial story around a velocity of 3.56 versus 4? Well even if you don't challenge the estimates, mandating an increased velocity is in essence the same thing. You are saying that people are not working hard enough, are in cruise-mode, etc.
Eventually the velocity will increase. The numbers will go up, they will even achieve 5, 6, or any number that you choose. Just by osmosis the team will learn about your "technique" and it will adapt. The estimates for every single item will go up. If an item was estimated to be 2 it will then be 3 (or even 20!). Then the velocity will increase not only to 4 but they will achieve a breath taking 35! Just imagine!
But seriously. If you are interested in working with the true numbers so that you have realistic and reliable plans that you can base important decisions on then you shouldn't mandate velocity.
A different question is certainly when there is a project that is very critical to the company. If you have treated your people with openness, honesty, and fairness, I'm sure they will understand if as an exception extraordinary measures needs to be taken. However, if those are used in every other release cycle, it becomes normal mode of operation and hence unsustainable.
It all boils down to how you select your "supplier". Are you interested in getting only the cheapest one, no matter what? Or are you interested in a reliabel supplier that continuously reduces the cost / increases the output / improves the quality? -As a customer as well as a manager this is your choice. (My take on this: Sacrificing quality has never ever paid off.)
You are certainly interested in any improvement of speed. So you are considering to say: What if you would plan for 4 per week? Or maybe you are even considering to mandate a velocity of 4. Would that be a smart idea?
Let's have a look back to the "good old days". A manager describes a piece of work to one of his engineers and asks for the estimated effort required to do the job. The engineer comes back and says that he can build it in 28 days. The manager thinks about it and then says: I think you can build it in 25 days. You will get 25 days to build it. I expect it by ... (fill in a date).
What's wrong with this? For one, if the engineer is not stupid next time he will add what he thinks will removed from his estimate (3 days) plus some (maybe 2 days) in case the cuts are higher next time. So the initial estimate might be 32 days. The manager is not stupid either. Next time he might think that the estimates are "inflated" anyways and that "sand bagging" happens anyways and sure people are playing ping-pong or darts anyways. So it's safe to reduce the estimates by 10% or even more. Just to get to the "real" numbers.
All of this leads to a situation where nobody works with the real numbers anymore. Every single estimate becomes unreliable and as a consequence every project plan built on them is flacky and unreliable on the outset. The engineer and - using the approach on a large scale - the entire team is set up for failure. (Unless you don't have a problem with "whipping" them a little harder and expect them to work crazy hours, which may not be a problem for as maybe you don't have family and don't know how it feels when you miss important events in the lives of your kids.)
There are certainly chances that the engineer may finish the work in 25 instead of 28 days. But at what price? If it really takes that long to do the job properly and if the engineer is really working at the maximum sustainable pace then what is left? Quality. The only choice the engineer is left with is degrading quality. This could mean that code is no longer reviewed. It could mean that design is no longer reviewed. It could man that some error handling code is left out (good chances it won't get caught in system testing). It could mean that fewer tests are written or no tests at all.
How is this related to the initial story around a velocity of 3.56 versus 4? Well even if you don't challenge the estimates, mandating an increased velocity is in essence the same thing. You are saying that people are not working hard enough, are in cruise-mode, etc.
Eventually the velocity will increase. The numbers will go up, they will even achieve 5, 6, or any number that you choose. Just by osmosis the team will learn about your "technique" and it will adapt. The estimates for every single item will go up. If an item was estimated to be 2 it will then be 3 (or even 20!). Then the velocity will increase not only to 4 but they will achieve a breath taking 35! Just imagine!
But seriously. If you are interested in working with the true numbers so that you have realistic and reliable plans that you can base important decisions on then you shouldn't mandate velocity.
A different question is certainly when there is a project that is very critical to the company. If you have treated your people with openness, honesty, and fairness, I'm sure they will understand if as an exception extraordinary measures needs to be taken. However, if those are used in every other release cycle, it becomes normal mode of operation and hence unsustainable.
It all boils down to how you select your "supplier". Are you interested in getting only the cheapest one, no matter what? Or are you interested in a reliabel supplier that continuously reduces the cost / increases the output / improves the quality? -As a customer as well as a manager this is your choice. (My take on this: Sacrificing quality has never ever paid off.)
Tuesday, April 15, 2008
More on MindManager
I just received a marketing email encouraging me to upgrade from MindManager 6 (MM6) to MindManager 7 (MM7).
Well, at least this time they don't try to sell me on all those performance improvements.
Still, why should I pay for an upgrade if I don't know whether the performance issues are fixed?
They claim that 91% of those customers who upgraded to MM7 are "satisfied" or "very satisfied" after they upgraded. What they didn't mention how many of the customers that are on version prior to MM7 are dissatisfied with the software and don't intend to upgrade any time soon.
Also, under the headline "MindManager Pro 7 by the numbers" they claim that there are over one million licenses of MindManager around the world. They didn't say whether this is just MM7 or whether this figure includes prior version. I assume the latter.
So overall, I do understand the intention. They would like to get people off old versions onto the current one, and that way they can collect the upgrade fees.
I sent an email to their sales department asking for a 90 days trial license key. I have upgraded to 5 then to 6, each time hoping that the performance issues are resolved. Each time I paid money then I got a license. This time I think it would be more than fair to do it the other way round. I get the license first - time-limited to 90 days - and if their product turns out to be as fantastic as they claim - which it is without doubt! (sarcasm!) - then I'll be happy to pay for the upgrade.
So let's see what they come back with. At the moment my confidence level regarding MindManager isn't very high.
Well, at least this time they don't try to sell me on all those performance improvements.
Still, why should I pay for an upgrade if I don't know whether the performance issues are fixed?
They claim that 91% of those customers who upgraded to MM7 are "satisfied" or "very satisfied" after they upgraded. What they didn't mention how many of the customers that are on version prior to MM7 are dissatisfied with the software and don't intend to upgrade any time soon.
Also, under the headline "MindManager Pro 7 by the numbers" they claim that there are over one million licenses of MindManager around the world. They didn't say whether this is just MM7 or whether this figure includes prior version. I assume the latter.
So overall, I do understand the intention. They would like to get people off old versions onto the current one, and that way they can collect the upgrade fees.
I sent an email to their sales department asking for a 90 days trial license key. I have upgraded to 5 then to 6, each time hoping that the performance issues are resolved. Each time I paid money then I got a license. This time I think it would be more than fair to do it the other way round. I get the license first - time-limited to 90 days - and if their product turns out to be as fantastic as they claim - which it is without doubt! (sarcasm!) - then I'll be happy to pay for the upgrade.
So let's see what they come back with. At the moment my confidence level regarding MindManager isn't very high.
Monday, February 04, 2008
Software Quality and Tools
Being curious I thought I'd google for "software quality tools". There were approximately 23.8 million hits. I don't know how many tools this represents, or the type of tools, or the effectiveness and efficiency of those tools.
23.8 million hits represents a large number. Apparently there is no shortage of information and tools for improving the software quality. Why is it that we still have significant bugs in many software products (recent examples: here, here, and here)? Why are still many customers dissatisfied with the quality of software (indicators are mentioned here, here, and here)? Well, sometimes customeres might be satisfied because they have such low expectations which in turn were cause by poor quality in the past.
So, how to address this problem? I don't know the complete solution. I do know, though, that the longest journey starts with the first step. And here are two suggestions for what those first couple of steps might be for your organization. Introduce the following two rules:
23.8 million hits represents a large number. Apparently there is no shortage of information and tools for improving the software quality. Why is it that we still have significant bugs in many software products (recent examples: here, here, and here)? Why are still many customers dissatisfied with the quality of software (indicators are mentioned here, here, and here)? Well, sometimes customeres might be satisfied because they have such low expectations which in turn were cause by poor quality in the past.
So, how to address this problem? I don't know the complete solution. I do know, though, that the longest journey starts with the first step. And here are two suggestions for what those first couple of steps might be for your organization. Introduce the following two rules:
- Continuous automated build. Make sure your software system, component, etc. is built multiple times a day. Automatically. With feedback to all developers working on the code base. And if the build is doesn't pass (it's broken) then lock down the source control system until the build passes again. This might be quite disruptive at the beginning. But imagine what behavioral changes this rule might cause? For one you will notice that your people will start to be much more adament about the quality of their change set. Who wants to be the reason for a broken build? And then it also triggers off a continuous thinking about how to improve the toolset, the code, the tests, etc. so that it becomes easier for every engineer to produce quality code.
- One automated per bug. Let's assume you are not happy with the number of bugs in your software. What if for every single bug your engineers fix (at least) one automated test has to be added to the automated test suite? What if that automated test reproduces the bug and when the bug is fixed the test passes?
This rule makes most sense if each automated test is added to a test suite that is run as part of the automated build (see rule 1).
With the above rules you start building a comprehensive set of automated tests. Some may say, we have always been doing that. That might be correct, but my experience tells me that some organizations simply run the suite of tests only when they test the entire system, after the "development phase" is complete and just before the new version is shipped.
Also in some cases people claim that rule 2 cannot be used with legacy systems because writing such tests is too expensive. Again, that might be correct. If that is the case it would be an additional indicator for the (lack of) design quality. The system is too hard to test. Enforcing rule 2 will help to enforce a refactoring or redesigning of the system towards better testability. Also, lacking organizational discipline (or time) - bugs are simply fixed without writing an automated test and then simply shipped after a quick manual assessment. This is far from what is ideal.
By adding one automated test at a time to your automated build - now including an automated test portion - your system will increase in quality over time. A single test won't make a difference. But as your test suite increases in size it will cover more and more of your system.
And here is another observation: When a bug is reported which areas do they tend to be in? Typically you'll find them in frequently used areas or in areas that are particularly flaky. By adding automated tests in these arease you actually target those areas where you get the biggest bang for the buck.
Note that the two rules can be used for both new development and legacy code bases. There is no excuse for not even trying to improve the quality. It doesn't require myriads of expensive and complex tools. Simple rules like the above can help improving organizational behavior towards better (software) quality.
Wednesday, December 26, 2007
Some Thoughts on Bugs
The proper application of test-driven development yields in fewer bugs. This experience is share by quite a few people in the agile community. Still, there are bugs. This is not because your engineers have relaxed their quality standards. I think it is because expectations are higher. Customers can be more picky.
A few years ago people didn't dare to demonstrate software if it was only a few days old. After all it didn't go through a proper testing cycle, right? And to a degree - in particular in more traditional organizations - they were right. Quality was poor in some cases. Crashes could enrage even a mild mannered reviewer. But even if it wasn't a crash, bugs tended to be quite substantial in their impact. So showing the software created more uncertainty rather than excitement.
In a newer environment it is possible to show "brand new" software. Depending on your environment "new" means it's just a few hours or days old. Or just take the latest successful build.
When I look at some of the bugs raised today I see that they fall into the category "aesthetics". In the past people wouldn't have bothered raising them because compared to others they would have been negligable in terms of severity.
Does that mean that we should be happy about this? I don't think so. A bug is a bug is a bug. We shouldn't be proud of any bug. Each bug is an indication of a piece of unfinished work. If some bugs are a question of taste then it might be we will never be able to reach zero defects. In that case we should at least see this as an ideal towards which we work. Toyota - despite it's recent hickups (recalls of vehicles, even Lexus) - is seen as the well-recognized quality leader in the automative industry. Does this mean their products are defect free? No, not even close. They are subject to Murphy as well. But they have their unique way of dealing with them. They fix them one at a time. And not only that. They also try to identify ways of how to avoid the same defect from reappearing.
So where from here?
We should take Toyota's behavior as an example. We should still strive for reducing defects wherever we can. And we should do so by avoiding them in the first place. Communicating and collaborating closely with the customer is one of the first and most important steps. Writing your tests first and use them to drive your implementation helps, too. Keep your code simple, readable and maintainable.
And with bugs: Each time one slips through, follow these guidelines:
A few years ago people didn't dare to demonstrate software if it was only a few days old. After all it didn't go through a proper testing cycle, right? And to a degree - in particular in more traditional organizations - they were right. Quality was poor in some cases. Crashes could enrage even a mild mannered reviewer. But even if it wasn't a crash, bugs tended to be quite substantial in their impact. So showing the software created more uncertainty rather than excitement.
In a newer environment it is possible to show "brand new" software. Depending on your environment "new" means it's just a few hours or days old. Or just take the latest successful build.
When I look at some of the bugs raised today I see that they fall into the category "aesthetics". In the past people wouldn't have bothered raising them because compared to others they would have been negligable in terms of severity.
Does that mean that we should be happy about this? I don't think so. A bug is a bug is a bug. We shouldn't be proud of any bug. Each bug is an indication of a piece of unfinished work. If some bugs are a question of taste then it might be we will never be able to reach zero defects. In that case we should at least see this as an ideal towards which we work. Toyota - despite it's recent hickups (recalls of vehicles, even Lexus) - is seen as the well-recognized quality leader in the automative industry. Does this mean their products are defect free? No, not even close. They are subject to Murphy as well. But they have their unique way of dealing with them. They fix them one at a time. And not only that. They also try to identify ways of how to avoid the same defect from reappearing.
So where from here?
We should take Toyota's behavior as an example. We should still strive for reducing defects wherever we can. And we should do so by avoiding them in the first place. Communicating and collaborating closely with the customer is one of the first and most important steps. Writing your tests first and use them to drive your implementation helps, too. Keep your code simple, readable and maintainable.
And with bugs: Each time one slips through, follow these guidelines:
- Try to understand the root cause for the bug. Prove your understanding by writing an automated test that fails because of the bug.
- Make the test pass.
- Refactor mercilessly.
- Identify similar scenarios in your code base. Check whether they contain the same bug. If they do, chances are that you have identified another opportunity for refactoring.
Here is another aspect of bugs. Compare a bug to an ordinary story in your backlog. What is the major difference? Generally speaking, a story tends to be easier to estimate in particular if it is similar to a past one. A bug can be anything from "user misunderstanding" to "several weeks" of work. The point is that the amount of uncertainty tends to be higher for bugs.
So if you compile a single, consolidated list of all your work items, you will end up with a mixture of stories and bugs. The larger the proportion of bugs the higher the uncertainty. The higher the uncertainty the bigger the impact if one or more of the bugs require more than the "average" fix time.
So if you are interested to become more reliable (read predictable) then the suggestion is to keep the bug level as low as possible.
Does this mean you can't have crash parties any more? Under all circumstances, if the crash party helps you to assess the quality of your product and helps you to find more bugs, and you if you see value in this, then you definitely should have them. Not doing them because it keeps the bug numbers low, is cheating. You would be lying to yourself.
Also, if you don't fix bugs but keep piling them up, while you at the same time your are "completing" story after story, you are actually reporting a velocity (in stories per week) that is too high. Expectations for the next release cycle will be higher, and it will be even harder to negotiate time for fixing bugs or doing other items.
And more, if your approach results in a similar number of bugs your stakeholders may start asking the question whether your approach is really better than the traditional one.
I'm sure you don't want to let it come to that. So regardless how you look at it: Not fixing bugs and finding ways for avoiding them in the future doesn't help at all. Instead it can be the start of a vicious cycle. And by keeping the number of bugs low, the question of how to deal with bugs becomes a no-brainer!
Monday, August 07, 2006
Two Hour Technical Task Too Much?
Our recruitment process includes a programming task, which we give candidates as a preparation for the first one hour interview. The task is typically very simple and it is possible to complete it within roughly two hours. The intention is to assess whether a candidate is a) able to solve a simple task independently, b) able to demonstrate know of how to ensure quality (e.g. automated tests), and c) willing to invest that time for joining a team of highly skilled people.
In over 100 applications in the recent months we have never had any complaint by the candidates. Instead we see good results with regards to the performance of the successful candidates.
However, there was one case where a candidate sent an almost "rude" response after he received the programming task:
In this particular case we decided to not invite the candidate based on him not being willing to comply with some simple rules which are part of the policy of our company.
I wonder who else is using technical tasks as part of the recruitment process. It would be great if you would be willing to share your experiences.
In over 100 applications in the recent months we have never had any complaint by the candidates. Instead we see good results with regards to the performance of the successful candidates.
However, there was one case where a candidate sent an almost "rude" response after he received the programming task:
"IMHO only a fool will spend 2 hours sitting for a technical test without the employer having gone through some trouble to have a pre-selection round first."And later he added
"I happen to be not only adept on the technical front, but also socially observant and capable of making (mostly correct) strategic decision."Do we expect too much? I know of several other companies who use technical tasks for engineering positions as well, for example ThoughtWorks.
In this particular case we decided to not invite the candidate based on him not being willing to comply with some simple rules which are part of the policy of our company.
I wonder who else is using technical tasks as part of the recruitment process. It would be great if you would be willing to share your experiences.
Saturday, August 05, 2006
Customers Writing Executable Specifications
About two years ago I worked with a team which had to live (or suffer?) from requirements specifications with hundreds of pages of prose. This kind of document is hard to read, understand, and maintain. If you want to change it, and your customer is external, you have to follow a change management process which typically includes exchanging and having drafted and approved even more documents.
Ideally, requirements would be written in a way that makes it easy for the development team to verify whether the system satisfies them. So why not making requirements executable? That way your team members can run them as often as needed, and they could stop immediately once the system passes those tests.
The way to go forward is therefore executable specifications, or story tests. The most prominent thought leader on this is probably Rick Mugridge, on whose web site you can also find further information.
One tool to create and maintain such customer tests is Fitnesse. It is available for many different languages.
The benefits are very compelling. You have a much tighter link between your customer (might also be a product manager) and your development team. Executable specifications are a means for improving communication. You also get a tool that reduces the gap between what your customer wants the system to do, and what the system really does.
From my experience, it's worth playing with this concept. It might turn out to be an excellent addition to your toolbox for agile project management.
Ideally, requirements would be written in a way that makes it easy for the development team to verify whether the system satisfies them. So why not making requirements executable? That way your team members can run them as often as needed, and they could stop immediately once the system passes those tests.
The way to go forward is therefore executable specifications, or story tests. The most prominent thought leader on this is probably Rick Mugridge, on whose web site you can also find further information.
One tool to create and maintain such customer tests is Fitnesse. It is available for many different languages.
The benefits are very compelling. You have a much tighter link between your customer (might also be a product manager) and your development team. Executable specifications are a means for improving communication. You also get a tool that reduces the gap between what your customer wants the system to do, and what the system really does.
From my experience, it's worth playing with this concept. It might turn out to be an excellent addition to your toolbox for agile project management.
Subscribe to:
Posts (Atom)
