This time I'm writing about an item that is admittedly very specific to software development. More than once when I spoke to a members of a development team I was told "yes, we have an automated test suite". And yet, further along the conversation it turned out that despite a significant test suite the resulting quality wasn't where all the efforts put into creating those tests indicated it should be. And in all these cases when we then took a closer look at the tests themselves it turned out that at least one key element was missing.
That begs the question: What makes up a good test? What key characteristics should a good test have?
Setup, Execute, Validate
The first key ingredient is that a test consists of three parts: The first parts sets up the data that is needed for the test. This could be restoring a particular database content, it can be setting up a few objects in your programming language, it can be launching a particular user interface (e.g. browser) and many more. The second part is the actual execution of the test. This means you invoke functionality that modifies data. In the final third party you validate whether you have the expected outcome, e.g. the actual data is equal to the expected one.
Occasionally I've found, though, that people forget about the third step. I don't have data but suspect that this happens when people come from a background where executing a piece of code without crashing is almost a success. Think early/mids 90s of last century. C and C++ were still very dominant in the PC industry. Exceptions in the midst of running a program were nothing completely out of the ordinary. (Maybe you are the only one who never experienced them?) However, we can do better. Just because it doesn't crash with a nasty null pointer exception doesn't mean it performs as expected. Therefore at the end of a test always validate the outcome! The typical tool for that are the various assertions that come as part of test tools.
Repeatable
Not strictly a requirement but there are quite a few scenarios where running the same test more than once reveals and thereafter prevents certain bootstrapping type issue from happening. Assume your service implementation does some sort of housekeeping upon startup. The first time you invoke an operation on the service everything is still fine. But then perhaps as you repeat the same test (or set of tests) using operations on that service things go off track. Maybe connections are not properly closed. Maybe the service cannot handle more than 10 open connections at a time (rightly or wrongly). By repeating the same test over and over again chances increase are that discover a hidden issue and resolve it before your product is shipped.
Random Order
Tests should not depend on each other. A test should not require a different test to run first. If they do changes to one test may trigger further changes to other tests in the suite thus making changes more expensive and time consuming. You don't want to lose time. You want to be fast.
For example, lets assume you are working on a system that has Project as a concept and the name of the project becomes a unique identifier for each project. If all tests use the same project name for their tests, then each test would have to check during setup whether the project already exists. If it doesn't it would create it. The alternative would be to use a generated name in each test such as a string with the value "ProjectName" + RandomNumberAsString(). That way you make the tests independent from each other.
A corollary to this is that you can run each test in isolation, meaning you can run just that test focusing on the specific task at hand. You don't have to run other tests first and you don't have to remember the sequence for those other tests. You can - and probably want - to run the entire suite anyways once you have finished the coding part of your task.
Fast Equals Cheap
Why do tests to be fast? To an engineer time is money. The longer a test or test suite needs to execute the less likely it is that people will run the suite or portions of it. It a suite runs takes 1 minute to complete you probably run it before each commit. If a suite takes 5 hours you won't run it before each commit. So keep them fast, for example work with the smallest possible dataset, avoid or mock costly operations like talking to remote systems, databases, filesystems, anything that requires mechanical parts to move. Use in-memory databases (e.g. SQLite) instead of client-server systems such as Microsoft SQL Server or Oracle.
You may also want to continuously refactor your tests as well. Keep them lean and mean. Split the test suites into the ones you definitely want to run each time before you commit and those that are more expensive in terms of duration. Platform tests or scalability tests fall into the latter category.
Automated And Integrated
Tests have a value only when they are executed. As long as they are just sitting in your version control system they are useless. Make them work. Make them work hard. Use all those machines that sit idle while the developers using them during daytime are at home enjoying live. Integrate the execution of your automated test suites into your development processes. When tests and their execution are automated and integrated they are executed more frequently and each time a change in the code base has been committed.
Are Automated Tests Orphans In Your Team?
Automated tests are first class citizens and equally valuable as the product that you ship. Don't even think for a second that they are just an unavoidable side affect. It's not a tax report that you do because law says so. Instead fully integrated automated testing is the mechanism that allows your team to operate at full speed. Depending on your team size maintaining the testing infrastructure can well turn into a full-time job for a motivated engineer.
Treat your testing infrastructure at least as well as the other parts of your development infrastructure. Make all of it part of an end-to-end integrated development process that starts with a customer suggestion and ends with a deployed system operational for the very same customer. Automated testing is a key ingredient and the rules are simple to follow. No excuse any more for not improving software quality. Happy testing!
Saturday, January 30, 2010
Saturday, December 12, 2009
Where to start?
A question that I've been asked several times is where to start with improving your software engineering practices. Based on my experience, my answer typically is "It depends." Let me explain.
In one of my previous roles the release cycle was very long. This was not because we didn't want to release more often. It was simply because the system was very large and there were only few customers. Typically the upgrades would require significant manual intervention anyways. On the other hand, quality was a real issue and starting the improvements in that direction was a good choice by introducing a set of new tools, new designs, new technologies, new equipment, and new process. It was very focused on the inside of the development team.
In my current role we moved from one major release per year to monthly release cycles earlier this year. And although there were some concerns with it we were able to mitigate the impact in such a way that our clients can choose their own upgrade cycle. Whenever there is a feature of interest in a new release they can upgrade from their current version to the latest version in a one-step process. There is no need any longer to upgrade to the any of the versions in between. So, while we still offer new features each month, no client is force to upgrade on a monthly basis. There are, though, some clients who do, and each month there are clients who upgrade to that monthly release.
With this background it has become clear that just moving to monthly releases wasn't enough. Instead we combined it with significant efforts to simplify the upgrade process for our clients. And the feedback we get from clients in different geographies is very positive. Therefore we will continue improving the upgrade process so that it becomes even easier for our clients to move to later and even better versions of our software. With this ever-improving upgrade process in place we have established a delivery mechanism that allows us shipping new features faster to the market place. New features are picked up sooner and we receive feedback and suggestions faster as well. Our clients benefit from earlier availability of new features that in the turn allow them to run their business more efficiently.
In the second example I focused on the delivery process first - short release cycle combined with easier upgrades - while in the first example I focused on an improved engineering environment first. In the second example think of this: What if you had the perfect product but it would be a nightmare to upgrade? It would be very difficult to get improved versions installed on client's sites. If on the other hand you have a very simple upgrade process (and hence delivery process) you can roll out product improvements much faster.
There is no hard and fast rule what to do in each scenario. The key learning is that you need to identify what the determining factors are for your team's environment. Then create options and see how they would address the biggest challenges in your situation. Start in one place, and start small. Then observe and take the next step. And make sure you line up your team members. In particular their creativity and innovation are key elements to selecting the right starting point and to successfully move from there.
In one of my previous roles the release cycle was very long. This was not because we didn't want to release more often. It was simply because the system was very large and there were only few customers. Typically the upgrades would require significant manual intervention anyways. On the other hand, quality was a real issue and starting the improvements in that direction was a good choice by introducing a set of new tools, new designs, new technologies, new equipment, and new process. It was very focused on the inside of the development team.
In my current role we moved from one major release per year to monthly release cycles earlier this year. And although there were some concerns with it we were able to mitigate the impact in such a way that our clients can choose their own upgrade cycle. Whenever there is a feature of interest in a new release they can upgrade from their current version to the latest version in a one-step process. There is no need any longer to upgrade to the any of the versions in between. So, while we still offer new features each month, no client is force to upgrade on a monthly basis. There are, though, some clients who do, and each month there are clients who upgrade to that monthly release.
With this background it has become clear that just moving to monthly releases wasn't enough. Instead we combined it with significant efforts to simplify the upgrade process for our clients. And the feedback we get from clients in different geographies is very positive. Therefore we will continue improving the upgrade process so that it becomes even easier for our clients to move to later and even better versions of our software. With this ever-improving upgrade process in place we have established a delivery mechanism that allows us shipping new features faster to the market place. New features are picked up sooner and we receive feedback and suggestions faster as well. Our clients benefit from earlier availability of new features that in the turn allow them to run their business more efficiently.
In the second example I focused on the delivery process first - short release cycle combined with easier upgrades - while in the first example I focused on an improved engineering environment first. In the second example think of this: What if you had the perfect product but it would be a nightmare to upgrade? It would be very difficult to get improved versions installed on client's sites. If on the other hand you have a very simple upgrade process (and hence delivery process) you can roll out product improvements much faster.
There is no hard and fast rule what to do in each scenario. The key learning is that you need to identify what the determining factors are for your team's environment. Then create options and see how they would address the biggest challenges in your situation. Start in one place, and start small. Then observe and take the next step. And make sure you line up your team members. In particular their creativity and innovation are key elements to selecting the right starting point and to successfully move from there.
Saturday, July 25, 2009
Mini SWAT Team
Similar to the build master role - I posted about that a few weeks ago - you may find occasionally that there are other topics that you want to overemphasize for a certain period. The build master is specifically taking care of continuously improving the engineering and build environment.
But you may have items that are either to beyond the available build master capacity or don't even fall under the build master's duties. Or you want to speed things up. Let me give you two examples.
Let's say you want to introduce a new testing tool that you haven't used in the past. The tool arrives. While rolling it out you find that there are challenges and issues that need to be addressed. Not that you are surprised but the amount of work is way beyond of what your build master (or build master team) can do within their time. So what do you do?
Or you want to migrate your build environment towards a substantially improved environment, more powerful, more flexible, maybe involving virtual machines, new technologies, and so fourth. A steep learning curve is guaranteed. What do you do?
In both cases you can create a "Mini SWAT Team". This can even be as small as one person. The mini SWAT team would be dedicate to one particular subject for a period of time and then wrapped up. In the first example the introduction of the testing tool could be sped up by working with the QA team and by working with the developers. In the second example the mini SWAT team could work with IT on the environment improvements.
Who would you put on your Mini SWAT Team? Certainly you'd look for someone who can work independently but also in a team. People who can work with different functions are certainly advantaged. But then, you also have a quite different option as well: Give it to a more junior member of your team. Provide a lot of support and you'll find that the person steps up and grows with the job, enjoys the responsibility and the opportunity to pull something through that is of value to the whole team.
Food for thought? I hope so. It's all about trying something new and then adapting as you go!
But you may have items that are either to beyond the available build master capacity or don't even fall under the build master's duties. Or you want to speed things up. Let me give you two examples.
Let's say you want to introduce a new testing tool that you haven't used in the past. The tool arrives. While rolling it out you find that there are challenges and issues that need to be addressed. Not that you are surprised but the amount of work is way beyond of what your build master (or build master team) can do within their time. So what do you do?
Or you want to migrate your build environment towards a substantially improved environment, more powerful, more flexible, maybe involving virtual machines, new technologies, and so fourth. A steep learning curve is guaranteed. What do you do?
In both cases you can create a "Mini SWAT Team". This can even be as small as one person. The mini SWAT team would be dedicate to one particular subject for a period of time and then wrapped up. In the first example the introduction of the testing tool could be sped up by working with the QA team and by working with the developers. In the second example the mini SWAT team could work with IT on the environment improvements.
Who would you put on your Mini SWAT Team? Certainly you'd look for someone who can work independently but also in a team. People who can work with different functions are certainly advantaged. But then, you also have a quite different option as well: Give it to a more junior member of your team. Provide a lot of support and you'll find that the person steps up and grows with the job, enjoys the responsibility and the opportunity to pull something through that is of value to the whole team.
Food for thought? I hope so. It's all about trying something new and then adapting as you go!
Thursday, July 02, 2009
Taking turns in being the build master
The build master is a quite important role in an software development team. Responsibilities include ensuring the build works all the time and chasing down people who are fixing a broken build. Other items may include improving the build process which should also include the execution of comprehensive test suites in all test beds.
While on one hand it is a big help if a person in this role has prior experience with these tasks. On the other hand - and that doesn't have to be a competing goal - it is also valuable if team members take turns.
Let me explain. In all teams that I have coached I have found that different people have different strengths. For example one person might be excellent in writing Perl scripts while a different person is extremely proficient in relational database systems. By taking turns each individual can emphasize their area of expertise in the build master role. That way, if the skill is important to the whole team, the skill is also important for the build master role. Eventually all areas of expertise will get addressed eventually.
One more benefit of taking turns is that all team members walk in "the build master's shoes". There is a much better understanding of and buy-in by the team for build master related tasks and challenges. The response to being chased down because of a broken build will be tainted quite differently than if the same person is in the build master role.
At the moment I'm experimenting with swapping the build master role at the end of each release cycle (currently one month). And already the above mentioned effects have become visible.
Also check out "Manfred's DotNet Blog"
While on one hand it is a big help if a person in this role has prior experience with these tasks. On the other hand - and that doesn't have to be a competing goal - it is also valuable if team members take turns.
Let me explain. In all teams that I have coached I have found that different people have different strengths. For example one person might be excellent in writing Perl scripts while a different person is extremely proficient in relational database systems. By taking turns each individual can emphasize their area of expertise in the build master role. That way, if the skill is important to the whole team, the skill is also important for the build master role. Eventually all areas of expertise will get addressed eventually.
One more benefit of taking turns is that all team members walk in "the build master's shoes". There is a much better understanding of and buy-in by the team for build master related tasks and challenges. The response to being chased down because of a broken build will be tainted quite differently than if the same person is in the build master role.
At the moment I'm experimenting with swapping the build master role at the end of each release cycle (currently one month). And already the above mentioned effects have become visible.
Also check out "Manfred's DotNet Blog"
Saturday, May 30, 2009
The Choice of Not Being an Elephant
Lou Gerstner described in his book Who Says Elephants Can't Dance?: Leading a Great Enterprise through Dramatic Change
how to make even a very large organization more agile, more adaptive to changing conditions. IBM had no other choice since some of their old business models stopped working. When I speak with people who were decision makers in the 1970s they unanimously told me that there was a time where the sales guys in dark blue dresses almost walked in to their customers and just said "sign down here". The customer had little choice.
Even if these stories are exaggerated - I'm not a sales guy but I'm sure their stories are always true! - there is one interesting aspect to this. I'm also mentioning this since I just came back from a trade show and there I spoke to a manager of one of the largest vendors in that particular industry. Somehow that person made me think of those stories from the past about IBM.
Like IBM in the IT industry this person's company is a key player in their industry. And the way the person came across was almost as if he was saying: "We know what is good for the industry, for you. Just do what we say, sign here and everything will be right." Sounds to me a little bit like "We are the center of the universe and all other companies better lign up around us."
The conversation left me wondering by when their CEO will write a book about how she had to turn their company around? Maybe they haven't noticed yet that the downturn in the industry probably requires them changing their attitude as well. But maybe I'm totally wrong and they still have full order books and the customers are lining up to the horizon. Not long ago, however the news were saying they are in the process to make a large number of people redundant.
That doesn't sound to me like full order books... The news of redundancies along with the attitude of the person I spoke to appear more like a company that isn't yet set up flexible and adaptive enough. Maybe reading Lou Gerstner's book helps.
Also check out "Manfred's DotNet Blog"
how to make even a very large organization more agile, more adaptive to changing conditions. IBM had no other choice since some of their old business models stopped working. When I speak with people who were decision makers in the 1970s they unanimously told me that there was a time where the sales guys in dark blue dresses almost walked in to their customers and just said "sign down here". The customer had little choice.
Even if these stories are exaggerated - I'm not a sales guy but I'm sure their stories are always true! - there is one interesting aspect to this. I'm also mentioning this since I just came back from a trade show and there I spoke to a manager of one of the largest vendors in that particular industry. Somehow that person made me think of those stories from the past about IBM.
Like IBM in the IT industry this person's company is a key player in their industry. And the way the person came across was almost as if he was saying: "We know what is good for the industry, for you. Just do what we say, sign here and everything will be right." Sounds to me a little bit like "We are the center of the universe and all other companies better lign up around us."
The conversation left me wondering by when their CEO will write a book about how she had to turn their company around? Maybe they haven't noticed yet that the downturn in the industry probably requires them changing their attitude as well. But maybe I'm totally wrong and they still have full order books and the customers are lining up to the horizon. Not long ago, however the news were saying they are in the process to make a large number of people redundant.
That doesn't sound to me like full order books... The news of redundancies along with the attitude of the person I spoke to appear more like a company that isn't yet set up flexible and adaptive enough. Maybe reading Lou Gerstner's book helps.
Also check out "Manfred's DotNet Blog"
Wednesday, May 20, 2009
How Pair Programming Reduces Distraction
Pair programming is not for everyone. Some people love it, some people hate it. It almost seems there is nothing in the middle.
I'm more in the camp of people preferring pair programming. For many different reasons but I keep the discussion for a different post.
In this one I'd like to describe an example for what happened to me yesterday. I was working with one or my fellow software engineers, Jason, and while we were looking through a nasty C++ link problem we discovered a small item in an adjacent code area that we felt should be cleaned up. So I took a card and wrote it down as a story and returned back to the problem at hand.
Jason was surprised and made comment saying that by taking a note we would actually avoid being distracted. Although I have worked like that for years my colleague is newer to pair programming and that was probably the reason why he noticed what I did. For me it was just normal mode of operation. I did it unconsciously.
Based on my colleagues observation I think there is a lesson to be learned: First there is the specific case. When you come across something that should be done then just take a note (create a 'story' card). Don't let yourself get distracted. That way there is only a small interruption and then you are back on the problem at hand. Secondly, don't assume that the way you work is normal in the sense that everybody knows all the techniques you are using. It pays off to make yourself aware of this and sometimes it takes a good observer like Jason making a comment to become aware of what you are actually doing. So keep your ears open!
I'm more in the camp of people preferring pair programming. For many different reasons but I keep the discussion for a different post.
In this one I'd like to describe an example for what happened to me yesterday. I was working with one or my fellow software engineers, Jason, and while we were looking through a nasty C++ link problem we discovered a small item in an adjacent code area that we felt should be cleaned up. So I took a card and wrote it down as a story and returned back to the problem at hand.
Jason was surprised and made comment saying that by taking a note we would actually avoid being distracted. Although I have worked like that for years my colleague is newer to pair programming and that was probably the reason why he noticed what I did. For me it was just normal mode of operation. I did it unconsciously.
Based on my colleagues observation I think there is a lesson to be learned: First there is the specific case. When you come across something that should be done then just take a note (create a 'story' card). Don't let yourself get distracted. That way there is only a small interruption and then you are back on the problem at hand. Secondly, don't assume that the way you work is normal in the sense that everybody knows all the techniques you are using. It pays off to make yourself aware of this and sometimes it takes a good observer like Jason making a comment to become aware of what you are actually doing. So keep your ears open!
Friday, May 01, 2009
How fast do you adapt?
Here is a quote from today's online edition of The Wall Street Journal:
(Source and more details: The Wall Street Journal, retrieved 01 May 09)
What caught my eye was the word "adapt". By the looks of it - I agree with the assessment - Chrysler didn't adapt fast enough. Now it has filed for Chapter 11, a bankruptcy process that protects it while going through a restructuring.
I'm not happy to see anything like that because many people will be severely affected in their personal lives. However, this is an (extreme) example of what can happen if you don't move fast enough as a company. Chrysler's product mix is no longer a good enough fit for what the market demands. The economic downturn just amplified that.
What can we learn from it? If we don't adapt fast enough to changing market demands and conditions it could mean very bad times for the company or the end of the company. Need any more incentive to adapt?
President Obama said that Chrysler has been "a pillar" of the industrial economy but that the company moved too slowly to adapt to a changing market.
(Source and more details: The Wall Street Journal, retrieved 01 May 09)
What caught my eye was the word "adapt". By the looks of it - I agree with the assessment - Chrysler didn't adapt fast enough. Now it has filed for Chapter 11, a bankruptcy process that protects it while going through a restructuring.
I'm not happy to see anything like that because many people will be severely affected in their personal lives. However, this is an (extreme) example of what can happen if you don't move fast enough as a company. Chrysler's product mix is no longer a good enough fit for what the market demands. The economic downturn just amplified that.
What can we learn from it? If we don't adapt fast enough to changing market demands and conditions it could mean very bad times for the company or the end of the company. Need any more incentive to adapt?
Subscribe to:
Posts (Atom)
