The Continuous Integration Build Game

Originally published 14 Sep 2007

I was reading Alistair Cockburn's Agile Software Development book the other night. In it, he described a game developed by Darin Cummins to reinforce good development practices. The game is described here. This inspired me to create a fun game for continuous integration builds.

As I described in a previous blog entry, I've had problems in the past with people breaking the build. To help, I've used techniques like the rotating Build Nazi and the put a dollar in the broken build jar, but these are all negative focused. How about something that rewards developers that don't break the build? How about rewarding developers for following the best practice of breaking their work into smaller chunks and checking in early and often?

So I'm thinking of a game where a developer gets, say 1 point for getting his name on a successful build. The number of files checked in is ignored. We want to discourage big check-ins, so you get more points for smaller grained check-ins since your name will show up on more successful builds. And on the other side, you get points taken away when you break the build. Now, we want to keep things simple, so we could probably stop right here, but I'm thinking that some failures are worse than others. So maybe we have something like:

Description	Reward Points
Check-in and build passes	1
One or more unit tests failed	-10
Compiler error (come on now)	-20
Big check-in caused build to remain in a broken state for hours	-40

It's a game, so we need something for the winner. Bragging rights is too lame, so maybe lunch on the team, or some kind of trophy kept on the winner's desk to remind everybody of his champion status.

Now, there are some negatives. Perhaps not everybody would want to play. Particularly, notorious build breakers wouldn't want everybody (specifically management) to see their poor results. In that case, I suppose we could only publicly display the leaders, or top half, but that wouldn't be as fun.

People could easily cheat too. Maybe, write a cron job that every hour checks out a file, changes a comment and checks back in. We'd have to look out for that kind of thing.

What about the analysis time required to keep score? I could easily see how a post processing ant task could be developed to update the points for developers on a successful build. But for a failure, I think you'd need human analysis. That's a negative because it requires time, so the job could be rotated. On the plus side, what I've noticed is that analyzing why the build failed brings awareness to issues. Issues like some people needing training, or a test requiring change because it's non-deterministic, or a hole in the pre-check-in process used to ensure a successful build.

To keep the game fresh and the developers motivated, we'd have to reset the points periodically. Iteration boundaries seem appropriate here.

Well, maybe I'll give it a shot and see what happens...

The todo Test Category

Originally published 26 Jan 2007

Recently, Elliotte Rusty Harold blogged about committing test cases for bugs to the build cycle before fixing them right away. He enumerated several good reasons for doing this such as the fix being difficult and time consuming. His point was that breaking the build (where breaking the build includes passing all the unit tests) is okay in some cases.

I disagree with this stance, but I think Harold has a really good idea there, which I'll get to in a moment. I think breaking the build does include passing all the unit tests and should be part of the continuous integration process. I've always liked the idea of the tests serving as a safety net when I add new functionality or refactor existing code. The tests not only specify the desired behavior, but act as a regression suite.

So what's the idea? I first learned of the concept of test categorization from Cedric Beust's TestNG framework and the writings of disco king Andrew Glover . Essentially, the concept is to break tests into groups such as unit, component, and system and run them at different intervals. Unit tests are run many times a day as code is developed, component tests less frequently, and system tests the least frequently of all. You can categorize tests in several different ways such as by file naming conventions, placing them in different directories, or using Java annotations.

Harold brings up a new category - the todo category. (I was originally thinking the fixme category, but I thought todo could serve more purposes.) The todo category contains all tests that serve as a reminder of work to do. They obviously fail or there would be nothing to do. When it's finally time, the test moves out of the todo category and into the unit, component, or system category as appropriate. Although you could have a todo category for each of the original, regression categories (todoUnit, todoComponent, and todoSystem), I think that would be overkill. Notice that I slipped the word regression into "original categories" above. I'm advocating that the original categories should pass at all times - they should break the build if they fail (well, at a minimum the unit category). Once tests get into the regression categories, they never go back to todo.

The obvious downside of this approach and of test categorization in general is the added complexity. It's more work to maintain categories and in this case, move tests into different categories than a simple one category fits all approach. Also, there's more work (at least initially) to setup the running of those categories. As always, I recommend weighing the options and deciding what's best for your particular situation. See Glover's writings for more benefits of test categorization.

Break, Then Leave and Pre-Check In Checklist

Originally published 16 Sep 2006

My build/SCM blog series (see part 1 and part 2) ends with a look at one of the most annoying things a member can do to his team and a checklist to follow for reducing the chances of breaking a build.

So what can really irritate the rest of team? How about checking in a bunch of changes that break the build and then leaving for the day? This is bad enough for small teams, but consider bigger, distributed teams developing complex applications. You could have team members who start their day after you've left. If the team follows the rule of not checking in on a broken build, what's the team supposed to do? Well, to continue progress, somebody else has to step in and fix the build. This can really slow a team down.

A good practice is to wait for a successful build after your last check in before leaving. After all, you know the most about the changes that would have broken the build. This practice implies some prerequisites: a continuous integration system that runs often and builds that run quickly.

Now that I've ranted enough, what can you do to reduce the chances of breaking a build in the first place? Listed below are steps to follow. It starts at the point where you're ready to check in. Essentially, you mimic the continuous integration system on your development workstation.

1. Update your view so you have the latest versions of all the files.
2. If you use configuration management software that allows hijacking and reserving files (like Rational ClearCase), now is the time to "unhijack" and reserve existing files with changes. By existing files, I mean files the CM software knows about (not new). You only need to reserve files if your team follows that practice. If nobody on the team reserves files, then skip this part, but it's important to reserve at this point if your team follows this practice to ensure all of your changes can get checked in.
3. If necessary, merge any existing files that have been updated and checked in by others since you checked out. You want your files to represent what will be checked in. Now, your files will contain your changes merged with the most recently checked in versions.
4. Make note of any new files that need to be added to source control and any deleted files that must be removed from source control. Forgetting to do this can cause a build failure.
5. Run the application to make sure your changes function and integrate well.
6. Do a clean build, compile and run all the unit tests. Consider a single ant target that does this for you.
7. Check in. Don't forget those new and deleted files!

Do I follow all of these steps on every check in? No way. That wouldn't be practical. You need to consider the size and impact of your changes. The bigger they are, the more rigorous you should be. If I made just a small change, I would probably just make sure the code compiled and unit tests for that change passed.

Well, that's it. Hope you enjoyed.

After the Build Breaks

Originally published 28 Aug 2006

The second part of the build/SCM blog series (see part 1) deals with what happens when a build breaks. This happens, hopefully infrequently, but it happens. Getting the build back on track should be one of the highest priorities - your build box teammate is down and you need to get him back on his feet.

Ideally, what should happen is the following:

1. Notification of a build failure is distributed by e-mail to the team. This is part of the publishing functionality of CI systems like CruiseControl.
2. Particularly, whoever checked in since the last successful build analyzes the results.
3. The person who broke the build mans up and replies to all, “This is me. I’m on it.”
4. Nobody checks in until the build is fixed unless the check in is for the purpose of fixing the build.
5. Somebody fixes the build, obtaining any help necessary.
6. (Optional) Once the build is believed to be fixed, the person from step 3 sends an e-mail saying the build should be fixed.

Let’s talk more about step 3. I know it could be embarrassing to break the build. The step is not to call anyone out or lay blame. The point is to communicate to the team, who may be large and distributed, that somebody is taking responsibility for fixing the build. Several others may be analyzing and trying to fix the problem and this communication is to reduce that wasted effort. I have been on teams where the build has been broken for hours and nobody knows if anybody is doing anything about it.

Sometimes nobody is doing anything about it. If this is a recurring problem, I recommend the use of a Build Nazi (in honor of the Seinfeld Soup Nazi). The job of the Build Nazi is to make sure somebody is responsible for fixing a broken build. The job is not to fix the build (unless help is needed). The Build Nazi job is not a fun one and therefore should be rotated. It is also a controversial role in an agile environment where the practice of collective code ownership is being followed. Ideally, the team is following the steps above and a Build Nazi is totally unnecessary. I’ve found I’ve had to resort to the role for short term periods in times of chaos when the build is breaking more than it is succeeding. Once things get back on track, the Build Nazi role typically becomes dormant.

The last step I’d like to elaborate on is step 4. One of the worst things to do is check in a bunch of changes that don’t have anything to do with fixing the build. Checking in causes the CI system to start another build that will result in another failure. Piling on check ins during a broken build prolongs the broken state of the build and therefore the feedback cycle that is all important in agile development. Another risk is that the fix necessary to correct the original problem must be applied to the check ins that occurred after the first failure, further prolonging the broken state.

I plan on finishing this series next time. Stay tuned.

Checklist for Finding Test Cases

Originally published 4 Jun 2006

Suppose I'm focused on specifying the behavior of a particular method. How do I know if I've specified everything? I'd like to share a checklist of things I think about to help accomplish this. The goal is to thoroughly specify the behavior of the method by building up a collection of tests following test-driven development.

I'm going to base my example on Spring's BeanFactory interface. Suppose I'm developing a Map backed BeanFactory and I am currently focused on implementing the following method of the BeanFactory interface:

Object getBean(String beanName) throws BeansException

Let's call the class I'm developing MapBackedBeanFactory. When getBean() is called, the implementation will look up the bean in its beanMap field.

When I think about test cases, I think about:

1. The state of the object under test (each field), i.e., the fixture. In the example, this would be the beanMap.
2. The state of the parameters of the method I'm calling on the object under test. In the example, there's only one: the beanName.

That was pretty obvious. Who doesn't think about these things? But what is the checklist? The checklist consists of thinking about the following for each of the above:

1. A good value. This will execute the normal flow and is the most obvious test. For beanMap, this means the beanMap will have a good bean mapping. For the beanName, a good value is a name that will be found in the beanMap. There may be multiple good values, not really in this case, but think about the classic bowling game. You'd want tests for a non-mark frame, a spare, and a strike. I'll also consider multiple good values for things like collections, maps, and arrays. That is, maybe I'm dealing with a  List and I need a List of two good things. However, I find that most times, one good value in the collection, map, or array is sufficient.  This is because I write the iteration code for the single case and a multiple collection test case wouldn't make me write any new code.  If I find I need multiple good values, I'll start with the simple cases and work towards the harder.
2. A negative case. In #1, I set up the fixture and parameters to get a good result.  Now, I want to think of the negative side.  So I found beans in #1, but now I want to not find a bean.  Pass in an unknown beanName.
3. A null value. What if the beanMap were null? That's probably not realistic, so I'm going to skip that. What if the beanName were null? That's possible and according to the BeanFactory JavaDoc, it looks like the contract specifies to throw a NoSuchBeanDefinitionException. I better write that test. Many times, if you don't handle null, you'll get a NullPointerException and that might be acceptable. In those cases, I wouldn't write a test (since it wouldn't require any code).
4. An empty value. The beanMap could have no mappings or the beanName could be "". I think #2 covered us here, so no test is required.
5. An invalid value. What if beanMap were invalid? What if it were a Map of Integers to bean Objects? I'm going to assume this is unrealistic in this case. We would have handled such a situation when the map was loaded.  What if the beanName were "invalid"? Well, we covered that in #2.
6. An exceptional case. I'm pretty much talking about handling exceptions here. Maybe we need to write a test case to see if we handle an exception properly. The example doesn't have such a case, but suppose the method under test were doing some I/O. Maybe we'd need to write a test to see if we handled that properly. In all honesty, even though I think about this, I don't usually write a test to handle exceptions. I won't get 100% test coverage, but I usually ignore writing the test because checked exceptions force me to write the necessary handling code.

That's a lot of stuff to think about for each object field/parameter. That's the biggest negative - thinking about all those things, all the possibilities, all the time it takes. Many times, as you saw in the example, most of those things just aren't realistic.

Sometimes I'll take a different approach. I'll start with the simple, good case. I'll write the test and get it to pass by writing the minimal code to do so.  In the example, I'd have:

return beanMap.get(beanName);

Then, I'll look at the method I just wrote word by word (or maybe token by token).  At each word, I'll think about special cases (things like in the checklist) and jot down test cases. Then, I'll go back to the test, write one of the jotted-down tests, get it to pass, and continue until I'm done. It's a different perspective that goes a little faster at the risk of not quite catching everything as the first approach.  I like this second approach and have been doing it more and more lately.

Coding By Intention

Originally published 10 Jan 2006

I thought I'd start blogging about something I think I'm pretty decent at - programming.  Yeah, yeah, I know, everybody thinks they are the best programmer, but I think I have some decent stuff to share.  I wanted to start with programming tips.  I'll try one tip per blog entry.
The first tip is coding by intention. The first time I learned of this was in college. I was reviewing some code with a professor of mine, Dave Binkley of Loyola College. We were reviewing a long method of mine and Professor Binkley said something equivalent to, “I like to extract a method here to make it more clear what I’m doing.” The light came on. I learned that programming was more than just a science, but an art. This was back in the early 90s, before the Extreme Programming/Agile folks coined terms like “extract method”, “refactor”, and “code by intention”.

Now, I'm a big fan of Test-Driven Development (TDD).  Since I've got to write a test first, I start there.  The real code doesn't exist yet.  The intent part starts here.  I think, "I've got this piece of functionality to implement.  What would I call if I could write anything?  What would the method name be and its arguments?  How would I call this functionality as a client so that what I'm doing is crystal clear? The world is my oyster."

Coding by intention from the point of view of the test is pretty obvious.  What I wanted to emphasize in this blog entry is to continue coding by intention all the way through until you've completely implemented everything.  So after you've created a new call in the test, you start implementing it.  It'll most likely be a public method.  When you're implementing this, you continue coding by intention.  You could just crank out a possibly long method to get the functionality implemented, but that's not what I'm talking about – that’s what I did in the early days of college.  The public method now becomes like another client.  You code by intention here, asking yourself what you would call if what you needed already existed.  You keep doing this with every method you need to implement, creating lots of small, but communicative and easy to understand methods.  Eventually, you'll get to a level where the implementation is small and simple and you're done.  What you've done is created some possibly reusable and very easy to maintain methods along the way.

It’s time for an example. Suppose I’m working with the usual Order/Line Items web site shopping example. One coding by intention technique I use when I need to process a collection like LineItems looks like this:

    public void processLineItems(List lineItems) {

        Iterator it = lineItems.iterator();

        while (it.hasNext()) {

            process((LineItem) it.next());

        }

    }

Instead of writing all the code in the while loop to process an item, I code by intention, pretending that the process(LineItem) method already exists.
Here’s another, albeit obviously contrived example. Suppose I’m working on Goldilocks’ Porridge class and need to implement the isJustRight() method. Right now I just have:

    public class Porridge {

        private double currentTemperature = 0.0;

       

        public Porridge(double currentTemperature) {

            this.currentTemperature = currentTemperature;

        }

    }

To implement isJustRight(), in coding by intention style, I would do:

        public boolean isJustRight() {

            return (notTooHot() && notTooCold());

        }

So why code by intention? There are several reasons I can think of off the bat. The first is, as I’ve stated above, you end up creating small, possibly reusable methods. My first tip was going to be “create small methods”, but I realized that coding by intention drives that. In my next entry, I’ll go into more detail on why I think creating small methods is really important. There’s a process to crafting elegant software and this is the first step.

Another reason is that the top level methods become really easy to understand.
They describe the algorithm in almost pseudo-code level. They communicate your intent of what you’re trying to accomplish step by step.

Yet another reason is that it is sometimes hard to get started on a task and by coding by intention you are delaying all the details, but still getting things done. That is, not everything you need is available, but you’re coding like it is. You’re working top down, drilling down into the details level by level until your done, making progress along the way.

Now, if you’re doing true TDD, you are writing just enough test to get a failing test, and just enough code to pass that test. Remember, compiler errors count as failures. I’m a little lenient on this practice. I like writing an entire test method for one test case scenario before going to the code. So in the isJustRight() method above, I’d first write a complete test method for a Porridge object that is too cold with all its calls and assertions. Then, I’d write just enough code to get the test to pass, but still coding by intention in the Porridge class. So, in this case, the isJustRight() method would just call notTooCold() for now.

I suppose if you truly wanted to follow TDD, and in particular, get a passing test as fast as you could, you could just write the code in isJustRight(), then extract a notTooCold() method. Just remember to do this. Sometimes, we as programmers get a little lazy. Still, I favor coding by intention top down from the beginning.

Well, that’s it for my first blog. Hope you enjoyed.