Originally published 13 Nov 2008
Subversion is the de facto version control system of the day, but Git is the rising star. More and more people are using Git, but I'm a bit concerned about its effects on Continuous Integration (CI).
First some background...
Subversion follows a centralized repository model. So does CVS and many others. There's one server that everybody commits to. Git can follow suit or be configured in a distributed fashion. In a fully distributed model, each developer has a private copy of the repository. Mercurial is an example of a distributed version control system.
My concern is really with the distributed model. One of the appeals of this model, and particularly with having your own private repository, is the ability to experiment and check-in/commit at a much finer-grained rate than with a centralized model. With a centralized model, you need to be more careful of your commits. Otherwise, you could break the build and everybody else. So consequently, you commit less frequently here.
But what's better in the context of CI? I have to be honest and admit that I've never used the distributed model on a project, so I'm being theoretical here. My gut feel is that with a distributed model, integration will be less frequent. I believe people will check in to private repositories more often, but will push those changes to the main integration branch less frequently than people committing straight to the main branch in a centralized model. In a centralized model, integration is in your face. You can't commit without thinking about it. In a distributed model, you can get carried away in your own little world. And we've learned in the agile community that we should be integrating early and often, haven't we?
I compare the distributed model with multiple repositories to a centralized model with multiple branches. If you can accept this analogy, you can probably see how integration would be less frequent. I fear with Git that people will adopt more of a distributed model and thus, CI will suffer. This is pure speculation on my part, and I'm interested to see how things play out.
So my point here is to be mindful of using lots of repositories with Git and its potential negative consequences on CI.
For some more comments on using Git and CI, particularly on larger teams, see this.
Showing posts with label scm. Show all posts
Showing posts with label scm. Show all posts
November 22, 2010
The Continuous Integration Build Game
Originally published 14 Sep 2007
I was reading Alistair Cockburn's Agile Software Development book the other night. In it, he described a game developed by Darin Cummins to reinforce good development practices. The game is described here. This inspired me to create a fun game for continuous integration builds.
As I described in a previous blog entry, I've had problems in the past with people breaking the build. To help, I've used techniques like the rotating Build Nazi and the put a dollar in the broken build jar, but these are all negative focused. How about something that rewards developers that don't break the build? How about rewarding developers for following the best practice of breaking their work into smaller chunks and checking in early and often?
So I'm thinking of a game where a developer gets, say 1 point for getting his name on a successful build. The number of files checked in is ignored. We want to discourage big check-ins, so you get more points for smaller grained check-ins since your name will show up on more successful builds. And on the other side, you get points taken away when you break the build. Now, we want to keep things simple, so we could probably stop right here, but I'm thinking that some failures are worse than others. So maybe we have something like:
It's a game, so we need something for the winner. Bragging rights is too lame, so maybe lunch on the team, or some kind of trophy kept on the winner's desk to remind everybody of his champion status.
Now, there are some negatives. Perhaps not everybody would want to play. Particularly, notorious build breakers wouldn't want everybody (specifically management) to see their poor results. In that case, I suppose we could only publicly display the leaders, or top half, but that wouldn't be as fun.
People could easily cheat too. Maybe, write a cron job that every hour checks out a file, changes a comment and checks back in. We'd have to look out for that kind of thing.
What about the analysis time required to keep score? I could easily see how a post processing ant task could be developed to update the points for developers on a successful build. But for a failure, I think you'd need human analysis. That's a negative because it requires time, so the job could be rotated. On the plus side, what I've noticed is that analyzing why the build failed brings awareness to issues. Issues like some people needing training, or a test requiring change because it's non-deterministic, or a hole in the pre-check-in process used to ensure a successful build.
To keep the game fresh and the developers motivated, we'd have to reset the points periodically. Iteration boundaries seem appropriate here.
Well, maybe I'll give it a shot and see what happens...
I was reading Alistair Cockburn's Agile Software Development book the other night. In it, he described a game developed by Darin Cummins to reinforce good development practices. The game is described here. This inspired me to create a fun game for continuous integration builds.
As I described in a previous blog entry, I've had problems in the past with people breaking the build. To help, I've used techniques like the rotating Build Nazi and the put a dollar in the broken build jar, but these are all negative focused. How about something that rewards developers that don't break the build? How about rewarding developers for following the best practice of breaking their work into smaller chunks and checking in early and often?
So I'm thinking of a game where a developer gets, say 1 point for getting his name on a successful build. The number of files checked in is ignored. We want to discourage big check-ins, so you get more points for smaller grained check-ins since your name will show up on more successful builds. And on the other side, you get points taken away when you break the build. Now, we want to keep things simple, so we could probably stop right here, but I'm thinking that some failures are worse than others. So maybe we have something like:
| Description | Reward Points |
|---|---|
| Check-in and build passes | 1 |
| One or more unit tests failed | -10 |
| Compiler error (come on now) | -20 |
| Big check-in caused build to remain in a broken state for hours | -40 |
It's a game, so we need something for the winner. Bragging rights is too lame, so maybe lunch on the team, or some kind of trophy kept on the winner's desk to remind everybody of his champion status.
Now, there are some negatives. Perhaps not everybody would want to play. Particularly, notorious build breakers wouldn't want everybody (specifically management) to see their poor results. In that case, I suppose we could only publicly display the leaders, or top half, but that wouldn't be as fun.
People could easily cheat too. Maybe, write a cron job that every hour checks out a file, changes a comment and checks back in. We'd have to look out for that kind of thing.
What about the analysis time required to keep score? I could easily see how a post processing ant task could be developed to update the points for developers on a successful build. But for a failure, I think you'd need human analysis. That's a negative because it requires time, so the job could be rotated. On the plus side, what I've noticed is that analyzing why the build failed brings awareness to issues. Issues like some people needing training, or a test requiring change because it's non-deterministic, or a hole in the pre-check-in process used to ensure a successful build.
To keep the game fresh and the developers motivated, we'd have to reset the points periodically. Iteration boundaries seem appropriate here.
Well, maybe I'll give it a shot and see what happens...
Break, Then Leave and Pre-Check In Checklist
Originally published 16 Sep 2006
My build/SCM blog series (see part 1 and part 2) ends with a look at one of the most annoying things a member can do to his team and a checklist to follow for reducing the chances of breaking a build.
So what can really irritate the rest of team? How about checking in a bunch of changes that break the build and then leaving for the day? This is bad enough for small teams, but consider bigger, distributed teams developing complex applications. You could have team members who start their day after you've left. If the team follows the rule of not checking in on a broken build, what's the team supposed to do? Well, to continue progress, somebody else has to step in and fix the build. This can really slow a team down.
A good practice is to wait for a successful build after your last check in before leaving. After all, you know the most about the changes that would have broken the build. This practice implies some prerequisites: a continuous integration system that runs often and builds that run quickly.
Now that I've ranted enough, what can you do to reduce the chances of breaking a build in the first place? Listed below are steps to follow. It starts at the point where you're ready to check in. Essentially, you mimic the continuous integration system on your development workstation.
Well, that's it. Hope you enjoyed.
My build/SCM blog series (see part 1 and part 2) ends with a look at one of the most annoying things a member can do to his team and a checklist to follow for reducing the chances of breaking a build.
So what can really irritate the rest of team? How about checking in a bunch of changes that break the build and then leaving for the day? This is bad enough for small teams, but consider bigger, distributed teams developing complex applications. You could have team members who start their day after you've left. If the team follows the rule of not checking in on a broken build, what's the team supposed to do? Well, to continue progress, somebody else has to step in and fix the build. This can really slow a team down.
A good practice is to wait for a successful build after your last check in before leaving. After all, you know the most about the changes that would have broken the build. This practice implies some prerequisites: a continuous integration system that runs often and builds that run quickly.
Now that I've ranted enough, what can you do to reduce the chances of breaking a build in the first place? Listed below are steps to follow. It starts at the point where you're ready to check in. Essentially, you mimic the continuous integration system on your development workstation.
- Update your view so you have the latest versions of all the files.
- If you use configuration management software that allows hijacking and reserving files (like Rational ClearCase), now is the time to "unhijack" and reserve existing files with changes. By existing files, I mean files the CM software knows about (not new). You only need to reserve files if your team follows that practice. If nobody on the team reserves files, then skip this part, but it's important to reserve at this point if your team follows this practice to ensure all of your changes can get checked in.
- If necessary, merge any existing files that have been updated and checked in by others since you checked out. You want your files to represent what will be checked in. Now, your files will contain your changes merged with the most recently checked in versions.
- Make note of any new files that need to be added to source control and any deleted files that must be removed from source control. Forgetting to do this can cause a build failure.
- Run the application to make sure your changes function and integrate well.
- Do a clean build, compile and run all the unit tests. Consider a single ant target that does this for you.
- Check in. Don't forget those new and deleted files!
Well, that's it. Hope you enjoyed.
After the Build Breaks
Originally published 28 Aug 2006
The second part of the build/SCM blog series (see part 1) deals with what happens when a build breaks. This happens, hopefully infrequently, but it happens. Getting the build back on track should be one of the highest priorities - your build box teammate is down and you need to get him back on his feet.
Ideally, what should happen is the following:
Sometimes nobody is doing anything about it. If this is a recurring problem, I recommend the use of a Build Nazi (in honor of the Seinfeld Soup Nazi). The job of the Build Nazi is to make sure somebody is responsible for fixing a broken build. The job is not to fix the build (unless help is needed). The Build Nazi job is not a fun one and therefore should be rotated. It is also a controversial role in an agile environment where the practice of collective code ownership is being followed. Ideally, the team is following the steps above and a Build Nazi is totally unnecessary. I’ve found I’ve had to resort to the role for short term periods in times of chaos when the build is breaking more than it is succeeding. Once things get back on track, the Build Nazi role typically becomes dormant.
The last step I’d like to elaborate on is step 4. One of the worst things to do is check in a bunch of changes that don’t have anything to do with fixing the build. Checking in causes the CI system to start another build that will result in another failure. Piling on check ins during a broken build prolongs the broken state of the build and therefore the feedback cycle that is all important in agile development. Another risk is that the fix necessary to correct the original problem must be applied to the check ins that occurred after the first failure, further prolonging the broken state.
I plan on finishing this series next time. Stay tuned.
The second part of the build/SCM blog series (see part 1) deals with what happens when a build breaks. This happens, hopefully infrequently, but it happens. Getting the build back on track should be one of the highest priorities - your build box teammate is down and you need to get him back on his feet.
Ideally, what should happen is the following:
- Notification of a build failure is distributed by e-mail to the team. This is part of the publishing functionality of CI systems like CruiseControl.
- Particularly, whoever checked in since the last successful build analyzes the results.
- The person who broke the build mans up and replies to all, “This is me. I’m on it.”
- Nobody checks in until the build is fixed unless the check in is for the purpose of fixing the build.
- Somebody fixes the build, obtaining any help necessary.
- (Optional) Once the build is believed to be fixed, the person from step 3 sends an e-mail saying the build should be fixed.
Sometimes nobody is doing anything about it. If this is a recurring problem, I recommend the use of a Build Nazi (in honor of the Seinfeld Soup Nazi). The job of the Build Nazi is to make sure somebody is responsible for fixing a broken build. The job is not to fix the build (unless help is needed). The Build Nazi job is not a fun one and therefore should be rotated. It is also a controversial role in an agile environment where the practice of collective code ownership is being followed. Ideally, the team is following the steps above and a Build Nazi is totally unnecessary. I’ve found I’ve had to resort to the role for short term periods in times of chaos when the build is breaking more than it is succeeding. Once things get back on track, the Build Nazi role typically becomes dormant.
The last step I’d like to elaborate on is step 4. One of the worst things to do is check in a bunch of changes that don’t have anything to do with fixing the build. Checking in causes the CI system to start another build that will result in another failure. Piling on check ins during a broken build prolongs the broken state of the build and therefore the feedback cycle that is all important in agile development. Another risk is that the fix necessary to correct the original problem must be applied to the check ins that occurred after the first failure, further prolonging the broken state.
I plan on finishing this series next time. Stay tuned.
Anti-Practice: Reserving Files
Originally published 20 Aug 2006
Here is the first blog in a series discussing some obsolete practices and just plain annoyances regarding builds and source code management. The context is an agile development environment following the best practices of continuous integration (CI) and collective code ownership. The team is of medium to large size, possibly distributed, and working different hours, possibly in different time zones.
The first obsolete practice is reserving (locking) a file by default when checking out the file. (I’m using ClearCase terms here). This is like pessimistic locking in the database world. The thought is, “I’m going to reserve this so that no one else will affect me. I’ll make my changes, then check in. If anybody else wants to modify the same file, they’ll have to wait for me to finish, then deal with merging in their changes.” That doesn’t sound like collective code ownership to me. Sharing the code base means sometimes having to deal with merge issues.
Reserving files prohibits others who are ready to check in. In the worst case, suppose a person with reserved files leaves for the day (or for vacation). Others are likely working different hours, so what do they do? They try to get in touch with you to see what’s up. They can’t get a hold of you, but they are ready to check in and move on. So after some time not hearing from you, do they unreserve the files? In some stricter environments, what if they don’t have rights? What if they can’t get in touch with the select few that have rights to unreserve files? (By the way, if you do unreserve a file, make sure you tell the person who reserved it that you did that.) The point is reserving files slows down the team.
There are of course exceptions. An important change or a change that has a big impact to the system might merit an early reserve on a check out. In this blog, I’m talking about reserving files by default.
The biggest positive of reserving early is finding out that somebody else is working on the same file. For example, I try to check out a file and reserve it right away and find that I can’t because Joe’s got the file reserved. Depending on the circumstances, I may want to know exactly what Joe’s doing. I’m not familiar with another way of getting this notification as easily without getting spammed by unwanted notifications. In other words, I might want to know, upon check out of a file, whom else has the file checked out. For me, this advantage doesn't outweigh the disadvantages mentioned above.
Next time I’ll talk about an annoyance or two that occurs when a build breaks.
Here is the first blog in a series discussing some obsolete practices and just plain annoyances regarding builds and source code management. The context is an agile development environment following the best practices of continuous integration (CI) and collective code ownership. The team is of medium to large size, possibly distributed, and working different hours, possibly in different time zones.
The first obsolete practice is reserving (locking) a file by default when checking out the file. (I’m using ClearCase terms here). This is like pessimistic locking in the database world. The thought is, “I’m going to reserve this so that no one else will affect me. I’ll make my changes, then check in. If anybody else wants to modify the same file, they’ll have to wait for me to finish, then deal with merging in their changes.” That doesn’t sound like collective code ownership to me. Sharing the code base means sometimes having to deal with merge issues.
Reserving files prohibits others who are ready to check in. In the worst case, suppose a person with reserved files leaves for the day (or for vacation). Others are likely working different hours, so what do they do? They try to get in touch with you to see what’s up. They can’t get a hold of you, but they are ready to check in and move on. So after some time not hearing from you, do they unreserve the files? In some stricter environments, what if they don’t have rights? What if they can’t get in touch with the select few that have rights to unreserve files? (By the way, if you do unreserve a file, make sure you tell the person who reserved it that you did that.) The point is reserving files slows down the team.
There are of course exceptions. An important change or a change that has a big impact to the system might merit an early reserve on a check out. In this blog, I’m talking about reserving files by default.
The biggest positive of reserving early is finding out that somebody else is working on the same file. For example, I try to check out a file and reserve it right away and find that I can’t because Joe’s got the file reserved. Depending on the circumstances, I may want to know exactly what Joe’s doing. I’m not familiar with another way of getting this notification as easily without getting spammed by unwanted notifications. In other words, I might want to know, upon check out of a file, whom else has the file checked out. For me, this advantage doesn't outweigh the disadvantages mentioned above.
Next time I’ll talk about an annoyance or two that occurs when a build breaks.
Subscribe to:
Posts (Atom)