Test Smell: Test Breaks After Good Refactoring

Originally published  17 Mar 2006

Goal: Write a thorough unit test that doesn’t require any changes after mercilessly refactoring the object it’s testing.

I’ve been noticing that I sometimes feel a need to go back and modify my test to be more in line with the object it’s testing. And I’m not alone. David Chelimsky recently wrote a related blog. That blog, in the context of my goal, focuses more on changing tests because they start drifting away from the code they’re testing.

In this blog, I want to talk about changes required because the tests break as a result of good refactoring. This isn’t something that occurs often, but it does happen, and I think it’s a code smell. One of the main advantages of having a test suite is that you can refactor to make the code better or try a different algorithm without worrying whether the final result broke existing functionality. When the tests break as a result of correct refactoring, I lose that safety net kind of feeling.

So what does it mean when a test breaks as a result of good refactoring? Well, what jumps out is that the test is too coupled to the main object it’s testing. However, it is a unit test and unit tests are traditionally white box tests and white box tests imply knowledge of the code and that implies coupling.

What about a black box approach to unit testing to reduce coupling? Elliotte Rusty Harold recently posted a comment on one of his blogs Experimental Programming, “The class is a black box which is accessed solely through its public interface. The internal details are deliberately opaque, and thus can be changed as necessary to improve performance or other desirable characteristics without breaking client code.”

Yeah, that’s what I’m talking about, but I think a total black box approach would be too coarse grained, particularly for ensuring 100% test coverage. Specifically, I’m thinking that touching every conditional statement would require a lot of black box test code. I think the first tip then is to find a balance between black box and white box testing; the sub-goal being to find the appropriate coupling to get 100% coverage with the least amount of test code. I’m trying to keep this in mind when I write tests.

Another technique to help achieve the goal is to shy away from mock objects, particularly strict mock objects. Martin Fowler wrote an article called “Mocks Aren’t Stubs”. In it, he describes the difference between mock objects and stubs. He also describes two styles of testing: state-based and interaction-based. Interaction-based testing uses lots of mocks, which really couples tests to the implementation details. This is not what I want if I want to achieve my goal. So I prefer the state-based approach Martin talks about.

Now, having said that, I do use mocks in certain situations. They’re good at making sure your object made a call on a secondary object:

   public void testInitRegisters() {
      // setup a mock object that expects a register call
      // and inject it into the object under test
      objectUnderTest.init();
      // verify the mock
   }

Typically, I find that I use mock objects when I’m testing void methods, just like the init() method above. In fact, thinking more about it, it might actually be when the method I need to call on the object I’m mocking is void, just like register() method above. I’ll have to pay more attention, but that seems to make sense. The point is to use mock objects only when they are more practical than the state-based approach, preferring the state-based approach otherwise to reduce test coupling.

Stubs are great for tests whose primary object calls methods that return values from secondary objects. I use mock libraries to create stubs, particularly for Interfaces and expensive-to-create classes, since it’s so easy to do so. I reduce test coupling by creating lenient stub objects that either ignore unexpected method calls or return empty values (0, null, false) for unexpected method calls. In EasyMock terms, I’m talking about nice controls.

I also use the Object Mother pattern as a factory to create the stub objects. This is so that the stub creation code can be shared among many test classes. I pass the values I want returned into object mother. The object mother creates the stub object with those values. In this way, the test has all the knowledge of the canned results. Then I use a state-based approach to verify everything. Here’s a quick example, putting this all together:

   private Order order = new Order();

   public void testGetTotal() {
      // LineItem is an interface.  LineItemMother uses a
      // mock library for creating a stub LineItem
      LineItem lineItem = LineItemMother.newItem(3, 2.50);
      order.add(lineItem);
      assertEquals(7.50, order.getTotal(), 0.001);
   }