Checklist for Finding Test Cases

Originally published 4 Jun 2006

Suppose I'm focused on specifying the behavior of a particular method. How do I know if I've specified everything? I'd like to share a checklist of things I think about to help accomplish this. The goal is to thoroughly specify the behavior of the method by building up a collection of tests following test-driven development.

I'm going to base my example on Spring's BeanFactory interface. Suppose I'm developing a Map backed BeanFactory and I am currently focused on implementing the following method of the BeanFactory interface:

Object getBean(String beanName) throws BeansException

Let's call the class I'm developing MapBackedBeanFactory. When getBean() is called, the implementation will look up the bean in its beanMap field.

When I think about test cases, I think about:

1. The state of the object under test (each field), i.e., the fixture. In the example, this would be the beanMap.
2. The state of the parameters of the method I'm calling on the object under test. In the example, there's only one: the beanName.

That was pretty obvious. Who doesn't think about these things? But what is the checklist? The checklist consists of thinking about the following for each of the above:

1. A good value. This will execute the normal flow and is the most obvious test. For beanMap, this means the beanMap will have a good bean mapping. For the beanName, a good value is a name that will be found in the beanMap. There may be multiple good values, not really in this case, but think about the classic bowling game. You'd want tests for a non-mark frame, a spare, and a strike. I'll also consider multiple good values for things like collections, maps, and arrays. That is, maybe I'm dealing with a  List and I need a List of two good things. However, I find that most times, one good value in the collection, map, or array is sufficient.  This is because I write the iteration code for the single case and a multiple collection test case wouldn't make me write any new code.  If I find I need multiple good values, I'll start with the simple cases and work towards the harder.
2. A negative case. In #1, I set up the fixture and parameters to get a good result.  Now, I want to think of the negative side.  So I found beans in #1, but now I want to not find a bean.  Pass in an unknown beanName.
3. A null value. What if the beanMap were null? That's probably not realistic, so I'm going to skip that. What if the beanName were null? That's possible and according to the BeanFactory JavaDoc, it looks like the contract specifies to throw a NoSuchBeanDefinitionException. I better write that test. Many times, if you don't handle null, you'll get a NullPointerException and that might be acceptable. In those cases, I wouldn't write a test (since it wouldn't require any code).
4. An empty value. The beanMap could have no mappings or the beanName could be "". I think #2 covered us here, so no test is required.
5. An invalid value. What if beanMap were invalid? What if it were a Map of Integers to bean Objects? I'm going to assume this is unrealistic in this case. We would have handled such a situation when the map was loaded.  What if the beanName were "invalid"? Well, we covered that in #2.
6. An exceptional case. I'm pretty much talking about handling exceptions here. Maybe we need to write a test case to see if we handle an exception properly. The example doesn't have such a case, but suppose the method under test were doing some I/O. Maybe we'd need to write a test to see if we handled that properly. In all honesty, even though I think about this, I don't usually write a test to handle exceptions. I won't get 100% test coverage, but I usually ignore writing the test because checked exceptions force me to write the necessary handling code.

That's a lot of stuff to think about for each object field/parameter. That's the biggest negative - thinking about all those things, all the possibilities, all the time it takes. Many times, as you saw in the example, most of those things just aren't realistic.

Sometimes I'll take a different approach. I'll start with the simple, good case. I'll write the test and get it to pass by writing the minimal code to do so.  In the example, I'd have:

return beanMap.get(beanName);

Then, I'll look at the method I just wrote word by word (or maybe token by token).  At each word, I'll think about special cases (things like in the checklist) and jot down test cases. Then, I'll go back to the test, write one of the jotted-down tests, get it to pass, and continue until I'm done. It's a different perspective that goes a little faster at the risk of not quite catching everything as the first approach.  I like this second approach and have been doing it more and more lately.