Posts Tagged ‘ Testing ’

Creative uses of Hamcrest matchers

The matcher API of Hamcrest is typically associated with assertThat() or mocks. I always knew other people would find good uses for it, but I never really knew what.

I particularly like these:

Collection processing

Håkan Råberg blogged about how Hamcrest can be used with iterators:

List<Integer> numbers = Arrays.asList(-1, 0, 1, 2);
List<Integer> positiveNumbers = detect(numbers, greaterThan(0)));

List<String> words = Arrays.asList("cheese", "lemon", "spoon");
List<String> wordsWithoutE = reject(words, containingString("e"));

Nothing rocket-sciencey about it. But simple and useful because it reduces boilerplate code and get to use the ever growing library of Hamcrest matchers.

On top of that, combining Hamcrest with a CGLib generated proxy, he has built a staticly typed query API:

List<Person> employees = ...;
List<Integer> allAges
= collect(from(employees).getAge());
List<Person> allBosses
= collect(from(employees).getDepartment().getBoss());
List<Person> allAccountants
= select(from(employees).getDepartment().getName(),
containingString("Accounts"));

This is nice alternative to a string based query language as you get your IDE completions, refactoring, compile time checking etc, without the noise of boilerplate code.

Web testing

Robert Chatley has taken some of the concepts of his LiFT framework and reimplemented them using Hamcrest and WebDriver for performing web testing.

public void testHasLotsOfLinks() {
goTo("http://some/url");
assertPresenceOf(greaterThan(15), links());
assertPresenceOf(atLeast(1), link().with(text(containingString("Sign in"))));

clickOn(link().with(text(containingString("Sign in"))));
assertPresenceOf(exactly(1), title().with(text(equalTo("Sign in page"))));
}

Now initially this seems a bit wordy and strange. Robert has designed this as a literate API. If you adjust the syntax highlighting of your API and make the Java keywords and syntax less visible, you get this:

goTo "http://some/url"
assertPresenceOf greaterThan 15 links
assertPresenceOf atLeast 1 link with text containingString "Sign in"

clickOn link with text containingString "Sign in"
assertPresenceOf exactly 1 title with text equalTo "Sign in page"

The motivation here is that the API usage is self documenting and could be useful to non-programmers. The flip-side to this is that it’s actually quite hard to write APIs like this and the usage can take quite a bit of getting used to.

Robert also introduced a Finder interface (the link() and title() methods return Finder implementations). This allows you to factor out your own UI specific components:

assertPresenceOf(atLeast(1), signInLink());
clickOn(signInLink());
assertPresenceOf(exactly(1),
blogLink().with(urlParameter("name", containingString("joe"))));

This is the bit I really like.

Allowing abstractions of components and matching rules to be combined in many different ways, so tests can check exactly what they need to, resulting in reduced less brittle tests that are easier to maintain.

Other uses

As I hear of other uses I’m listing them on the Hamcrest wiki.

When it goes bad

Of course, like any technology, it’s easy to get carried away.

Here’s an example of Hamcrest gone bad:

assertThat(myNumber, anyOf(equalTo(0), allOf(greaterThan(5), lessThan(10))));

I’m not a LISP programmer, so I find that really hard to understand. Just because we have an assertTHAT() method, we don’t have to use it all the time. In this case it’s much simpler to use plain old assertTRUE():

assertTrue("myNumber should be 0 or between 5 and 10",
myNumber == 0 || (myNumber > 5 && myNumber < 10));

Even though the non-Matcher version is longer (it could be shortened by leaving out the message and using a shorter variable name, but that would make it harder to understand), I find it much easier to understand.

But, what if you actually needed to use a matcher (e.g. for the web testing or collection processing examples above)?

One approach is you could use higher level matcher that are composed of other matchers:

matcher = anyOf(equalTo(0), allOf(greaterThan(5), lessThan(10)))
// simplifies to
matcher = anyOf(equalTo(0), between(5, 10))

Complete tangent: An alternative to between(5, 10) is between(5).and(10). The latter makes for more literate code, but is harder to implement – again a design tradeoff.

Another approach is to create a one-off anonymous matcher implementation:

matcher = new CustomMatcher() {
public boolean matchesSafely(Integer n) {
return n == 0 || (n > 5 && n < 10);
}
}

What are you doing with Hamcrest?

Updates:

  1. JUnit 4.4 now comes with Hamcrest and assertThat().

Hamcrest 1.1 released

http://code.google.com/p/hamcrest

Testing on the Toilet

At Google we have pretty good internal documentation, tutorials and places to find good tips. If you know you don’t know something, it won’t take long to find the answer.

However, it’s slightly tougher to place something to be read, when the target readers don’t know they don’t know it. This was a problem the testing group were finding, as they wanted to improve sharing of practical testing techniques.

So, Testing on the Toilet was started. A regular weekly(ish) tip posted in toilet cubicles and above urinals. Short enough to be read whilst doing your business.

Soon after, many visitors started noticing these postings and we got requests to make these available to put up in offices of other development teams.

So, we have.

http://googletesting.blogspot.com/

Each episode will be made available as a toilet friendly PDF.

Building testable AJAX apps (Does my button look big in this?)

Last week, Adam Connors and I presented “Does my button look big in this? Building testable AJAX applications.” at the Google London Test Automation Conference.

Unfortunately the code is unclear on the video, so you can also download the slides separately (13mb!).

Flexible JUnit assertions with assertThat()

Over time I’ve found I end up with a gazillion permutation of assertion methods in JUnit: assertEquals, assertNotEquals, assertStringContains, assertArraysEqual, assertInRange, assertIn, etc.

Here’s a nicer way. jMock contains a constraint library for specifying precise expectations on mocks that can be reused in your own assertion method (and that’s the last time I’m going to mention mocks today, I promise – despite the frequent references to the jMock library).

By making a simple JUnit assertion method that takes a Constraint, it provides a replacement for all the other assert methods.

I call mine assertThat() because I think it reads well. Combined with the jMock syntactic sugar, you can use it like this:

assertThat(something, eq("Hello"));
assertThat(something, eq(true));
assertThat(something, isA(Color.class));
assertThat(something, contains("World"));
assertThat(something, same(Food.CHEESE));
assertThat(something, NULL);
assertThat(something, NOT_NULL);

Okay, that’s nice but nothing radical. A bunch of assert methods have been replaced with different methods that return constraint objects. But there’s more…

h3. Combining constraints

Constraints can be chained making it possible to combine them in different permutations. For instance, for virtually every assertion I do, I usually find that I need to test the negative equivalent at some point:

assertThat(something, not(eq("Hello")));
assertThat(something, not(contains("Cheese")));

Or maybe combinations of assertions:

assertThat(something, or(contains("color"), contains("colour")));

h3. Readable failure messages

The previous example can be written using the vanilla JUnit assert methods like this:

assertTrue(something.indexOf("color") > -1 || something.indexOf("colour") > -1);

Fine, the constraint based one is easier to read. But the real beauty is the failure message.

The vanilla JUnit assert fails with:

junit.framework.AssertionFailedError:

Useless! Means you have to put an explicit error message in the assertion:

assertTrue(something.indexOf("color") > -1 || something.indexOf("colour") > -1,
"Expected a string containing 'color' or 'colour'");

But the jMock constraint objects are self describing. So with this assertion:

assertThat(something, or(contains("color"), contains("colour")));

I get this useful failure message, for free:

junit.framework.AssertionFailedError:
Expected: (a string containing "color" or a string containing "colour")
but got : hello world

h3. Implementing it

The simplest way is to grab jMock and create your own base test class that extends MockObjectTestCase. This brings in convenience methods for free (I’m still not talking about mocks, honest). If you don’t want to extend this class, you can easily reimplement these methods yourself – it’s no biggie.

import org.jmock.MockObjectTestCase;
import org.jmock.core.Constraint;

public abstract class MyTestCase extends MockObjectTestCase {

protected void assertThat(Object something, Constraint matches) {
if (!matches.eval(something)) {
StringBuffer message = new StringBuffer("nExpected: ");
matches.describeTo(message);
message.append("nbut got : ").append(something).append('n');
fail(message.toString());
}
}

}

Now ensure all your test cases extend this instead of junit.framework.TestCase and you’re done.

h4. Defining custom constraints

Creating new constraints is easy. Let’s say I want something like:

assertThat(something, between(10, 20));

To do that I need to create a method that returns a Constraint object, requiring two methods; eval() for performing the actual assertion, and describeTo() for the self describing error message. This is something that can live in the base test class.

public Constraint between(final int min, final int max) {
return new Constraint() {
public boolean eval(Object object) {
if (!object instanceof Integer) {
return false;
}
int value = ((Integer)object).intValue();
return value > min && value < max;
}
public StringBuffer describeTo(StringBuffer buffer) {
return buffer.append("an int between ").append(min).append(" and ").append(max);
}
}
}

This can be combined with other constraints and still generate decent failure messages.

assertThat(something, or(eq(50), between(10, 20));
junit.framework.AssertionFailedError:
Expected: (50 or an int between 10 and 20)
but got : 43

In practice I find I only need to create a few of these constraints as the different combinations gives me nearly everything I need.

More about this in the jMock documentation.

h4. Summary

Since using this one assert method I’ve found my tests to be much easier to understand because of lack of noise and I’ve spent a lot less time creating ‘yet another assertion’ method for specific cases. And in most cases I never need to write a custom failure message as the failures are self describing.

h4. Updates

  1. The matchers from jMock have been pulled out into a new project, Hamcrest.
  2. A follow up to this post shows some creative uses of matchers, and talks a bit about when you shouldn’t use them.
  3. JUnit 4.4 now comes with assertThat()!

JUnit tip: Setting the default timezone with a TestDecorator

A problem I was finding when testing XStream is that many of the tests were timezone dependent. Initially I had some code in each setUp()/tearDown() method to set the default timezone and reset it afterwards.

This lead to a lot of duplication. Putting the common code in a super class was an option but this lead to a fragile base class.

Using composition allows this commonality to be put in one case and applied to the relevant test cases. This gets out of the ‘single inheritance’ issue.

JUnit provides an (often overlooked) decorator class to help with this. Here’s my TimeZoneTestSuite:

  public class TimeZoneTestSuite extends TestDecorator {

private final TimeZone timeZone;
private final TimeZone originalTimeZone;

public TimeZoneTestSuite(String timeZone, Test test) {
super(test);
this.timeZone = TimeZone.getTimeZone(timeZone);
this.originalTimeZone = TimeZone.getDefault();
}

public void run(TestResult testResult) {
try {
TimeZone.setDefault(timeZone);
super.run(testResult);
} finally {
TimeZone.setDefault(originalTimeZone); // cleanup
}
}

}

To use it, you need to override the default test suite behavior by adding a method to your TestCase:

  public class MyTest extends TestCase {
public static Test suite() {
Test result = new TestSuite(MyTest.class); // default behavior
result = new TimeZoneTestSuite("EST", result); // ensure it runs in EST timezone
return result;
}

TestDecorators are a very powerful feature of JUnit – don’t forget about them.

Is 100% test coverage a BAD thing?

I’m a huuuge advocate of TDD and high test coverage, and I will often go to great lengths to ensure this, but is 100% such a good thing?

I recently heard Tim Lister talking about risk in software projects and the CMM (powerpoint slides).

The ‘ultimate’ level of CMM ensure that everything is documented, everything goes through a rigorous procedure, blah blah blah. Amusingly, Tim pointed out that no CEO in their right mind would ever want their organization to be like that as they would not be effectively managing risk. You only need this extra stuff when you actually need this extra stuff. If there’s little risk, then this added process adds a lot of cost with no real value – you’re just pissing away money.

This also applies for test coverage. There are always going to be untested parts of your system but when increasing the coverage you have to balance the cost with the value.

With test coverage, you get the value of higher quality software that’s easier to change, but it follows the Law of diminishing returns. The effort required to get from 99% to 100% is huge… couldn’t that be spent on something more valuable like adding business functionality or simplifying the system?

Personally, I’m most comfortable with coverage in the 80-90% region, but your mileage may vary.

NMock article in MSDN Magazine

An article about NMock has appeared in the October issue of MSDN Magazine. Chuffed :)

Mock Objects to the Rescue! Test Your .NET Code with NMock

Unit Testing Asynchronous Code

I try to avoid using code that instantiates threads from unit-tests. They’re awkward to write, brittle and I would rather extract the controlling thread, using the test as the controller. However there are times when it’s unavoidable.

Here’s an example. A PriceFinder is a class that goes and retrieves a price for a symbol asyncronously, returning immediately. Sometime later it receives a response and performs a callback on a handler. I’ve left out the implementation of how it actually does this (maybe a web-service, database call or JMS message).

public class PriceFinder {
 public void findPrice(String symbol, PriceHandler handler) { ... }
}

public interface PriceHandler {
 void foundPrice(Price price);
}

To test this, a handler can be used from the test case that records what is passed in. The test can then wait for a specified time and assert that the correct result is received. If the asyncronous code does not complete within the specified time, the test will fail (suggesting the code is either running very slowly or is never going to complete).

public class PriceFinderTest extends TestCase {

 private PriceFinder finder = new PriceFinder();
 private Price receivedPrice;

 public void testRetrievesAValidPrice() throws Exception {
  finder.findPrice("MSFT", new PriceHandler() {
   public void foundPrice(Price price) {
    receivedPrice = price;
   }
  });

  // Smelly!
  Thread.sleep(2000); // Wait two seconds.
  assertNotNull("Expected a price", receivedPrice);
 }

}

However, this sucks as it will slow your test suite right down if you have loads of tests using Thread.sleep().

A less time consuming way to do it is by using wait() and notify() on a lock. The handler notifies the lock and the test waits for this notification. In case this notification never happens, a timeout is used when calling wait().

public class PriceFinderTest extends TestCase {

 private PriceFinder finder = new PriceFinder();
 private Price receivedPrice;
 private Object lock = new Object();

 public void testRetrievesAValidPrice() throws Exception {
  finder.findPrice("MSFT", new PriceHandler() {
   public void foundPrice(Price price) {
    receivedPrice = price;
    synchronized(lock) {
     lock.notify();
    }
   }
  });

  synchronized(lock) {
   lock.wait(2000); // Wait two seconds or until the
   // monitor has been notified.
   // But there's still a problem...
  } 
  assertNotNull("Expected a price", receivedPrice);
 }

}

This optimistic approach results in fast running tests while all is good. If the PriceFinder is behaving well, the test will not wait any longer than the PriceFinder takes to complete its work. If PriceFinder has a bug in it and never calls the handler, the test will fail in at most two seconds.

However, there’s still a subtle issue. In the case that the PriceFinder is really fast, it may call notify() before the test starts wait()ing. The test will still pass, but it will wait until the timeout occurs.

This is where threading and synchronization get messy and beyond me. Doug Lea has a nice little class called Latch in his concurrency library (available in JDK 1.5 as CountDownLatch). A latch can only be locked once, once released it will never lock again.

public class PriceFinderTest extends TestCase {

 private PriceFinder finder = new PriceFinder();
 private Price receivedPrice;
 private Latch latch = new Latch();

 public void testRetrievesAValidPrice() throws Exception {
  finder.findPrice("MSFT", new PriceHandler() {
   public void foundPrice(Price price) {
    receivedPrice = price;
    latch.release();
   }
  });

  latch.attempt(2000); // Wait until the latch is released
  // or a timeout occurs.
  assertNotNull("Expected a price", receivedPrice);
 }

}

That’ll do it.

OT2004 : Personal Practices Map

In the evening, everyone had a chance to run a Birds of a Feather session. I thought I’d give it a go and repeat the session from XtC last year.

It was a chance to talk to other developers about the little things we do that help guide development.

It was a calm and enlightening session. I felt I learned a lot from the attendees. Techniques that I would probably otherwise never know about unless I actually paired with them.

I look forward to doing this again at ADC.

Romilly generated a topic map from the results: (click here)