Pain killing the pingponger's decease

(or how we got our ci builds green again)

Everyone (at least almost everyone I know) writing Selenium or webdriver tests knows it's not easy to keep all these tests stable. Most of the time, they run perfectly fine on your development machine and start failing after a while in your CI build (and I'm not talking about regression here).

I've felt this pain already in quite some projects and it's not always easy to find what's causing your tests to fail. On my current project, again we are having some pingpong playing webdriver tests (for those who don't understand: it means they run sometimes and sometimes they don't).

Of course your tests should be reproducible and consistently working or failing. And that should be your first step in solving the problem: find the cause of the instability in your test and fix it. Unfortunately when webdriver tests are involved this can be sometimes hard to achieve. The webdriver framework highly depends on references to elements that can become stale. When AJAX calls are involved, this can become a real headache-causer. I'm not going in too deep on all the possible causes of instability, since this is not the main subject of this blog.

So, how do you make your build less dependent upon some hickup of your CI system? At our current project we didn't have an auto-commit for about 3 months because of pingponging webdriver tests. Our 'solution' was, when a build had run and some tests failed, run them locally and if they pass, you can tag your build manually. It cost us a lot of time that could be better spend on more fun stuff.

While trying to find a solution for this problem, we had this idea: maybe we could just ignore the pingpong-playing webdriver tests by excluding them from the main build and running them separately. In that case our main build is no longer dependent upon the vagaries of our tests. That way we would have more auto-commits again, but we would introduce the risk that one of the pingpong-playing tests this time fails for the right reasons. When deploying this strategy, one could ask himself whether you shouldn't throw away the pingpong tests entirely, since you would be completely ignoring them.

Then, we came up with another solution which turned out to be our salvation. What is this magic drug we are using? It's quite simple actually: we created our own (extended) version of the JUnit runner we used before and let it retry pingponging tests. To accomplish this, we mark pingpong tests with the @PingPong annotation, using a maxNumberOfRetries property to define how many times the tests should be retried (default is one). The @PingPong annotation can be used both at method and class level to mark a single test or all tests in a test class as playing pingpong.

An example of a test class using the @PingPong annotation looks like this.

@RunWith(MyVeryOwnTestRunnerRetryingPingPongers.class)
public class MyTest {

  @PingPong(maxNumberOfRetries = 2)
  @Test
  public void iWillSucceedWhenSuccessWithinFirstThreeTries() {
    // ...
  }
}

With MyVeryOwnTestRunnerRetryingPingPongers defined like this.

public class MyVeryOwnTestRunnerRetryingPingPongers extends SomeTestRunner
    implements StatementFactory {

  public MyVeryOwnTestRunnerRetryingPingPongers (Class aClass)
      throws InitializationError {
    super(aClass);
  }

  @Override
  protected Statement methodBlock(FrameworkMethod frameworkMethod) {
    return new StatementWrapperForRetryingPingPongers(frameworkMethod, this);
  }

  @Override
  public Statement createStatement(FrameworkMethod frameworkMethod) {
    return super.methodBlock(frameworkMethod);
  }
}

You still need the implementation of StatementWrapperForRetryingPingPongers. You can find this one here.

I am conscious this is just a pain killer, but it's one that helped us getting our builds green again and gives us extra time to fix our instable pingpong tests more thoroughly.

Please let me know if this post was helpful to you. What do you think about our solution to our instability problem?

When mocking builders... (using Mockito)

Most of you already know I'm not really keen about mock testing. It gets even worse when mocking builders. As a matter of fact, I'm not really crazy about builders either (so now I find myself looking for a better way to use two of the things I always try to avoid). At least not in production code. Builders in tests are great for quickly creating full-fledged domain objects. For production code however, always ask yourself whether you wouldn't be better of using a factory.

Anyway, in case you are using builders and you want to mock them out, don't do it this way (my example is overly simplistic, but that's not the point here):

  @Test
  public void createLogbookRecordIsNotReallyTested() {
    LogbookRecordBuilder builderMock = mock(LogbookRecordBuilder.class);
    when(builderMock.withType(any(LogbookRecordType.class)))
      .thenReturn(builderMock);
    when(builderMock.withPersonName(anyString()))
      .thenReturn(builderMock);
    when(builderMock.withAccountNumber(anyString()))
      .thenReturn(builderMock);
    when(builderMock.build()).thenReturn(LOG_RECORD);

    LogbookRecord result = client.createLogbookRecord(builderMock);
    assertThat(result).isEqualTo(LOG_RECORD);
  }

Of course we were testing this immensely vital production code:

  @Test
  public LogbookRecord createLogbookRecord(LogbookRecordBuilder builder) {
    return builder
      .withType(LogbookRecordType.DEPOSIT)
      .withPersonName(personName)
      .withAccountNumber(accountNumber)
      .build();
  }

What's wrong with the test? Well, it actually tests nothing. Ok, i'm exagerating a bit, it just tests the build method was called and the result is returned.
So you could change your production code into this:

  public LogbookRecord createLogbookRecord(LogbookRecordBuilder builder) {
    return builder.build();
  }

and your test would still work just fine. Reason is of course that you are stubbing your with-calls, the test is not forcing them to be called.
To fix that, we'll add verify methods to check whether the right methods are called on your builder:

  @Test
  public void createLogbookRecordIsTestedWithStubbingAndInOrderVerifications() {
    LogbookRecordBuilder builderMock = mock(LogbookRecordBuilder.class);
    when(builderMock.withType(any(LogbookRecordType.class)))
      .thenReturn(builderMock);
    when(builderMock.withPersonName(anyString()))
      .thenReturn(builderMock);
    when(builderMock.withAccountNumber(anyString()))
      .thenReturn(builderMock);
    when(builderMock.build()).thenReturn(LOG_RECORD);

    LogbookRecord result = client.createLogbookRecord(builderMock);
    assertThat(result).isEqualTo(LOG_RECORD);

    InOrder inOrder = Mockito.inOrder(builderMock);
    inOrder.verify(builderMock).withType(LogbookRecordType.DEPOSIT);
    inOrder.verify(builderMock).withPersonName(PERSONNAME);
    inOrder.verify(builderMock).withAccountNumber(ACCOUNTNUMBER);
    inOrder.verify(builderMock).build();
    verifyNoMoreInteractions(builderMock);
  }

I don't really like this approach because of the DRY principle (you're stubbing and verifying the same methods). Did you also notice the inOrder checking of the method calls? I'm not really proud about this, but that way I make sure the build method was called last.

One solution to stop repeating yourself might be using a default answer (ReturnSelfWhenTypesMatchAnswer) for your mocked builder which just returns the mock itself when the method's result type is some supertype of your mocked builder (so all your with-methods will return the mocked builder). That way your test looks like this:

  @Test
  public void createLogbookRecordIsTestedUsingDefaultAnswer() {
    LogbookRecordBuilder builderMock = 
      mock(LogbookRecordBuilder.class, new ReturnSelfWhenTypesMatchAnswer());
    when(builderMock.build()).thenReturn(LOG_RECORD);

    LogbookRecord result = client.createLogbookRecord(builderMock);
    assertThat(result).isEqualTo(LOG_RECORD);

    InOrder inOrder = Mockito.inOrder(builderMock);
    inOrder.verify(builderMock).withType(LogbookRecordType.DEPOSIT);
    inOrder.verify(builderMock).withPersonName(PERSONNAME);
    inOrder.verify(builderMock).withAccountNumber(ACCOUNTNUMBER);
    inOrder.verify(builderMock).build();
    verifyNoMoreInteractions(builderMock);
  }

The implementation of ReturnSelfWhenTypesMatchAnswer is quite simple:

public class ReturnSelfWhenTypesMatchAnswer implements Answer<Object> {

  @Override
  public Object answer(InvocationOnMock invocation) throws Throwable {
    Class<?> returnType = invocation.getMethod().getReturnType();
    if (returnType.isInstance(invocation.getMock())) {
      return invocation.getMock();
    }
    return null;
  }
}

Question I ask myself: why didn't anybody test the code like this?

  @Test
  public void createLogbookRecordIsTestedTheRightWayWithoutMocks() {
    LogbookRecordBuilder builder = new LogbookRecordBuilder();

    LogbookRecord result = client.createLogbookRecord(builder);

    assertThat(result.getType()).isEqualTo(LogbookRecordType.DEPOSIT);
    assertThat(result.getAccountNumber()).isEqualTo(ACCOUNTNUMBER);
    assertThat(result.getPersonName()).isEqualTo(PERSONNAME);
  }

For me, the only reasonable answer to the above question would be: because our builder is doing incredibly complex stuff which we don't want to re-test. Did you also notice the last test is the shortest and most readable version of all without copy-pasting any production code?

Ok, so I resulted back in discouraging you from writing mock tests. My original intention however was to show that using a default answer for mocks can sometimes be a more elegant solution than stubbing lots of stuff.