Recent Comparison of jMock 1.1 and EasyMock 2.2

My Colleges and me are in the process of figureing out wich mock frameworks to use. One of my colleges has experiences with jMock and I have used diffrent versions of easymock. We wanted a detailed comparison to help us decide what the strengths and weaknesses are of the diffrent Mockframeworks. During a web search we discovered this comparison between jMock and Easymock. Although, regarding the authors background not necessarily, but to me most likely, this comparison of Jmock to an unspecified version of Easymock is biased towards jMock.
Many diffrences/disadvantages in this comparison are obsoleted by more recent versions of Easymock, and speculating on the version used in favour of the author would imho result in 1.3 wich dates way back to 2005.

The desription of the link to the jMock project on the easymock project page, states that jMock opposed to easymock uses

another approach to define expectations that results in more flexibility to set expectations (you may specify almost everything imaginable) at the cost of less flexibility when refactoring (refactorings like method renaming and parameter reordering will break the tests) and loss of code completion when defining expectations or test-driving code.

Well this note is definitely true for older version of easymock, and probably stems from the same era as the comparision hosted on the jMock page. Since version 2.0 easymock started to look a lot more like jMock in terms of behaviuor specification. I found that the easymock approach helped me doing test first programming, in cases where I was not only specifying the class under test but also the mocked interfaces, because modern IDEs like eclipse 3.1 and higher may come up with some fancy hinting/auto fixing feature, that will create non existing methods and classes.

Back to the jMock hosted comparison of easymock vs jMock, one drawback the authour of the comparison states is:

EasyMock lets you change the way that arguments are matched on a call-by-call basis, but the syntax is awkward and same matcher applies to all arguments.

Luckily this is not true for Easymock versions higher than 1.3, but let's take closer look anyway.
As example of how (not awkward and brittle) one specifies the beahaviuor in jmock this code fragment send on its mission to insult its readers intelligence:

mock.expects(once()).method("method1").with( eq(a) );
mock.expects(once()).method("method2").with( same(b1), ANYTHING )
.will(returnValue(method2Result));

Did I just read that a mock object(wich really is NOT the mock object) expects an invocation of a local method called once()(wich it does not), and that the result of the expecation is some object that returns probably a method (wich it also doesnt) with something equal to "a"?
Of course I do understand what this behaviour specification is all about, but to be able to write such a specification, one has to learn a syntax/api not as Java-ish as EasyMock, where the only necessary quirk is the distinction between behaviour recording and verification(wich was clear to me at once when I first saw it).

The author continues to state that:

This means that expectations are more verbose but precisely and clearly specify the expected behaviour of the object.By precisely specifying expected behaviour you get flexible tests¹ that break only when the actual behaviour is different from expected behaviour, and do not break when you make unrelated changes to application code.

To my knowledge it possible - without extensive API knowledge or awkward syntax - to specify any behaviour with abitrarily varied levels of precision with easymock. In jMock you were spoiled with this kind of flexibility right from the beginning, but this flexibilty is achieved only by introducing some drawbacks i.e.:

• test development/refactoring is harder because mock-method invocation errors will only show up during runtime
• after i.e. adding a parameter to method1 one has even more problems fixing this than merely changing the name of method1

The last point is especially dangerous in a multi developer team. The brittleness of the test cases is exactly what I want. There is added value in compile time checks.

Now let's have a look at the Easymock (2.0 and higher) version of the code snippet:

mock.method1(eq(a));
expect(mock.method2(same(b), anyInt())).andReturn(method2result);

Notice the subtile diffrences, in jMock one provides the method parameters in a non typesafe way through the "with" method, while in easymock one provides the arguments matchers where the actual parameters would be written. Also through the use of Java 5 generics in easymock the behaviuor definitions can be checked at compile time for type related mistakes.

Another great diffrence between the two examples are that in the jmock example "mock" is a control object and in the easymock example, "mock" is the mock object, that implements the interface we wich to "mock".

One drawback of easymock is that because method1 returns void we have to call "expectLastCall()" to get an object that allows us to specify some more expectation
like the number of invocations of the last method:

mock.method1(eq(a));
expectLastCall().anyTimes();

expect(mock.method2(same(b), anyInt())).once().andReturn(method2result);

Another more recent comparison dated around march 2006, seems to be done with a newer version of easymock, but it seems that also the author did not (yet) regard/compare more than the absolute minimum mock framework features one might want to use. The key point in favour of easymock were

• method identification through actual method of the proxy
• no extension of testcase base class
• mixing of class mocking and interface mocking is very easy (you just use another implementation of MockControl)
• very low API complexity

The points in favour of jMock were:

• No control object is required
• no replay/verify steps are necessary

Lets shed some more light upon this comparison:
The main point, that no control object is required in jMock is imho false. One exclusively operates on the control object and invokes a getter for the proxy. That is the reason just why
one cannot use the actual methods to specify behaviour. Easymock achieves refactoring safe actual method specification by introducton of its replay/verify concept. Furthermore, by exploiting new Java 1.5 features(static imports, generics, varargs) easymock almost completely frees the user from the burdern of being aware of this dichotomy in his daily work, and it even frees the user from the burden of subclassing some special test case base class as in jMock.

JMock on the other hand allows the user to specify behaviuor without having the recording/playback states, and mock behaviuor is automgically verified by base class methods after the method finishes. Also mock control object can be instantiated by a base class member method.
While this is for sure some helpful syntactic sugar, it is not so hard to do any of this with easymock because of the clever design of the easymock MockControl class and the usage of static imports.


Also the possibility to specify argument matchers was judged as "difficult" in easymock and easy in jMock.

Finally, I would like to point to an interesting article done by Martin Fowler, who explains some interesting aspects of overall applicability of Mock/Stubs/Dummies and Fakes, and tries to open a more general perspective on test doubles of all kinds. He also includes a small comparison of jMock and EasyMock. He also compares jMock to a very outdated version of Easymock it seems, as the code he provides as example, does not make use of the "import static" feature of Java 1.5, as well as some other features that could have been used to make the code alot more like typical jMock code. The article can be found here.
Although this article was updated 02 Jan 07
the easymock example he provides is also based on an obsolete version of easymock. Exactly why the author values mentioning some diffrences between jMOck and easymock is not clear to me. The article focuses on explaining the distinction between stubs and mock, not on comparing mock frameworks.
I will use this opportunity to rewrite this example of easymock:

public class OrderEasyTester extends TestCase {
private static String TALISKER = "Talisker";

private MockControl warehouseControl;
private Warehouse warehouseMock;

public void setUp() {
warehouseControl = MockControl.createControl(Warehouse.class);
warehouseMock = (Warehouse) warehouseControl.getMock();
}

public void testFillingRemovesInventoryIfInStock() {
//setup - data
Order order = new Order(TALISKER, 50);

//setup - expectations
warehouseMock.hasInventory(TALISKER, 50);
warehouseControl.setReturnValue(true);
warehouseMock.remove(TALISKER, 50);
warehouseControl.replay();

//exercise
order.fill(warehouseMock);

//verify
warehouseControl.verify();
assertTrue(order.isFilled());
}

public void testFillingDoesNotRemoveIfNotEnoughInStock() {
Order order = new Order(TALISKER, 51);

warehouseMock.hasInventory(TALISKER, 51);
warehouseControl.setReturnValue(false);
warehouseControl.replay();

order.fill((Warehouse) warehouseMock);

assertFalse(order.isFilled());
warehouseControl.verify();
}
}

Here is the rewrite:

import static org.easymock.EasyMock.*;

public class OrderEasyTester extends TestCase {
private static String TALISKER = "Talisker";

private Warehouse warehouseMock;

public void setUp() {
warehouseMock = createMock(Warehous.class);
}

public void testFillingRemovesInventoryIfInStock() {
//setup - data
Order order = new Order(TALISKER, 50);

//setup - expectations
expect(warehouseMock.hasInventory(TALISKER, 50).andReturn(true);
warehouseMock.remove(TALISKER, 50);
replay(warehouseMock);

//exercise
order.fill(warehouseMock);

//verify
verify(warehouseMock);
assertTrue(order.isFilled());
}

public void testFillingDoesNotRemoveIfNotEnoughInStock() {
Order order = new Order(TALISKER, 51);

warehouseMock.hasInventory(TALISKER, 51).andReturn(false);
replay(warehouseMock);

order.fill((Warehouse) warehouseMock);

assertFalse(order.isFilled());
verify(warehouseMock);
}
}

to be continued...