Feeds:
Posts
Comments

Archive for the ‘Test Frameworks’ Category

Flaky user interface tests don’t generally flake a second time unless there is a real problem.  Unfortunately the standard unit test runners don’t offer an option to double-check test failures.  This can result in a high build failure rate.  After being hit with a large number of suddenly flaky UI tests when a new version of Chrome came out we wrote our own test runner to double-check test failures.

Scooter is a simple NUnit console replacement.

It is compatible with NUnit attribute names out of the box but the attribute names can be replaced via command line configuration switches if your tests are marked with different attributes.

Unique features:

– Scooter double checks all test failures/errors by running the test a second time and only counting a failure if the second attempt fails.

– You can attribute an alternative method (via the -afta command line switch, e.g. -afta=AfterEachFailedTestAttribute) to be called after each test failure. This can be used for capturing screenshots, test failure notification, etc.

	[AfterEachFailedTest] 
	public void After_each_failed_test(string testFixtureName, string testMethodName) {}

– You can also create TestSuiteSetUpAttribute and TestSuiteTearDownAttribute (or equivalent) attributes and make a method for each to perform an action at the beginning and end of the test run respectively.

	[AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = true)]
	public class TestSuiteTearDownAttribute : Attribute
	{
	}
	
	[Test]
	[TestSuiteTearDownAttribute]
	public void Close_all_the_browsers()
	{

– Scooter can optionally run the tests in parallel (-parallel) and/or shuffled (-shuffle).

Usage:

Replace the call to NUnit console in your build script with one that calls Scooter.Console.exe instead.
All TestFixtures in your assemblies will be run unless marked with an Ignore or Explicit attributes.

License: MIT License.

Read Full Post »

We finally decided to do something about the time we spend waiting for the pre-check-in build to run. Our check-in dance starts with running a rake script to compile our code, run the associated unit and integration tests, and create deployment Zip files. This entire process takes ~96 seconds on an Intel Core i7-2600K CPU as follows:

  • Compile – 21 seconds
  • Run tests – 58 seconds
  • Build Zip files – 17 seconds

NUnit is clearly the biggest time consumer. A quick check of the task manager while the tests are running shows the following.

From blog images

A Google search for ready made solutions found PNUnit (a distributed version of NUnit), MBUnit (requires marking assembly, test fixtures, and/or tests as [Parallelizable]), and this StackOverflow question which led us to an inactive open source project called NUnit.MultiCore on Codeplex. We decided to fork the latter.

We modified the code as follows:

  • Make it run all test fixtures in parallel by default – no test project code changes required
  • Use C# Parallel.ForEach in place of explicit threading
  • Change Main to return a result code so it works with Continuous Integration
  • Make the output more closely resemble that of NUnit

Introducing this tool into our build process reduced our average run time to 66 seconds and the test portion to 28 seconds by making more effective use of the processor cores.

From blog images

The main problem we encountered in converting our test projects to run in parallel involved RhinoMocks which, it turns out, isn’t thread safe. So we forked a copy from Ayende and made the appropriate changes.

We also noticed that the albacore msbuild task doesn’t support configuring MSBuild to use the /m switch to parallelize the compilation. So we forked a copy of albacore and added that capability.

This reduced our compile time from 21 seconds to 19 seconds.

We’re quite happy with the speed up we’ve gained so far and we have identified additional time savings that can gained in the Zip file creation portion.

Read Full Post »

While reading Karl Seguin’s post entitled Stop Using Mocks I was reminded of a pattern we use at work to reduce test tax. Test tax is the time you spend fixing “broken tests” that result from tests, and in particular mocks, that know too much about your implementation.

Consider the interactions in the sample method from Karl’s post:

public User FindByCredentials(string username, string password)
{
	var user = _store.FindOneByNamedQuery("FindUserByUserName", username);
	if (user == null) { return null; }
	return _encryption.CheckPassword(user.Password, password) ? user : null;
}

At a higher level it looks like

This method has calls to two injected tools, a branch after each, and the second is dependent on the output of the first. Tests for this kind of method can become brittle fast. You might easily find yourself fixing half a dozen “broken tests” that are specific to the second injected tool because you changed the method prototype of the call to the first in a way that wasn’t accounted for in the tests. And this is a simple method. Ugh!

We’ve tried a number of different patterns to reduce this test tax. Our current best solution is to rewrite this kind of method into what we call a coordinator. Coordinator methods have no branches and are tested by integration tests, that means real objects instead of mock objects. The process they are coordinating is broken down into a series of steps that can be unit tested in isolation. The step methods do not generally appear on the interface of the coordinating class.

Karl’s method might be implemented using the coordinator pattern as follows:

[CanBeNull]
public User FindByCredentials([NotNull] string username, [NotNull] string password)
{
	var user = GetUserByUserName();
	return GetUserIfPasswordMatches(user.ToNonNull(), password);
}

with subordinate steps

[CanBeNull]
public User GetUserByUserName([NotNull]string username)
{
	var user = _store.FindOneByNamedQuery("FindUserByUserName", username);
	return user;
}

[CanBeNull]
public User GetUserIfPasswordMatches([NotNull] User user, [NotNull] string password)
{
	bool passwordMatches = _encryption.CheckPassword(user.Password, password);
	return passwordMatches ? user : null;
}

This lets us write detailed tests around GetUserByUserName and GetUserIfPasswordMatches without worrying about a change in the implementation of one method unexpectedly breaking the tests for the other. At the integration test level we can verify broader behaviors and interactions with real objects.

Read Full Post »

I encountered this NUnit message in a test fixture that had the following method:

	[TearDown]
	private void AfterEachTest()
	{
		_repository.VerifyAllExpectations();
	}

NUnit doesn’t allow methods decorated with [SetUp] or [TearDown] attributes to be private. Change the access to public or protected to resolve the problem.

Read Full Post »

This is a sample unit test written using the FluentAssert BDD framework.

	[Test]
	public void Given_a_business_object_with_no_natural_keys()
	{
		Test.Verify(
			with_a_business_object_that_has_no_natural_keys,
			when_lookup_is_called,
			should_return_a_valid_notification);
	}

When my cursor is on one of the test actions and I hit alt-enter Resharper is usually smart enough to suggest creating a method.

Other times, however, Resharper doesn’t offer the option we need.

My solution to this problem is a Visual Studio macro that creates a void parameterless method for the “word” the cursor is on. This necessitates determining where the new method should be created e.g. top of the file, bottom of the file, just before the method, just after the method. I like to keep the new method close to the test when I’m first creating it so that means above or below the method. Initially I tried it above the method but learned that in my workflow I expect the new method to be added below the test. So how do we do that? We’ll write a Visual Studio macro to do the work.

First we need to copy the proposed method name. We’ll grab the word under the cursor and put it in a variable. Also, since we don’t want to affect the user’s copy-paste buffer we’ll do it within an undo context.

        DTE.UndoContext.Open("AddBddMethod")

        DTE.ExecuteCommand("Edit.SelectCurrentWord")

        Dim name As String
        name = DTE.ActiveDocument.Selection.Text

Next to move to the end of the test method we’ll search for ‘}’ and close the find dialog when we’re done.

        DTE.ExecuteCommand("Edit.Find")
        DTE.Find.Target = vsFindTarget.vsFindTargetCurrentDocument
        DTE.Find.FindWhat = "}"
        DTE.Find.MatchCase = True
        DTE.Find.MatchWholeWord = False
        DTE.Find.Backwards = False
        DTE.Find.MatchInHiddenText = True
        DTE.Find.PatternSyntax = vsFindPatternSyntax.vsFindPatternSyntaxLiteral
        DTE.Find.Action = vsFindAction.vsFindActionFind
        If (DTE.Find.Execute() = vsFindResult.vsFindResultNotFound) Then
            Throw New System.Exception("vsFindResultNotFound")
        End If
        DTE.Windows.Item("{CF2DDC32-8CAD-11D2-9302-005345000000}").Close()

We want one newline between the test method and the new method so we move the cursor one position to the right to get past the ‘}’ then add two newlines.

        DTE.ActiveDocument.Selection.CharRight()
        DTE.ActiveDocument.Selection.NewLine()
        DTE.ActiveDocument.Selection.NewLine()

Next we’ll introduce the new method.

        DTE.ActiveDocument.Selection.Text = "private void "
        DTE.ActiveDocument.Selection.Text = name
        DTE.ActiveDocument.Selection.Text = "()"
        DTE.ActiveDocument.Selection.NewLine
        DTE.ActiveDocument.Selection.Text = "{}"

Our method now looks like:

	private void when_lookup_is_called()
	{
	}// <- cursor here

Now we want to set up to add contents to the new method so we move up between the ‘{}’ and add a newline.

        DTE.ActiveDocument.Selection.LineUp()
        DTE.ExecuteCommand("Edit.BreakLine")

Our method now looks like:

	private void when_lookup_is_called()
	{
		// <- cursor here
	}

Finally we need to close the undo context.

        DTE.UndoContext.Close()

That’s it.

Next we need to add it as a Visual Studio macro. First open the macro editor.

Create a new module called “RecordingModule” under MyMacros.

Then replace the entire contents of the default macro with the macro we built:

Option Strict Off
Option Explicit Off
Imports System
Imports EnvDTE
Imports EnvDTE80
Imports EnvDTE90
Imports System.Diagnostics

Public Module RecordingModule


	Sub Add_BDD_method()
        DTE.UndoContext.Open("AddBddMethod")
        DTE.ExecuteCommand("Edit.SelectCurrentWord")

        Dim name As String
        name = DTE.ActiveDocument.Selection.Text

        DTE.ExecuteCommand("Edit.Find")
        DTE.Find.Target = vsFindTarget.vsFindTargetCurrentDocument
        DTE.Find.FindWhat = "}"
        DTE.Find.MatchCase = True
        DTE.Find.MatchWholeWord = False
        DTE.Find.Backwards = False
        DTE.Find.MatchInHiddenText = True
        DTE.Find.PatternSyntax = vsFindPatternSyntax.vsFindPatternSyntaxLiteral
        DTE.Find.Action = vsFindAction.vsFindActionFind
        If (DTE.Find.Execute() = vsFindResult.vsFindResultNotFound) Then
            Throw New System.Exception("vsFindResultNotFound")
        End If
        DTE.Windows.Item("{CF2DDC32-8CAD-11D2-9302-005345000000}").Close()
        DTE.ActiveDocument.Selection.CharRight()
        DTE.ActiveDocument.Selection.NewLine()
        DTE.ActiveDocument.Selection.NewLine()
        DTE.ActiveDocument.Selection.Text = "private void "
        DTE.ActiveDocument.Selection.Text = name
        DTE.ActiveDocument.Selection.Text = "()"
	DTE.ActiveDocument.Selection.NewLine
        DTE.ActiveDocument.Selection.Text = "{}"
        DTE.ActiveDocument.Selection.LineUp()
        DTE.ExecuteCommand("Edit.BreakLine")
        DTE.UndoContext.Close()
    End Sub
End Module

Finally, bind a keystroke to the macro and you’ll be able to create test actions very quickly.

Read Full Post »

I’ve been using FluentAssert for my test-driven development because it abstracts away the test details leaving only a highly readable test. I also find that it enforces a certain level of separation of concerns and moistness in the tests without spreading portions of the tests through an inheritance stack. For example:

[TestFixture]
public class When_asked_if_a_string_is_null_or_empty_with_optional_trim
{
	[Test]
	public void Given_a_null_string_and_trim_is_false()
	{
		Test.Static()
			.When(asked_if_a_string_is_null_or_empty)
			.With(a_null_string)
			.With(trim_set_to_false)
			.Should(return_true)
			.Verify();
	}
}

with test details

	private void asked_if_a_string_is_null_or_empty()
	{
		_result = _input.IsNullOrEmpty(_trim);
	}

	private void a_null_string()
	{
		_input = null;
	}

	private void trim_set_to_false()
	{
		_trim = false;
	}

	private void return_true()
	{
		_result.ShouldBeTrue();
	}

Today I was reading about features of the Factor language, specifically the part about naming code not values and that it is stack based instead of function based, and it occurred to me that a DSL that has no parameters is a lot like a contactenative language except for the extra visual bits like . and () joining the method calls together. This started me to thinking how I might be able to further reduce the ceremonial aspects of the test framework.

One nice thing about a DSL is it can be coded to be context sensitive, meaning it might not let you put an assertion call (e.g. Should()) before an arranging call (e.g. With()). A lower ceremony framework wouldn’t necessarily offer that level of protection but in return we might get tests that are even easier to read and that don’t have an execution command appended. That’s one thing that bothers me about the current framework, the command of execution, .Verify(), is required at the end of each test chain in order to fire off the framework test runner. If you leave it off the NUnit test runs but the BDD test doesn’t and you get a false green. I wondered if I could address that annoyance as well as make something more concatenative. Here’s the result:

[TestFixture]
public class When_asked_if_a_string_is_null_or_empty_with_optional_trim
{
	[Test]
	public void Given_a_null_string_and_trim_is_false()
	{
		Test.Verify(
			with_a_null_string,
			with_trim_set_to_false,
			when_asked_if_a_string_is_null_or_empty,
			should_return_true
			);
	}
}

Instead of chaining methods with Action parameters we’re passing all the Actions as a params array. The DSL methods .When(part_A) .Should(part_B) etc. are simply replaced with their parameters part_A, part_B but we can change the Action names to include the When, Should etc. in order to retain readability in the test and its output. The call to Verify becomes the only framework method we call and we can give it the test Actions to run in the order we want to run them instead of having to give the Action for the method under test (When()) first – this is about as close as I think I can get to a Forth style Word command line in C#.

I’ve integrated this variation with the existing test result reporting so if you start your Action names with known keywords you’ll continue to get uppercased keyword output, e.g.:

WITH a null string
WITH trim set to false
WHEN asked if a string is null or empty
SHOULD return true - FAILED

  Expected: True
  But was:  False

I’ve added this as an alternative DSL path in the FluentAssert framework on GitHub if you want to try it out.

Enjoy!

Read Full Post »

The gist of BDD (Behavior Driven Develoment) is being able to write tests in language that a non-programmer can read and verify. Until now I’ve used a style that I adapted from J P Boodhoo that looks like the following:

An outer class for all tests of a particular class

public class CollectionTests
{
}

that contains TestFixture classes for each action (method) on the class under test

	[TestFixture]
	public class When_asked_to_Save
	{
	}

	[TestFixture]
	public class When_asked_to_Delete
	{
	}

each of which contains test methods that describe the desired behavior when its action is called with certain inputs.

	[Test]
	public void Should_set_the_Id_if_save_succeeds()
	{
	}

	[Test]
	public void Should_return_an_error_if_save_fails()
	{
	}

This makes tests that are highly readable in NUnit with test run results that can be coerced into a Word or Excel report. The main problem is the bodies of the Setup TearDown and particularly the Test methods get so cluttered with Mock expectations and other goo that even developers can have a hard time looking at them and verifying their correctness not to mention fixing breakage.

	var person = new Person
	{
		FirstName = "Loreena",
		LastName = "McKennitt"
	};
	_people.Save(person);
	person.Id.ShouldNotBeNull();

Did Save() succeed? Was it supposed to?

Then Jon Kruger sent out this tweet describing his desired test framework and sent me down this really interesting path:

	Given(person)
	.With(valid_first_name)
	.With(valid_last_name)
	.When(save_succeeds)
	.Should(populate_the_id_property)

So let’s figure out how to do it.

The first thing is we don’t want to have to inherit from a base class to get access to Given() so that implies some kind of starter class. It could be an extension method, or a class that has to be new’d up, but I prefer a static class that starts a DSL for type T.

public static class Test
{
	public static TestDefinition<T> For<T>()
	{
		return new TestDefinition<T>();
	}
}

Next we add the Given method to TestDefinition to capture an instance of the class under test. In most cases this will end up being a Mock. We’ll just pass the T along and return a new container.

public class TestDefinition<T>
{
	public TestGivenClause<T> Given(T item)
	{
		return new TestGivenClause<T>(item);
	}
}

Given() has two possible next paths With() – a method that, according to the example, can be called multiple times to modify the item under test

	public TestGivenClause<T> With(...)
	{
		// ...
		return this;
	}

and When() which transitions us to the next stage in the DSL

	public TestWhenClause<T> When(...)
	{
		return new TestWhenClause<T>(...);
	}

The interesting question is what goes in the …’s? Well, notice when looking at the desired language

	Given(person)
	.With(valid_first_name)
	.With(valid_last_name)
	.When(save_succeeds)
	.Should(populate_the_id_property)

that the things passed to With(), When() and Should() don’t have parentheses… and they aren’t lambdas either. If we want this to exactly match the example the only things left are Func and Action. In the case of With() we’re going to want to take a T and make a change to it but we don’t need to return anything so that implies Action. Based on the example we also want to be able to call With() multiple times so internally we’ll store the Actions in a List:

	private readonly List<Action<T>> _setupActionsForItem = new List<Action<T>>();

	public TestGivenClause<T> With(Action<T> setupForItem)
	{
		_setupActionsForItem.Add(setupForItem);
		return this;
	}

When() can also take an action because our implementation of save_succeeds can store its result in our test class for later processing, so:

	public TestWhenClause<T> When(Action<T> action)
	{
		return new TestWhenClause<T>(_item, action, _setupActionsForItem);
	}

Which implies the constructor:

	public TestWhenClause(
		T item, 
		Action<T> action, 
		List<Action<T>> setupActionsForItem)
	{
		_item = item;
		_action = action;
		_setupActionsForItem = setupActionsForItem;
	}

Now the example shows only one Should() clause but we’ll want that to be repeatable too (although in practice I generally only have one assertion per test method) so that means the implementation is just a variation on With()

	public TestWhenClause<T> Should(Action<T> verify)
	{
		_verificationActions.Add(verify);
		return this;
	}

The only thing remaining is a command to fire off the test. Let’s call it Verify() . In Verify() we’ll have to perform all the With() actions on the class under test

	public void Verify()
	{
		foreach (var setupItem in _setupActionsForItem)
		{
			setupItem(_item);
		}

then perform the When() action

		_action(_item);

and finally, perform all the Should() verifications

		foreach (var verify in _verificationActions)
		{
			verify(_item);
		}
	}

Does that look familiar? It should. We’ve just written a wrapper around the AAA (Arrange Act Assert) pattern that lets us abstract all the details out of the actual test.

Now to use it we write the test in language as close to business-speak as possible. The first difference is we’re going to be testing a Repository, not a Person (because we’re separating concerns) so the test comes out as:

[Test]
public void Should_populate_the_person_id_if_save_succeeds()
{
	Test.For<MongoCollection<Person>>()
		.Given(_personRepository)
		.When(Save_succeeds)
		.Should(Populate_the_id_property_of_the_person)
		.Verify();
}

Now that we can see the final product the .For<T> part is a bit icky to look at and certainly wouldn’t be useful to a business person. Let’s clean that up a bit. We can get the T by induction if we just pass the object so we’ll add Given() to Test too:

	public static TestGivenClause<T> Given<T>(T item)
	{
		return new TestGivenClause<T>(item);
	}

and that will make the repository test look like

[Test]
public void Should_populate_the_person_id_if_save_succeeds()
{
	Test.Given(_personRepository)
		.When(Save_succeeds)
		.Should(Populate_the_id_property_of_the_person)
		.Verify();
}

Much better.

Next we’ll use Resharper (or whatever) to stub the Action methods and finally fill in the details

private void Save_succeeds(MongoCollection<Person> repository)
{
	_person = new Person
	          	{
	          		FirstName = "Loreena",
	          		LastName = "McKennitt"
	          	};

	repository.Save(_person);
}

private void Populate_the_id_property_of_the_person(MongoCollection<Person> repository)
{
	_person.Id.ShouldNotBeNull();
}

That was too easy. Let’s try another one:

[Test]
public void Should_not_populate_the_person_id_if_save_fails()
{
	Test.Given(_personRepository)
		.When(Save_fails)
		.Should(Not_populate_the_id_property_of_the_person)
		.Verify();
}

private void Save_fails(MongoCollection<Person> repository)
{
	_person = new Person
	          	{
	          		FirstName = null,
	          		LastName = null
	          	};

	Assert.Throws<ArgumentException>(()=> repository.Save(_person));
}

private void Not_populate_the_id_property_of_the_person(MongoCollection<Person> repository)
{
	_person.Id.ShouldBeNull();
}

Sweet.

This framework has the side benefit of forcing us to break up our Actions into composable chunks and keeps our test classes moist and maintainable. Now one could go so far as to split the parts among the [Setup] (For, Given, and When) [Test] (Should) and [TearDown] (Verify) but I think that takes us away from the goal – tests that are so easy to read that a non-developer can understand and verify their correctness and detect/suggest missing test cases.

One more thing to think about: Given a project where many tests are written in this style it wouldn’t be too difficult to write a tool to crawl the source code for methods marked [Test] that start with the Test.Given() construct, grab their innards and drop them all in some kind of document that could be reviewed by an analyst. Hmmm.

This framework is available as part of the FluentAssert project on GitHub.

Edit: Try any of my BDD Kata if you’d like to see the current version of this framework in action.

Read Full Post »

%d bloggers like this: