Facilitating test creation with macros

This is a sample unit test written using the FluentAssert BDD framework.

	[Test]
	public void Given_a_business_object_with_no_natural_keys()
	{
		Test.Verify(
			with_a_business_object_that_has_no_natural_keys,
			when_lookup_is_called,
			should_return_a_valid_notification);
	}

When my cursor is on one of the test actions and I hit alt-enter Resharper is usually smart enough to suggest creating a method.

Other times, however, Resharper doesn’t offer the option we need.

My solution to this problem is a Visual Studio macro that creates a void parameterless method for the “word” the cursor is on. This necessitates determining where the new method should be created e.g. top of the file, bottom of the file, just before the method, just after the method. I like to keep the new method close to the test when I’m first creating it so that means above or below the method. Initially I tried it above the method but learned that in my workflow I expect the new method to be added below the test. So how do we do that? We’ll write a Visual Studio macro to do the work.

First we need to copy the proposed method name. We’ll grab the word under the cursor and put it in a variable. Also, since we don’t want to affect the user’s copy-paste buffer we’ll do it within an undo context.

        DTE.UndoContext.Open("AddBddMethod")

        DTE.ExecuteCommand("Edit.SelectCurrentWord")

        Dim name As String
        name = DTE.ActiveDocument.Selection.Text

Next to move to the end of the test method we’ll search for ‘}’ and close the find dialog when we’re done.

        DTE.ExecuteCommand("Edit.Find")
        DTE.Find.Target = vsFindTarget.vsFindTargetCurrentDocument
        DTE.Find.FindWhat = "}"
        DTE.Find.MatchCase = True
        DTE.Find.MatchWholeWord = False
        DTE.Find.Backwards = False
        DTE.Find.MatchInHiddenText = True
        DTE.Find.PatternSyntax = vsFindPatternSyntax.vsFindPatternSyntaxLiteral
        DTE.Find.Action = vsFindAction.vsFindActionFind
        If (DTE.Find.Execute() = vsFindResult.vsFindResultNotFound) Then
            Throw New System.Exception("vsFindResultNotFound")
        End If
        DTE.Windows.Item("{CF2DDC32-8CAD-11D2-9302-005345000000}").Close()

We want one newline between the test method and the new method so we move the cursor one position to the right to get past the ‘}’ then add two newlines.

        DTE.ActiveDocument.Selection.CharRight()
        DTE.ActiveDocument.Selection.NewLine()
        DTE.ActiveDocument.Selection.NewLine()

Next we’ll introduce the new method.

        DTE.ActiveDocument.Selection.Text = "private void "
        DTE.ActiveDocument.Selection.Text = name
        DTE.ActiveDocument.Selection.Text = "()"
        DTE.ActiveDocument.Selection.NewLine
        DTE.ActiveDocument.Selection.Text = "{}"

Our method now looks like:

	private void when_lookup_is_called()
	{
	}// <- cursor here

Now we want to set up to add contents to the new method so we move up between the ‘{}’ and add a newline.

        DTE.ActiveDocument.Selection.LineUp()
        DTE.ExecuteCommand("Edit.BreakLine")

Our method now looks like:

	private void when_lookup_is_called()
	{
		// <- cursor here
	}

Finally we need to close the undo context.

        DTE.UndoContext.Close()

That’s it.

Next we need to add it as a Visual Studio macro. First open the macro editor.

Create a new module called “RecordingModule” under MyMacros.

Then replace the entire contents of the default macro with the macro we built:

Option Strict Off
Option Explicit Off
Imports System
Imports EnvDTE
Imports EnvDTE80
Imports EnvDTE90
Imports System.Diagnostics

Public Module RecordingModule


	Sub Add_BDD_method()
        DTE.UndoContext.Open("AddBddMethod")
        DTE.ExecuteCommand("Edit.SelectCurrentWord")

        Dim name As String
        name = DTE.ActiveDocument.Selection.Text

        DTE.ExecuteCommand("Edit.Find")
        DTE.Find.Target = vsFindTarget.vsFindTargetCurrentDocument
        DTE.Find.FindWhat = "}"
        DTE.Find.MatchCase = True
        DTE.Find.MatchWholeWord = False
        DTE.Find.Backwards = False
        DTE.Find.MatchInHiddenText = True
        DTE.Find.PatternSyntax = vsFindPatternSyntax.vsFindPatternSyntaxLiteral
        DTE.Find.Action = vsFindAction.vsFindActionFind
        If (DTE.Find.Execute() = vsFindResult.vsFindResultNotFound) Then
            Throw New System.Exception("vsFindResultNotFound")
        End If
        DTE.Windows.Item("{CF2DDC32-8CAD-11D2-9302-005345000000}").Close()
        DTE.ActiveDocument.Selection.CharRight()
        DTE.ActiveDocument.Selection.NewLine()
        DTE.ActiveDocument.Selection.NewLine()
        DTE.ActiveDocument.Selection.Text = "private void "
        DTE.ActiveDocument.Selection.Text = name
        DTE.ActiveDocument.Selection.Text = "()"
	DTE.ActiveDocument.Selection.NewLine
        DTE.ActiveDocument.Selection.Text = "{}"
        DTE.ActiveDocument.Selection.LineUp()
        DTE.ExecuteCommand("Edit.BreakLine")
        DTE.UndoContext.Close()
    End Sub
End Module

Finally, bind a keystroke to the macro and you’ll be able to create test actions very quickly.

Read Full Post »

A Low Ceremony Test Driven Development Framework

I’ve been using FluentAssert for my test-driven development because it abstracts away the test details leaving only a highly readable test. I also find that it enforces a certain level of separation of concerns and moistness in the tests without spreading portions of the tests through an inheritance stack. For example:

[TestFixture]
public class When_asked_if_a_string_is_null_or_empty_with_optional_trim
{
	[Test]
	public void Given_a_null_string_and_trim_is_false()
	{
		Test.Static()
			.When(asked_if_a_string_is_null_or_empty)
			.With(a_null_string)
			.With(trim_set_to_false)
			.Should(return_true)
			.Verify();
	}
}

with test details

	private void asked_if_a_string_is_null_or_empty()
	{
		_result = _input.IsNullOrEmpty(_trim);
	}

	private void a_null_string()
	{
		_input = null;
	}

	private void trim_set_to_false()
	{
		_trim = false;
	}

	private void return_true()
	{
		_result.ShouldBeTrue();
	}

Today I was reading about features of the Factor language, specifically the part about naming code not values and that it is stack based instead of function based, and it occurred to me that a DSL that has no parameters is a lot like a contactenative language except for the extra visual bits like . and () joining the method calls together. This started me to thinking how I might be able to further reduce the ceremonial aspects of the test framework.

One nice thing about a DSL is it can be coded to be context sensitive, meaning it might not let you put an assertion call (e.g. Should()) before an arranging call (e.g. With()). A lower ceremony framework wouldn’t necessarily offer that level of protection but in return we might get tests that are even easier to read and that don’t have an execution command appended. That’s one thing that bothers me about the current framework, the command of execution, .Verify(), is required at the end of each test chain in order to fire off the framework test runner. If you leave it off the NUnit test runs but the BDD test doesn’t and you get a false green. I wondered if I could address that annoyance as well as make something more concatenative. Here’s the result:

[TestFixture]
public class When_asked_if_a_string_is_null_or_empty_with_optional_trim
{
	[Test]
	public void Given_a_null_string_and_trim_is_false()
	{
		Test.Verify(
			with_a_null_string,
			with_trim_set_to_false,
			when_asked_if_a_string_is_null_or_empty,
			should_return_true
			);
	}
}

Instead of chaining methods with Action parameters we’re passing all the Actions as a params array. The DSL methods .When(part_A) .Should(part_B) etc. are simply replaced with their parameters part_A, part_B but we can change the Action names to include the When, Should etc. in order to retain readability in the test and its output. The call to Verify becomes the only framework method we call and we can give it the test Actions to run in the order we want to run them instead of having to give the Action for the method under test (When()) first – this is about as close as I think I can get to a Forth style Word command line in C#.

I’ve integrated this variation with the existing test result reporting so if you start your Action names with known keywords you’ll continue to get uppercased keyword output, e.g.:

WITH a null string
WITH trim set to false
WHEN asked if a string is null or empty
SHOULD return true - FAILED

  Expected: True
  But was:  False

I’ve added this as an alternative DSL path in the FluentAssert framework on GitHub if you want to try it out.

Enjoy!

Read Full Post »

A simple C# BDD test framework

The gist of BDD (Behavior Driven Develoment) is being able to write tests in language that a non-programmer can read and verify. Until now I’ve used a style that I adapted from J P Boodhoo that looks like the following:

An outer class for all tests of a particular class

public class CollectionTests
{
}

that contains TestFixture classes for each action (method) on the class under test

	[TestFixture]
	public class When_asked_to_Save
	{
	}

	[TestFixture]
	public class When_asked_to_Delete
	{
	}

each of which contains test methods that describe the desired behavior when its action is called with certain inputs.

	[Test]
	public void Should_set_the_Id_if_save_succeeds()
	{
	}

	[Test]
	public void Should_return_an_error_if_save_fails()
	{
	}

This makes tests that are highly readable in NUnit with test run results that can be coerced into a Word or Excel report. The main problem is the bodies of the Setup TearDown and particularly the Test methods get so cluttered with Mock expectations and other goo that even developers can have a hard time looking at them and verifying their correctness not to mention fixing breakage.

	var person = new Person
	{
		FirstName = "Loreena",
		LastName = "McKennitt"
	};
	_people.Save(person);
	person.Id.ShouldNotBeNull();

Did Save() succeed? Was it supposed to?

Then Jon Kruger sent out this tweet describing his desired test framework and sent me down this really interesting path:

	Given(person)
	.With(valid_first_name)
	.With(valid_last_name)
	.When(save_succeeds)
	.Should(populate_the_id_property)

So let’s figure out how to do it.

The first thing is we don’t want to have to inherit from a base class to get access to Given() so that implies some kind of starter class. It could be an extension method, or a class that has to be new’d up, but I prefer a static class that starts a DSL for type T.

public static class Test
{
	public static TestDefinition<T> For<T>()
	{
		return new TestDefinition<T>();
	}
}

Next we add the Given method to TestDefinition to capture an instance of the class under test. In most cases this will end up being a Mock. We’ll just pass the T along and return a new container.

public class TestDefinition<T>
{
	public TestGivenClause<T> Given(T item)
	{
		return new TestGivenClause<T>(item);
	}
}

Given() has two possible next paths With() – a method that, according to the example, can be called multiple times to modify the item under test

	public TestGivenClause<T> With(...)
	{
		// ...
		return this;
	}

and When() which transitions us to the next stage in the DSL

	public TestWhenClause<T> When(...)
	{
		return new TestWhenClause<T>(...);
	}

The interesting question is what goes in the …’s? Well, notice when looking at the desired language

	Given(person)
	.With(valid_first_name)
	.With(valid_last_name)
	.When(save_succeeds)
	.Should(populate_the_id_property)

that the things passed to With(), When() and Should() don’t have parentheses… and they aren’t lambdas either. If we want this to exactly match the example the only things left are Func and Action. In the case of With() we’re going to want to take a T and make a change to it but we don’t need to return anything so that implies Action. Based on the example we also want to be able to call With() multiple times so internally we’ll store the Actions in a List:

	private readonly List<Action<T>> _setupActionsForItem = new List<Action<T>>();

	public TestGivenClause<T> With(Action<T> setupForItem)
	{
		_setupActionsForItem.Add(setupForItem);
		return this;
	}

When() can also take an action because our implementation of save_succeeds can store its result in our test class for later processing, so:

	public TestWhenClause<T> When(Action<T> action)
	{
		return new TestWhenClause<T>(_item, action, _setupActionsForItem);
	}

Which implies the constructor:

	public TestWhenClause(
		T item, 
		Action<T> action, 
		List<Action<T>> setupActionsForItem)
	{
		_item = item;
		_action = action;
		_setupActionsForItem = setupActionsForItem;
	}

Now the example shows only one Should() clause but we’ll want that to be repeatable too (although in practice I generally only have one assertion per test method) so that means the implementation is just a variation on With()

	public TestWhenClause<T> Should(Action<T> verify)
	{
		_verificationActions.Add(verify);
		return this;
	}

The only thing remaining is a command to fire off the test. Let’s call it Verify() . In Verify() we’ll have to perform all the With() actions on the class under test

	public void Verify()
	{
		foreach (var setupItem in _setupActionsForItem)
		{
			setupItem(_item);
		}

then perform the When() action

		_action(_item);

and finally, perform all the Should() verifications

		foreach (var verify in _verificationActions)
		{
			verify(_item);
		}
	}

Does that look familiar? It should. We’ve just written a wrapper around the AAA (Arrange Act Assert) pattern that lets us abstract all the details out of the actual test.

Now to use it we write the test in language as close to business-speak as possible. The first difference is we’re going to be testing a Repository, not a Person (because we’re separating concerns) so the test comes out as:

[Test]
public void Should_populate_the_person_id_if_save_succeeds()
{
	Test.For<MongoCollection<Person>>()
		.Given(_personRepository)
		.When(Save_succeeds)
		.Should(Populate_the_id_property_of_the_person)
		.Verify();
}

Now that we can see the final product the .For<T> part is a bit icky to look at and certainly wouldn’t be useful to a business person. Let’s clean that up a bit. We can get the T by induction if we just pass the object so we’ll add Given() to Test too:

	public static TestGivenClause<T> Given<T>(T item)
	{
		return new TestGivenClause<T>(item);
	}

and that will make the repository test look like

[Test]
public void Should_populate_the_person_id_if_save_succeeds()
{
	Test.Given(_personRepository)
		.When(Save_succeeds)
		.Should(Populate_the_id_property_of_the_person)
		.Verify();
}

Much better.

Next we’ll use Resharper (or whatever) to stub the Action methods and finally fill in the details

private void Save_succeeds(MongoCollection<Person> repository)
{
	_person = new Person
	          	{
	          		FirstName = "Loreena",
	          		LastName = "McKennitt"
	          	};

	repository.Save(_person);
}

private void Populate_the_id_property_of_the_person(MongoCollection<Person> repository)
{
	_person.Id.ShouldNotBeNull();
}

That was too easy. Let’s try another one:

[Test]
public void Should_not_populate_the_person_id_if_save_fails()
{
	Test.Given(_personRepository)
		.When(Save_fails)
		.Should(Not_populate_the_id_property_of_the_person)
		.Verify();
}

private void Save_fails(MongoCollection<Person> repository)
{
	_person = new Person
	          	{
	          		FirstName = null,
	          		LastName = null
	          	};

	Assert.Throws<ArgumentException>(()=> repository.Save(_person));
}

private void Not_populate_the_id_property_of_the_person(MongoCollection<Person> repository)
{
	_person.Id.ShouldBeNull();
}

Sweet.

This framework has the side benefit of forcing us to break up our Actions into composable chunks and keeps our test classes moist and maintainable. Now one could go so far as to split the parts among the [Setup] (For, Given, and When) [Test] (Should) and [TearDown] (Verify) but I think that takes us away from the goal – tests that are so easy to read that a non-developer can understand and verify their correctness and detect/suggest missing test cases.

One more thing to think about: Given a project where many tests are written in this style it wouldn’t be too difficult to write a tool to crawl the source code for methods marked [Test] that start with the Test.Given() construct, grab their innards and drop them all in some kind of document that could be reviewed by an analyst. Hmmm.

This framework is available as part of the FluentAssert project on GitHub.

Edit: Try any of my BDD Kata if you’d like to see the current version of this framework in action.

Read Full Post »