Thursday, December 20, 2007

Creating Custom Code Analysis Rule

In VS2005, code analysis can be run very easily selecting the rules individually. There might be occasions that these rules are not enough. The rule libraries in VS2005 are created using Fxcop and you can easily create your own coding rules. I will give an example on creating a rule that will force the application to use "Test" as the prefix in unit test methods.

Fxcop dlls are located under C:\Program Files\Microsoft Visual Studio 8\Team Tools\Static Analysis Tools\FxCop. While writing the rules FxCopSdk and Microsoft.Cci dlls need to be references. Introspection library has been introduced in Fxcop but I could not see documentation enough. With the reflection and file assembler add-on, it is very easy to open the dlls and see the rule project as a whole in c# code. You will have lots of examples to look at.

I will summarize the necessary steps

1) Create a class library, UnitTestsRules class library

2) Add a base class for your unit test rules, UnitTestIntrospectionRule.cs

using Microsoft.FxCop.Sdk;
using Microsoft.FxCop.Sdk.Introspection;
using System;
namespace Microsoft.FxCop.Rules.Naming
{
internal abstract class UnitTestIntrospectionRule : BaseIntrospectionRule
{
protected UnitTestIntrospectionRule(string name)
: base(name, "UnitTestRules.UnitTestRules", typeof(UnitTestIntrospectionRule).Assembly)
{
}
internal Resolution GetNamedResolutionInternal(string id, params string[] args)
{
return this.GetNamedResolution(id, args);
}
public override TargetVisibilities TargetVisibility
{
get
{
return TargetVisibilities.All;
}
}
}
}
As seen in the example the constructor refers to a resource. This resource is an xml file with the name UnitTestRules.xml. It should be selected as an embedded resource in the project. Since the default namespace is UnitTestsRules, it is possible to access to the resource as UnitTestRules.UnitTestRules. If you have any directories in place for the resource you need to add them to the path as well.

3) Add the rule class, MethodsMustStartWithTest.cs

In this class we will write the rule to make sure all the test methods start with "Test" To do that we will overriding the method checking for members. If you are doing any code analysis for types etc you need to override the appropriate one. To test if a test method starts with Test, we first get the method info and check the method's attributes to see whether it contains TestMethodAttribute attribute or not. If there is a problem we add a resolution to the problems collection. Since it is a problem in a member, name the resolution as Member. It will match to an entry in the xml file that we created earlier. We will pass the name of the method that does not conform to the standard in the resolution.
using Microsoft.Cci;
using Microsoft.FxCop.Sdk;
using Microsoft.FxCop.Sdk.Introspection;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;
using System.Collections;
using System.Globalization;
using System.Reflection;
namespace Microsoft.FxCop.Rules.Naming
{
internal sealed class MethodsMustStartWithTest : UnitTestIntrospectionRule
{
public MethodsMustStartWithTest()
: base("MethodsMustStartWithTest")
{
}
public override ProblemCollection Check(Member member)
{
//Check if the member being processed is a method
Method method = member as Method;
if (member.NodeType != Microsoft.Cci.NodeType.Method)
{
return base.Problems;
}
//Chekc if the method is a test method
object[] info = method.GetMethodInfo().GetCustomAttributes(typeof(TestMethodAttribute), true);
if (info == null info.Length == 0)
{
return base.Problems;
}
//It is a test method, so check the name of the method to see if it starts with 'Test'
string name = member.Name.Name;
if (!name.StartsWith("Test"))
{
//Create a problem using 'Member' resolution
Problem problem = new Problem(this.GetNamedResolution("Member", new string[] { name }), name);
base.Problems.Add(problem);
}
return base.Problems;
}
}
}

4) Create the test rule content in UnitTestRules.xml

We will give a friendly name to the rule collection. It will be displayed in VS2005 using this name. We will also map the rule's type name to the name of the class that we just, MethodsMustStartWithTest.
For each rule give a different id. In this case it is UNITTEST0001. The other attributes are already self explanotory. We have a resolution named as Member and expects one paramaterwhich is the method name as explained in the previous step.

<Rules FriendlyName="Custom UnitTest Rules">
<Rule TypeName="MethodsMustStartWithTest" Category="UnitTests.Naming" CheckId="UNITTEST0001">
<Name>Methods Must Start With Test</Name>
<Description>Methods Must Start With Test</Description>
<Url>/UnitTesting/MethodsMustStartWithTest.html</Url>
<Resolution Name="Member">Method {0} does not start with Test</Resolution>
<Email>
</Email>
<MessageLevel Certainty="95">Error</MessageLevel>
<FixCategories>Breaking</FixCategories>
<Owner />
</Rule>
</Rules>

5) Deploy the rule dll

After building the dll, drop it under C:\Program Files\Microsoft Visual Studio 8\Team Tools\Static Analysis Tools\FxCop\Rules. It ius ready to use from VS2005!

TDD Standards, Naming Conventions


Standard naming conventions applied for code should also be applied for the test codes. But there might be some extra naming conventions in place like starting test methods with Test or having a specific keyword in the test class library name itself.

This page covers basic guidelines about the naming standards.
http://weblogs.asp.net/rosherove/archive/2005/04/03/TestNamingStandards.aspx



  • Test name should express a specific requirement

  • Test name should include the expected input or state and the expected result for that input or state

  • Test name should be presented as a statement or fact of life that expresses workflows and outputs

  • Test Name should only begin with Test if it is required by the testing framework or if it eases development and maintenance of the unit tests in some way.

  • Test name should include name of tested method or class


Fxcop is being used in VS2003 as a code analysis tool and in VS2005 it is already integrated to the development environment. You can enable code analysis and select specific rules. It can also be done in nightly builds to make sure your builds do not break your conventions. What if you need a new one? For example how do you make sure all the test methods start with "Test" if you have such a rule? I will explain how to create a custom rule in a different blog entry.

Wednesday, December 12, 2007

TDD Standards, Code Coverage


Code coverage defines the percentages of the methods tested by unit tests. Ideally it should be 100%. In this case all the lines in the code should be exercised. It does not tell about if all the possible scenarios are tested or not. With code coverage we try to execute every piece of code and test it. We can also use other coverage techniques to improve the quality of the tests. They are nicely written in this link. http://en.wikipedia.org/wiki/Code_coverage How many tests should we write then? Ideally all the tests should be written so that there is nothing left to break the code.


Thursday, December 06, 2007

What Should Be Tested in Unit Tests


Unit tests must be written to test general functionality and boundary conditions. Anything that can break the code should be tested. If there is an uncertainty for a piece of code, we should write a test method. It is better to use judgement while deciding about writing a unit test or not. It is not easy to test GUI. To be able to cover as much code as possible, GUI codes should be kept very small and business classes should be aimed at the tests. It also forces us to keep everything in the business classes and let the GUI do only the presentation work.


What to test:
- We should write test methods for all the public methods.
- Private methods should be tested via public methods. If a private method is very critic we can write a specific test method for it. In VS2005 it is very easy to add a unit test method. Visual Studio itself creates the unit test template using reflection.
- For each piece of requirement we should write a unit test. More than one requirement may be in one public method but usually each requirement represents one possible path in the code and each path should be tested.
-Expected or possible exceptions should be tested as well.
-Before fixing a bug, a unit test can be written first. We can then write the piece of code to fix the issue. It makes us sure that the code is bug-free.

What not to test
-There is no need to test accessors (to getvalues) or mutators (to set values). They are very simple get and set methods. They are unlikely to break the code.

Hints:
- Unit tests for each class must be placed in a separate test class.
- To test abstract classes implement abstract test classes. Respectively for the concrete classes, concrete test classes inheriting from the abstract ones should be written
- We should write the test methods for dependencies.