Protosystem Blog

By

In my day job at JDA Software I have been looking at code coverage options for determining the effectiveness of our testing.  My team uses four types of tests to test the software we write.

  • Unit Tests - Tests focused on a single component of the application.  These tests are MSTests that exercise a specific software component and typically mock out any dependant components.
  • Integration Tests -  Tests focused on multiple components of the application.  These tests are MSTests that exercise a component and it's dependencies to ensure the components work together.
  • Manual Tests - Tests that are executed by are testers manually from a GUI interface. 
  • Automated functional test - Tests that are scripted in a fashion that can be repeated build after build to ensure the build is still functional.  Sometimes referred to regression testing.

The goal of implementing a code coverage process was to determine the effectiveness of the types of tests listed above.  Visual Studio 2005 Team Edition (for testers and developers) provides some nice code coverage features we wanted to tap into.  Code coverage of unit tests and some integration tests worked fine from the visual studio IDE.  But for the integration, manual and automation tests that used web services we began to run into problems.  The web services that where hosted under IIS where not getting covered.  After some research I determined that code coverage under ISS was not going to work.  So I started looking into alternatives.  One thing I noticed is that web service projects that where not hosted under IIS had no problem getting covered.  These types of projects used the development web server that comes with ASP.NET 2.0.   So I decided to look into using the same development web server in place of IIS for code coverage purposes.

 

The generic steps for establishing code coverage for web services is as follows:

  1. Turn on instrumentation for the binaries that you want to cover
  2. Start the coverage monitor
  3. Start the development web server for a specified unused port pointing to the folder that is supposed to represent the web service.
  4. Execute your tests
  5. Stop the development web server
  6. Stop the coverage monitor
  7. Review the results in Visual Studio 2005

For the following commands use the Visual Studio Command prompt.

 

Turning on instrumentation of assemblies from the command line is done by the following command:

vsinstr -coverage myassembly.dll

 

To start the coverage monitor you use the follwoing command:

start vsperfmon -coverage -output:mytestrun.coverage

 

To start the development web server use the following command:

start WebDev.WebServer /port:8080 /path:c:\mypath

 

To stop the development web server simply right click on the task bar icon for the web server and select stop

 

To shutdown the coverage monitor use the following command:

vsperfcmd -shutdown

 

We were now able to do code coverage of all types of tests that we planned on implementing.  Some of our test runs are going to executed by the build and some will be executed by the testers.  Since VS.Net 2005 supports the ability to merge code coverage results this should not be a problem to combine all runs into a single report that now shows how effective our tests really are.

 

Some references to articles that help me come to this conclusion:

Command Line code coverage:

http://blogs.msdn.com/ms_joc/articles/406608.aspx

 

Using WebDev.WebServer from the command line:

http://www.devsource.com/article2/0,1895,1886246,00.asp

Comments


Comments are closed