Test Automation and Execution Framework (TAEF)

I know I’d said I was going to do a series of posts explaining how to use TAEF. This is no longer necessary. Documentation for TAEF has been posted on the MSDN. Instead I’ll use this post to explain why TAEF is my favorite testing framework.

TAEF is developed by a team within the Windows org specifically dedicated to improving and standardizing test writing and execution within Windows. While developed by the Windows org, TAEF is also gaining popularity in a lot of different teams across Microsoft.

At this point you may be wondering, why write another testing framework? There are already a number of popular frameworks such as NUnit, MSTest, Boost’s Unit Test Library, etc.

In my opinion, the biggest reason is the flexibility TAEF gives you. TAEF was designed from the ground up to provide consistent programming, execution and features for tests writing in .Net, C++ and scripts. Additionally, it provides a simple interface for building Unit Tests while providing additional capabilities useful in writing end to end tests.

I encourage everyone interested in writing test code to read the MSDN docs. In the rest of this posting I’m simply going to list out some of the highlights of TAEF.

TAEF can execute MS Test based tests

Writing tests in managed code for TAEF is simple. You merely create a DLL with a class that has attributes to indicate it’s a test. TAEF supports uses the test markup attributes used by MS Test as defined in Microsoft.VisualStudio.TestTools.UnitTesting. If you don’t wish to take a dependency on the Visual Studio unit test dll, the attributes are also defined in WEX.TestExecution.Markup.

Except for the namespace declaration, usage of the attributes is identical. If you’re using MS Test now, your tests can be immediately executed by TAEF. No need to redo any of your existing tests.

The C++ markup is nearly identical to the Managed markup

Writing tests C++ for TAEF is simple. You merely create a DLL with one or more classes. You create a header file to define the interface to the class. In the interface you’ll use some macros to markup your test methods and test properties. Then simply implement the class in a source file.

The macros mirror the managed attributes as closely as possible.

TAEF supports multiple data driven testing mechanisms

• Runtime Parameters
• Table Data Source – XML file containing data rows
• WMI Query – You can run a WMI Query and the test will be executed for each row returned by the query.
• Light Weight Data Driven – If you don’t want a full XML file, you can provide data as part of the testcase markup. If you have more than one data element, it will execute the test for each combination of data elements.
• PICT – Pairwise Independent Combinatorial Testing

Rebooting within Tests

TAEF supports writing tests which require a reboot as part of the test.

TAEF allows you to control the threading model of the test (MTA, STA, None)

Title says it all

Parallel Tests

TAEF allows you to flag tests as being Parallelizable. When you provide the Parallel attribute for a test TAEF will execute tests with the Parallel flag concurrently. However, any test without this flag will be executed by itself.

Cross Machine Execution

TAEF supports a /runon flag. When used it will copy the test binaries and execute the test on the machine specified.

RunAs

TAEF supports running a test Elevated, Restricted, as System or as a Low Integrity Process. You can pass the RunAs flag when calling TAEF or you can provide it as an attribute on each test. For example, you could have 5 tests in a single DLL. You could have some of those tests run elevated while the others run restricted.

Even more interesting, you can provide the RunAs flag to individual rows in a Table Data Source based test. This will allow you to run the exact same test using different execution models in a single call to TAEF.

Note: This feature does require you to install the TE Service. This install is extremely simple to do and is very light weight.

User Mode Monitor

TAEF can monitor a running process during test execution. When monitoring a process it will automatically log events such as modules loaded, debugee output and crashes.

Debug Ability Features

I left the best for last. TAEF provides a lot of mechanisms for debugging test failures.

• Mini Dump on Error – TAEF can take a mini dump any time an error is logged in a test.  You can initiate a minidump in code without logging an error if you choose.
• Screen Shot on Error – TAEF can take a screenshot any time an error is logged. You can take a screenshot any time an error is logged if you choose.
• StackTraceOnError – TAEF can write out the stack trace whenever an error is logged.
• BreakOnCreate and BreakOnInvoke – You can cause the test to immediately break into the debugger before a test class is instantiated or before a test method is executed. This will allow you to attach the debugger and step through a test, even if you aren’t running it from the VS IDE.
• BreakOnError – Breaks into th e debugger when an error is logged in a test.

Summary

I didn’t list all the features of TAEF by a long shot. I strongly encourage everyone writing tests for Windows based code to take a good look at TAEF.