Automated Unit Testing with
Xcode 3 and Objective-C

OCUnit is a mature and robust unit testing framework that integrates easily into Xcode. With only a few minutes worth of work you can configure your Cocoa projects to automatically run any test cases that you write whenever you compile your project, making it easy to continuously test and re-test with no additional effort. In this article, you’re going to set up an Xcode project for unit testing, write and run a unit test, and then setup your project for unit testing using the debugger.

What Is Unit Testing?

The term unit test refers to any test designed to validate the functionality of a single “unit” of source code. In object-oriented languages like Objective-C and C++, a “unit” is generally defined as a single method.

The main goal of unit testing is to make sure that each part of your program functions properly before you use that functionality anywhere else in your program. In addition to helping you identify many bugs earlier in the development process, automated unit testing also makes it possible to know when future changes to your code break previously-working functionality.

Unit testing does not eliminate the need for user testing and you’ll never be able to write automated unit tests to cover every possible problem, but it can help to ensure a final software product with fewer bugs. As new bugs are discovered, new unit tests can be added so that if the same bug gets introduced again, it will get caught immediately.

A unit testing framework is a set of code libraries designed to simplify the process of creating automated unit tests. Although there are many different unit testing frameworks, and the specifics vary quite a bit, the general idea is that you can issue a single command to cause all of your tests to run and, if any of your unit tests fail, you will know immediately. Many unit testing frameworks, including OCUnit, the unit testing framework that ships with Xcode, allow you to incorporate unit test execution into the build process so that each time you compile your code, the appropriate tests are run and you are immediately notified if any of the tests break.

Setting Up Xcode for OCUnit

Although OCUnit, which was written by Mark Scheurer of Sen:te software and released under an open source license, does ship with Xcode 3, it is not enabled by default when you create a new project. To show the steps involved, let’s create an application project in Xcode and set it up to use OCUnit.

Open up Xcode and select New Project… from the File menu, or select ⌘⇧N to bring up the New Project Assistant. Select Application from under the Mac OS X heading in the left pane, and then click on the Cocoa Application icon (Figure 1). Press the Choose… button and, when prompted for a name, type in OCT.

Figure 1  Selecting Cocoa Application in the New Project Assistant.

The Xcode project you just created has one target, called OCT, which is designed to generate your application. You need to add a second target to hold your unit tests. When you compile your project with this new target, Xcode will first compile your application, and then look for any unit tests you’ve written and run them. If any of your unit tests fail, the build will fail.

Select New Target… from the Project menu. When the New Target Assistant comes up, select Cocoa from under the Mac OS X heading in the left pane, then select the Unit Test Bundle icon from the upper right pane (Figure 2).

Figure 2 Adding a new Unit Test bundle target to our project.

When prompted for a name for the new target, type in Unit Tests and then press the Finish button. Once Xcode is done creating the new target, the Target Info window for Unit Tests will appear. Select the General tab in the Info Window, if it’s not already selected (Figure 3).

Figure 3 The General tab in the Info Window.

Under the Target Info’s General tab, there are two panes, the top of which is labeled Direct Dependencies. You have to make your original application target a direct dependency of this new Unit Test target so that compiling the Unit Test target triggers the application target to compile first. By doing that, you ensure that you’re always running your unit tests against the most current version of our application code.

Press the plus icon just below the Direct Dependencies window, select the target named OCT from the sheet that drops down, then press the Add Target button (Figure 4).

Figure 4 The Target Info window for the Unit Tests target

Now switch to the Build tab and scroll down through the build settings until you find an entry called Test Host under the Unit Testing heading. This setting identifies the application to be tested, and you need to point this to your application executable. The path you need to provide here is not the one to the application bundle but the path to the actual executable file inside of the bundle.

There are two ways to edit an entry. Single-click on the Test Host row to select it. Then single-click in the column to the right of the words Test Host. This allows you to edit the field in-line. Alternatively, you can double-click in the column to the right of the words Test Host and a sheet will drop down to allow you to edit the field. The benefit of the latter approach is that you have a much larger space in which to type.

No matter your approach, edit the field and enter the following:

$(BUILT_PRODUCTS_DIR)/OCT.app/Contents/MacOS/OCT

Before you hit the okay button, type ⌘A to select the text you just typed, then press ⌘C to copy it to the clipboard. You have to use the same value for another build setting, so this will save you a little typing.

Now look for a build setting called Bundle Loader under the Linking heading. Edit the field in the right column and press ⌘V to paste in the value you copied a moment ago. This setting tells your Unit Test target to load the specified application as if it were a framework, which will allow your testing classes, which won’t be part of the application, to test code that is contained in the application.

Using OCUnit

Your project is now properly configured for unit testing and you can create your unit tests. All of the unit test classes you create will be added to the Unit Test target exclusively, while all your other classes will be added to the original application target only (the one with the same name as your project).

When OCUnit is ready to do its thing, the unit test bundle is injected into the application and then your unit tests are run, as if they were part of the application, while both the application and the unit tests are in memory. Because of that, the unit tests will have access to your application code without you having to add your application source code files to the Unit Test target.

Because the unit tests are only added to the unit test target, when you are ready to release your application, you select the original application target (in this case, the one called OST), and your application will get created without any of the test cases being compiled in.

The end result is that all your tests run every time you compile when you’ve got the Unit Test target selected, but your final application will ship without being burdened by any of that additional test code.

Creating a Simple Model Object

Now you’re going to create a very simple Objective-C class around which you can write a test case. Make sure that the Classes folder is selected, then press ⌘N or select New File… from the File menu. When the new file assistant comes up, select Cocoa from under the Mac OS X heading in the left pane, then select Objective-C class in the right pane. Press the Next button and give this file the name RightTriangle.m. Make sure the checkbox to create the header file is checked so that it will automatically create RightTriangle.h as well.

Also, make sure that in the Targets box, OCT is checked, and that the Unit Tests target is not, as in Figure 5.

Figure 5 Adding the RightTriangle class to our application.

Single-click RightTriangle.h and replace the contents with the code that follows:

#import <Cocoa/Cocoa.h>

@interface RightTriangle : NSObject {
	double	lengthA;
	double	lengthB;
	double	lengthC;
}
@property double lengthA;
@property double lengthB;
@property double lengthC;
-(id)initWithLengthA:(double)inA andLengthB:(double)inB;
@end

Switch over to RightTriangle.m and replace the contents with this code:

#import "RightTriangle.h"

@implementation RightTriangle
@synthesize lengthA;
@synthesize lengthB;
@synthesize lengthC;

-(id)initWithLengthA:(double)inA andLengthB:(double)inB
{
	if (self = [super init])
	{
		self.lengthA = inA;
		self.lengthB = inB;
		self.lengthC = sqrt((inA * inA) * (inB * inB));
	}
	return self;
}
-(void)dealloc
{
	[super dealloc];
}
@end

This is a pretty simple, straightforward class that represents a right triangle. The initializer takes the two shorter sides of the triangle as inputs and calculates the length of the long side using the Pythagorean Theorem. Of course, if you look closely, you’ll see that the formula is wrong. The Pythagorean theorem is A2 + B2 = C2, but the method multiplied the squares of A and B rather than adding them.

Obviously, this class is much simpler than most of the projects you’ll be testing, but it will do just fine for demonstrating how unit tests work with OCUnit and Xcode.

Creating a Test Case

Let’s create a test case designed to test the RightTriangle class. Once again,with the Classes folder still selected, press ⌘N or select New File… from the File menu to create a new file. Cocoa should still be selected in the left pane, so in the right pane, scroll down until you see Objective-C test case class and select it. Press the Next button and when prompted for a name, use RightTriangleTestCases.m, making sure the checkbox to create the header file is checked. This time, you need to add the class to the Unit Test target and not to the OCT target, as shown in Figure 6.

Figure 6 Adding the test case class to the Unit Tests target.

Select the RightTriangleTestCases.h class and replace its contents with the following code:

#import <SenTestingKit/SenTestingKit.h>

#define kSideA	5
#define kSideB	5
#define kExpectedResult sqrt(50.0)
@interface RightTriangleTestCases : SenTestCase {
}
-(void)testCreateRightTriangle;
@end

There are a few things worth noticing here. First, three constants are defined that will be used to test the class. This model is a known quantity: a right triangle where sides A and B are both 5. The length of side C of such a triangle will be the square root of 50, which is also defined as a constant.

The class is defined as a subclass of SenTestCase. It is very important that you subclass this class when writing your test case methods. The name SenTestCase comes from Marco Scheurer of Sen:te software, the author of OCUnit.

This class declares a single test method. Notice that the name of the method starts with “test”. That’s important. The system knows which methods to run when you build the Unit Tests target by looking for methods that begin with “test” in classes that are subclasses of SenTestCase. If your test case methods do not begin with “test” or if they are not contained in classes that are subclasses of SenTestCase, then those unit tests will never execute. Switch over to RightTriangleTestCases.m and replace the contents of that file with the following:

#import "RightTriangleTestCases.h"
#import "RightTriangle.h"

@implementation RightTriangleTestCases
-(void)testCreateRightTriangle
{	
	RightTriangle *testTriangle = [[RightTriangle alloc] initWithLengthA:kSideA andLengthB:kSideB];
	STAssertNotNil(testTriangle, @"Triangle not created successfully");
	STAssertTrue(testTriangle.lengthC == kExpectedResult, @"Side C calculated incorrectly. Expected %f, got %f", kExpectedResult, testTriangle.lengthC);

	[testTriangle release];
}
@end

Since you are testing the RightTriangle class, you have to import the RightTriangle.h header file. Then you have to implement the test method you declared in the header file a moment ago. The first thing the test method does is allocate and initialize a RightTriangle object, based on the defined values.

The next line uses one of many macros that are part of OCUnit. This particular macro, STAssertNotNil, checks the value of the first argument to make sure it is not nil. If the triangle object is nil, then the build would stop and Xcode would show an error at this line of code. The error message would be the second argument passed in.

The third statement uses another macro built into OCUnit, STAssertTrue, which makes sure that the expression passed in as the first parameter evaluates to true. There are several STAssert macros, and they all allow you to pass format strings, as was done here, rather than just passing a static string. When testTriangle.lengthC == kExpectedResult evaluates to NO, then Xcode will print an error that includes both of the values being compared. The message string works exactly like NSString’s initWithFormat: method, and you can pass as many additional parameters into these functions as long as you include a replacement token for each of them in the message string. In this example, the message includes two replacement tokens (%f), with two more arguments after the message to supply their values.

The following table shows the various macros available in OCUnit. Each of these will cause your build to fail under different circumstances, and you should be able to find a macro for any situation you need to test.

OCUnit Macro	Purpose
STAssertNil(a1, description, ...)	Generates an error if a1 is not nil.
STAssertNotNil(a1, description, ...)	Generates an error if a1 is nil.
STAssertTrue(expression, description, ...)	Generates an error if expression does not evaluate to true
STAssertFalse(expression, description, ...)	Generates an error if expression does not evaluate to false
STAssertEqualObjects(a1, a2, description, ...)	Generates an error if a1 is not equal to a2. Both must be Objective-C objects.
STAssertEquals(a1, a2, description, ...)	Generates an error if a1 is not equal to a2. Both must be C scalar values.
STAssertEqualsWithAccuracy(left, right, accuracy, description, ...)	Generates an error if a1 and a2 are not within a certain amount of each other. Primarily for use with floats and doubles to take into account small rounding errors due to the way they store values.
STAssertThrows(expression, description, ...)	Generates an error if expression does not throw an exception.
STAssertThrowsSpecific(expression, specificException, description, ...)	Generates an error if expression does not throw an exception of a specified class.
STAssertThrowsSpecificNamed(expr, specificException, aName, description, ...)	Generates an error if expression doesn’t throw an exception with a specific name.
STAssertNoThrow(expression, description, ...)	Generates an error if expression throws an exception.
STAssertNoThrowSpecific(expression, specificException, description, ...)	Generates an error if expression throws an exception of a specified class.
STAssertNoThrowSpecificNamed(expr, specificException, aName, description, ...)	Generates an error if expression throws an exception with a specific name.
STFail(description, ...)	Generates an error.
STAssertTrueNoThrow(expression, description, ...)	Generates an error if expression is false or if it throws an exception.
STAssertFalseNoThrow(expron, description, ...)	Generates an error if expression is true or if it throws an exception

Table 1: Macros Available in OCUnit

In the upper left corner of your project’s window, use the popup menu to change the target from OCT to Unit Tests (Figure 7).

Figure 7 Changing to the Unit Tests target.

Now, you are ready to run your test. Select Build from the Build menu or press ⌘B. If you didn’t make any typos, your build should should fail because of that error in the RightTriangle class (Figure 8).

Figure 8 Build error caused by unit test failing.

As long as the active target is Unit Tests, the build will not finish successfully until all your unit tests pass. You can still compile your application by switching to the Debug or Release targets, but you won’t get a successful build with Unit Tests until and unless you’ve fixed all the bugs causing your unit tests to fail.

Using the Debugger with OCUnit

In this case, the source of the bug is obvious, but it won’t always be, so let’s look at how to debug when a unit test fails. If you were to set a breakpoint at the first line in testCreateRightTriangle and then select Build and Debug from the Debug menu, it wouldn’t work. Your Unit Tests would fire, but the debugger would not stop at your breakpoint.

Because of the way the unit test bundle is injected into your application, the debugger won’t work unless you do a little bit of extra setup.

Setting up OCUnit to Allow Debugging

When you compile the Unit Tests target, it builds the underlying application, launches it, then uses a private framework to inject your unit test bundle into the application. In order for the debugger to work, you have to set up the executable so that when it is launched from Xcode, it will inject the test bundle into itself and then run the tests.

One important thing to note is that the configuration options you’re about to set will affect your application any time it is launched from Xcode, not just when the Unit Tests target is selected.

In the Groups & Files pane of Xcode, double-click the topmost item. It should be called OCT. This should bring up the project info window. If the General tab is not selected, select it now. Look for the option called Place Intermediate Build Files In:. Change it from Default intermediates location to Build products location. Once you’ve done that, you can close the info window.

Next, expand Executables in the Groups & Files pane. There should be one item underneath it, also called OCT. Double-click that to bring up the executable info window. In the executable info window, select the Arguments tab.

There are two panes in this tab (Figure 9). The top one is for arguments that will be passed to the program when it is launched from Xcode.

Figure 9 Adding arguments and environment variables to the executable info window.

You need to add one argument, which will tell your program to execute all unit tests once it is launched. Click the plus icon below the arguments window, and it will add a row to the arguments table. Edit the new row and type in

-SenTest All

The bottom pane is used to set environment variables for your application. You need to add four environment variables here to ensure that your Unit Test bundle is injected into our application and that the application is able to find all the frameworks it needs to inject and run the bundle. Press the plus button at the bottom of the info window to add four rows to the arguments table. Set the four rows to the following values:

Name	Value
DYLD_INSERT_LIBRARIES	$(DEVELOPER_LIBRARY_DIR)/PrivateFrameworks/DevToolsBundleInjection.framework/DevToolsBundleInjection
DYLD_FALLBACK_FRAMEWORK_PATH	$(DEVELOPER_LIBRARY_DIR)/Frameworks
XCInjectBundle	Unit Tests.octest
XCInjectBundleInto	OCT.app/Contents/MacOS/OCT

The first value, DYLD_INSERT_LIBRARIES, tells the application the location of the private framework that will be used to inject the testing bundle into our application. The second environment variable, DYLD_FALLBACK_FRAMEWORK_PATH, tells the application where to find the OCUnit framework needed by our unit tests. The third environment variable, XCInjectBundle, tells the insert libraries which bundle needs to get injected, and you set this to point to your testing bundle. The last environment variable, which is a new requirement with Xcode 3, tells the injection framework what application executable the bundle should be injected into. You set this path to your project’s application executable.

It should be noted that arguments set here will not affect your shipped application. They will only be used when launching your application from within Xcode. When you are done debugging your unit tests, just uncheck the values you just added, and it will stop injecting and running your unit tests whenever your application is launched from Xcode.

One more step and you’re ready to run. As you know, Xcode uses gdb as its debugger. Because of the default way that gdb interacts with its target application, via the spawning of a Unix shell, Xcode will not be able to inject our test bundle properly unless we make a slight adjustment. We’ll do this using Terminal.app.

Launch Terminal. First, make sure you are at your home directory by typing:

							
cd ~

Then type:

echo '' >> .gdbinit 
echo 'set start-with-shell 0' >> .gdbinit	

These commands will append the necessary configuration to the end of your existing .gdbinit file if you have one or it will create a new .gdbinit file in your home directory if you don’t already have one.

Debugging with OCUnit

Now you are all set up to debug your unit tests, but there’s one more little gotcha. You can’t use Build and Debug. You have to do this as a two step process. First, select Build from the Build menu. It will compile the underlying application, then your test cases, then it will run your tests and you will see your error again. Once it has finished, then you can then select Debug from the Run menu or from Xcode’s toolbar and it will launch your tests with the debugger.

This time, it should stop at any breakpoint you set in your test cases and let you step through the code just as you would when debugging your application code. Make sure to turn off the arguments and environment variables when you’re done debugging, otherwise or it will continue to inject and run the tests every time you launch your program from within Xcode

Unit Testing Strategies

You’ve seen how to add a test bundle to your project, and to set up your project both for automatically running tests at build time and for debugging your test cases. You are ready to move forward with unit testing your Cocoa projects. Here are a few very general guidelines to keep in mind when designing your unit tests.

• Test the creation of any new object you write. Make sure you test using every initializer as well as every one of your class convenience methods.
• Make sure you test the use of every new method you write that’s not a straight accessor/mutator.
• Write boundary tests whenever your code includes a loop or numeric comparisons. You want to always make sure that all values that are supposed to fall inside the loop or comparison do, and that all values that should fall outside also do.
• Every time you fix a bug, whether found by your or one of your customers, add a new unit test specifically to test for that bug.

Updated: 2009-01-30