Open Radar

 
Summary:
In order to set up an environment for some acceptance tests, I would like to start and stop a server before and after my tests are run. In Xcode 3.x, I would do this by using Run Script build phases, which, because tests were executed by building the bundle and running a script, worked.

Due to the way in which tests are run in Xcode 4.0 (separately from the act of building the test bundle), I can no longer use build phases. It appears that the logical place to do this is in the Test pre and post actions, using a Run Script action.

However, it would appear that the post-action is actually run before the tests have finished running, making it essentially useless (in my case, the server I started in my pre-action would be terminated while the tests were still running causing them to fail).

Steps to Reproduce:
Create a new Xcode 4 project with unit tests. Create a simple test - it doesn't matter if it passes or fails, and insert a sleep() to cause that test to ensure the test run lasts a certain duration.

Something as simple as:

- (void)testPostActions
{
   sleep(3);
   STAssertTrue(YES);
}

Now open the scheme editor and add a Run Script pre-action and post action. To make it clear when the scripts are being run, we'll use the built-in "say" command.


My pre-action shell script is:

say "Running pre action"

My post-action shell script is:

say "Running post action".

Now select Product > Test. The test bundle and app target will be built, you will hear "Running pre action" and then almost immediately, "Running post action". If you look at the logs, you will see that, because of our sleep, the tests are still running and won't finish until the sleep times out.

Expected Results:
The post action should not be run until the test run is completely, no matter how long it takes. We use sleep() here as a quick and dirty way to force a longer test run, but any larger test suite could easily take several minutes to run. If we need to do something important when those tests are finished, the post action should not be run until the tests have finished running.

Actual Results:
Post action appears to be run as soon as the test bundle is executed.