Xcode Test Navigator does not support +[XCTestCase testInvocations] well

Originator:benchatelain
Number:rdar://26029875 Date Originated:30-Apr-2016 04:46 PM
Status:Open Resolved:
Product:Developer Tools Product Version:Xcode 7.3 (7D175)
Classification:Serious Bug Reproducible:Always
 
This is a duplicate of rdar://26028557

Summary:

Add the following file (also available at https://gist.github.com/modocache/fb651b97e34b5badb56f585570d4d56b) to an XCTest test suite. Follow the instructions in the comments, which demonstrate the following bugs:

 1. `XCTestCase` subclasses that do not define any test methods are not displayed in test output, but *are* displayed in the Test Navigator. As a result, the Test Navigator provides misleading information when users define `XCTestCase` subclasses that are meant to provide shared test logic.
 2. The Xcode Test Navigator displays test invocations that were run before, even if those invocations would no longer be returned by `+[XCTestCase testInvocations]`.
 3. Xcode Test Navigator only displays test invocations returned by `+[XCTestCase testInvocations]` only once they've all been run.
 4. Xcode Test Navigator does not display all test invocations returned by `+[XCTestCase testInvocations]`.
 5. Xcode crashes the XCTest test suite when attempting to re-run a test invocation that no longer exists.

```
#import <XCTest/XCTest.h>
#import <objc/runtime.h>

/**
 Running a test suite composed of only this file results in the following output,
 which is exactly what I expect as a user:

    ```
    Test Suite 'All tests' started at 2016-04-30 15:32:11.912
    Test Suite 'InvocationStationTests.xctest' started at 2016-04-30 15:32:11.913
    Test Suite 'StringLengthOfFourTestCase' started at 2016-04-30 15:32:11.913
    Test Case '-[StringLengthOfFourTestCase parameterizedTest_blop]' started.
    Test Case '-[StringLengthOfFourTestCase parameterizedTest_blop]' passed (0.000 seconds).
    Test Case '-[StringLengthOfFourTestCase parameterizedTest_blip]' started.
    Test Case '-[StringLengthOfFourTestCase parameterizedTest_blip]' passed (0.000 seconds).
    Test Case '-[StringLengthOfFourTestCase parameterizedTest_blam]' started.
    Test Case '-[StringLengthOfFourTestCase parameterizedTest_blam]' passed (0.000 seconds).
    Test Suite 'StringLengthOfFourTestCase' passed at 2016-04-30 15:32:11.915.
    Executed 3 tests, with 0 failures (0 unexpected) in 0.001 (0.002) seconds
    Test Suite 'StringLengthOfThreeTestCase' started at 2016-04-30 15:32:11.916
    Test Case '-[StringLengthOfThreeTestCase parameterizedTest_foo]' started.
    Test Case '-[StringLengthOfThreeTestCase parameterizedTest_foo]' passed (0.000 seconds).
    Test Case '-[StringLengthOfThreeTestCase parameterizedTest_bar]' started.
    Test Case '-[StringLengthOfThreeTestCase parameterizedTest_bar]' passed (0.000 seconds).
    Test Case '-[StringLengthOfThreeTestCase parameterizedTest_baz]' started.
    Test Case '-[StringLengthOfThreeTestCase parameterizedTest_baz]' passed (0.000 seconds).
    Test Case '-[StringLengthOfThreeTestCase parameterizedTest_flo]' started.
    Test Case '-[StringLengthOfThreeTestCase parameterizedTest_flo]' passed (0.000 seconds).
    Test Suite 'StringLengthOfThreeTestCase' passed at 2016-04-30 15:32:11.917.
    Executed 4 tests, with 0 failures (0 unexpected) in 0.001 (0.002) seconds
    Test Suite 'InvocationStationTests.xctest' passed at 2016-04-30 15:32:11.918.
    Executed 7 tests, with 0 failures (0 unexpected) in 0.002 (0.005) seconds
    Test Suite 'All tests' passed at 2016-04-30 15:32:11.918.
    Executed 7 tests, with 0 failures (0 unexpected) in 0.002 (0.006) seconds


    Test session log:
    /Users/bgesiak/Library/Developer/Xcode/DerivedData/InvocationStation-dhushsztyfdzejhkwcfzltheeaug/Logs/Test/FBD56B53-AB7C-4B6B-9826-A488847CDE2A/Session-2016-04-30_15:32:09-1ezFHI.log

    Program ended with exit code: 0
    ```

 However, this file demonstrates five problems with Xcode's Test Navigator when
 used in conjunction with `+[XCTestCase testInvocations]`:
 
 1. `XCTestCase` subclasses that do not define any test methods are not
    displayed in test output, but *are* displayed in the Test Navigator.
    As a result, the Test Navigator provides misleading information when users
    define `XCTestCase` subclasses that are meant to provide shared test logic.
 2. The Xcode Test Navigator displays test invocations that were run before,
    even if those invocations would no longer be returned by
    `+[XCTestCase testInvocations]`.
 3. Xcode Test Navigator only displays test invocations returned by
   `+[XCTestCase testInvocations]` only once they've all been run.
 4. Xcode Test Navigator does not display all test invocations returned by
   `+[XCTestCase testInvocations]`.
 5. Xcode crashes the XCTest test suite when attempting to re-run a test
    invocation that no longer exists.
 
 Each of these problems is explained in detail below.
 */

#pragma mark - Parameterized test implementation using `+[XCTestCase testInvocations]`

/**
 A parameterized test case allows you to make the same expectation against
 multiple input parameters. The expectation is specified by subclassing
 `ParameterizedTestCase` and overriding `+[ParameterizedTestCase parameterizedTest]`.
 The parameters can be specified by overriding `+[ParameterizedTestCase parameters]`.
 */

/**
 Expectations are blocks that are given a parameter. They may make
 expectations based on that parameter by using an `XCTAssert` assertion.
 Because `XCTAssert` macros require `self` to be defined as an instance of
 an `XCTestCase`, one is passed to the block.
 */
typedef void (^ParameterTestBlock)(XCTestCase *self, id parameter);

/**
 Parameterized tests execute the same expectation on many different
 parameters. If the expectation fails for one of those many parameters,
 we want to know which one. So the user provides a name for each parameter
 test.
 */
typedef NSDictionary<NSString *, id> TestNameAndParameter;
static NSString * const TestNameKey = @"TestName";
static NSString * const TestParameterKey = @"TestParameter";

/**
 Subclass in order to provide a parameterized test and a set of parameters.

 Problem #1: This is meant to be an abstract base class used to define tests,
             not an actual test case. However, there is no way to keep it
             from being displayed in the Xcode Test Navigator.
             Screenshot: ParameterizedTestCase_appears_in_Test_Navigator.png

             One solution would be providing an API for an `XCTestCase`
             to not be displayed in the Xcode Test Navigator.
 */
@interface ParameterizedTestCase : XCTestCase

/** Return an expectation block to be made against each parameter. */
+ (ParameterTestBlock)parameterizedTest;

/**
 Return a list of test names and parameters. Each of these will be tested
 against the expectation block from `+[ParameterizedTestCase parameterizedTest]`.
 */
+ (NSArray<TestNameAndParameter *> *)parameters;

@end

@implementation ParameterizedTestCase

// Subclasses must override this method to return an expectation to be made
// against each parameter from `+[ParameterizedTestCase parameters]`.
+ (ParameterTestBlock)parameterizedTest { return ^(XCTestCase *self, id parameter){}; }

// Subclasses must override this method to return a set of parameters to be
// passed to the expectation from `+[ParameterizedTestCase parameterizedTest]`.
+ (NSArray<TestNameAndParameter *> *)parameters { return @[]; }

// A helper method that generates a selector for the given parameterized test name.
+ (SEL)_selectorForTestWithName:(NSString *)name {
    NSString *testName = [NSString stringWithFormat:@"parameterizedTest_%@", name];
    return NSSelectorFromString(testName);
}

+ (void)initialize {
    // We add an instance method for each parameterized test.
    for (TestNameAndParameter *testNameAndParameter in [self parameters]) {
        SEL name = [self _selectorForTestWithName:testNameAndParameter[TestNameKey]];
        IMP imp = imp_implementationWithBlock(^(ParameterizedTestCase *self){
            [[self class] parameterizedTest](self, testNameAndParameter[TestParameterKey]);
        });
        const char *types = [[NSString stringWithFormat:@"%s%s%s",
                              @encode(id), @encode(id), @encode(SEL)] UTF8String];
        class_addMethod(self, name, imp, types);
    }
}

// Apple XCTest allows us to define tests dynamically, using
// `+[XCTestCase testInvocations]`. We return one `NSInvocation` for each
// `+[ParameterizedTestCase parameters]` parameter we pass to the
// `+[ParameterizedTestCase parameterizedTest]` expectation.
//
// Problem #2: `+[XCTestCase testInvocations]` is the only API available
//             for dynamically defining test methods in Apple XCTest.
//             However, it is *not called* when running a single test via
//             the Xcode Test Navigator (done by clicking on the gray arrow
//             next to a test name). This is not ideal--after all, what if
//             `+testInvocations` returns a list of invocations that no
//             longer includes the single test that Xcode is attempting to
//             run? Currently, Xcode does not respect that fact, and happily
//             runs the test anyway--a test that no longer exists, according
//             to `+testInvocations`.
//
//             One solution would be to provide an API for an `XCTestCase` to
//             list which tests should be displayed in the Xcode Test Navigator.
//             Xcode could use this API to determine what to display.
+ (NSArray<NSInvocation *> *)testInvocations {
    NSMutableArray<NSInvocation *> *invocations = [NSMutableArray array];
    for (TestNameAndParameter *testNameAndParameter in [self parameters]) {
        // We create an invocation based on the added instance method and add
        // it to our list of test invocations.
        SEL testSelector = [self _selectorForTestWithName:testNameAndParameter[TestNameKey]];
        NSMethodSignature *signature = [self instanceMethodSignatureForSelector:testSelector];
        NSInvocation *invocation = [NSInvocation invocationWithMethodSignature:signature];
        invocation.selector = [self _selectorForTestWithName:testNameAndParameter[TestNameKey]];
        [invocations addObject:invocation];
    }
    return [invocations copy];
}

@end

#pragma mark - Examples of using parameterized tests that demonstrate Xcode Test Navigator bugs

@interface StringLengthOfThreeTestCase : ParameterizedTestCase
@end

@implementation StringLengthOfThreeTestCase

// Problem #3: The Xcode Test Navigator does display these tests--but only
//             after they've been run once. That also means there's no way to
//             run just one of these tests until after running all of them.
//             For large test suites, this can become prohibitively expensive.
//
//             One solution would be to provide an API for an `XCTestCase` to
//             list which tests should be displayed, and thus are capable of
//             being run, in the Xcode Test Navigator. This would allow an
//             `XCTestCase` to list its test methods without those tests being
//             run first.
//
//             (Xcode is capable of listing test methods that are *not*
//             dynamically generated via `+[XCTestCase testInvocations]`,
//             presumably by parsing the source code. Dynamically generated
//             tests, however, are not always clearly present in source code.)
//
// Problem #4: The Xcode Test Navigator doesn't accurately display all of these
//             tests. Run the test suite, and you'll see the following:
//             Only_three_tests_from_StringLengthOfThreeTestCase_appear_in_Test_Navigator.png.
//             Look at the logged XCTest output, and you'll see that four tests
//             are being executed, but only three appear in the Test Navigator.
//             It actually does appear for a split second, but disappears for
//             some reason.
+ (NSArray<TestNameAndParameter *> *)parameters {
    return @[
        @{TestNameKey: @"foo", TestParameterKey: @"foo"},
        @{TestNameKey: @"bar", TestParameterKey: @"bar"},
        @{TestNameKey: @"baz", TestParameterKey: @"baz"},
        @{TestNameKey: @"flo", TestParameterKey: @"flo"},
    ];
}

+ (ParameterTestBlock)parameterizedTest {
    return ^(XCTestCase *self, NSString *parameter){
        XCTAssertEqual([parameter length], 3);
    };
}

@end

@interface StringLengthOfFourTestCase : ParameterizedTestCase
@end

@implementation StringLengthOfFourTestCase

+ (NSArray<TestNameAndParameter *> *)parameters {
    return @[
        // Problem #5: The following steps cause Xcode to crash the test suite:
        //             1. Run the entire test suite in Xcode (⌘U).
        //             2. Comment out the "blop" test below. This means it will
        //                no longer be defined dynamically as a test in this
        //                test suite. **It is still displayed in the Xcode
        //                Test Navigator, even after being commented out.**
        //             3. Using the Test Navigator, click on the green check
        //                mark next to the `StringLengthOfFourTestCase > parameterizedTest_blop`
        //                test. This will re-run this test. However, this test
        //                is no longer defined in this test suite. Xcode calls
        //                `-[XCTestCase initWithSelector:]`, which calls
        //                `+[NSInvocation _invocationWithMethodSignature:frame:]`
        //                passing a nil `NSMethodSignature`. The test suite
        //                crashes with: "*** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '+[NSInvocation _invocationWithMethodSignature:frame:]: method signature argument cannot be nil'".
        //
        //             Again, a solution would be to provide an API for an
        //             XCTestCase to list the test methods that should be
        //             displayed in the Xcode Test Navigator, to prevent
        //             undefined tests from being run.
        @{TestNameKey: @"blop", TestParameterKey: @"blop"}, // Comment this line out in step 2 of "Problem #5" above.
        @{TestNameKey: @"blip", TestParameterKey: @"blip"},
        @{TestNameKey: @"blam", TestParameterKey: @"blam"},
    ];
}

+ (ParameterTestBlock)parameterizedTest {
    return ^(XCTestCase *self, NSString *parameter){
        XCTAssertEqual([parameter length], 4);
    };
}

@end
```

Steps to Reproduce:
1. Read the comments in the source file above and follow the instructions.
2. Confirm the five problems listed in this radar (and in the comments of the source file).

Expected Results:
 1. `XCTestCase` subclasses that do not define any test methods are not displayed in test output, and are also not displayed in the Test Navigator.
 2. The Xcode Test Navigator does not display test invocations that are no longer defined by the test suite.
 3. The Xcode Test Navigator is capable of displaying test invocations returned by `+[XCTestCase testInvocations]`, without the need to run the entire test suite first.
 4. The Xcode Test Navigator displays all test invocations returned by `+[XCTestCase testInvocations]`.
 5. Xcode does not crash the XCTest test suite when attempting to re-run a test invocation that no longer exists.

Actual Results:
 1. `XCTestCase` subclasses that do not define any test methods are not displayed in test output, but *are* displayed in the Test Navigator. As a result, the Test Navigator provides misleading information when users define `XCTestCase` subclasses that are meant to provide shared test logic.
 2. The Xcode Test Navigator displays test invocations that were run before, even if those invocations would no longer be returned by `+[XCTestCase testInvocations]`.
 3. Xcode Test Navigator only displays test invocations returned by `+[XCTestCase testInvocations]` only once they've all been run.
 4. Xcode Test Navigator does not display all test invocations returned by `+[XCTestCase testInvocations]`.
 5. Xcode crashes the XCTest test suite when attempting to re-run a test invocation that no longer exists.

Regression:
This occurs in every version of Xcode that has shipped with a Test Navigator (which, to my knowledge, is every version after Xcode 5.)

Notes:
Provide additional information, such as references to related problems, workarounds and relevant attachments.

Comments


Please note: Reports posted here will not necessarily be seen by Apple. All problems should be submitted at bugreport.apple.com before they are posted here. Please only post information for Radars that you have filed yourself, and please do not include Apple confidential information in your posts. Thank you!