XCTest notification expectations incorrectly fulfilled more than once

Originator:defagos
Number:rdar://36902658 Date Originated:January 26 2018
Status:Open Resolved:
Product:Xcode Product Version:8.3
Classification:Bug Reproducible:Always
 
Summary:

XCTest expectations must be fulfilled at most once. Since Xcode 8.3, expectation notifications can be fulfilled more than once, though. This was not the case until Xcode 8.2.1, and can lead to subtle issues.

Steps to Reproduce:

I attached a test case implementation file. You can simply add it to any test target to run the single test it contains.

Here is how the test works:

1. A built-in XCTest notification expectation is registered at the beginning for "clock" notifications. This expectation is fulfilled (`YES` is returned) once a clock notification has been received.
1. A similar custom-made notification is also registered in parallel. This custom implementation uses standard XCTest expectations in its implementation.
1. Another expectation, fulfilled after 5 seconds, is also registered. This way we are able to watch what happens during 5 seconds.
1. With all these expectations set, a timer is created, emitting one "clock" notification each second. We then wait until all expectations are fulfilled.

For each notification expectation version (built-in and custom), we check how many times the expectation handler is called. After all expectations have been fulfilled, we check how many times each handler was called.

Expected Results:

Since the example notification expectation handlers return `YES` and are therefore fulfilled the first time they are executed, we expect that each handler has been called once once the test finishes.

Actual Results:

On Xcode 8.3 and above, the built-in notification expectation handler is called 5 times. On Xcode 8.2.1, the handler is called only once.

The problem also occurs with the newest Xcode 9.3 beta 1.

Version/Build:

Xcode 8.3.

Configuration:

-

Code sample:

#import <XCTest/XCTest.h>

static NSString * const ClockNotification = @"ClockNotification";

@interface NotificationExpectationBugTestCase : XCTestCase

@end

@implementation NotificationExpectationBugTestCase

// An expectation which is fulfilled after some time interval has elapsed.
- (XCTestExpectation *)expectationForElapsedTimeInterval:(NSTimeInterval)timeInterval withHandler:(void (^)(void))handler
{
    XCTestExpectation *expectation = [self expectationWithDescription:[NSString stringWithFormat:@"Wait for %@ seconds", @(timeInterval)]];
    dispatch_after(dispatch_time(DISPATCH_TIME_NOW, (int64_t)(timeInterval * NSEC_PER_SEC)), dispatch_get_main_queue(), ^{
        [expectation fulfill];
        handler ? handler() : nil;
    });
    return expectation;
}

// Custom expectation for notification. Based on standard exceptations.
- (XCTestExpectation *)customExpectationForNotification:(NSNotificationName)notificationName object:(id)objectToObserve handler:(XCNotificationExpectationHandler)handler
{
    NSString *description = [NSString stringWithFormat:@"Expectation for notification '%@' from object %@", notificationName, objectToObserve];
    XCTestExpectation *expectation = [self expectationWithDescription:description];
    __block id observer = [[NSNotificationCenter defaultCenter] addObserverForName:notificationName object:objectToObserve queue:nil usingBlock:^(NSNotification * _Nonnull notification) {
        void (^fulfill)(void) = ^{
            [expectation fulfill];
            [[NSNotificationCenter defaultCenter] removeObserver:observer];
        };
        
        if (handler) {
            if (handler(notification)) {
                fulfill();
            }
        }
        else {
            fulfill();
        }
    }];
    return expectation;
}

// Test that exception notifications are fulfilled at most once. This proves that the built-in notification expectation can
// be incorrectly fulfilled several times.
- (void)testIncorrectDoubleExpectation
{
    __block NSInteger builtInTestNotificationEpectationFulfilledCount = 0;
    [self expectationForNotification:ClockNotification object:nil handler:^BOOL(NSNotification * _Nonnull notification) {
        ++builtInTestNotificationEpectationFulfilledCount;
        return YES;
    }];
    
    __block NSInteger customInTestNotificationEpectationFulfilledCount = 0;
    [self customExpectationForNotification:ClockNotification object:nil handler:^BOOL(NSNotification * _Nonnull notification) {
        ++customInTestNotificationEpectationFulfilledCount;
        return YES;
    }];
    
    [self expectationForElapsedTimeInterval:5 withHandler:nil];
    
    [NSTimer scheduledTimerWithTimeInterval:1. repeats:YES block:^(NSTimer * _Nonnull timer) {
        [[NSNotificationCenter defaultCenter] postNotificationName:ClockNotification object:nil];
    }];
    
    [self waitForExpectationsWithTimeout:10. handler:nil];
    
    XCTAssertEqual(builtInTestNotificationEpectationFulfilledCount, 1);         // Fails, 5 != 1
    XCTAssertEqual(customInTestNotificationEpectationFulfilledCount, 1);        // Succeeds, 1 == 1
}

@end

Comments

Follow up

This bug is related to rdar://36902658.


Please note: Reports posted here will not necessarily be seen by Apple. All problems should be submitted at bugreport.apple.com before they are posted here. Please only post information for Radars that you have filed yourself, and please do not include Apple confidential information in your posts. Thank you!