Parallel testing: is not much faster than running on single simualtor

Originator:vytis.s.lt
Number:rdar://42334523 Date Originated:18-Jul-2018 05:00 PM
Status:Open Resolved:
Product:Developer Tools Product Version:Xcode 10.0 beta 4 (10L213o)
Classification:Security Reproducible:Always
 
Summary:
When running a test suite containing many test classes the time taken to complete testing with multiple simulators is not significantly less than running on a single device.

Steps to Reproduce:
1. Download the project’s code from this revision: https://github.com/wireapp/wire-ios-data-model/archive/c511a7a41289fd8a862d6051fda5a2d6f28cf579.zip
2. Build its dependencies with `carthage carthage bootstrap --platform iOS`
3. Build with command `xcodebuild -scheme WireDataModel -enableCodeCoverage YES -destination 'platform=iOS Simulator,name=iPhone 6,OS=latest' -derivedDataPath DerivedData build-for-testing`
4. Run tests in parallel with command `time xcodebuild -scheme WireDataModel -enableCodeCoverage YES -destination 'platform=iOS Simulator,name=iPhone 6,OS=latest' -derivedDataPath DerivedData -parallel-testing-enabled YES -parallel-testing-worker-count 4  test-without-building`
5. Run tests on single simulator with command `time xcodebuild -scheme WireDataModel -enableCodeCoverage YES -destination 'platform=iOS Simulator,name=iPhone 6,OS=latest' -derivedDataPath DerivedData -parallel-testing-enabled NO test-without-building`

Expected Results:
6. Tests running in parallel finish much quicker than with parallel testing turned off

Actual Results:
6. Tests with 4 simulators finish only 20% quicker than running on a single simulator.

Version:
Xcode 10.0 beta 4 (10L213o)

Notes:
It seems that even for moderately large test suites (134 test classes with 1789 tests) the improvement is not significant (tested on i7 2016 MacBookPro):
4 worker: 191 s (-20%)
3 worker: 180 s (-25%)
2 workers: 179 s (-25%)
1 worker: 231 s (-3%)
Parallel tests disabled: 238 s

Attaching .xcresult of all combinations

Comments


Please note: Reports posted here will not necessarily be seen by Apple. All problems should be submitted at bugreport.apple.com before they are posted here. Please only post information for Radars that you have filed yourself, and please do not include Apple confidential information in your posts. Thank you!