Open Radar

 
Summary:
In parallel runs, seems that:

xcodebuild builds once for all destinations, creating all the x86_64 binaries once.
When building, also the assets catalogs is compiled, and universal PDF files are converted to their 2x or 3x versions, depending on the target device.
What seems to be happening with parallel builds is that:

xcodebuild is building, using the first destination as the target.
As a result, the generated assets match the required ones for the first destination (2x or 3x). Thus, running the tests in the first destination succeeds.
When running the tests in the second destination, it may happen that the assets do not match the required ones: in case this second destination requires different assets than the first destination (ex: first destination requires 2x assets, and second destination requires 3x assets). Thus, the second destination uses assets that are not 100% adapted for it, which produce differences in the borders when showing the icons/images on screen (because its resolution is not the proper one), and the test fails.

(from: https://github.com/uber/ios-snapshot-test-case/issues/37#issuecomment-399801243)

Steps to Reproduce:
1. Write tests that depend on assets (PDFs without “Preserve Vector Data” or regular images)
2. Run `xcodebuild test -destination 'platform=iOS Simulator,name=iPhone SE,OS=11.4' -destination 'platform=iOS Simulator,name=iPhone 6,OS=11.4' -destination 'platform=iOS Simulator,name=iPhone X,OS=11.4' -destination 'platform=iOS Simulator,name=iPhone SE,OS=12.0' -destination 'platform=iOS Simulator,name=iPhone 6,OS=12.0' -destination 'platform=iOS Simulator,name=iPhone X,OS=12.0’`


Expected Results:
The tests would use the appropriate asset on each device. They should produce the same results as running the tests without parallelization or running one destination at a time (multiple commands)

Actual Results:
The tests fail because the wrong asset is sometimes used

Version:
Xcode 10.0

Notes:
There’re some workarounds that were provided by Brian Croom (@aikoniv) on Twitter:

1) Run xcodebuild with the -disable-concurrent-destination-testing flag to cause the destinations to be run in serial. In that configuration, a new incremental build is performed before testing on each destination, which should produce the correct assets for the current device.

2) First run xcodebuild build-for-testing with "-destination generic/platform=iOS" to produce a build targeted at any device, followed by xcodebuild test-without-building with the actual destinations you want to run tests on.

(We actually had to use "-destination generic/platform=iOS Simulator” though)