Xcode UI Testing: Test suites should not have to relaunch the application for each individual test

Originator:designatednerd
Number:rdar://23115791 Date Originated:14-Oct-2015 04:48 PM
Status:Addressed Resolved:Closed
Product:Developer Tools Product Version:Xcode 7.0.1 (7A1001)
Classification:UI/Usability Reproducible:Always
 
Summary:
XCUI testing presently requires relaunching the application for every single test in a suite rather than once per suite. This is a giant pain for applications with larger UI hierarchies. 

If I need to test a screen that is displayed to the user after I’ve registered and added a credit card but before I get into my main application, I have to go through getting to the registration page, filling out the registration form, filling out the credit card form just to get to the screen I need to test for every. single. test. 

This exponentially increases how long each test takes to run, and makes it impossible to test multiple things which share similar state without just resorting to convenience methods and manual cleanup rather than individual tests. 


Steps to Reproduce:
1. Write XCUI tests 
2. Notice that the app has to relaunch for every test
3. Realize that XCUIApplication().launch() is in the instance setUp() and not the class setup
4. Move XCUIApplication().launch() to the static class setUp()
5. Try to run test suite (ie, all the tests in one class). 
6. https://www.youtube.com/watch?v=pR0iRf_2j7k
7. Have sad.

Expected Results:
If I put XCUIApplication().launch() in the static class setUp(), it should only launch the application once at the beginning of the test suite, allowing navigation to one part of the app for the test suite, and not requiring re-navigating to that portion of the app again for other tests in the same suite. 

Actual Results:
App must relaunch and re-navigate for every. single. test. 

Regression:
UI testing was just introduced, so it’s not really a regression unless you’re comparing to *gasp* third party tools. 

Notes:
This is a major dealbreaker for me in actually using XCUI testing because of how badly it inflates the time it takes to run UI tests.

Comments

This has been fixed in Xcode 9 with the activate method on XCUIApplication, which you should use instead launch if you don't want to relaunch every time.

By designatednerd at June 8, 2017, 11:44 p.m. (reply...)

And I responded to their response by sending a modified version of https://github.com/designatednerd/GoCubs and this:

Thank you for the fast response. The video is intended to show that the application never launches.

I have attached the sample application I made for a talk on this subject. At the top of the GoCubsUITests.swift file, there is a commented out static setUp() method. If you run it as is, you'll see everything works fine with the relaunch prior to each test. If you comment the launch line in the instance setUp() and uncomment the static setUp(), you will see that the application does not launch at all. I have included in the zip the log output when I do this as well.

I agree that in an ideal world starting from scratch is a best practice and I definitely start each suite from scratch, but when I've tried to start every test from scratch in the past, my UI test targets for even moderately complex apps have grown to take over half an hour to run very quickly. I've found a real point of diminishing returns because it makes people less likely to actually run the damn tests if they take forever to run, which defeats the point of having the tests in the first place.

'radar_23115791_Go_Cubs.zip' was successfully uploaded.

By designatednerd at Oct. 15, 2015, 11:48 p.m. (reply...)

Apple responded today:

We need more information to investigate this issue.

Putting the launch() in the class setUp() should do what you want - launch it once for all the tests in that class. It’s not clear from the video what the problem is. Does the setUp never get called or does it fail to launch? If you could reduce this to a small project and provide more information about what’s happening, we’d be glad to investigate.

As far as launching once for each test, that is not intended to be a requirement, but it is definitely a “best practice”, despite the additional time it requires. The reason for this is the complex level of state that applications maintain; if you have a large set of tests that each build on top of the previous, then a failure at one point in the chain will almost always result in every subsequent test failing, even if the thing they are testing is actually working correctly.

It is up to you - or should be, unless we have a bug in setUp()/launch() - but our own experience has shown that the time tradeoffs are worth the finer grained test results.

Please provide your response or results by updating your bug report.

By designatednerd at Oct. 15, 2015, 11:47 p.m. (reply...)

Please note: Reports posted here will not necessarily be seen by Apple. All problems should be submitted at bugreport.apple.com before they are posted here. Please only post information for Radars that you have filed yourself, and please do not include Apple confidential information in your posts. Thank you!