Thirty Seconds to Pass: Performance Comparison of iOS UI Testing Frameworks

Mobile development has changed tremendously in recent years with iOS and Android receiving new hardware features and application updates more than ever. This means that QA engineers are constantly being faced with new challenges to improve automated testing development. 

At Devexperts, we design and manage test cases for financial applications – both manually and using automated testing frameworks. The testing framework choice is critical to testing execution performance.

UI tests are the slowest among automated ones. If we pick the fastest framework we will greatly decrease the testing pyramid execution time. It is very important to speed up our CI/CD as we will be able to run more tests, more often. You have probably already heard something like, “framework A is faster than framework B.” The problem is that “faster” is not precisely defined – is it 10% faster? Or twice as fast?

In this article, we will compare different iOS UI testing frameworks’ performance. Let us begin with an introduction to the “usual suspects.”

• XCTest ((Apple Developers Tools Team. “XCTest | Apple Developer Documentation”, Apple Developer website, Apple Inc., 2019, developer.apple.com/documentation/xctest)) is a framework developed by Apple for testing Xcode projects. XCTest runs the unit, performance, and UI tests written in Swift or Objective-C. Its API for UI testing iOS apps was first released at WWDC 2015¹. This Apple software product is capable of performing black-box UI Testing based on a VoiceOver accessibility feature.
• Appium² is the most popular framework which supports black-box UI tests’ automation for mobile apps. Appium is an open-source framework and supports many programming languages and testing tools. It is possible to automate UI testing for Android and iOS apps of all kinds – native, web, and hybrid. Appium uses XCTest’s UI testing feature to automate iOS test cases.
• Katalon Studio³ is a rising star in testing automation. It’s an all-in-one solution providing tools for mobile, Web, and API automation. Test cases may be recorded or created in just a few clicks. It is also possible to write test code in Groovy. Appium is required for black-box mobile automation tests in Katalon Studio.
• EarlGrey⁴ is an iOS automation framework and sometimes is called the “twin sibling of Espresso”. EarlGrey and Espresso were developed by Google. EarlGrey integrates into the app code itself to speed up testing execution. Quality assurance engineers can write tests both in Swift and Objective-C directly in Xcode and execute test runs from there. The first release of EarlGrey is white-box while the second release is grey-box. The second version of EarlGrey is still in beta.

About Performance Evaluation

Test cases describe sequences of steps required to verify the application features. Apart from launching and terminating the application, test cases specify which actions need to be done with the screen’s elements. If we count these actions in real-world test cases we will get a distribution very similar to the ones shown on the pie chart below.

To evaluate the performance of a testing framework, we should test how fast it is in tapping, swiping, and determining the element’s existence.

The elements are usually embedded in views like Collections, Tables, and Scroll Views⁵. We will measure how much time it takes to find an element in these views and assert its uniqueness among 100 elements in view.

To make our evaluation fair we will not use the real-world application because of many hard-to-control conditions like network speed and the app’s responsiveness. We have created a dedicated application that models common behavior.

• The first tab of the app shows the Collection View with 100 cells. Each cell has a unique accessibility identifier (identifiers are used to distinguish different elements of UI hierarchy in testing) and a text label. We will measure the time of finding View #0 multiple times and compare average values.
• The second tab of the app shows three Scroll Views with 100 emoji each. Emoji are grouped by category – facial expressions, food, and animals. Each emoji has a unique accessibility identifier. We will evaluate frameworks by measuring the time it takes to find the first emoji in the first Scroll View.
• The third tab of the app is similar to the first one but shows the Table View. Each one of 100 cells in the table has a unique accessibility identifier and a text label. As with Collection View, we will measure the time it takes to find the first cell.

It is important to mention that offscreen elements of Collection View do not appear in VoiceOver elements hierarchy which means the search for an element in the first tab should be much faster than in Scroll View and Table View. This should be noticeable in XCTest, Appium, and Katalon Studio results. Also, the Table View cells contain accessory elements like horizontal lines and buttons on the right, and we expect that the element’s search will be slower.

• The fourth tab only displays a tappable button and a counter of taps. The counter of taps is required to determine if any tap was not properly sent by the testing framework. A taps counter is required to get the average time required to perform a tap.
• Finally, the fifth tab tracks swipes. The text on the screen shows the count of swipes.

In addition to these actions, we will measure the average time each framework needs to launch and terminate the application.

The evaluation is performed using the latest versions of each framework (as of September 2019). The test host is MacBook Pro running macOS 10.14.6 Mojave and the test target is iPhone Xʀ Simulator running iOS 12.2.

Results

The results of finding a single element in a different container view are shown in the following chart. Notice that the value axis (X-axis) scale is logarithmic.

The numbers of each evaluation will surely be different if we vary the hardware but the overall trend will remain unchanged. Without any doubt, EarlGrey delivers the fastest performance in these particular tests. XCTest is an order of magnitude slower than EarlGrey. Appium is another order slower than XCTest. And finally, Katalon Studio is another order slower than Appium.

Imagine you need to assert the existence of 10 elements in large Table View. It will take about one hundred milliseconds using EarlGrey 1.0 and about 2 minutes using Katalon Studio – more than a thousand times slower!

Let’s compare the remaining results shown in the following chart. Notice that the value axis scale is still logarithmic.

Katalon Studio is much slower in launching and terminating the application because it automatically resets all the data on the iOS Simulator. While this action is good for stability, it is bad for performance. This automatic reset could be turned off, but not in the latest release of Katalon Studio because of a bug⁶. EarlGrey 1.0 is the fastest in launching and terminating – because of its deep integration into the application itself. Appium is the fastest in tapping. Katalon Studio is a little bit worse in tapping than other frameworks and much worse in swiping.

Conclusion

While the Katalon Studio is the most feature-rich framework among the evaluated ones, it is not as powerful as iOS-only frameworks, especially in finding the elements.

Appium is much better at performance and it gets close to XCTest, which is the best performing framework among black-box testing frameworks.

The second release of EarlGrey is another big step in performance, yet the first release is the best at performance among all compared frameworks.

Which frameworks do you use to automate iOS UI testing? Do you consider switching from one to another and why?

1. Turner, Wil, and Brooke Callahan. “UI Testing in Xcode – WWDC 2015.” AppleDeveloper website, Apple Inc., 2015, developer.apple.com/videos/play/wwdc2015/406 [↩]
2. JS Foundation. “Appium: Mobile App Automation Made Awesome”, Appium website, The JS Foundation, 2019, appium.io [↩]
3. Katalon, Inc. “Katalon Studio | The #1 Codeless Automation Tool”, Katalon Studio website, Katalon, Inc., 2018, katalon.com/katalon-studio [↩]
4. Google Inc. “EarlGrey Reference”, EarlGrey website, Google Inc., 2017, google.github.io/EarlGrey [↩]
5. Apple Developers Tools Team. “Views and Controls | Apple Developer Documentation”, Apple Developer website, Apple Inc., 2019, developer.apple.com/documentation/uikit/views_and_controls [↩]
6. liaomeng. “The customized desired capability doesn’t work – Katalon Studio / Mobile Testing – Katalon Community”, Katalon Community website, Katalon, Inc., 2018, forum.katalon.com/t/the-customized-desired-capability-doesnt-work/10754 [↩]