This week I heard some chatter in certain iOS groups that I belong to about whether testing your iOS, macOS, watchOS, or tvOS applications provided any immediate value in the development life cycle. As opposed to developing features without testing and getting them to market right away and then iterating upon them in a highly agile methodology. Coming from a background where I like to write a lot of automated tests and system tests for software that I write, I have to admit, this point of view really made me pause for a moment to try and see the perspective from both sides. I tried to see this point of view from my consulting background where I have been in many situations where there has been a very very tight deadline and a need to get a functional product to market as fast as possible. However, even from my consulting point of view, I can only understand half of this argument, and cannot understand the other half in that agile iteration will out-weigh the need for testing an application before it is released to market. In the end I look at testing iOS software from two different perspectives, one is a consulting point of view and the other is a product point of view. Each situation can have a unique set of constraints surrounding the development and testing methodology that it used and testing can be approached a bit differently in each situation. In the following post I will briefly explain the testing methodologies I have used in the past in the context of a consulting or a product based development. Then, to provide a bit of context to the discussion, I will show a couple of brief examples using XCTest and how it can benefit you in your iOS project, no matter what environment you are currently working in.
Testing in a Consulting Environment
In a consulting environment where deadlines are tough and functionality has to be developed quickly there is still room for different styles of testing. One of those styles can be small and fast test driven development (TDD) cases. In this style small bits of functionality can be developed and then small test cases can be written right away to assert the validity of this functionality. In a high pressure deadline driven environment these TDD cases could be written for just for the critical pieces of functionality and things like selector callbacks could wait for future iterations. Another style of testing is system testing where you would work with a QA engineer or possibly a third party outside of the project to hand test critical functionality in the project and assert whether a certain piece of functionality passes, fails, or creates an unknown anomaly. In my experience with high pressure deadline environments, system tests can be more effective at asserting the validity of critical functionality rather than unit tests because system tests will usually uncover most user interface and user experience functionality that is not uncovered by unit testing.
Testing in a Product Environment
In a product development environment testing is absolutely paramount. What is the difference between developing in a product environment and a consulting environment? A product environment is usually what I consider to be an application that is being built to serve as a platform or is expected to grow and expand over a long period of time. Where as a consulting environment often a application is built for small segmented purpose and is only expected to be used for a short window of time. When I am leading the development of a product environment I will use a combination of the two styles mentioned above along with the additional UIAutomation tests to strengthen the test converge. This will allow for the product to be validated at each major iteration and system testing will often make sure that no regressions have been introduced into the product. What are regressions? Regressions are bugs that get introduced into your code base that use to work in your product. Now because of code that you added in the development of new features bugs may have been introduced and old functionality that used to work no longer does and this is called a regression. How system testing combats this is by starting with a checklist of critical features and adding to this checklist as the product grows. Each time a system test are run on the product the old tests are run on the product to validate that tests that used to pass, still pass and have not been regressed.
XCTests
XCTest is the deafult testing framework that comes out tof the box with Xcode. XCTest can be used to write small unit tests, tests for user interface elements, and it can even be used to execute performance testing on different parts of your application. To illustrate using XCTest in the context of an iOS application I created a small sample application that uses Core Data and a custom UITableViewCell to add a new table row every time a UIBarButton is tapped. You might recognize this as an out of the box template that comes standard with Xcode. Another thing you may recognize is the opportunity that this template gives us to write a few tests to validate some of the functionality that is taking place in this sample application.
In the code below I provided an overview of the MasterViewController in my sample application. The first thing to notice is that there is a UIBarButton being added to the navigation bar in the ViewDidLoad function. This button has a selector that calls back to a function called insertNewObject. The insertNewObject function is responsible for creating a new CoreData Event object, setting fields on that object, saving it to the NSMangedObjectContext, and then once the Event object is saved, to trigger the reload of the UITableView. The code below is not the entire MasterViewController but just the necessary pieces to illustrate the functionality taking place.
import UIKit import CoreData class MasterViewController: UITableViewController, NSFetchedResultsControllerDelegate { var detailViewController: DetailViewController? = nil var managedObjectContext: NSManagedObjectContext? = nil override func viewDidLoad() { ... // Adds a button to the navigation bar and calls back to the insertNewObject when tapped let addButton = UIBarButtonItem(barButtonSystemItem: .add, target: self, action: #selector(insertNewObject(_:))) self.navigationItem.rightBarButtonItem = addButton ... } ... // Inserts a new event into CoreData and triggers an event to reload the UITableView func insertNewObject(_ sender: Any) { let context = self.fetchedResultsController.managedObjectContext let newEvent = Event(context: context) // If appropriate, configure the new managed object. newEvent.timestamp = NSDate() // Getting an random string as the title. // This is an opportunity for a test! newEvent.title = self.getRandomString(stringLength:10) // Save the context. do { // Saving the managedObjectContext triggers the following functions: // func controllerWillChangeContent(_ controller: NSFetchedResultsController<NSFetchRequestResult>) // func controller(_ controller: NSFetchedResultsController<NSFetchRequestResult>, didChange anObject: Any, at indexPath: IndexPath?, for type: NSFetchedResultsChangeType, newIndexPath: IndexPath?) try context.save() } catch { let nserror = error as NSError fatalError("Unresolved error \(nserror), \(nserror.userInfo)") } } // Called by insertNewObject to generate a random string based upon a set length func getRandomString(stringLength:Int) -> String { let letters:String = "abcdefghijklmnopqrstuvwxyz1234567890" // The character array let characterArray = [Character](letters.characters) // Random string about to be generated var randomString: String = "" // For loop to generate a string based upon the passed in size // The string is generated at random based off the letters string for _ in (0..<stringLength) { let rand = Int(arc4random_uniform(UInt32(characterArray.count))) randomString += String(characterArray[rand]) } // Return the new string return randomString } ... // Triggered when the NSManagedObjectContext is updated in any way func controllerWillChangeContent(_ controller: NSFetchedResultsController<NSFetchRequestResult>) { self.tableView.beginUpdates() } // Triggered when the NSManagedObjectContext is updated in any way func controller(_ controller: NSFetchedResultsController<NSFetchRequestResult>, didChange anObject: Any, at indexPath: IndexPath?, for type: NSFetchedResultsChangeType, newIndexPath: IndexPath?) { switch type { case .insert: tableView.insertRows(at: [newIndexPath!], with: .fade) case .delete: tableView.deleteRows(at: [indexPath!], with: .fade) case .update: self.configureCell(tableView.cellForRow(at: indexPath!)! as! CustomTableCell, withEvent: anObject as! Event) case .move: tableView.moveRow(at: indexPath!, to: newIndexPath!) } } ... // Triggered as a result of the update to the NSManagedObject context and creates a CustomTableCell, sets data to this cell and returns it override func tableView(_ tableView: UITableView, cellForRowAt indexPath: IndexPath) -> UITableViewCell { let customCell: CustomTableCell = tableView.dequeueReusableCell(withIdentifier: "Cell") as! CustomTableCell customCell.layoutMargins = UIEdgeInsets.zero customCell.preservesSuperviewLayoutMargins = false let event = self.fetchedResultsController.object(at: indexPath) self.configureCell(customCell, withEvent: event) return customCell } // Call from the above function to set the properties of the CustomTableCell func configureCell(_ cell: CustomTableCell, withEvent event: Event) { cell.titleLabel!.text = event.title cell.dateLabel!.text = event.timestamp!.description cell.rowImage!.image = UIImage(named: "swift-bird")! } ... } // // CustomTableCell.swift // Custom UITableViewCell called CustomTableCell // import UIKit class CustomTableCell: UITableViewCell { @IBOutlet weak var rowImage: UIImageView? @IBOutlet weak var titleLabel: UILabel? @IBOutlet weak var dateLabel: UILabel? override func awakeFromNib() { super.awakeFromNib() } override func setSelected(_ selected: Bool, animated: Bool) { super.setSelected(selected, animated: animated) } }
Getting Started Using XCTest
Getting started with XCTests is very easy and most of the time included in your project from the start if you leave the testing check boxes ticked when you create a new Xcode project. A complete overview of XCTest is outside the scope of this post but just know that as long as you have a XCTestCase target added to your project, you can added tests and remove tests as you see fit.
So, how do we go about using XCTest to test our applications functionality in the context of a consulting environment and a product environment? Well, as I mentioned above, in a fast paced, deadline driven environment it is important to pick out the critical functionality to test first. Then, as time allows, or in a future iteration, we can go back and layer on a more tests as we see fit.
So in a consulting environment one of the first critical pieces of functionality that needs to be tested is the creation of a new Event object. If we cannot create a new Event object and save it to our NSMangedObjectContext then we have essentially all of the functionality that takes place in our sample application fails. Below you will see that I created two test classes. One called CoreDataTestClass that creates a NSPersistentContainer and then an actual test class called TestEventCreation that inherits from the CoreDataTestClass and asserts whether creating a new Event object is successful or not.
// // CoreDataTestClass.swift // XCTestExample // // Created by Matt Eaton on 10/8/16. // Copyright © 2016 AgnosticDev. All rights reserved. // import XCTest import CoreData class CoreDataTestClass: XCTestCase { // Create a local NSManagedObjectContext var managedObjectContext: NSManagedObjectContext? = nil // In iOS 10 and Swift 3 this is all I need to create my core data stack lazy var persistentContainer: NSPersistentContainer = { let container = NSPersistentContainer(name: "XCTestExample") container.loadPersistentStores(completionHandler: { (storeDescription, error) in if let error = error as NSError? { fatalError("Unresolved error \(error), \(error.userInfo)") } }) return container }() override func tearDown() { super.tearDown() managedObjectContext = nil } override func setUp() { super.setUp() // Set the NSManagedObjectContext with the view Context managedObjectContext = self.persistentContainer.viewContext } func testExample() { // This is an example of a functional test case. // Use XCTAssert and related functions to verify your tests produce the correct results. } func testPerformanceExample() { // This is an example of a performance test case. self.measure { // Put the code you want to measure the time of here. } } } // // TestEventCreation.swift // XCTestExample // // Created by Matt Eaton on 10/8/16. // Copyright © 2016 AgnosticDev. All rights reserved. // import XCTest import CoreData @testable import XCTestExample class TestEventCreation: CoreDataTestClass { var newEvent:Event? override func setUp() { super.setUp() let newEventEntity = NSEntityDescription.entity(forEntityName: "Event", in: managedObjectContext!) newEvent = Event(entity: newEventEntity!, insertInto: managedObjectContext) } override func tearDown() { super.tearDown() } func testNewEvent() { XCTAssertNotNil(self.newEvent, "Cannot create a new event!") } override func testPerformanceExample() { // This is an example of a performance test case. self.measure { // Put the code you want to measure the time of here. } } }
After we have asserted that creating an Event object can successfully be done we then should turn our attention to the next critical piece of functionality and that is randomly generating a string for our Event title. In a fast paced consulting environment I would say that after testing the random string creation all of the critical pieces of functionality have been tested and we could then make a decision upon whether to move on with other development or to continue writing tests for this feature.
Below is a XCTestCase class that tests the creation of a random string based upon the passed in length. The goal of this test is to assert that the string returned to us is the same length as the passed in integer in the getRandomString function.
// // TestTitleGeneration.swift // XCTestExample // // Created by Matt Eaton on 10/8/16. // Copyright © 2016 AgnosticDev. All rights reserved. // import UIKit import XCTest @testable import XCTestExample class TestTitleGeneration: XCTestCase { let master = MasterViewController() override func setUp() { super.setUp() // Put setup code here. This method is called before the invocation of each test method in the class. } override func tearDown() { // Put teardown code here. This method is called after the invocation of each test method in the class. super.tearDown() } func testRandomTitle() { // Declare two constant lengths for random string generation length let firstTestLength = 10 let secondTestLength = 15 // Get the first string let firstString = master.getRandomString(stringLength: firstTestLength) XCTAssertEqual(firstString.characters.count, firstTestLength, "First string characters should equal \(firstTestLength)") // Get the second string let secondString = master.getRandomString(stringLength: secondTestLength) XCTAssertEqual(secondString.characters.count, secondTestLength, "Second string characters should equal \(secondTestLength)") } func testPerformanceExample() { // This is an example of a performance test case. self.measure { // Put the code you want to measure the time of here. } } }
Transitioning away now from a consulting environment and thinking now about a product environment there are a couple of extra things that could potentially be tested in regards to the sample application functionality. One that comes to mind right away is if we can correctly create a CustomTableCell object to be used in place of a UITableViewCell object. Another could potentially be the call back from the UIBarButton to the insertNewObject function. These are things that I consider to be an extra layer of protection when creating a product. Not only do they allow you to validate functionality but creating these tests also allows us to assert that something has not been regressed in the future when more code has been added to the project.
Below is just a simple example of a UITableView registering the CustomTableCell and then our test case asserts that it can successfully be created using the dequeueReusableCell method. This will be the last example that I provide but certainly as mentioned above there could be room for more testing in regards to asserting that the UIBarButton calls back to the insertNewObject function correctly.
// // TestCustomCell.swift // XCTestExample // // Created by Matt Eaton on 10/8/16. // Copyright © 2016 AgnosticDev. All rights reserved. // import UIKit import XCTest @testable import XCTestExample class TestCustomCell: XCTestCase { var testTable = UITableView() override func setUp() { super.setUp() // Put setup code here. This method is called before the invocation of each test method in the class. testTable.register(CustomTableCell.self, forCellReuseIdentifier: "Cell") } override func tearDown() { // Put teardown code here. This method is called after the invocation of each test method in the class. super.tearDown() } func testCustomCell() { let customCell: CustomTableCell = testTable.dequeueReusableCell(withIdentifier: "Cell") as! CustomTableCell XCTAssertNotNil(customCell, "No Custom Cell Available") } func testPerformanceExample() { // This is an example of a performance test case. self.measure { // Put the code you want to measure the time of here. } } }
In conclusion I think that testing code is a good practice no matter what environment you are in. No mater if you are in a consulting or a product based environment that is based on a mobile device, the web, or a desktop computer, all code should be tested and, in my opinion, no matter how fast your agile development process is it cannot make up for the benefits tests will give you in the long run. Testing is like an investment in your code. You put in the time up front and this investment will pay you back down the road.
Please let me know if there is any feedback, questions, or comments based upon this post. I always love hearing from readers. The sample Xcode project that I referenced in this post is up on my Github account here. Thanks!
Comments
how do you verify image exist in XCTest
Hello,
Thank you for the blog. In your test I did not see how do you verify that the image shows up correctly. Is there a way to test that the image is displayed (using maybe its identifier or label)?
Accessibility Label Validation
AJ, thank you for the feedback and I am glad you enjoyed the post!
In terms of validating an image that exists on a UIImageView, this is a good question. I added an accessibilityLabel to the rowImage property on the cell to text for the existence of this label in my project, but did not have any luck with this in Xcode 9, iOS 11. Xcode actually crashed, but I am still investigating my implementation.
Another way of validating the existence of an image to an UIImageView was to directly check the object's existence. Here is a sample of my testing code:
class TestUIImage: XCTestCase {
var image: UIImage?
var imageView: UIImageView?
...
func testImageExists() {
image = UIImage(named: "swift-bird.png")
imageView = UIImageView(image: image)
if let unwrappedImageView = imageView {
XCTAssert(unwrappedImageView.image != nil)
} else {
XCTAssert(false)
}
}
}
Testing third party application
Hi there! Congrats for your post, its very interesting an complete. I'd like to ask you if is it possible to automate the UI tests of a third party app, with no access to source code, only the .ipa. Its a scenario that occurs to me with some frequency, so it would be great using the same tool with or wothout the source code.
Best regards!
Thank you and great question!
Thank you for the response and for the great question. In terms of testing a general IPA without an Xcode project files this would probably be something best left to manual testing or possibly an automated testing framework like Selenium. There is an interesting product from Sauce Labs called Appium that does something like this making use of Selenium. Check it out: https://saucelabs.com/resources/automated-testing/selenium-for-ios