Assertions

Learn how to use Tuneup’s assertions

While the UIAutomation API provides a good interface to the UI, it doesn’t provide a very succinct or elegant way to assert the state of your UI. That’s where tuneup really shines.

The Basics

At its core, tuneup’s assertion mechanism is built on throwing and catching JavaScript exceptions. The test function catches any exceptions thrown, logs the message of the exception and marks the test as a failure. All of the assertions can be found in the assertions.js file if you’re interested in the nitty-gritty of it all.

The two core functions are fail and assertTrue. The first takes a string as its sole argument. The second one accepts any JavaScript expression that evaluates to something "truthy" and a string message. All of the other assertions are built on these two functions.

fail(message)

Immediately throws a FailureException with the message given as the sole argument to the function

assertTrue(expression, message)

Asserts that the given expression is true and throws an AssertionException with a default message, or the given one (if provided).

assertFalse(expression, message)

Asserts that the given expression is not true. If the expression is true (according to assertTrue), an AssertionException is thrown with a default message, or a custom one (if provided).

assertEquals(expected, received, message)

Asserts that the received object matches the given object using the == operator. If they are not equal, an AssertionException is thrown with a default message, or a custom one (if provided).

assertNotEquals(expected, received, message)

Asserts that the expected object does not match the received object using the != operator. If it is equal, an AssertionException is thrown with a default message, or custom one (if provided).

assertMatch(regExp, expression, message)

Asserts that the given regular expression successfully applies against the result of the given expression. If it fails, an AssertionException is thrown with a default message, or a custom one (if provided).

assertNull(expression, message)

Asserts that the given expression is either null or is a UIAElementNil, which is commonly used by UIAutomation as a placeholder for conceptually null objects. If this is not the case, an AssertionException is thrown with a default message or custom one (if provided).

assertNotNull(expression, message)

Asserts the opposite of assertNull.

Screen Assertions

A common theme in writing integration tests for "screen flows" is the repetitive cycle of making several assertions on a screen, then engaging some user-control after all of the assertions pass. With the UIAutomation API as it is, it's easy to lose sight of this structure when bogged down in the syntax of asserting and navigating the user interface.

To make this cycle more obvious, and cut down on unnecessary verbosity, use the assertWindow function. It works by applying a given JavaScript object literal to the current main window (UIAWindow instance). Properties are nested in the object literal in the same hierarchy as you would access them via UIAutomation’s API.

test("my test", function(target, app) {
  mainWindow = app.mainWindow();
  navBar = mainWindow.navigationBar();
  leftButton = navBar.leftButton();
  rightButton = navBar.rightButton();

  assertEquals("Back", leftButton.name());
  assertEquals("Done", rightButton.name());

  tableViews = mainWindow.tableViews();
  assertEquals(1, tableViews.length);
  table = tableViews[0];

  assertEquals("First Name", table.groups()[0].staticTexts()[0].name());
  assertEquals("Last Name", table.groups()[1].staticTexts()[0].name());

  assertEquals("Fred", table.cells()[0].name());
  assertEquals("Flintstone", table.cells()[1].name());
});

With assertWindow, you can boil it down to this:

test("my test", function(target, app) {
  assertWindow({
    navigationBar: {
      leftButton: { name: "Back" },
      rightButton: { name: "Done" }
    },
    tableViews: [
      {
        groups: [
          { name: "First Name" },
          { name: "Last Name" }
        ],
        cells: [
          { name: "Fred" },
          { name: "Flintstone" }
        ]
      }
    ]
  });
});

Note how the nested properties in the object literal match object-access path of objects in the UIAutomation API.

For each property you wish to make an assertion about, you can specify a string, number, regular expression or function. String and number values are asserted using the assertEquals function. Regular expressions are asserted using the assertMatch function. If a function is supplied, it is simply evaluated. You can use any of the other assertion functions to match inside of your custom function.

For UI properties that result in arrays (e.g. tableViews()), you need to provide an array object with a matcher for each element in the array. These can be a mix of strings, numbers, regular expressions or functions.

You can also pass null to match a property. This signals to tuneup that you don’t care to match the property, but may be required to do so given the structure for the UI. A common example of this is when you have an array of UI elements and you only care to make an assertion about a few of them.

If the object literal you pass to assertWindow provides an onPass property (pointing to a function), it will be invoked if all of the matching assertions have passed.

assertWindow({
   navigationBar: {
     leftButton: { name: "Back" }
   },
   onPass: function(window) {
     var leftButton = window.navigationBar().leftButton();
     leftButton.tap();
   }
 });

Window Assertions for Univeral Applications

If you have a Universal Application and want to maintain a single set of test files, you can mark specific properties to match by adding a "~ipad" or "~iphone" extension to the property name. When you do this, you need to quote the property name instead using a literal, like so:

test("my test", function(target, app) {
  assertWindow({
    "navigationBar~iphone": {
      leftButton: { name: "Back" },
      rightButton: { name: "Done" }
    },
    "navigationBar~ipad": {
      leftButton: null,
      rightButton: { name: "Cancel" }
    },
  });
});

Note that the "~iphone" extension should work for iPod Touch devices also.

This convention is derived for how device-specific images are loaded on both iPad and iPhone/iPod devices. Hopefully it looks somewhat familiar.

Screenshots assertion

Tuneup can compare captured screen images against provided reference images and generate diff images for them. This function relies on compare utility from ImageMagick. Steps to activate this feature:

  1. brew install imagemagick.
  2. In your test script create an ImageAsserter:

    /**
     * tuneup_folder     - folder with tuneup sources
     * output_folder     - folder with test results
     * ref_images_folder - folder with you reference images
     **/
    createImageAsserter('tuneup_folder', 'output_folder', 'ref_images_folder');
  3. Assert current screen against reference image with the assertScreenMatchesImageNamed() function. The first argument is the name of the image on-disk found in the ref_images_folder you specified in the previous step. The second argument is an optional custom message if the screen doesn’t match the given image.

    One thing that can be confusing is specifying the path to the tuneup_js folder. The path you specify here is going to be different than the one you use in your #import statements. Instead of the path being relative to your test file, it's relative to the working directory in which you run your tests.

  4. Generated diff images are located in screens_diff subfolder of the output folder.

Capturing Screenshots

To capture a screenshot to use later in your application, you can add this code snippet in your test:

UIATarget.localTarget().captureScreenWithName("MainScreen");

Then, in your test output folder, you should find the directory for your latest run (typically something like Run 23) and in there you should find the MainScreen.png. You simply need to copy that image to your folder of reference screenshots and use them later in calls to assertScreenMatchesImageNamed.

Note that image-comparison will fail if the sizes of the images do not match, even if the content is an exact match. In some cases your screencapture may include the 20-point status bar at the top so you may need to edit your expectation images in your image-editor of choice and chop off the top part of the image.

Image-Matching Tolerance

Note that the createImageAsserter function can take an optional fourth argument to override the default image-tolerance when comparing images. The default value should work in most cases, but may be useful to override if you are getting unexpected test failures.

Retrying Assertions

There are times in UIAutomation when you there is an indeterminate amount of time between when you exercise the app and state change has been made (think of tapping a table cell to navigate to sub-screen). This can result in false test failures because your assertions are being executed too quickly.

UIAutomation does provide a timeout mechanism and the ability to pause test script execution, but this is pretty hacky and a bit unwieldy. Tuneup provides a retry() function that allows you to package up assertions in a JavaScript function that will be retried a number of times until it passes or until it timesout. This results in more readable code, and potentially quicker test execution times.

The retry() function takes a number of optional arguments. The first is a JavaScript function to execute. In here, you make whatever assertions you like. The second and third optional arguments specify the maximum number of retries and the time to wait between each attempt, relatively. The default value is three attempts with a delay of 0.5 seconds between each invocation.

Here’s an example:

retry(function() {
  assertEquals("Login", window.navigationBars()[0].name());
});