Subliminal

An understated approach to iOS integration testing.

inkling
756
54
Objective-C

Subliminal

Build Status
Gitter chat

Subliminal is a framework for writing iOS integration tests. Subliminal provides
a familiar OCUnit/XCTest-like interface to Apple’s UIAutomation framework,
with tests written entirely in Objective-C. Subliminal also provides a powerful
mechanism for your tests to manipulate your application directly.

[
Features •
Getting Started •
Requirements •
Usage •
Continuous Integration •
Contributing •
Contact •
License
]

Features

Seamless Integration

Write your tests in Objective-C, and run them from Xcode. See rich-text logs
and screenshots in Instruments. Use UIAutomation to simulate user interaction.
Subliminal lets you use familiar tools, no dependencies required.

Full Control

By using UIAutomation, Subliminal can simulate almost any interaction–without
resorting to private APIs. From navigating in-app purchase dialogs, to putting
your app to sleep, Subliminal lets you simulate complex interaction like a user.
And when you want to manipulate your app directly, Subliminal will help you do
that too.

Scalable Tests

Define Objective-C methods to help set up and tear down tests. Leverage native
support for continuous integration. Take confidence in Subliminal’s complete
documentation and full test coverage. Subliminal is the perfect foundation
for your tests.

How to Get Started

Running the Example App

  1. Clone the Subliminal repo: git clone https://github.com/inkling/Subliminal.git.
  2. cd into the directory: cd Subliminal.
  3. If you haven’t already, set up Subliminal: rake install.
  4. Open the Example project: open Example/SubliminalTest.xcodeproj.
  5. Switch to the “Integration Tests” scheme.
    You may also see a scheme called “Subliminal Integration tests”–make sure you choose “Integration Tests.”
  6. Choose Product > Profile (⌘+I).
  7. Under the User Templates, choose Subliminal.

Installing Subliminal

For an installation walkthrough, refer to Subliminal’s wiki.

Requirements

Subliminal supports projects built using Xcode 5.1 and iOS 7.x SDKs,
and deployment targets running iOS 6.1 through 7.1.

For iOS 5.1 support, use Subliminal 1.1.0 (found in the
Releases section or on
CocoaPods). To test in the iOS 5.1 Simulator, you will
need to run OS X 10.8 and manually add the iOS 5.1 Simulator to Xcode 5.1,
as described here.

Usage

Subliminal is designed to be instantly familiar to users of OCUnit/XCTest.
In Subliminal, subclasses of SLTest define tests as methods beginning with test.
At run-time, Subliminal discovers and runs these tests.

Tests manipulate the user interface and can even manipulate the application directly.
Here’s what a sample test case looks like:

@implementation STLoginTest

- (void)testLogInSucceedsWithUsernameAndPassword {
	SLTextField *usernameField = [SLTextField elementWithAccessibilityLabel:@"username field"];
	SLTextField *passwordField = [SLTextField elementWithAccessibilityLabel:@"password field" isSecure:YES];
	SLElement *submitButton = [SLElement elementWithAccessibilityLabel:@"Submit"];
	SLElement *loginSpinner = [SLElement elementWithAccessibilityLabel:@"Logging in..."];
	
    NSString *username = @"Jeff", *password = @"foo";
    [usernameField setText:username];
    [passwordField setText:password];

    [submitButton tap];

	// wait for the login spinner to disappear
    SLAssertTrueWithTimeout([loginSpinner isInvalidOrInvisible], 
    						3.0, @"Log-in was not successful.");

    NSString *successMessage = [NSString stringWithFormat:@"Hello, %@!", username];
    SLAssertTrue([[SLElement elementWithAccessibilityLabel:successMessage] isValid], 
    			@"Log-in did not succeed.");
    
    // Check the internal state of the app.			
    SLAssertTrue(SLAskAppYesNo(isUserLoggedIn), @"User is not logged in.")
}

@end

For more information, see Subliminal’s wiki.

Continuous Integration

Subliminal includes end-to-end CI support for building your project, running its tests on the appropriate simulator or device, and outputting results in a variety of formats.

For example scripts and guides to integrate with popular CI services like Travis and Jenkins, see Subliminal’s wiki.

Comparison to Other Integration Test Frameworks

  • How is Subliminal different from other integration test frameworks?

    Most other integration test frameworks fall into two categories: entirely
    Objective-C based, or entirely UIAutomation-based.

    Frameworks that are entirely Objective-C based, like KIF,
    Frank, etc., must hack the application’s
    touch-handling system, using private APIs, to simulate user interaction.
    There is thus no guarantee that they accurately simulate a user’s input.
    Moreover, these frameworks can only simulate interaction with the application,
    as opposed to interaction with the device, other processes like in-app purchase
    alerts, etc.

    Frameworks that are entirely based on Apple’s UIAutomation framework require
    cumbersome workflows–writing tests in JavaScript, in Instruments–which do not
    make use of the developer’s existing toolchain. Moreover, they offer the developer
    no means of manipulating the application directly–it is a complete black box
    to a UIAutomation-based test.

    Only Subliminal combines the convenience of writing tests in Objective-C
    with the power of UIAutomation.

  • How is Subliminal different than UIAutomation?

    Besides the limitations of UIAutomation described above, it is extremely
    difficult to write UIAutomation tests. This is because UIAutomation requires
    that user interface elements be identified by their position within the
    “element hierarchy”, like

    var cell = UIATarget.localTarget().frontMostApp().mainWindow().tableViews()[0].cells()["foo"];

    These references are not only difficult to read but are also difficult to write.
    To refer to any particular element, you have to describe its entire ancestry,
    while including only the views that UIAutomation deems necessary (images, yes;
    accessible elements, maybe; private UIWebView subviews, sure!).

    UIAutomation-based tests are not meant to be written, but to be “recorded”
    using Instruments. This forces dependence on Instruments, and makes the tests
    difficult to modify thereafter.

    Subliminal allows developers to identify elements by their properties,
    independent of their position in the element hierarchy:

    SLElement *fooCell = [SLElement elementWithAccessibilityLabel:@"foo"];

    Subliminal abstracts away the complexity of UIAutomation scripts to let developers focus on writing tests.

    Subliminal also fixes several bugs in UIAutomation and the instruments CLI tool,
    such as instruments’ lack for true device support.
    And, last but not least, Subliminal rewrites instruments’ output using human-friendly formatting
    and ANSI colors:

Contributing

Subliminal welcomes pull requests! Check out the contributing guidelines to learn how to set up Subliminal for development and how to make a successful pull request.

Credits

Created by Jeff Wear, made possible by Inkling,
with help from:

and Subliminal’s growing list of contributors.

Contact

If you’d like to chat, we hold “office hours” on Gitter
3-4pm Pacific Time, Tuesdays and Thursdays.

Lastly, you can follow Subliminal on Twitter for news and tips (@subliminaltest).

Copyright and License

Copyright 2013-2014 Inkling Systems, Inc.

Licensed under the Apache License, Version 2.0 (the “License”);
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an “AS IS” BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.