AppThwack and Appium Tutorial: Running Appium locally and on the AppThwack device cloud

Run Appium tests on AppThwack's device cloud

Appium Overview

With a compelling testing philosophy and an already thriving community, Appium is a relatively new mobile app automation framework that uses the familiar open standard JSON wire protocol to drive automated tests. Appium is a flexible framework in that tests can be written in many different languages; right now bindings exist for Python, Java, .NET, Objective-C, PHP and Ruby. If you have written tests in Selenium then you will be right at home with Appium.

Appium uses a client/server model where clients are test scripts written in your preferred language, and the server is an open source NodeJS application that manages test runs against a device. Appium can test both native mobile and hybrid applications available through Apple’s App Store or Google Play, as well as mobile web apps that run in the browser.

An Appium test can run entirely on your local machine, and that same test will easily run on AppThwack, enabling you to quickly test on multiple physical devices. This Appium tutorial will show you how to run a test locally and then run that test on AppThwack with just a small change to your test configuration and no changes to the test itself.

Getting started with Appium

The Appium server is written in NodeJS, meaning you’ll need to locally install NodeJS before starting the tutorial. We will write our sample test in Ruby, so you’ll need Ruby and Rubygems as well. RVM is a great tool for installing and managing Ruby. You’ll also need to be able to access your local device through adb.

$ adb devices
List of devices attached
c0808bc73583621 device

Looks good. Let’s create a folder for this tutorial and get this party started.

$ mkdir appium-tutorial
$ cd appium-tutorial

Install Appium using the npm tool bundled with NodeJS and start the server.

$ npm install -g appium
$ appium &

Make sure you have the Ruby client libraries.

$ printf "source 'https://rubygems.org'\ngem 'appium_lib'\n" >> Gemfile
$ bundle

Our test will reference an APK that is included in the Appium GitHub repository. This APK shows a button at the bottom of the screen, and this test will prove that the button exists. Grab the file and drop it into your appium-tutorial directory.

Now we’ll set up our sample script. Create a file called simple_test.rb that contains the following ruby test code.

require 'rubygems'
require 'appium_lib'

APP = "./ContactManager.apk"
opts = { "caps" => { "platformName" => "android", "app" => APP } }
driver = Appium::Driver.new(opts)
driver.start_driver

if exists { button('Add Contact') }
  puts('We have the "Add Contact" button')
else
  puts('No button was found')
end

driver.driver_quit

This test simply verifies the existence of a button with text “Add Contact” using the exists method from the Appium ruby library. The exists method is just one method of many and the full API is explained in more detail on the GitHub project page. You could also use the selenium-webdriver ruby gem if you want to use something less ruby-idiomatic and closer to the original JSON wire protocol. Converting this script to other languages is a matter of just adjusting for the language syntax.

Now, you can run your test:

$ bundle exec ruby simple_test.rb
Debug is: false
We have the "Add Contact" button

Appium on AppThwack

Once you have your Appium test running locally, you can run that same test on any of the hundreds of devices in our device lab.

Upload the APK into the AppThwack service using the API if you have not already done so. Be sure to use your API key instead of the placeholder.

$ curl -X POST -u "<your_api_key>:" https://appthwack.com/api/file \
  -F "name=ContactManager.apk" -F "save=true" \
  -F "file=@./ContactManager.apk"

{"file_id": 84773}

The API call returns a file_id descriptor that references the APK inside of AppThwack. AppThwack will automatically delete file uploads after the first test run unless you use the save=true option to maintain the file on the AppThwack servers. Using this option allows you to reuse the same file_id in subsequent test runs without having to reupload the APK.

Modify the opts object in simple_test.rb to include the AppThwack server URL and a few specific additions to the device capabilities for testing against specific devices within the AppThwack device lab.

opts = {
  "appium_lib" => {
    "server_url" => "https://appthwack.com/wd/hub"
  },
  "caps" => {
    "platformName" => "android",
    "platformVersion" => "4.4.2",                                                                                                                                         
    "automationName" => "Appium",
    "deviceName" => "LG Nexus 5",
    "app" => "84773",
    "apiKey" => "<your_api_key>",
    "project" => "20861",
    "device" => "android"
  }
}
Appium on a project page

Note that we’ve replaced the app value, formerly the APK path, with the result from our file upload, “84773.” We’ve also added the apiKey, which can be found on your Profile page, and a project ID that you can find on the project settings for an existing project inside your AppThwack account.

It is very important that the deviceName and platformVersion match exactly as they appear when listed using the AppThwack API. For example, the Samsung Galaxy Nexus S 4G would have a deviceName of Samsung Galaxy S 4G and a platformVersion of 4.1.1.

Rerunning the now modified test script will show the same output, but the test will run inside our device lab on a real Nexus S 4G device.

$ bundle exec ruby simple_test.rb 
Debug is: false
We have the "Add Contact" button
Appium Test Results

When execution is finished, the AppThwack project will display the results of this test. The results are associated with the project we assigned in the device capabilities within our test script and you will see a new report for each test run we perform.

Limitations

If you are using Appium on AppThwack, there are a few caveats to note.

  1. Since Appium tests execute client-side, we generate reports using only data from the device: memory consumption, thread utilization and CPU usage, device logs, etc. You’ll need to determine pass/fail results locally.

  2. AppThwack does not include screenshots in the online report for the same reason, though, like tests results, these can be viewed on the client side. For example, in Ruby, use the screenshot method on the driver object, documented in more detail in this code.

  3. If you request a device that is currently unavailable, Appium will wait for the device or timeout the request. We recommend that you adjust your client timeout to 15 minutes. That’s the maximum time it will ever take, and it’s unlikely you’ll ever wait that long.

Conclusion

Appium leverages the powerful and familiar JSON wire protocol to offer a compelling new tool for mobile developers. With AppThwack and just a few changes to your test configurations, you can run your Appium tests on hundreds of real, physical devices in the cloud. Please let us know if you have questions, ideas, or feedback in general. Happy testing!

Continuous Integration for Mobile Apps with AppThwack and Jenkins (Part 2)

Jenkins AppThwack plugin

This is a two part blog post about continuous integration for mobile apps. Be sure to check out Part 1 for getting started with AppThwack and Jenkins.


Back in September, we announced the initial release of the official AppThwack Jenkins plugin. Since its inception, we’re pleased to see many developers seamlessly integrate AppThwack and testing on real mobile devices into their development workflow; testing on a per-commit and nightly-build basis.

While the plugin was perfect for capturing build artifacts and kicking off automated tests, it lacked Jenkins UI integration for displaying results. Customer feedback showed that this is an extremely important feature and we couldn’t agree more. The benefits of continuous testing are diminished if developers cannot quickly view results and historical trends. This is essential data for promptly diagnosing problems which facilitates a quick development process.

We’ve been adding features over the past few months and as of today (v1.8) of our plugin, we now have real-time AppThwack results right in your Jenkins UI. We have plenty more cool Jenkins features up our sleeves, but we’re happy with the initial release and hope you are too! Let’s jump right in and check it out!


Instant Feedback

Screen Shot 2014-04-30 at 11.13.50 PM

It’s important to see the status of your builds as quickly as possible. The AppThwack plugin now attaches to your project view on the Jenkins homepage to display the status of the latest build.

Choose your own Adventure

Screen Shot 2014-04-30 at 11.12.28 PM

Previously, users were required to enter the names of the project and device pool they wanted to use. This wasn’t very user friendly and was prone to typos. Enough of that foolishness! The project and device pool entry fields are now drop-down selections populated automatically for you.

Interested in testing on real devices for free? Check out this blog post for more information about our Intel sponsored devices.

Gotta catch ‘em all

It’s your data, use it when you need it. Continuous testing generates tons of logs, screenshots and important device specific data imperative for debugging. Screen Shot 2014-04-30 at 11.06.03 PM

All of these test artifacts are now automatically downloaded and stored as part of every Jenkins build, available for deeper analysis whenever you need it.

Bell-bottoms? JNCO jeans? Macarena?

Screen Shot 2014-04-30 at 11.05.52 PM

Stay hip with the latest trends of your build results.

Easily view and identify trends in test passes, failures and warnings for every build and see high level results for your most recent AppThwack runs, right on your Jenkins project homepage.

Screen Shot 2014-04-30 at 11.06.33 PM

Individual Results

You can also drill down and view the highlights of a single AppThwack run. This will display all the high level result data you’ve come to expect, including latest trends in performance data (CPU, Memory, Threads, Network Traffic, FPS, etc).

Screen Shot 2014-04-30 at 11.07.42 PM

Individual tests are grouped by result (pass/fail/warn) and then by each device, enabling you to quickly identify which test and/or device(s) are encountering errors.

Additionally, each test links back to its own report on AppThwack, allowing for even deeper analysis.

Getting Started

Interested in mobile Continuous Integration using real devices but not sure where to start? Check out our partner CloudBees who specializes in cloud based CI.


Feedback

Using our Jenkins plugin? Love it? Hate it? Let us know!

We’d love to hear success stories and feedback about how we can improve our integration into your workflow.

Having a problem or want to contribute? The plugin is open source! Check out the Github page for more details!

Mobile App Testing with AppThwack at PyCon 2014

appthwack-pycon

We are super excited to announce that AppThwack was selected as a member of Startup Row at PyCon 2014 in Montreal. We sit among a large class of very intriguing startups built on the shoulders of the Python community and are honored to be one of the select few to participate.

Attending PyCon this year? Come visit us and eleven other companies in Startup Row! We’ll be demoing AppThwack (powered almost entirely by Python) all day on Friday, April 11th and we’d love to meet you! This is our first time attending PyCon and we look forward to meeting all who contribute towards building such a wonderful development community.

Run XCTest and OCUnit Tests on Real Devices with AppThwack

AppThwack iOS Device Lab

We are excited to announce support for two iOS test frameworks that are very popular with app developers: OCUnit and XCTest. As of today, you can run your OCUnit and XCTest tests on our cloud of real, non-jailbroken iOS devices.

What is OCUnit/XCTest?

OCUnit is a testing framework from Sente that was first integrated into Xcode 2.1. XCTest was introduced in Xcode 5 as a replacement for OCUnit, but both frameworks are still supported in the latest version of Xcode. Both frameworks are very developer friendly. They not only allow for writing automated tests in Objective-C, but are also tightly integrated with the development environment. Setting up your tests in Xcode is easy and Apple’s documentation contains a quick guide for adding a new test target using Xcode’s Test Navigator.

KIF, Kiwi, and more

A number of popular frameworks and libraries have been built on top of OCUnit that make development and maintenance of tests even easier, and AppThwack supports all of them. Here is a short list of the most notable ones:

KIFKeep It Functional, an integration test framework that leverages accessibility labels for UI automation of an app.

Kiwi – Behavior Driven Development (BDD) library for iOS.

Specta – Another BDD framework for Objective-C.

OCMock – An excellent library that implements Objective-C mock objects. Can be used in combination with any of the frameworks listed above.

Running OCUnit and XCTest on AppThwack

Running your existing tests on AppThwack is very straight forward and doesn’t require you to make any changes to your code or binaries. Simply build your project, upload the bundles via our UI or API, select some devices, and hit Go! Here is a quick step-by-step guide based on the KIF sample Calculator project:

  1. 1) Build your tests by selecting ‘Product’->’Build For’->’Testing’.

    Build your OCUnit or XCTest bundles
  2. 2) Locate your application and test bundles using ‘Show in Finder’.

    Location bundles
  3. 3) Archive both bundles so they can be uploaded to AppThwack.

    Archive bundles
  4. 4) Go to your AppThwack iOS project, upload your application bundle ZIP file, select OCUnit and upload our test bundle ZIP, select devices and hit the the ‘Go’ button.

    OCUnit in AppThwack

That’s it! Your tests are now running on AppThwack’s device cloud and you will see your results in a few minutes. Your AppThwack report will contain results for all of your tests as well as performance data for the devices that you selected. All of the test failures will be displayed in the highlights view:

Report highlights

Once you go into the results for a particular device, you will see detailed performance data, all test results, device console logs, and crash reports (if any):

Detailed report

You can see a complete sample report here.

Continuous Integration with AppThwack

You can also kick off your tests using our RESTful API. Using cURL, this can be done in a few simple steps:

  1. 1) Upload your app bundle ZIP file:

    curl -X POST -u "<your_api_key>:" https://appthwack.com/api/file -F
    "name=myApp.zip" -F "file=@<path_to_your_app_bundle_zip>"

  2. 2) Upload your test bundle ZIP file:

    curl -X POST -u "<your_api_key>:" https://appthwack.com/api/file -F "name=myTests.zip" -F "file=@<path_to_your_test_bundle_zip>"

  3. 3) Schedule your tests:

    curl -X POST -u "<your_api_key>:" https://appthwack.com/api/run -F "project=<your_project_id>" -F "name=<test_run_name>" -F "app=<app_upload_id>" -F "ocunit=<tests_upload_id>"

Support for OCUnit and XCTest in our AppThwack Jenkins plugin is coming soon too.

Free App Testing on Intel Devices

AppThwack and Intel Software parter to offer free app testing

Intel Software and AppThwack are happy to announce free app testing on Intel Atom-powered devices.

More and more Android devices have Intel processors inside, and now, with our partnership with Intel Software, you can access Intel-based Android devices to test your apps for free. In addition, you can find great tools, documentation, and support to write, port, or optimize your Android apps at the Intel Developer Zone.

Which Devices Are Included?

The Lenovo K900 is one device provided for free app testing

To start, we have a modest pool with some of the latest and greatest Intel Atom devices.

  • Asus MeMO Pad FHD 10
  • Dell Venue 7
  • Dell Venue 8
  • Lenovo IdeaPhone K900
  • Motorola Droid RAZR i
  • Samsung Galaxy Tab 3 10

Tests on these devices do not deduct minutes from your account and are 100% free.

How Do I Get Started?

An Intel Atom device pool is provided automatically

To get started, simply select Intel Atom devices when scheduling a test, either individually or via the pre-populated pool “Intel Atom.” The estimated test completion time will indicate how many minutes will and will not count towards your account.

As always, you can test your app using our scriptless compatibility tests or take full control with any of the automation frameworks we support, such as Robotium, Calabash, or the brand new Espresso. For a complete list see our FAQ.

Enjoy!

Moving the AppThwack Device Lab

Update Dec 1 7:00am PST iOS testing is once again available! Everything appears to be running in a stable fashion. We’ll be watching closely to make sure your tests run without a hitch. Happy to be in our new home!

Update Nov 30 11:00pm PST Android testing is once again available! iOS to come online shortly. Expect a hiccup here and there over the next couple days as we work out some kinks in the new setup. Once again, thanks for your patience!

Update Nov 30 7:00am PST We’re moving now and testing will be available in approximately 24 hours! We’ll keep this and our Twitter feed up to date if the plan changes. Thanks for your patience!

We’re moving the device lab on Saturday, November 30th. Because this requires physically transporting the devices, servers, and other infrastructure the testing portion of AppThwack will be unavailable for approximately 24 hours. The viewing of existing test results will be uninterrupted.

Why are we moving?

We’ve been fortunate to be part of the Portland Incubator Experiment, or PIE as it’s more commonly known, and have been running operations in its facilities since July of 2012. We’re forever grateful to the team at PIE and the companies we’ve worked with side-by-side. Seriously, it’s not easy sitting in the same room as several hundred devices played Nyan Cat with full volume (Yes, this happened).

As we’ve expanded both our team and our lab this past year, it became clear we need some breathing room. A dedicated space is a must, especially given our unique lab requirements, so a search for a new home began earlier this year.

What changes?

From a testing perspective, nothing is changing. We have a bunch of cool features in the works, but they’re all independent of the lab move we’re announcing today. There are a couple high-level advantages to the new move that directly benefits you, though.

Network stability

Although we made huge strides in WiFi connectivity over the past few months, there’s always room for improvement. The new space will have a more reliable Internet connection, and our WiFi setup is getting a refresh. The result is better overall network stability for your tests.

Security

Our current space is keycard protected during off hours, but the new space ups the security measures substantially. The lab will exist in its own, physically isolated space, with access restricted to just AppThwack employees. In short, nobody will catch a glimpse of your latest work.

Timing

We’ll provide updates on Twitter throughout the move. We’ll start around 7am Pacific Time on November 30th and will resume services Sunday, December 1, unless we update earlier of course.

Thanks for your patience, and thanks for testing on AppThwack!

AppThwack Partners with In-Q-Tel

We’re excited to announce a partnership with In-Q-Tel today. In summary, it enables us to accelerate the expansion of useful features for testing your apps on real phones, tablets, and other form factors, and reinforces the utility and longevity of the platform as a whole.

As always, thank you for testing your apps on AppThwack; we love that we get to play a role in your success!

AppThwack Partners with IQT to Expand its App Testing Technology

IQT selects AppThwack’s enterprise-grade app testing platform to validate apps on real devices

September 17, 2013

AppThwack, a leader in cloud-based app testing, announced it has entered into a strategic partnership and technology development agreement with In-Q-Tel (IQT). IQT is an independent strategic investment firm that identifies innovative technology solutions to support the missions of the U.S. Intelligence Community (IC). The investment enables further development of AppThwack’s enterprise-grade app testing platform, and helps IC customers thoroughly and quickly test app compatibility, functionality, and performance before deployment of an app.

“Today’s apps must function and perform across myriad devices, OS versions, network configurations, and other variables, and we see more and more companies turning to automation to help them navigate this difficult landscape,” said AppThwack co-founder Trent Peterson. “We’re excited to work with IQT in enhancing our app testing platform and extending this capability to the U.S. Intelligence Community.”

AppThwack provides cloud-based access to real, physical device labs for automated Android and iOS app testing through its website, tool integrations, and API. Test reports are generated in minutes and provide high-level trends, low-level logs, high-resolution screenshots, and detailed performance profiles. AppThwack enables companies to focus their resources on development and mission-critical activities by removing the complexities of maintaining device labs, building automation infrastructure, and analyzing scores of disparate data.

“Due to the wide adoption of apps, both for external and internal use, along with increasingly popular internal app markets and bring-your-own-device (BYOD) policies, there’s a great need to test and vet apps within the enterprise,” says Jay Emmanuel, Senior Vice President of IQT’s Mobility practice. “AppThwack’s technology provides an excellent user interface and easy to read reports, making it easier to quickly identify problems before deploying mobile applications. We’re excited to help extend this capability to our Intelligence Community partners.”

Increased Test Concurrency and Timeouts on AppThwack

One of the reasons test results appear quickly on AppThwack relates to how we handle test concurrency (parallelized tests) and timeouts. In this article you’ll get a brief overview of what these settings are, how they work, and how you can use them to your advantage while testing your Android and iOS applications.

Test Concurrency

AppThwack Device Lab

We have a lot of devices, and if you were to run a 20 minute test on just 200 of them one after another you’d have to wait nearly three days to get a report. Yikes!

Luckily, our back-end queues tests on each device individually, and every account allows some number of tests to run in parallel at any given time. Given the example above, this means AppThwack can run a 20 minute test on one device or 200, and the complete results are available in the same amount of time – about 20 minutes.

To accommodate many people running tests at once with very short wait times, each account receives a maximum concurrency number that dictates how many individual devices can run at a time. This is so that numerous people can run simultaneously, each receiving some results in a couple minutes while receiving the rest over the next 10-20 minutes. You can still schedule tests on as many devices as you’d like at the same time; the limit only applies during actual test execution.

Test concurrency isn’t just for load balancing AppThwack and providing a consistent user experience, either. If your app is media-heavy, has license limitations, or is in some other way dependent on limited remote resources, you can use concurrency to throttle testing to accommodate your unique situation. For instance, adjusting concurrency all the way down to 1 results in serial execution.

Test Timeouts

In order to prevent runaway tests and draining device minutes accidentally, we have a default test timeout that varies between 5 and 20 minutes depending on the account type. Once that timeout is reached the test is forcibly terminated and the test’s result summary will explain that it errored due to exceeding the maximum time allowed.

Since we’re talking automation here, most tests are very, very fast, and the vast majority of custom automation completes well before the default maximum is reached. If not, it’s pretty common to break the scripts down into smaller pieces by category or type. This can be done without restructuring tests by specifying the tests you’d like to run. Both JUnit/Robotium and Calabash allow for this through class and tags respectively, and AppThwack supports filtering in this way when scheduling tests, both in the user interface and the API.

That said, there are certain types of tests that absolutely benefit from larger timeouts. Performance tests, stress tests, and apps that require significant amounts of assets to be downloaded are all candidates for higher timeouts.

Adjusting Concurrency and Test Timeouts

Account settings

You can now take control of how many of your tests run in parallel, as well as set the maximum timeout after which a runaway test will be terminated. From your account page you can adjust the maximum concurrency and maximum test duration to fine-tune AppThwack’s behavior to your app and tests.

Announcing Increased Maximums for Test Concurrency and Timeouts

As we continue to build out our device labs and refine our queuing algorithms, we’ll make changes to the maximum allowed concurrency and timeouts. For example, today we bumped maximum concurrency from 35 to 50 tests at a time and duration from 10 to 30 minutes.

If you need any more than that, please don’t hesitate to contact us at . And, as always, if you have any questions at all please reach out.

Continuous Integration for Mobile Apps with AppThwack and Jenkins

Jenkins AppThwack plugin

This is part one of a blog post series about continuous integration for mobile apps. Be sure to check out Part 2 for checking out advanced plugin features including results integration and performance graphs.


Today, we’re happy to announce the initial release of our Jenkins plugin, which introduces AppThwack into your Continuous Integration cycle.

As developers ourselves, we understand the importance of seamless tool integrations within the development workflow. Last week we announced the release of our Android Studio/Gradle plugin, which enables testing of Android apps on hundreds of devices directly from your IDE.

Ready to get started? Here’s a quick tutorial to get started with AppThwack, Jenkins and Continuous Integration for mobile apps. Once you’re up and running, you’ll be able to kick off tests on our device lab from your own Jenkins server!


Installation

The AppThwack plugin lives in the official Jenkins-CI maven repository. That means you can download/install it directly from within your running Jenkins server.

Starting at the Jenkins homepage, navigate to the Manage Jenkins > Manage Plugins page. From the plugin view, click the Available tab and scroll to find the AppThwack plugin.

Jenkins AppThwack Plugin Select Box

Finally, scroll to the bottom and click Install.

System Settings

AppThwack Account API Key

Navigate to Manage Jenkins > Configure System.

Scroll down to the AppThwack settings section to add your API Key. This key is used to authenticate your Jenkins server with AppThwack and is used across all of your Jenkins projects.

Now that your API key added, it’s time to navigate to a Jenkins project and configure the post-build action.

Project Settings

Navigate to a project of your choice and click Jenkins Project Configure Button from the left-hand side menu.

Add Post-build action

Click the Add post-build action button, select Run Tests on AppThwack Jenkins Post Build Action and you’ll notice a new, AppThwack specific settings section appear. All remaining configuration will happen here.

Choose your Project and Device Pool

First up is selecting your AppThwack project and selecting which devices to test on. Jenkins Project Application Settings

Find your Application

Next up is configuring the plugin to find your newly built mobile app. The application field allows for standard Jenkins (ant) pattern matching with expandable environment variables. Please note that this pattern is relative to the Jenkins workspace currently being used by the project/build.

Test Settings

The next step is selecting which tests you wish to run. If you don’t have custom tests written for your app (yet), you can still run the AppThwack built-in compatibility test suites. These suites will Install, Launch, Explore and Stress your app with no code changes necessary.

Let’s walk-through configuring some tests for both Android and iOS.

Built-in Test Suites

AppThwack provides a built-in compatibility test suite for both Android and iOS.

The built-in Android test suite supports additional options for configuring our AppExplorer. These are optional and should configured on a case-by-case basis.

Jenkins Android Built-in Test Suite

Calabash

Jenkins Calabash Settings

You can run your custom Calabash scripts for both Android and iOS. Please note that the features field supports the same pattern matching and environment variables as the application one above.

JUnit/Robotium

Jenkins JUnit/Robotium Settings

Have a JUnit/Robotium project building with your app? Same pattern matching rules apply.

KIF

Jenkins KIF iOS

Functional tests on iOS with KIF are supported too; no configuration required.

MonkeyTalk

jenkins-monkey-talk

Using MonkeyTalk for Android? Just upload an archive of your tests.

UIA

jenkins-uia

Last, but certainly not least, iOS UI Automation. Same pattern rules apply for finding your Javascript tests.

Click Save as the configuration is complete!

Build/Test

To manually kick off a build from a Jenkins project, click the Jenkins Project Build Button button from the left-side menu.

Once your app builds successfully, an examination of the Console Output will yield the following:

...
[AppThwack] Using Project 'demoproject-ios'
[AppThwack] Using DevicePool 'hawker-test-pool-ios'
[AppThwack] Archiving artifact 'IOSTestApp.ipa'
[AppThwack] Using App '/home/ahawker/src/appthwack-jenkins/work/jobs/test/builds/2013-08-30_17-54-09/archive/IOSTestApp.ipa'
[AppThwack] Archiving artifact 'calabash.zip'
[AppThwack] Using 'calabash' test content from '/home/ahawker/src/appthwack-jenkins/work/jobs/test/builds/2013-08-30_17-54-09/archive/calabash.zip'
[AppThwack] Scheduling 'calabash' run 'IOSTestApp.ipa (Jenkins)'
[AppThwack] Congrats! See your test run at https://appthwack.com/project/demoproject-ios/run/24549
Finished: SUCCESS

You’re done! You can now schedule tests on AppThwack from your own Jenkins Continuous Integration server!

What’s Next?

Stay tuned as we have more features in the pipeline, including reports/results integrated directly in your Jenkins dashboard.


Using the AppThwack Jenkins plugin (or any of our tools) in your development process? Let us know! We’d love to hear success stories and feedback about how we can improve our integration into your workflow.

Having a problem or want to contribute? The plugin is open source! Check out the Github page for more details!

Android app testing with Android Studio (Gradle) and AppThwack

AppThwack invades your IDE!

Here’s a quick tutorial to get started with AppThwack, Android Studio and Gradle. Once you’re up and running you’ll be able to kick off tests on our device lab from the IDE or command line. You can run our built-in compatibility tests or run your own (JUnit/Robotium, Calabash, MonkeyTalk, etc), giving you full control over when, what, and where you’re testing your app.

Having a problem or interested in contributing? This plugin is open source! Check out the Github page for more details.


Configure Gradle

Android Studio Project Layout

To add AppThwack support to an existing Android Studio project, we must modify the project specific build.gradle file. Be sure to select your app specific build.gradle file as the top-level file is for configuration options across all projects.

Apply the plugin

Adding any custom plugin to Gradle is a simple call to apply plugin.

apply plugin: 'appthwack'

Add plugin dependencies

Add the AppThwack plugin+dependencies to the classpath so it can be built.

dependencies {
    //...
    classpath 'com.sun.jersey:jersey-core:1.8'

    // Version needs to match current version of the plugin (See the github readme for current version)
    classpath 'com.appthwack:gradle:1.3'
}

AppThwack settings

All AppThwack specific configuration happens within the appthwack closure of your build.gradle file. The first step is adding your account API Key, the target project to use and a pool of devices to test on.

appthwack {
    apiKey='...' //See: https://appthwack.com/user/profile
    project='My Android App Project'
    devicePool='Top 10 devices'
    //...
}

With the account and project configured, now it’s time to configure the tests. Within the same appthwack closure, we can configure JUnit/Robotium, Calabash or AppExplorer test runs.

JUnit/Robotium Tests

Selecting a JUnit run will automatically use the test APK which is built with your application.

appthwack {
    //...
    junit (
        'com.appthwack.MyApp.tests#TestFoo' //testFilter: Choose specific test cases. (optional)
    )
}

Note: The testFilter parameter is optional and should be omitted if you wish to run the entire JUnit test suite built with your app.

Calabash Tests

Selecting a Calabash run will test your newly built application with the selected features.

appthwack {
    //...
    calabash (
        '/src/app/tests/calabash/features.zip', //path: Path to Calabash features zip. (required)
        '@uitest' //tags: Pass cucumber based tags to Calabash. (optional)
    )
}

Note: The tags argument is optional and can be omitted.

AppExplorer Tests

Selecting an AppExplorer run will test your newly built application with the AppThwack AppExplorer and Exerciser Monkey.

appthwack {
    //...
    appexplorer (
        'test_account', //username: Value to use if encounters sign-in form. (optional)
        'testpassword123', //password: Value to use if encountered sign-in form. (optional)
        '...', //launchdata: Data URI to pass to application launch activity. (optional)
        1000, //eventcount: Number of Exerciser Monkey events. (optional)
        0 //monkeyseed: Seed value for randomizing the Exerciser Monkey. (optional)
    )
}

Note: All arguments are optional and can be omitted.

That’s it! Now that our build.gradle is complete, let’s setup a quick task in Android Studio to upload and run our tests.


Configure Android Studio

We’ll need to add a new Run Configuration in Android Studio so we can quickly start our test runs.

Add a new Run Configuration

Add New Gradle Run Configuration

Select Run > Edit Configurations from the Android Studio menu bar.

In the Run/Debug Configurations dialog, click the Add New Configuration in the top left-hand corner and add a new Gradle configuration.

Add the appthwackUpload task to your new Run Configuration. AppThwack Gradle Run Configuration

That’s it for Android Studio! Now it’s time to run our tests.


Running Tests

Now that our configuration is complete, we can kick off AppThwack test runs from Android Studio or directly through Gradle.

Android Studio

Select our Run Tests on AppThwack configuration from the drop down on the menu bar and hit play! Run Tests on AppThwack

Gradle

All Android Studio projects are packaged with the GradleWrapper, which gives us Gradle command line access from any platform.

$ cd /src/MyApp
$ ./gradlew appthwackUpload

Results

On a successful run from Android Studio or Gradle, your output will look something like this.

...
:GradleTestApp:appthwackUpload
[AppThwack] Using Project 'demoproject'.
[AppThwack] Using DevicePool 'Top 10 devices'.
[AppThwack] Uploading apk 'GradleTestApp-debug-unaligned.apk'.
[AppThwack] Uploading test apk 'GradleTestApp-test-unaligned.apk'.
[AppThwack] Scheduling 'junit' run 'GradleTestApp-debug-unaligned.apk (Gradle)'.
[AppThwack] Congrats! See your test results at https://appthwack.com/project/demoproject/run/24219.

Now you can kick off AppThwack test runs from your shell or IDE.