We are super excited to announce that AppThwack was selected as a member of Startup Row at PyCon 2014 in Montreal. We sit among a large class of very intriguing startups built on the shoulders of the Python community and are honored to be one of the select few to participate.
Attending PyCon this year? Come visit us and eleven other companies in Startup Row! We’ll be demoing AppThwack (powered almost entirely by Python) all day on Friday, April 11th and we’d love to meet you! This is our first time attending PyCon and we look forward to meeting all who contribute towards building such a wonderful development community.
We are excited to announce support for two iOS test frameworks that are very popular with app developers: OCUnit and XCTest. As of today, you can run your OCUnit and XCTest tests on our cloud of real, non-jailbroken iOS devices.
What is OCUnit/XCTest?
OCUnit is a testing framework from Sente that was first integrated into Xcode 2.1. XCTest was introduced in Xcode 5 as a replacement for OCUnit, but both frameworks are still supported in the latest version of Xcode. Both frameworks are very developer friendly. They not only allow for writing automated tests in Objective-C, but are also tightly integrated with the development environment. Setting up your tests in Xcode is easy and Apple’s documentation contains a quick guide for adding a new test target using Xcode’s Test Navigator.
KIF, Kiwi, and more
A number of popular frameworks and libraries have been built on top of OCUnit that make development and maintenance of tests even easier, and AppThwack supports all of them. Here is a short list of the most notable ones:
KIF – Keep It Functional, an integration test framework that leverages accessibility labels for UI automation of an app.
Kiwi – Behavior Driven Development (BDD) library for iOS.
OCMock – An excellent library that implements Objective-C mock objects. Can be used in combination with any of the frameworks listed above.
Running OCUnit and XCTest on AppThwack
Running your existing tests on AppThwack is very straight forward and doesn’t require you to make any changes to your code or binaries. Simply build your project, upload the bundles via our UI or API, select some devices, and hit Go! Here is a quick step-by-step guide based on the KIF sample Calculator project:
1) Build your tests by selecting ‘Product’->’Build For’->’Testing’.
2) Locate your application and test bundles using ‘Show in Finder’.
3) Archive both bundles so they can be uploaded to AppThwack.
4) Go to your AppThwack iOS project, upload your application bundle ZIP file, select OCUnit and upload our test bundle ZIP, select devices and hit the the ‘Go’ button.
That’s it! Your tests are now running on AppThwack’s device cloud and you will see your results in a few minutes. Your AppThwack report will contain results for all of your tests as well as performance data for the devices that you selected. All of the test failures will be displayed in the highlights view:
Once you go into the results for a particular device, you will see detailed performance data, all test results, device console logs, and crash reports (if any):
Intel Software and AppThwack are happy to announce free app testing on Intel Atom-powered devices.
More and more Android devices have Intel processors inside, and now, with our partnership with Intel Software, you can access Intel-based Android devices to test your apps for free. In addition, you can find great tools, documentation, and support to write, port, or optimize your Android apps at the Intel Developer Zone.
Which Devices Are Included?
To start, we have a modest pool with some of the latest and greatest Intel Atom devices.
Asus MeMO Pad FHD 10
Dell Venue 7
Dell Venue 8
Lenovo IdeaPhone K900
Motorola Droid RAZR i
Samsung Galaxy Tab 3 10
Tests on these devices do not deduct minutes from your account and are 100% free.
How Do I Get Started?
To get started, simply select Intel Atom devices when scheduling a test, either individually or via the pre-populated pool “Intel Atom.” The estimated test completion time will indicate how many minutes will and will not count towards your account.
As always, you can test your app using our scriptless compatibility tests or take full control with any of the automation frameworks we support, such as Robotium, Calabash, or the brand new Espresso. For a complete list see our FAQ.
Update Dec 1 7:00am PST iOS testing is once again available! Everything appears to be running in a stable fashion. We’ll be watching closely to make sure your tests run without a hitch. Happy to be in our new home!
Update Nov 30 11:00pm PST Android testing is once again available! iOS to come online shortly. Expect a hiccup here and there over the next couple days as we work out some kinks in the new setup. Once again, thanks for your patience!
Update Nov 30 7:00am PST We’re moving now and testing will be available in approximately 24 hours! We’ll keep this and our Twitter feed up to date if the plan changes. Thanks for your patience!
We’re moving the device lab on Saturday, November 30th. Because this requires physically transporting the devices, servers, and other infrastructure the testing portion of AppThwack will be unavailable for approximately 24 hours. The viewing of existing test results will be uninterrupted.
Why are we moving?
We’ve been fortunate to be part of the Portland Incubator Experiment, or PIE as it’s more commonly known, and have been running operations in its facilities since July of 2012. We’re forever grateful to the team at PIE and the companies we’ve worked with side-by-side. Seriously, it’s not easy sitting in the same room as several hundred devices played Nyan Cat with full volume (Yes, this happened).
As we’ve expanded both our team and our lab this past year, it became clear we need some breathing room. A dedicated space is a must, especially given our unique lab requirements, so a search for a new home began earlier this year.
From a testing perspective, nothing is changing. We have a bunch of cool features in the works, but they’re all independent of the lab move we’re announcing today. There are a couple high-level advantages to the new move that directly benefits you, though.
Although we made huge strides in WiFi connectivity over the past few months, there’s always room for improvement. The new space will have a more reliable Internet connection, and our WiFi setup is getting a refresh. The result is better overall network stability for your tests.
Our current space is keycard protected during off hours, but the new space ups the security measures substantially. The lab will exist in its own, physically isolated space, with access restricted to just AppThwack employees. In short, nobody will catch a glimpse of your latest work.
We’ll provide updates on Twitter throughout the move. We’ll start around 7am Pacific Time on November 30th and will resume services Sunday, December 1, unless we update earlier of course.
Thanks for your patience, and thanks for testing on AppThwack!
We’re excited to announce a partnership with In-Q-Tel today. In summary, it enables us to accelerate the expansion of useful features for testing your apps on real phones, tablets, and other form factors, and reinforces the utility and longevity of the platform as a whole.
As always, thank you for testing your apps on AppThwack; we love that we get to play a role in your success!
AppThwack Partners with IQT to Expand its App Testing Technology
IQT selects AppThwack’s enterprise-grade app testing platform to validate apps on real devices
September 17, 2013
AppThwack, a leader in cloud-based app testing, announced it has entered into a strategic partnership and technology development agreement with In-Q-Tel (IQT). IQT is an independent strategic investment firm that identifies innovative technology solutions to support the missions of the U.S. Intelligence Community (IC). The investment enables further development of AppThwack’s enterprise-grade app testing platform, and helps IC customers thoroughly and quickly test app compatibility, functionality, and performance before deployment of an app.
“Today’s apps must function and perform across myriad devices, OS versions, network configurations, and other variables, and we see more and more companies turning to automation to help them navigate this difficult landscape,” said AppThwack co-founder Trent Peterson. “We’re excited to work with IQT in enhancing our app testing platform and extending this capability to the U.S. Intelligence Community.”
AppThwack provides cloud-based access to real, physical device labs for automated Android and iOS app testing through its website, tool integrations, and API. Test reports are generated in minutes and provide high-level trends, low-level logs, high-resolution screenshots, and detailed performance profiles. AppThwack enables companies to focus their resources on development and mission-critical activities by removing the complexities of maintaining device labs, building automation infrastructure, and analyzing scores of disparate data.
“Due to the wide adoption of apps, both for external and internal use, along with increasingly popular internal app markets and bring-your-own-device (BYOD) policies, there’s a great need to test and vet apps within the enterprise,” says Jay Emmanuel, Senior Vice President of IQT’s Mobility practice. “AppThwack’s technology provides an excellent user interface and easy to read reports, making it easier to quickly identify problems before deploying mobile applications. We’re excited to help extend this capability to our Intelligence Community partners.”
One of the reasons test results appear quickly on AppThwack relates to how we handle test concurrency (parallelized tests) and timeouts. In this article you’ll get a brief overview of what these settings are, how they work, and how you can use them to your advantage while testing your Android and iOS applications.
We have a lot of devices, and if you were to run a 20 minute test on just 200 of them one after another you’d have to wait nearly three days to get a report. Yikes!
Luckily, our back-end queues tests on each device individually, and every account allows some number of tests to run in parallel at any given time. Given the example above, this means AppThwack can run a 20 minute test on one device or 200, and the complete results are available in the same amount of time – about 20 minutes.
To accommodate many people running tests at once with very short wait times, each account receives a maximum concurrency number that dictates how many individual devices can run at a time. This is so that numerous people can run simultaneously, each receiving some results in a couple minutes while receiving the rest over the next 10-20 minutes. You can still schedule tests on as many devices as you’d like at the same time; the limit only applies during actual test execution.
Test concurrency isn’t just for load balancing AppThwack and providing a consistent user experience, either. If your app is media-heavy, has license limitations, or is in some other way dependent on limited remote resources, you can use concurrency to throttle testing to accommodate your unique situation. For instance, adjusting concurrency all the way down to 1 results in serial execution.
In order to prevent runaway tests and draining device minutes accidentally, we have a default test timeout that varies between 5 and 20 minutes depending on the account type. Once that timeout is reached the test is forcibly terminated and the test’s result summary will explain that it errored due to exceeding the maximum time allowed.
Since we’re talking automation here, most tests are very, very fast, and the vast majority of custom automation completes well before the default maximum is reached. If not, it’s pretty common to break the scripts down into smaller pieces by category or type. This can be done without restructuring tests by specifying the tests you’d like to run. Both JUnit/Robotium and Calabash allow for this through class and tags respectively, and AppThwack supports filtering in this way when scheduling tests, both in the user interface and the API.
That said, there are certain types of tests that absolutely benefit from larger timeouts. Performance tests, stress tests, and apps that require significant amounts of assets to be downloaded are all candidates for higher timeouts.
Adjusting Concurrency and Test Timeouts
You can now take control of how many of your tests run in parallel, as well as set the maximum timeout after which a runaway test will be terminated. From your account page you can adjust the maximum concurrency and maximum test duration to fine-tune AppThwack’s behavior to your app and tests.
Announcing Increased Maximums for Test Concurrency and Timeouts
As we continue to build out our device labs and refine our queuing algorithms, we’ll make changes to the maximum allowed concurrency and timeouts. For example, today we bumped maximum concurrency from 35 to 50 tests at a time and duration from 10 to 30 minutes.
If you need any more than that, please don’t hesitate to contact us at . And, as always, if you have any questions at all please reach out.
As developers ourselves, we understand the importance of seamless tool integrations within the development workflow. Last week we announced the release of our Android Studio/Gradle plugin, which enables testing of Android apps on hundreds of devices directly from your IDE.
The AppThwack plugin lives in the official Jenkins-CI maven repository. That means you can download/install it directly from within your running Jenkins server.
Starting at the Jenkins homepage, navigate to the Manage Jenkins > Manage Plugins page. From the plugin view, click the Available tab and scroll to find the AppThwack plugin.
Finally, scroll to the bottom and click Install.
Navigate to Manage Jenkins > Configure System.
Scroll down to the AppThwack settings section to add your API Key. This key is used to authenticate your Jenkins server with AppThwack and is used across all of your Jenkins projects.
Now that your API key added, it’s time to navigate to a Jenkins project and configure the post-build action.
Navigate to a project of your choice and click from the left-hand side menu.
Add Post-build action
Click the Add post-build action button, select Run Tests on AppThwack and you’ll notice a new, AppThwack specific settings section appear. All remaining configuration will happen here.
Choose your Project and Device Pool
First up is selecting your AppThwack project and selecting which devices to test on.
Find your Application
Next up is configuring the plugin to find your newly built mobile app. The application field allows for standard Jenkins (ant) pattern matching with expandable environment variables. Please note that this pattern is relative to the Jenkins workspace currently being used by the project/build.
The next step is selecting which tests you wish to run. If you don’t have custom tests written for your app (yet), you can still run the AppThwack built-in compatibility test suites. These suites will Install, Launch, Explore and Stress your app with no code changes necessary.
Let’s walk-through configuring some tests for both Android and iOS.
Built-in Test Suites
AppThwack provides a built-in compatibility test suite for both Android and iOS.
The built-in Android test suite supports additional options for configuring our AppExplorer. These are optional and should configured on a case-by-case basis.
You can run your custom Calabash scripts for both Android and iOS. Please note that the features field supports the same pattern matching and environment variables as the application one above.
Have a JUnit/Robotium project building with your app? Same pattern matching rules apply.
Functional tests on iOS with KIF are supported too; no configuration required.
Using MonkeyTalk for Android? Just upload an archive of your tests.
Click Save as the configuration is complete!
To manually kick off a build from a Jenkins project, click the button from the left-side menu.
Once your app builds successfully, an examination of the Console Output will yield the following:
[AppThwack] Using Project 'demoproject-ios'
[AppThwack] Using DevicePool 'hawker-test-pool-ios'
[AppThwack] Archiving artifact 'IOSTestApp.ipa'
[AppThwack] Using App '/home/ahawker/src/appthwack-jenkins/work/jobs/test/builds/2013-08-30_17-54-09/archive/IOSTestApp.ipa'
[AppThwack] Archiving artifact 'calabash.zip'
[AppThwack] Using 'calabash' test content from '/home/ahawker/src/appthwack-jenkins/work/jobs/test/builds/2013-08-30_17-54-09/archive/calabash.zip'
[AppThwack] Scheduling 'calabash' run 'IOSTestApp.ipa (Jenkins)'
[AppThwack] Congrats! See your test run at https://appthwack.com/project/demoproject-ios/run/24549
You’re done! You can now schedule tests on AppThwack from your own Jenkins Continuous Integration server!
Stay tuned as we have more features in the pipeline, including reports/results integrated directly in your Jenkins dashboard.
Using the AppThwack Jenkins plugin (or any of our tools) in your development process? Let us know! We’d love to hear success stories and feedback about how we can improve our integration into your workflow.
Having a problem or want to contribute? The plugin is open source! Check out the Github page for more details!
Here’s a quick tutorial to get started with AppThwack, Android Studio and Gradle. Once you’re up and running you’ll be able to kick off tests on our device lab from the IDE or command line. You can run our built-in compatibility tests or run your own (JUnit/Robotium, Calabash, MonkeyTalk, etc), giving you full control over when, what, and where you’re testing your app.
Having a problem or interested in contributing? This plugin is open source! Check out the Github page for more details.
To add AppThwack support to an existing Android Studio project, we must modify the project specific build.gradle file. Be sure to select your app specific build.gradle file as the top-level file is for configuration options across all projects.
Apply the plugin
Adding any custom plugin to Gradle is a simple call to apply plugin.
apply plugin: 'appthwack'
Add plugin dependencies
Add the AppThwack plugin+dependencies to the classpath so it can be built.
// Version needs to match current version of the plugin (See the github readme for current version)
All AppThwack specific configuration happens within the appthwack closure of your build.gradle file. The first step is adding your account API Key, the target project to use and a pool of devices to test on.
With the account and project configured, now it’s time to configure the tests. Within the same appthwack closure, we can configure JUnit/Robotium, Calabash or AppExplorer test runs.
Selecting a JUnit run will automatically use the test APK which is built with your application.
'com.appthwack.MyApp.tests#TestFoo' //testFilter: Choose specific test cases. (optional)
Note: The testFilter parameter is optional and should be omitted if you wish to run the entire JUnit test suite built with your app.
Selecting a Calabash run will test your newly built application with the selected features.
'/src/app/tests/calabash/features.zip', //path: Path to Calabash features zip. (required)
'@uitest' //tags: Pass cucumber based tags to Calabash. (optional)
Note: The tags argument is optional and can be omitted.
Selecting an AppExplorer run will test your newly built application with the AppThwack AppExplorer and Exerciser Monkey.
'test_account', //username: Value to use if encounters sign-in form. (optional)
'testpassword123', //password: Value to use if encountered sign-in form. (optional)
'...', //launchdata: Data URI to pass to application launch activity. (optional)
1000, //eventcount: Number of Exerciser Monkey events. (optional)
0 //monkeyseed: Seed value for randomizing the Exerciser Monkey. (optional)
Note: All arguments are optional and can be omitted.
That’s it! Now that our build.gradle is complete, let’s setup a quick task in Android Studio to upload and run our tests.
Configure Android Studio
We’ll need to add a new Run Configuration in Android Studio so we can quickly start our test runs.
Add a new Run Configuration
Select Run > Edit Configurations from the Android Studio menu bar.
In the Run/Debug Configurations dialog, click the in the top left-hand corner and add a new Gradle configuration.
Add the appthwackUpload task to your new Run Configuration.
That’s it for Android Studio! Now it’s time to run our tests.
Now that our configuration is complete, we can kick off AppThwack test runs from Android Studio or directly through Gradle.
Select our Run Tests on AppThwack configuration from the drop down on the menu bar and hit play!
All Android Studio projects are packaged with the GradleWrapper, which gives us Gradle command line access from any platform.
$ cd /src/MyApp
$ ./gradlew appthwackUpload
On a successful run from Android Studio or Gradle, your output will look something like this.
[AppThwack] Using Project 'demoproject'.
[AppThwack] Using DevicePool 'Top 10 devices'.
[AppThwack] Uploading apk 'GradleTestApp-debug-unaligned.apk'.
[AppThwack] Uploading test apk 'GradleTestApp-test-unaligned.apk'.
[AppThwack] Scheduling 'junit' run 'GradleTestApp-debug-unaligned.apk (Gradle)'.
[AppThwack] Congrats! See your test results at https://appthwack.com/project/demoproject/run/24219.
Now you can kick off AppThwack test runs from your shell or IDE.
Going to Google I/O this year? We’d love to meet you! AppThwack will be demoing in the developer sandbox Wednesday through Friday on the 3rd floor in the Android section. It’ll be great to put faces with names, answer questions, and talk shop about test automation.
Our demo will cover how you can test your Android, web, or (shhh) iOS apps on over 200 real devices hosted in our state-of-the-art device lab. We’ll show you how reports generate in minutes, as well as how you can run your own tests written in popular mobile automation frameworks like JUnit, Robotium, Calabash, and more. Basically, we’ll show you how to save a bunch of time so you can focus on building great apps.
In addition to the demo we’ll have some cool new shirts and stickers showcasing our logo many of you seem to enjoy (designed by the amazing Lloyd Winters).
We’re super stoked to be invited to Google’s flagship conference! We’re even more stoked to meet you! Founders Pawel and Trent will be on hand Wednesday through Thursday along with the newest Thwackers (just trying it out) to join our team.
P.S. – We’re sponsoring a pre-I/O party on Tuesday, May 14th. It’s likely sold out at this point, but you can find out more at meatup.io. If you like nerding out about Android development and eating Brazilian barbecue this is pretty much the perfect event.
While you’ve been building great apps we’ve been hard at work making AppThwack your one-stop-shop for automated app testing on real devices in the cloud. We’re excited to announce that AppThwack now supports automated testing of Android, iOS, and Web apps. It’s clear that today’s reality involves developing customer experiences that transcend device, platform, OS version, and so on, so we’ve adapted AppThwack to better serve your needs. The speed, flexibility, and intelligent reporting you’ve come to expect has simply been expanded so you can ensure a quality experience for your customers regardless of which platform or device they’ve chosen.
Official iOS Support
The lab is now a little more Apple-y to the tune of 20+ devices and counting. Just as we don’t root Android devices, none of our iOS devices are jailbroken. They’re exactly what your customers are using and provide a mixture of hardware versions, OS versions, and carriers. Check out the device lab for the specifics.
Flexible Automation and Intelligent Reporting
There are many popular automation frameworks for mobile app testing, but no clear winner. In fact, it’s becoming more and more common to use specific frameworks depending on the type of test being automated. To this end we now support JUnit/Robotium, Calabash, and MonkeyTalk for Android and UI Automation, KIF, and Calabash for iOS. We take care of running your tests in parallel, parsing your results for high-level analysis, collecting logs, pulling screenshots, and tracking performance. The result is a consistent and easy-to-understand report that focuses in on the problem areas; we don’t just dump a pile of data on you and expect you to figure out what to do with it.
Fast. Very Fast.
One of our goals has always been to provide a fast and responsive service that saves you hours, if not days of time every test cycle. Our queuing algorithms have been revamped so you’ll get results as soon as they’re ready, even if some of the devices you’ve selected are busy running other customers’ tests. We’re also monitoring device usage and installing duplicates of those devices that are extra busy. You’ll rarely find yourself in a queue waiting for results, whether you select a couple or a couple hundred devices.
New Subscriptions and Passes
After tons of conversations and much thought, we’re moving away from tiering plans on the market popularity of devices like Top 10 and Top 25. We understand your app’s most popular devices may differ from the overall market, so rather than dictate to you what you can test you can now choose from any of the devices in our ever-expanding lab. Check out the new subscriptions on the pricing page.
In addition, we’ve done away with a la carte runs and switched to a more powerful day-pass model. You can buy 1- and 3-day passes with full access to our device lab in the cloud. For those of you looking to test just before a major release this is ideal.
AppThwack in Your Build System
It’s now super easy to kick off nightly tests via our API. Subscriptions Medium and up include the ability to schedule tests automatically through a simple RESTful interface. A detailed post is in the works, but for now check out the API docs for more info.
Just like you, we’re cranking away on new features and better ways to serve our customers. There’s a ton of stuff on the way, and if you have specific requests please don’t hesitate to hit up our ideas page and let us know.