How to Manage Your Mobile Test Device Inventory
One of the greatest challenges in mobile app testing and website testing is assembling the right device/OS mix to make sure that your verification effort matches the installed base of your audience. Testing against every possible combination is not practical as the vast majority represent only a trivial percentage of your installation base. Your best approach is to study your analytics to pare the list down to a more reasonable array of equipment. This places a particular burden on QualityLogic’s efforts to provide a wide variety of devices and operating systems for our customers.
QualityLogic strives to provide effective coverage of devices and OSs in support of a broad testing capability. To maintain the best possible, and yet practical, coverage, we have evolved the following policies:
iOS (Apple) Device Inventory Policies
- Coverage is based on the various iOS versions.
- Taking any device to a previous iOS version is not possible nor is it possible to upgrade a previous minor version (i.e., an iPhone5 running 6.1.1 to 6.1.3 once 7.0.x is released). Apple wants you to upgrade your device to the latest new major version. This makes it critical to actively manage your inventory of devices and iOS versions.
- Our principle effort is directed toward providing coverage of at least one iPhone and one iPad device for each major iOS release (a major iOS release is defined as 6.x.x, 7.x.x, 8.x.x, etc.) and cover minor releases only for the current version. For example, just prior to when iOS 7 was released, all iOS 6.0.x and 6.1.x devices were updated to the most current version, leaving at least one device behind on the most recent version of 6.1.x. QualityLogic does not attempt to cover all minor releases (6.1.1, 6.1.2, 6.1.3, etc.) for non-current major versions.
- As soon as Apple releases a new device, which up to this point also includes a new major version of iOS, QualityLogic purchases the new devices with the latest version of iOS. This allows us to support our customers’ needs by testing their software on the latest devices.
- QualityLogic also participates in pre-order programs for new devices, ensuring that the newest devices are available for testing at the earliest possible moment. An example of this includes the release of the iPhone X. QualityLogic received our first iPhone X devices as soon as they were released.
Android Device Policies
- Android test requests come in two different varieties: specific device driven and OS version driven.
- To maintain coverage, QualityLogic purchases the most popular devices, as well as devices specifically requested by customers, that support the most popular Android OS releases. The intent is to have at least one popular phone and one popular tablet covering each major Android OS version.
- The Android market tends to adopt new OS revisions much more slowly than iOS revisions. QualityLogic attempts to maintain a balance of offering new device models with the newest Android OS as well as upgrading some existing test devices to the new OS.
- Android updates are handled in the same manner as Apple updates, updating most of the devices for each new release, leaving at least one device for each major prior release.
The resulting device/OS mix has been remarkable in its coverage of our customers’ test requests. While we occasionally need to purchase a device to meet the requirements of a specific test program, we typically find that our customers are very pleased to have access to this equipment array for their projects.