While these tools are a great addition to any agile testing team, it’s still crucial to assess and understand the activity of your app on an actual device. This means fewer false positives and more complete, reliable coverage.
Additionally, there are specific situations where testing on a real device beats all other methods:
Testing for interruptions such as incoming calls and texts.
Testing the functionality of your mobile applications in various conditions such as bright sunlight, rain, and in both day and night settings.
Ensuring the application runs smoothly and the UI and UX are convenient for real users.
Stress testing including working for long, continuous hours and any battery issues that may arise.
Best operating systems to test on
There are a plethora of app options. And for good reason.
And watch for Apple’s 13 beta version announced in June. Available today, it won’t be available for public consumption until September.
If you use an iPhone, you are well aware of the “Software Update, upgrade now or upgrade tonight?” notifications. iOS devices tend to skew towards the latest OS version. Android OS support has a different situation due to the fractured nature of models and manufacturers.
9.0 Pie (18.61%)
8.1 Oreo (18.06%)
6.0.1 Marshmallow (15.16%)
7.0 Nougat (10.34%)
5.1 Lollipop (9.42%)
Android Q is expected to be released in August 2019. Compatible Android Q beta phones include all Pixel phones to the OnePlus 6T.
It’s pretty cut and dry what operating systems you should test on then, right? Not necessarily.
Considering Huawei has its own app store and with Android open source, it’s possible we’ll be talking about a third popular OS you should be testing next year. But that’s next year. So for now, we’ll still stick to Android and iOS.
Everyone knows that iOS downloads indicate an Apple device. However, thanks to the plethora of smartphones running on Android, it’s a bit more harrowing to figure out which device those users have.
Most app teams rely on analytics to track user actions and engagement. These analytics packages also track the device and operating system of users. You definitely want to adjust your device coverage to match your users; as they may tend to skew one way or another.
For example, corporate executives tend to use the newest devices on the latest operating systems, while teenagers might be using phones that were handed down from their parents. You should skew your device selection based on your target audience, and even better, your actual audience (based on usage analytics).
Next, trends are important – if you have a photo or video app, then it may behoove you to test on a device with the best camera.
But trends are not the end-all, be all and legacy devices and operating systems are as important as the latest gadgets. You should also consider tracking OS version analytics and support for different screen sizes.
Top 15 mobile app testing devices for Android and iPhone
According to the same data, the second most popular model currently for 2019 is the iPhone 6s
With 12.7 million units sold, the iPhone X was the top-selling smartphone in the first 3 months of 2018 and ended the year with 11.57 percent of the market.
Google Pixel 3
Launched in Q4 of 2018, the Google Pixel 3 has received rave reviews for its security, speed, and – of course – its camera. This was the first Google phone to break into the top 3 smartphones in the US market. It’s also a top 5 premium phone in Western Europe.
Samsung Galaxy S10
For the most discerning users, the Samsung Galaxy S10 – with a fantastic 6.4-inch display, the largest battery capabilities, and a triple-lens rear camera – is currently the best Samsung phone on the market.
Not released until late October 2018, the iPhone XR ended the year claiming less than two percent of Apple’s market. That said, Greg Joswiak – Apple’s VP of product marketing – said it was the company’s most popular phone every day since.
iPhone XS and iPhone XS Max
Samsung Galaxy S8 and S8 Plus
If global users is your goal, the Galaxy S8 and S8 Plus rank in the top 10 in most Western Europe countries. In fact, Samsung’s Galaxy series has consistently been a top contender globally.
Samsung Galaxy A5
Newer isn’t always better. The more affordable and just as popular Samsung Galaxy A5 is still widely praised for its performance and durability. At over 2 years old, it still maintains a top 10 position in most countries.
Samsung Galaxy S9 and S9 Plus
Though the Galaxy S9 plus has 2GB more than its smaller counterpart, Android phones generally use RAM to keep multiple apps easily accessible in the background. Ranking in the top 10 most popular phones in the US and much of Western Europe, it’s a good decision to have these phones in your testing arsenal.
Huawei P8 Lite
Having already overtaken Apple in market share in 2017, Huawei has been eyeing the position of the world’s second-largest smartphone vendor for a while. If you’re going to test on one of the Chinese maker’s devices – and you should – it should be this one as it currently holds steady as a top smartphone in 10 countries.
Xiaomi Mi 9
Xiaomi is one of the more popular Android device manufacturers. With one of the highest-rated cameras and a Tech Advisor award winner at MWC 2019, Mi 9 is one of the best phones in the world… and costs users a fraction of the price of its closest competitors.
These are just a fraction of the phones you could use to test your mobile app. If it seems daunting, Testlio’s mobile app testing experts have these devices in hand. Don’t approximate performance, contact us today and get 100 percent physical device coverage.
This highly-competitive supply and demand means you must ensure that the quality, usability, and security of your mobile app not only meets expectations but exceeds them.
This is the power of mobile app testing.
But it takes organization and planning to ensure you can iterate through the software development life cycle quicker and – ultimately – out to market sooner. Knowing the types of mobile app testing and their functions can help.
10 mobile app testing types
1. Functional Testing
Functional software testing ensures that the application is, well, functioning, correctly. This type of testing focuses on the main purpose and flow of the app.
In addition to the mobile app’s specific functionality, there are other scenarios one should test for to limit errors, including but not limited to:
The application installs and launches correctly
The users can sign-up and login
Text boxes and buttons function properly
Push notifications render correctly
Only 4 out of 100 unhappy customers will complain directly to a company — the other 96 will churn without providing feedback. Since it’s 6-7 times more expensive to acquire a new customer than keep an existing one, unlocking that silence is key. – thinkJar
2. Usability Testing
Known as user experience testing, usability testing checks how user-friendly the app is in terms of ease of use and intuitiveness. Ideally, usability testing revolves around the entire app-driven customer experience with insights that include the identification of bugs and recommendations for ways to improve the customer experience, both in and out of the app.
Engineers, marketers and product people all want to test whether or not the end-to-end “app-driven” experience is world-class. To that end, it’s important for app usability testing to be done with real people, on real devices to quickly identify and fix usability issues prior to app release.
This type of testing is more art than science and requires skilled usability pros and facilitate the tests and capture insights from QA testers that mirror actual users or customers of the app.
Because usability testing is subjective, you should understand your target end-users and their preferences. Consider asking them to test the product themselves.
Other best practices for usability testing include:
The thoughtful design of usability test scripts and feedback questionnaires
Incorporate usability questionnaires within test cycles so testers understand the usability testing instructions, can access the online questionnaires and can provide feedback as part of their testing tasks.
Analyze results and create feedback summary with actionable insights and recommendations for improving the overall customer experience
3. Compatibility Testing
Compatibility testing is a type of non-functional testing and is critical as it ensures your mobile app works on various operating systems, a plethora of devices and applications, network environments, and with particular internal hardware specifications.
Specifically, you should know if:
The app is compatible with different operating systems and their various versions (iOS, Android, Windows, etc)
The app performs well with varying networks and their parameters (bandwidth, operating speed, etc)
The app is compatible with different browsers (Google, Firefox, Safari, etc)
The app is compatible with different devices (screen size, data storage, etc)
There are also two types of compatibility testing to consider:
Backwards: testing the mobile app behavior with older software versions
Forwards: testing the mobile app behavior with new — including beta — software versions
From Tinder to travel apps, some applications ask for user’s personal information. If yours does, too, you absolutely must guarantee confidentiality, authenticity, and integrity of the app.
6. Installation Testing
Installation testing is used to check the mobile application is installing and uninstalling properly.
Additionally, installation testing ensures updates are also uninterrupted and error-free. This includes understanding what happens if a user doesn’t update an app.
7. Localization Testing
Consumers routinely skip past apps that don’t consider graphical or UI elements aligned with their culture, language, or device accessibility. And when an app attempts to translate, it may sound awkward to a native speaker.
At the same time, localization testing continues to be a challenge as half of all QA teams lack access to the resources needed to test localization.
When understanding your end-user, you must consider the cultural and geographic aspects of your audience. For example, consumers routinely skip past apps that don’t consider graphical or UI elements aligned with their culture, language, or device accessibility.
From translating in multiple languages to converting to local currencies and adhering to local regulations and legal requirements, it’s important to ensure the app is accessible and usable in a wide variety of markets. Equally important to check that localization efforts haven’t caused new breaks.
Mobile app testing is a complex process. Sometimes, real human experience can deliver the results you want.
QA teams use manual testing to ensure that the final product really works as intended. With a specific role to play, manual testing is used to explore use-cases that may not be all that obvious.
Additionally, we simply can’t automate some types of tests… and shouldn’t. These include:
Physical interface tests
9. Automated Testing
As we’ve pointed out before, there are some cases where manual testing is the better option. However, some QA tests are either too tedious or too complex for human testers. That’s why many apps are now practicing automated testing, executed continuously and alongside manual tests, to assure quality and release better products, faster.
A few automated testing best practices and challenges include:
The thoughtful design, build, and maintenance of accurate test scripts
The alignment and integration of existing engineering workflows with your automated testing process
The creation and maintenance of your test automation framework, including infrastructure
The management of test runs and setups
Rigorous reviews to validate test results and defects
Careful monitoring and rapid response to noise and flakey tests
10. Mobile-Device Testing
Mobile apps would not exist without hardware and operating systems. So, we also need to think about mobile-device testing. There are several testing types specific to mobile including:
Interruptions – Interrupt testing evaluates how an app reacts to interruptions and if it resumes to its prior state. Common mobile app interruptions include loss of battery power, in incoming phone call or text, notifications, and app updates.
Location-based Services (LBS) – Using geo-data from a mobile device, location-based services provide real-time information, entertainment or security. They are also used by consumers to “check in” while experiencing life on the go, like a visit to the local Starbucks or while attending a concert.
Biometric – Mobile devices often include biometric sensors that include face recognition, fingerprint and hand geometry, iris recognition, and even DNA or insulin levels.
NFC payments – Near Field Communications (NFC) allows mobile devices to communicate with a payment terminal enabling contactless payments.
The result of intelligent effort
As author John Ruskin exclaimed, “Quality is never an accident; it is always the result of intelligent effort.”
Now that you have a better understanding of different mobile application testing types, it’s important to create a plan of action. If you’re ready to put in the effort and take your testing to a whole new level, schedule a quick demo today.
The goal of any mobile product is to create one that’s innovative and new. But there are specific, necessary steps you must accomplish between crafting a clear vision for your app and creating a mobile application.
As explained in the Step-by-step mobile application testing process, it’s imperative to understand and resolve any requirement contradictions before finalizing the development phase. The decisions you make about functionality, compatibility and data carry long-term ramifications if they aren’t tended to purposefully.
Your 9 Step App Testing Strategy Checklist
Step 1: Cross-platform testing
Sometimes extra fixes are required if your app is interacting with others.
Gathering and understanding project requirements, business objectives, various language platforms, and user needs is essential to create an appropriate cross-platform testing strategy.
Step 2: Feature functionality
Mobile apps usually interact with a number of features – both built into devices and built into the app. These interactions should be noted and thoroughly tested.
It’s not necessary – and not advisable – to run functional testing across a plethora of mobile devices. Instead, test on a single device and then conquer various platforms during compatibility testing.
Step 3: Type of application
There are three main types of mobile apps: native, mobile-web, and hybrid.
Mobile-web: The website opened in the device through a web browser.
Native: The application is developed specifically for individual platforms
Hybrid: A mix between native and mobile-web applications.
Before you can create an effective testing strategy, you must understand the pros and cons of these three application types and how yours will be built.
Step 4: Front-end testing
Front-end testing checks anything that is visible client-side also known as Graphical User Interface (GUI).
Testers need a solid grasp of business goals to perform this type of testing.
Other things to check for in front-end testing include:
Changes or updates to app files that might break front-end functionality
Step 5: Back-end testing
Back-end – or database testing – checks the server side of your mobile app. Anything that is entered and/or stored in the front-end is tested in the back-end.
This is also where you check the security and performance of the mobile application.
The ever-growing popularity of smartphones and IoT devices has led to an explosion of different brands and platforms. While it’s impossible to perform every test on all possible devices, mobile compatibility testing is indispensable.
This process should include tests such as:
Install and uninstall
Be sure to test different versions of the same major hardware platforms including iOS and Android.
Step 7: Storage
Today’s mobile devices don’t have enough storage for the vast amounts of games, music streaming services, and hi-res photos competing for space.
From how much data your app requires to how this might affect monthly data plans, keep these limitations in mind during your mobile app testing.
When software developers and testers get together to build a new feature, it’s easy for them to get bogged down in the details. I once worked on a project that would make your phone flash rapidly while taking pictures. It looked like the paparazzi was after you. It was great, and it didn’t work. The reason it didn’t work, was because it had only been tested in normal lighting. By the time it made it to the wild, it was quickly realized that the flashes didn’t match up with the taking of pictures. Cool scene in a club? Doesn’t work. Taking a candid set of shots in a dim restaurant? Nope. Wedding parties? Sorry.
The thing is, everything was tested correctly. It passed all of the mobile QA tests we gave it, but once it gets in the hand of the user, your original idea is no longer yours. Users will use what you’ve built the way they want to and you need to adjust as new use-cases come up.
If you’ve ever used a butter knife to tighten a screw, you know what I’m talking about.
Using an iron in college to make a grilled cheese? Guilty. But that led to the invention of the sandwich maker. It’s very difficult to realize what will happen with your product or feature when it gets into the hands of the general public. That’s why the software development life cycle (SDLC) requires a more agile way of producing what the user wants.
QA to follow suit
QA testers need to look at what is happening to a product and adjust their tests accordingly. By asking for feedback from actual users, testers can better predict which ways their product is actually being used and adopt future test for additional features. A new slimmed down version of the butter knife? We should probably test to see if it will still work as a screwdriver. Not because it was intended to be a screwdriver, but because that’s how users are using it and it’s what they’ve come to expect.
If it isn’t broken, don’t fix it
In a nutshell, the ultimate goal of qa software testing is to keep your product from annoying your users. Sounds simple, but as I’ve pointed out, you won’t catch every use-case. So what should you do? How can you be more proactive in your approach to agile QA testing and design?
Bring your QA team in early! I know I say this a lot, but that’s because it’s so important. Most development teams start testing late in the process, to the frustration of QA managers. The QA perspective is worth its weight in gold and if you make the QA team a part of the requirement reviews or the designs, you’ll be able to spot potential problems at a much earlier stage. Developers also have the added advantage of getting extra eyes on the project. This ultimately leads to a much better final product and a better experience for users. By implementing test cases earlier – before the code is completed – you’ll be able to reduce costly fixes to the code.
So what is QA testing? The answer isn’t as cut and dry as you may think.
Traditional software QA testing is the process of designing a set of tests that need to be executed to prove the code does what it says on the box. The problem is, with QA teams usually coming in late, they are often forced to use a subset of the tests available. Teams then determine the most critical cases to test to get a result.
Proactive test design starts on the other side. Actively prioritizing risk and building tests from the most to the least risk. Instead of writing test cases late in the process and then testing and sorting based on risk, the new, and arguably better way, is to analyze risk first and work down.
If you’ve worked in software development, you know when you hit a showstopper bug. These are the major ones that can kill a users good-will and enthusiasm for your product. They can render your software useless.
Traditional requirements testing will only test the design. Often missing showstoppers in the process. If you miss it in the requirements, by design, you will miss it in the testing phase. Proactive design looks at the risks, and designs test accordingly. They are also implemented much earlier, so there’s a better chance that you’ll spot the design flaws before they become an expensive fix.
The new norm in testing is to automate aggressively. It makes sense from a time-management perspective. Automated testing allows teams to increase testing cadence and volume by avoiding repetitive manual task load that piles up each release, in turn saving vast amounts of time and money, and getting to market faster. These benefits have us looking for more and more opportunities to automate, and can even create pressure to automate “everything”.
Many times, a trained human eye is the best way to get the results you want. The art of mobile app testing is complex. Despite the many advantages of automation, humans are still the best at manually evaluating the vast number of bugs and errors that inevitably crop up. There are many specific cases where automated testing might not be the best option. So how do you know where that line is?
The increasing prevalence of the Internet of Things (IoT) brings the real world into software testing more often. Fitness trackers, health monitors, voice-activated devices, automotive diagnostics, and media players are all used to interact with the real world, making their interactions difficult to simulate with automated test scripts.
User interactions with a physical device vary from person-to-person. Things like the force one uses to press a button, or the duration spent holding it down, varies wildly. Plus each individual will have a different perception of the usability. These are all difficult factors to program into a script.
Any type of manual manipulation of physical objects should be tested by people, at least for functional testing. Of course, if you are testing the durability of a button, you don’t want a person to press it 10,000 times (perfect for a robot). For most functional and usability testing with software that interacts in the real world, manual testing can’t be beaten.
When you’re writing scripts for automation, I’ve often said to keep it simple. In fact, “hermetic tests” or “atomic checks” are considered a test automation best practice. Writing elaborate scripts, especially scripts that span multiple platforms, takes far more time. And these scripts are prone to flakiness.
By breaking up complex testing into shorter scripts, you can then re-use those scripts on later tests. It increases the shelf-life of your existing scripts with minimal maintenance. So, before creating an elaborate end-to-end system-level automated test, try to see if simpler tests can help the team be more effective.
Testing across Apps
Switching between apps during testing can be difficult in most test automation frameworks. For example, imagine testing a marketplace app, where a buyer & seller each have to take some action in their respective apps to complete a transaction. This scenario can be automated, but often the investment to make a repeatable test is pretty high.
If you’ve ever changed your profile picture on Facebook and then used that same picture on Twitter, you’ll see why this has to be done manually. You’re interacting with different platforms, each with their own parameters. In order to ensure that your media works correctly with each, you’ll need to test manually to find the optimal format. Media formats and display surfaces vary widely, so it’s much easier and more cost-effective to test for each format manually.
The human eye is still the most effective visual processor on the planet. Once you’ve established a visual baseline, there are tools that help check that your UI is consistent across different platforms – but it’s the human that needs to establish that baseline.
Very early in your product’s cycle
You have a great idea for a product (or feature in an existing product), so you build it. In the very early stage, you believe its a great idea but you don’t know if your users or customers share your enthusiasm. In this early phase of the product, it’s most important to get your product into the hands of some customers and get their feedback. In feature/product ideation and growth phase, your product is about to change very often based on rapid feedback from the customers or by the team itself. By introducing automation testing investment in this phase of the SDLC, you will probably double the engineering cost to areas that will change drastically – would you be ok to maintain changes or even refactor on weekly basis your feature and automated test scripts simultaneously?
Exploratory testing is usually the optimal approach to rapidly test the app. Another factor is that you are usually enhancing the product rapidly based on customer feedback, and keeping an automation suite up to date with a very dynamic design is expensive. Your test automation strategy may benefit by waiting for when the product is more stable and regression testing becomes a factor.
Say yes to a combination
It is not about manual versus automation testing, but instead, it is a question of IF and WHEN to automate a test case. In many instances, a manual test makes more sense due to cost or testing cadence, but that doesn’t mean you should avoid test automation. Both manual testing and automation testing are investment decisions, therefore ROI should be kept in mind when building combined strategy. There are many scenarios where automated testing is preferable for stable and routine testing tasks. It can be a logical step forward from manual testing. By combining manual and automation into single strategy teams can experience an increase in overall test coverage by the number of tests and devices used in a single test session while keeping costs relatively low.
A nice blend of both automation (to increase coverage and depth of testing without increasing overall costs,) along with a set of manual tests (for more complex and exploratory testing), is the best way to achieve engineering excellence that improves the end user experience.
If you’re ready to take your testing to a new level, contact us today for a demo. Testlio leads the way in hybrid manual and automation testing services. We’d love to talk.
The new norm in testing is to automate everything. It makes sense from a time-management perspective. Teams are able to quickly test new scenarios, sometimes even using old scripts, in turn saving vast amounts of time and money.
But just as in development, a trained eye is sometimes the best way to get the results you want.
The art of mobile app testing is complex. Despite the many advantages of automation, humans are still the best at manually evaluating the vast number of bugs and errors that inevitably crop up. There are many specific cases where automation shouldn’t or can’t be used. So how do you know where that line is?
Any type of manual manipulation of physical objects should be tested manually. User interactions with a physical device vary from person-to-person. Things like the force one uses to press a button, or the duration spent holding it down, varies wildly. Writing a script to test all of the variables, would be difficult and should be avoided where possible. It should be tested manually.
When you’re writing scripts for automation, I’ve often said to keep it simple. Writing elaborate scripts takes far more time. It may also render your results too difficult to understand. By breaking up complex testing into shorter scripts, you can then re-use those scripts on later tests. It increases the shelf-life of your existing scripts with minimal maintenance. This also goes for new features, where a script won’t understand the changes.
When you need to switch between apps during testing, you may struggle to automate your process. Nevermind that you’re actually moving between two different pieces of software. If you truly want to see what a user would experience when switching between apps, these tests need to be done manually.
If you’ve ever changed your profile picture on Facebook and then used that same picture on Twitter, you’ll see why this has to be done manually. You’re interacting with different platforms; each with their own parameters. In order to ensure that your media works correctly with each, you’ll need to test manually to find the optimal format. Media formats vary widely, so it’s much easier and more cost-effective to test for each format manually.
Unit Testing is killing your developers
Unit testing is usually done by developers because it involves scripts. Developers tend to make these scripts the last item on their to-do list. Scripts are essential to automation and should be taken seriously in both QA and development teams, but they usually aren’t.
If you can’t find enough slack in your development team to write scripts for testing, you can choose to do manual testing with your QA team instead. Many QA teams are learning basic programming skills to help take some of the burdens from the development teams. Look at your team’s skill set and decide if it’s something you can do.
Say yes to a combination
In these instances, a manual test is obviously better, but that doesn’t mean you should avoid automation completely. There are many scenarios where automated testing makes complete sense and will actually help your team achieve more in a shorter amount of time. If you’re aware of what type of testing should be done in each situation, it will help in the decisions of your team and reduce failure rates.
A nice blend of both automation for speed and cost savings, along with a set of manual tests for the more complex testing, is the best way to go.