TestCraft is a test automation platform for regression and continuous testing, as well as monitoring of web applications. Our revolutionary AI technology eliminates maintenance time and cost as it automatically overcomes changes in the app.
At TestCraft, we take QA very seriously. With the proper team, tools, and processes, we believe that QA is essential to delivering high-quality software that propels businesses forward.
Whether you’re just starting out in QA or have been in the industry for many years, there are some terms that you will need to use as part of your everyday vocabulary. Here are some of the most important QA terms you should know about:
Manual testing is the process of testing software physically, without the assistance of other tools, to find bugs or defects. A manual tester assumes the role of an end user, mimicking different ways they would use the software to make sure everything is working properly.
Manual testing is an essential part of the QA process for multiple reasons. First, manual testing brings a human element to software testing. The software can have certain quirks that, while technically correct, can seem strange to the human eye. Pointing out these issues is essential to creating a good user experience. Second, testers can use manual testing to bridge the gap between software development and business objectives. While developers are responsible for creating the software and its functionalities, testers are the ones who normally have a better understanding of larger business goals. This business knowledge will help inform manual tests that will prioritize what’s most important to the business.
Another important term that is related to manual testing is “exploratory testing”. This is one major area where manual testing can make a large impact. In exploratory testing, testers use their experience with the software to innovate new and better tests that will support the software and move the product forward. With exploratory testing as the main objective, manual testing is something that will drive software testing forward.
Unlike manual testing, test automation uses tools to run software tests without human assistance. The automation can range from running the same tests on different browsers and in different versions, to running complicated scenarios to make sure everything works properly.
As companies are looking to achieve shorter and more frequent software releases, test automation speeds up this process by taking care of more mundane and repetitive tasks. This not only saves time and resources but also gives testers the freedom to run more creative (and generally more enjoyable) tests.
There are many areas where test automation can be helpful, but it’s important to note that test automation is not a blanket solution for doing faster and more efficient software testing. Test automation is something that QA teams need to adapt strategically in order to get the most out of it. QA teams need to factor in things such as project size, the amount of maintenance necessary and how much data you need to test.
In addition, QA teams need to factor which test automation tools to use in order to run these tests. While some companies may be happy to work directly with testing platforms like Selenium, others may find that codeless test automation solutions are a better fit for their team’s needs. Yet when applied correctly with the right tool, test automation can do a lot to shorten release cycles in a short amount of time.
One type of testing that is a great candidate for test automation is continuous testing. Continuous testing is an ongoing testing process that happens in each stage of the software development lifecycle to get feedback on changes as quickly as possible. This happens by embedding various types of software testing (unit, functional, non-functional) at every stage of software development.
This type of testing is very effective when building quality software. It allows QA teams to be more proactive with integrating testing into the DevOps pipeline. By automating and testing these scenarios as much and as early as possible, it also reduces the risk of finding vulnerabilities post-release.
Not only does this push teams to create better software, but continuous testing is also a cost-effective option. It costs companies much more to fix tests later on in the development cycle. In fact, it costs 6 times more to fix a bug during implementation than during design.
Regression testing is another type of testing that benefits from automation. When companies release new code, regression testing ensures that changes in code do not affect previously created test scenarios.
Instead of spending time going over old code after a new release, regression tests make sure that everything is working as it should when these changes happen. This can be done either manually or through test automation, but this task is often automated because it’s considered more routine.
When companies are looking to automate regression tests, there are two types of tools that do this: record-and-playback tools or codeless solutions that offer test modeling. The major benefit of record-and-playback tools is that they are simple to use. All someone needs to do is record a tester’s actions and then play it back to make sure that everything is working properly. Where record-and-playback tools fall short is when it comes to running regression tests after changes. When there is any change in the flow, the test needs to be recorded all over again.
To overcome the challenges presented by record-and-playback tools, there are other solutions that allow for creating a model of the test instead of a recording. With a test model, QA teams can add or change steps within the test flows, even as they’re running. This makes automated regression testing significantly easier and much less time-consuming.
When anyone discusses software testing, one of the topics that are sure to come up is how to do proper test maintenance. This means that when changes happen to your software, QA teams need to keep their tests up-to-date in order to make sure that it runs properly.
Test maintenance is considered one of the larger bottlenecks associated with automated testing. Tests that aren’t maintained properly are prone to breaking easily, which means that this is something that QA teams need to stay on top of in order to ensure effective testing. It is a time-consuming process where testers often need to go back into the code, find the issue that made the test break and make the proper fixes in the hopes of it not happening again.
To increase test stability, there are test automation tools that use artificial intelligence to take care of a lot of areas of test maintenance. Using AI can be especially helpful when you want to make sure that the test scenario identifies the right element while it is running, even after there are changes within the element itself (e.g. a login button that changes color, a free trial button that increases in size, etc.). These tests are not changing in logic; rather they are testing software that has changed slightly from the original code.
While this may seem small, the number of tests that can break from these changes add up over time, taking away time and resources from your team. The AI-based technology found in leading automation testing tools can overcome these changes with incredible accuracy, making sure that your tests are giving you the most value through their stability and repeatability.
DevOps is one of those terms that people in the QA world will often say and use, but not know exactly what it means. In his lengthy discussion of this term, Ernest Mueller of The Agile Admin defines DevOps as “‘Agile Software Delivery and Operations,’ which should similarly work in concert with others working on larger organizational initiatives, but without losing sight of its primary value proposition for the organization.”
Essentially, it provides a set of principles and best practices for teams (development and operations teams, but this extends to other teams in IT as well), to do their jobs better and faster. It offers a framework for which teams can release faster, adapt more quickly to changes and align more closely with larger business objectives.
DevOps and QA are often working side by side, but these two teams can often feel at odds as QA works to function in as agile a manner as the dev team. Things such as stronger communication, using automation where it’s necessary and creating for opportunities for collaboration are great ways to bridge this gap.
Future-Proofing Your Skills and Humanizing AI in Our World of Tech
It’s an exciting time to be in the test arena. Manual testing seems to be going by the wayside as automated and AI testing are coming to the forefront. The traditional lines between development and testing continue to blur.
What’s a tester to do to stay relevant, engaged, and employed in this ever-changing landscape? I’m so glad you asked!
Test Automation and Human Impact
There currently are and certainly will be more facets of testing that will be automated, especially the more monotonous and repetitive aspects. To be clear, manual and beta testing are not going away for the foreseeable future. The good news is there are aspects of testing – including collaborating with customers and internal stakeholders – will increase in both importance and urgency.
The skills we should cultivate – and really it’s about “re-learning” – are those that humanize the role of beta and test management. In the unfortunate trend of the past decade or more, the increasing emphasis on programming languages and skills has resulted in the lost art of human-to-human communications. That’s not just a tech company problem, it’s prevalent in nearly every Fortune 100 company that I’ve advised over the years.
Instead of feeling nervous, we should be excited! One of my missions is to humanize the role of testing and add greater value to what we’re doing. The net result of these changes, most of which is already happening now, is incredibly positive. If you strip away what AI and machine learning will inevitably become better at than humans, what’s left over is an even larger opportunity that beta and test managers must grasp onto. The impact of humans plus AI will far exceed what has been previously accomplished.
It all comes down to the roles and responsibilities that humans have over machines. Most importantly it starts with empathy, which enables us to understand our users and customers at a strategic level. Humans are also better at critical thinking, contextualizing problems, and collaborating across people, technology, and processes to drive success. Developing these skills is critical for continued success in the testing arena.
How Humans and AI Can Work Together
I know this all sounds daunting. Let me give you a real-life example:
I was recently at a conference in Las Vegas and noticed that Uber now allows you to opt into a self-driving car. But because self-driving Ubers are not thoroughly reliable and proven yet, they put two people in the front seats; one is an operator and the other is an engineer that makes real-time fixes. So as this program evolves, it needs guidance and assistance from humans. It takes both humans and machines, interacting in real time and with increasing amounts of data, to optimize the systems. You don’t need to change your personality; you need to enhance your skill set.
Technology has in part led to the deterioration of our communication skills – text, Messenger, Slack, Twitter, and more. Therefore, we need to bring back the human element of communicating; emotional intelligence, effective listening, body language, and situational awareness. For example, pick up a book or research online into body language. Know how to interpret body language; find out what people are telling you but not telling you. These are skills we don’t often learn in school yet are critical in the way we communicate and collaborate.
Become a knowledge worker. Instead of coming in every day with a set of tasks, come to the table with knowledge. The way to do this is by becoming a continual learner. Technology is changing so fast that what works today may not work tomorrow. So, get used to doing your own research, talking to people who know more than you, finding mentors, and learning how to communicate more effectively.
Keep Testing Agile with Critical Thinking
Use critical thinking skills to understand user profiles at a deeper level – who they are, what they want, when and where they use your product, and how they buy. It also involves assessing how your product stacks up against competitors, as well as using critical thinking and analysis to create product differentiation and advantages. As testers, if we can provide granular insights to internal teams about users and competitors, this is guaranteed to help drive better decisions amongst our executives.
Not only that, but also look at the holistic, end-to-end customer experience across the customer journey of pre-purchase, purchase, and post-purchase. Put yourself in your customers’ shoes and understand the true problem they’re facing. Once you’ve mastered design thinking, you can iterate quickly and validate quickly. In other words, it enables you to become quicker at the fails, which pushes the organization towards greater velocity and constantly delivering greater value for customers.
At the end of the day, technologists with consultative skills are much more effective at their jobs – they communicate effectively, contextualize for relevance, think critically, solve problems, and adapt dynamically to different situations. Time and time again I’ve seen two testers interviewing for the same position, and inevitably the tester with the better human understanding and contextual empathy for the problem that the client needs to solve wins the position. All things being equal, and in some instances, not equal; a lesser skilled tester with the more empathetic and humanized view wins out over a higher skilled tester with poor communication skills. Another important skill is learning to fight the attention span deficit by using more visual tools to tell a story and break through the clutter. If you can do all of that, no matter what tech disruptions are on the horizon, you will be valuable to any organization.
Driving QA Transformation in Agile Organizations [Webinar Recap]
Last week, we had the exciting opportunity to host a webinar with Tanya Kravtsov, Sr. Director of QA at Audible and founder of the DevOpsQA NJ meetup group. In her webinar, “Breaking the Bottleneck: Driving QA Transformation in Agile Organizations,” she discussed different strategies for overcoming bottlenecks that prevent QA teams from becoming more agile.
As companies move from waterfall to agile methodologies, Tanya stressed that QA has been wrongly accused of preventing this agile transformation from happening. Speaking from her own experiences with driving QA transformations in agile organizations, Tanya offered insights into how QA teams can identify and alleviate true bottlenecks in the software delivery lifecycle. She recommended a wide range of tools and techniques to help with this daunting task, offering innovative ways to achieve the best transformation for your company. We at TestCraft are also passionate about this, as a test automation tool that can help identify and address bottlenecks early in the development lifecycle.
Addressing the Bottleneck: Switching from Waterfall to Agile
Tanya started her talk by giving an exact definition of a bottleneck, to help everyone better understand what is necessary to transform their QA teams. She took a quote from The Phoenix Project which states, “Any improvements made anywhere besides the bottleneck are an illusion.” Essentially, anything solved before or after a bottleneck may fix an issue temporarily, but it will not provide a permanent solution that will really allow a QA team to scale. For Tanya, this continuous addressing of bottlenecks is necessary for a full QA transformation.
“Anything solved before or after a bottleneck may fix an issue temporarily, but it will not provide a permanent solution that will allow a QA team to truly scale.”
She then went on to explain what a QA transformation actually entails: transitioning from a waterfall methodology to an agile methodology. In a waterfall environment, developers generally do one or two releases per year, while QA generally takes a month or two afterward to test. Speaking from personal experience, she explained that this causes an environment where QA is a bottleneck. Either QA needs to compromise on the quality of testing that they do, or they don’t test within an acceptable timeframe.
In an agile environment, QA is able to function a lot differently. Instead of dealing with one to two releases per year, QA is testing releases that can come out weekly, if not daily. With this new reality, QA has been transforming themselves as well to fit this new mold. Different ways QA has started to do this is through risk assessment, compromising on which tests are most important, and resigning themselves to the fact they will release code with issues in order to get feedback as quickly as possible.
Differences between agile and waterfall methodologies. Source: CRMsearch
The Different Components of Being Agile
Tanya then went into different areas where QA can address bottlenecks in order to integrate better into their newly agile environments. She broke it down into five components, under the acronym CALMS:
She then delved into each area in more detail, focusing on different strategies that teams can do in each component to achieve a full QA transformation. As an example, when discussing “Culture” as a QA bottleneck, Tanya emphasized the importance of understanding all the different players and handovers in the delivery pipeline. She then recommended strategies such as the Speed Boat Game and other collaboration games, as well as running retrospectives to better understand and learn from different teams within the delivery pipeline. The only way to achieve true change is if everyone is through this type of understanding.
Other strategies she gave focused on different automation tools available to make QA more of an agile process. Of those who attended the webinar, 48% of them said their environment was their biggest software delivery bottleneck, while 36% attributed the bottleneck to testing. For environment-related issues, Tanya recommended tools like Chef or Ansible to help document everything related to environment creation, combined with a virtualized environment. For testing-related issues, she recommended not only test automation tools, but also service virtualization and test data management solutions that help make testing processes faster and more efficient.
“48% of webinar attendees said their environment was the biggest software delivery bottleneck, while 36% attributed the bottleneck to testing.”
Focus on QA Problems to Get Better QA Solutions
No matter which tool you use, Tanya’s biggest piece of advice to close the webinar was to focus on the problems that stand in the way of a QA transformation, not the solution. There are hundreds of tools for every problem, and she did recommend many different tools to use in an agile environment. Yet she also reminded everyone that your team’s needs will change over time, and with that the tools you will need to help you can change as well.
As long as your team focuses on the problems that prevent QA from integrating fully into the SDLC, they will have the opportunity to go through an agile transformation. Focusing on these issues in-depth and in their entirety will ultimately be what allows QA to help deliver software features constantly and more quickly.
Q&A: All Things QA, Agile and Machine Learning
At the end of the webinar, Tanya answered questions from the audience. For your convenience, you can read the questions and her answers below:
Theintroductionof smart testing vs. AI testing. Is smarttestingtool-dependent?
Smart testing can start out as a manual process by determining the test scope based on the scope of changes, associated risks and the time available for testing. Leveraging tools including AI will allow you to scale that process and potentially turn it into self-service, reducing the dependency on QA teams and further enabling shared ownership of quality.
How does QA factor into release management and CI/CD?
In agile organizations, QA is a critical part of the CI/CD and release management processes. The only way to ensure the reliability of CI/CD is to embed quality at every step and continuously monitor the outcomes. While traditionally, operations teams own the release management process, this can often shift to development or QA in agile organizations. A big advantage of the QA team owning or at least overseeing the release process is that it gives them an end-to-end view of product delivery pipeline. It also allows them to easily identify and address bottlenecks.
How can QA take advantage of machine learning technology?
Machine learning technology can prove a valuable tool for every phase of Quality Assurance. It can be leveraged for a more robust and reliable automation framework that requires minimal maintenance. It can also be used as part of your automated risk-based analysis to determine the scope of testing, as well as help out in troubleshooting and preventing defects. Whether or not you are using machine learning technology today, my advice is to start capturing baselines and collecting all relevant data upfront.
Continuous testing aims to identify business risks associated with code changes and, such testing is triggered through Jenkins or other schedulers. This is not a new statement.
In addition, and again not a new fact, continuous testing is enabled through mature, robust, and reliable test automation. The problem that arises with these practices that are required and critical for maturing DevOps is: “how to ensure that these test cases are still valuable?”
Which Tests Should I Automate?
This is one of the most common and old questions that practitioners and test managers often ask. This question is related to the value of the tests as well as the ROI that comes from developing and executing these test cases.
Before addressing the above question, it is imperative to understand that whatever test scenario an engineer develops and adds to the regression suite/CI, is an additional weight or load from a test maintenance perspective. When done continuously, such load rises and might add delays, flakiness and additional resources like devices/desktop browsers to meet strict test execution schedules.
With the above in mind, it is important to go through the following or similar exercises when deciding which tests a team and individuals should automate – the time spent on such “justification” will pay itself in the long run.
The following practice takes test scenarios that are manual and can be divided into functional areas, and “questions” each of them from the following criteria:
As the above categories state, Gut feeling stands for the initial aim to automate or not to automate a given scenario. Risk focuses on the probability of such a failure to occur, as well as the impact of when it does. Value in my mind is one of the most critical aspects and addresses the question of whether this test scenario adds new information and contributes to risk identification. Does it come as a duplicate of other unit or functional test cases? Cost relates to the time it takes the engineers/developers to automate, maintain and the ability to automate it as end-to-end scenario (efficiency). Lastly, History relates to the volume of historical failures around this functional area or feature – this should be a data-driven decision with defects of historical evidence.
Fig 1: Decision Table for Test Automation Scenario Selection. Source: Angie Jones
Test Automation Aging in Continuous Testing
While we defined earlier in this blog which tests should be automated, this is not a static definition, and cannot be taken as a “once and for all” decision. The product and the platforms on which the product runs evolve over time, and the tests should also follow this evolution.
Test maintenance practices are essential to ensure that test automation continuously adds value.
Executing test automation within CI has a purpose: provide value, identify business and quality risks, and to meet this objective, these tests must always stay updated and relevant.
It is a fact and common knowledge that after the execution of the same test code too long without this test, finding new or any defects is a waste of time and resources. As a matter of fact, practitioners often add layers of exploratory testing to mitigate this reality and add “out of the box” testing to identify new and hidden defects. In addition, when executing too many tests at scale and some percentage of them are invaluable, the level of test data that is growing adds time at the end of the cycle for test report analysis and debugging. This is redundant and unnecessary noise.
Test automation code should be treated and maintained as production code, and that means, refactoring, updating and yes, retiring obsolete test scenarios to continuously add value through testing.
Best Practices in Test Automation and Continuous Testing
To address the above challenges, teams need to follow some basic practices that can be categorized accordingly:
Test automation development – the decision-making process
Test automation maintenance as an ongoing practice
Test data analysis and trending via ML/AI
Let’s explore the above categories.
Test Automation Development Process
As highlighted earlier in this blog, deciding on each test automation scenario shall be given priority, since each new test scenario that is added into the pipeline has a “price”. The price is calculated as a factor of development costs, maintenance costs, execution across platform cost per night, per code commit. This is simply at the core of such a decision. Teams often chase the size of the test suite as a metric of their success, and they should focus on the value over time such test will provide.
As the above visual suggests, rushing into baking too many tests without stabilizing them, ensuring their value-add to the overall pipeline results in huge failure from both cost investments and quality objectives, hence, great disappointments.
Test Automation Maintenance
Test automation value must be measured over time and not only for the few first cycles. As mentioned above, platforms and product evolve and that has significant implications on the value and “price tag” that was initially placed on the test scenario. That’s why teams must have a structured and well-defined process to judge whether a test automation scenario is still relevant and shall continue running from within the CI. One way of addressing the aging of test automation code is by merging existing tests into a newer and advanced scenario that is either based on a newly developed exploratory test that is fresh, can find regression defects and extend functional coverage.
Test Data Analysis and Trending via ML/AI
It’s clear that teams cannot improve and measure data that they can’t visualize and understand. That’s why test data analysis that can be easily sliced and diced come into play and can be a game changer in determining test scenario’s relevancy, efficiency, etc.
In addition, test data analysis can also team teams whether the test automation scenario is reliable and stable enough or if it needs to be excluded and undergo updates to increase its stability and reliability.
Continuous Testing Needs a Refresh
Test automation infrastructure in the era of continuous testing must always stand the test of time.
To make continuous testing valuable and not a liability, teams need to refresh existing processes, bake into their daily processes some of the above practices, and most importantly, understand the implications of test automation suite size and maintenance.
There is a lot of hype that surrounds automation testing. Not only does it minimize test creation time and run time, but test automation can also ensure that testers are working on tasks that require more careful attention. Not only is this more productive, but testers can also find their work less repetitive and more enjoyable.
While automation testing definitely has the potential to greatly improve software testing as we know it, automation isn’t a be-all-end-all solution to your testing issues. There is no such thing as 100% test automation, and companies that try to do this will find themselves faced with larger costs and problems down the line.
To make the most of test automation, it is important to consider the appropriate scenarios for when it can make a positive impact. Below you will find some of the top use cases for automation testing, which we hope will help when you’re looking to incorporate it into your manual testing operations.
Regression testing is one major area where test automation can help, especially in an agile environment. As companies are looking to do more code releases to provide better user experiences, regression testing ensures that changes in code do not affect previously created test scenarios. This is critical, as it would be counterproductive to make improvements to one application feature just to have older features fail in the process.
Yet with the ever-increasing number of releases, it’s also become important for companies to shorten release cycles to meet their business needs. Due to this software testing “need for speed,” regression testing can often feel like a QA bottleneck.
Test automation bridges this gap, allowing testers to maintain their high-quality work while keeping up with shorter release cycles. Instead of spending time worrying about and checking old code, automation testing reviews this automatically and ensures that everything is working as it should after a new release. Testers can then rededicate their time to focusing on creating more complex test scenarios for future web application improvements.
Increase Test Coverage
In addition to regression testing, QA teams are looking to test as many elements of their web applications as possible. Yet manual testing does not offer the option to test at scale, leading them to neglect possible area where testing is especially important.
With automation testing, QA has the power to execute hundreds or thousands of test scenarios during each test run. These tests can span different browsers and configurations, and all happen simultaneously. The coverage that you can achieve from manual testers absolutely pales in comparison.
Not only does automation allow for more test coverage in terms of quantity, but it helps increase test coverage in terms of quality as well. Depending on the tool you select, your test automation can also handle more complex test case scenarios and work with components such as iFrames and ‘if’ conditions. When you consider test coverage both in terms of quantity and quality, investing in test automation can bring in an even larger return on investment.
Codeless test automation offers even more benefits when it comes to testing coverage. Instead of turning to developers (or testers with the relevant coding skills) to script tests, testers of all levels can author the initial versions of these scripts visually. QA teams can then add these tests easily to the CI toolchain for continuous testing, giving them more time to focus on the tests that they do not wish to automate. This increases test coverage by simplifying the test creation process.
Testing Early in Production
Another area where automation testing can help is when a company is looking to “shift left”, or test as early as possible in the software development lifecycle (SDLC). This can help with many areas of product development, including catching bugs earlier and preventing future defects.
This not only applies to the first launch of a product but also at the early stages of product component development. According to IBM Security, vulnerabilities cost $80 on average to fix if they’re found early in development, as opposed to $7600 if they’re found during production1. This shows that testing early in production not only a technical best practice but also a financial one. The earlier you find a defect, the easier and more cost-effective it is to fix.
Automation testing makes this process even easier by integrating with notification platforms such as text, e-mail or Slack. Not only can you run these critical tests to catch bugs early in the SDLC, but you can also find out about them as quickly as possible in a way that integrates easily with the rest of your testing environment. This ensures that all the relevant people on your team are kept up to date regarding these issues, and allows you to resolve them more efficiently.
Reduce Software Testing Cost and Time Consumption
With the increased focus on speed of delivery, testers can’t afford to be the reason that software development is slowing down. Incorporating automation testing at times when you have calculated that it will reduce your overall cost and time spent will prevent this from happening.
The question you should then ask yourself is, “How do we determine if test automation is reducing my overall costs?” Here are a few points to consider:
Automation testing can be especially helpful for large projects, or for tests that you expect to run for a long time. For these types of projects, test automation can help reduce the number of resources you need, thereby reducing costs in the long run.
According to test consultancy Abstracta, automated tests run on average five times faster than manual tests2. For automated tests that cover multiple browsers and versions, it can also help execute a larger amount of tests relative to the cost of test setup and maintenance.
One of the major obstacles that stand in the way of integrating QA into software development is the amount of time spent on test maintenance. This can become a major drain in terms of time, and will ultimately cost the company as a whole if test maintenance prevents QA from keeping up with shorter releases.
When considering automation testing, it’s important to think about how much time testers will spend doing test maintenance. Investing in a codeless test automation tool can reduce this amount of time tremendously when they offer AI-based maintenance. By automatically modifying tests to overcome changes in the application, these machine learning features allow testers to spend time building and running new test scenarios instead of maintaining existing ones. Companies who use automation testing in these cases can shorten their release cycles dramatically while maintaining (or even reducing) their existing resources.
Amount of Data Being Tested
In addition to large projects, automation testing can also reduce time and costs when you are working with a large amount of data for your tests. This is especially important in the age of personalizing user experiences across a range of industries.
As an example, a login feature is probably one of the most basic, but also the most essential elements to test on a regular basis. Yet to make sure that this works, you need to use a large amount of data to test different permutations of the login information. In addition, you need to also consider that you will need to test this data on different browsers and work environments. For this and other data-driven testing initiatives, automation testing can reduce major time-related costs.
Test Automation Leads to Testing Innovation
Automation testing can help in a variety of ways, but each of the use cases discussed above all fall under one major theme. When automation testing is applied to the more menial and repetitive aspects of software testing, that is where it can make the most positive impact.
Adopting test automation can seem scary at first, especially for manual testers who will fear that automation will make their jobs obsolete. Yet these use cases show that automation testing does the exact opposite, and merely takes away the tasks that are time-consuming and an overall drain on productivity. When applied correctly, test automation will give QA teams the time and resources necessary to take their testing efforts to the next level.
Test automation is continuing to grow in popularity. According to a recent survey of testers and QA professionals, roughly 50% of those who responded expect to make a big leap towards test automation in the next year. All of the respondents were from mid-size and large enterprises, which also means the market has potential to grow significantly.
There is a wide range of automation testing tools that can help QA teams and their companies achieve faster testing that keeps up with their SDLC. However, codeless automation provides distinct advantages that make it the logical choice for test automation.
One of the major ways where codeless automation stands out is that it caters to QA teams with diverse skill sets. While more traditional test automation tools assume a certain level of coding experience, codeless automation takes away this assumption entirely.
In fact, test maintenance is a major area where codeless automation testing tools can make a large impact. Since updating test code can be a very manual and time-consuming task, this is a particularly significant value-add. Let’s review how this works:
Codeless Automation and Test MaintenanceThe Challenge:
Web applications have many different elements, such as a login button, username and password boxes, and a checkout button. Over time, websites often make subtle changes to these attributes to improve the user experience. While this optimization is important, it also creates a lot of work for QA to make sure that these elements still work properly.
Before codeless automation tools that incorporate AI into their platforms, maintaining these tests required going into the code manually, identifying the changing element and updating the test flow accordingly. Unless you had a working knowledge of the tests’ code, it would be difficult to change these tests effectively. Tests would then break frequently, creating a backlog of manual work to keep them running smoothly.
To overcome these challenges, companies only had two choices: teach their QA team how to sift through and change the code or hire an automation engineer. Either one requires a large cost when it came to time, resources, or both, making them both less attractive options.
How Codeless Automation Helps:
Codeless automation testing tools bridge this skills gap in two ways. First, many codeless automation tools can overcome these changes automatically by incorporating machine learning into their test automation platforms. With machine learning, tests break less because it takes into account all of the different attributes for every test element. If there is a change, you don’t need to drill down into the code and apply the changes within the code itself.
Second, codeless automation allows testers to work in a visual interface, which makes it easier to make test flow adjustments upon changes in the application logic. Even without coding skills, testers can see exactly where their tests break and reconnect these broken elements during runtime. In addition, these changes will automatically reapply to all other tests that are using this process. In each of these ways, codeless automation tools can help reduce test maintenance time and resources.
Other Skills Gaps Where Codeless Automation Can Help
Although codeless automation most notably (and obviously) bridges this coding skills gap, it also can help teams who don’t have skills in other important elements of the software testing process. Here are some examples of other skills-related challenges that codeless automation can overcome.
No Need to Build Your Own Test Automation Framework
Without codeless test automation, QA teams are tasked to build their own framework from scratch in addition to creating the tests themselves. This requires many steps, as well as experience with solutions such as build tools, browser automation tools, and IDEs. More often than not, companies bring in a specialist to build the framework due to its complexity.
To build a framework, the person building the framework needs to create a base project, as well as the modules and dependencies that will go into it. Modules need to be set up for multiple elements, including page objects and classes that facilitate creating automated tests. In addition, they need to add the dependencies that go between each module in the framework. Without an existing knowledge of tools like Java and Maven, this initial setup will prove to be quite difficult.
Second, testers then need to add the actual code to this framework. This is problematic for a few reasons. A more obvious issue is that many testers that are tasked to create and execute these tests, often manual testers, don’t have previous coding skills. Additionally, this process also doesn’t allow for creating tests before the code has been written. This has many repercussions for those companies that are trying to shift left and become more agile.
With a codeless test automation tool, the framework is set up for you from the get-go. Plus, many codeless solutions are already integrated with tools that are likely to be part of your CI/CD pipeline, such as JIRA, Jenkins, and TeamCity. This makes test creation and incorporating test automation into your environment much easier.
By going codeless, testers also have the opportunity to create test scenarios before new features are even coded. Using a virtual canvas, testers can design abstract versions of the tests based on the product flow or wireframe. Once the code is ready, all you need to do is run the test and bind the element.
No Legacy Tool Experience Needed
With codeless test automation tools, you don’t need to invest a lot of time and resources into extensive training. This not only bridges another crucial skills gap but also allows QA to conduct more agile testing.
When searching for a test automation tool, many companies first consider legacy solutions from larger, more established enterprises. Perhaps this is because the company is already highly invested in this enterprise, or many test flows were already built using these tools.
While legacy tools may tout a highly recognized brand name, they also come with a heavy cost. In addition to their hefty price tag, many legacy tools also require a lot of maintenance due to their complexity. Instead of creating new test scenarios, testers will instead spend a lot of their time learning the tools themselves.
While codeless automation tools require some training to acclimate to a new solution, they offer a considerably shorter learning curve. Testers can create test scenarios based on what they actually see in the web application instead of within the code. This will free up time for testers to create new test scenarios, and give more flexibility to create tests in a more agile way. This seems like a much better alternative to spending time learning the ins and outs of a legacy solution.
Communicating with the Rest of Your Team
Another important skills gap that codeless automation tools can assist with is effective communication. While a much softer skill in comparison to test maintenance or building a framework, it is one that QA teams should not overlook.
According to a Cloverpop study, decisions made and executed by diverse teams delivered 60% better results. Therefore, it isn’t just a nice idea to invest in tools that allow for better collaboration; it’s essential for business.
Photo credit: ThePartnership.org
Unlike traditional automation testing tools, codeless automation creates an even playing field for all team members. Instead of tests that cater only to QA or to developers with coding skills, everyone can see and understand the test scenarios clearly. This also allows product managers to get involved in the testing process, which adds even more context and diversity to the conversation.
This creates ample opportunity for stronger communication, as well as more productive knowledge-sharing. Testers, developers and product managers question software testing differently, which is a strong asset when they can communicate well. With the opportunity to ask each other the right questions, the opportunity for faster, and higher quality growth is virtually endless.
Last week, we had the distinct pleasure of hosting a webinar with Chris Kenst entitled, “Participating in Code Reviews as a Tester”. In this webinar, Kenst showed that even without coding skills, testers can – and should – take part in code reviews.
At TestCraft, we were excited to host this webinar because we are passionate about testers getting more involved in the software development process. There are many ways to bridge the gap that exists between testers and developers. Along with codeless automation, doing code reviews helps testers play a more active role in software development without previous coding skills.
As a codeless test automation platform, we also believe in investing in processes that help testers adopt a “shift-left” mindset. Using TestCraft’s visual way of creating automated test scenarios, QA teams can test even before an application is ready. Both creating automated test scenarios and participating in code reviews can help with catching bugs early in the process, resulting in increased savings for the company. The earlier your team catches bugs, the lower the overall cost. Together, codeless automation and code reviews can help make software testing faster, more efficient and more productive.
In the first part of the webinar, Kenst discussed why he is encouraging testers to participate in code reviews in the first place. There is a major difference between reading code and writing code. While many testers have little to no code-writing experience, it is much simpler to learn how to read code. At the very least, testers are able to familiarize themselves with markers in code that can help them look for bugs more easily.
He continued that code reviews are not only very common in development, but are a great opportunity for testers to get more involved in the software development process. There is a big learning process involved when looking over a set of changes in code. Since testers question differently from developers, it’s a great way for them to become part of this discussion.
Participating in code reviews has another, less obvious benefit for testers. Not only can testers play a stronger role in the SDLC, but they can improve their own testing efforts as well. With the additional context of code reviews, testers can actually adapt their scenarios in response to what they learned. Collaborating with developers in this way allows testers to work more efficiently and become more technical in their testing approach.
Asking the Right Questions to Understand Changes
Understanding context is an invaluable asset that testers have during code reviews. In Kenst’s personal experience, he started receiving pull requests specifically because he understood the bugs in a larger context. To use that context effectively during a code review, it’s important to ask the right questions.
Kenst presented a list of possible questions that can help testers better understand the changes happening in a pull request. Examples of questions included asking about text changes to taking a mental inventory of the number of files that changed. He also suggested thinking of code changes in terms of categories: documentation, bug fixes, maintenance, refactors and new features. This can help testers better determine which bug fixes require a simple two-line change and which need more attention.
Another important point that he mentioned was to pay attention to the third-party services that your development team is using as part of their CI system. First, having access to these third-party services can provide data that you can use later for testing. Second, you can see where they are used, which adds to your testing because you can get a fuller understanding of how they function within an app.
Ultimately, getting into code reviews takes practice, a lot of trial and error and constantly asking for more information. Whether you get started by searching on Google or pairing with a developer, this will all give you the increased contextual awareness you need to do an effective code review. You will both bridge the gaps in your own knowledge and fill in the developers’ gap in their knowledge.
How to Get Started with Code Reviews
To end the webinar, Kenst went through some practical examples of code reviews that spanned different types of scenarios. Working in a Github repository, he went through the pull requests tab and gave everyone a glimpse of how to read the code, search for the issue and make a suggestion to fix it. He also offered different best practices, such as using the split setting to get a better visual comparison of the changes that are happening in the code.
He then offered recommendations for how testers can get started with code reviews on their own. Pairing with someone who does code reviews at your job, whether a developer or someone else, is a great place to start. This will help you better understand how to conduct code reviews in your specific environment. He also recommended that any GitHub project is a great opportunity to start learning how to do code reviews. Finally, he invited everyone to do code reviews his own OS list of software testing conferences, TestingConferences.org.
At the end of the webinar, Kenst answered a few questions from the audience. For your convenience, you can read the questions and his answers below:
Q: How do you approach introducing code reviews to a team?
A: For a testing team, start introducing this one tester at a time. When I started doing code reviews, I became a big advocate of everyone else taking part. Once I learned how to do code reviews on my own, I would pair with other testers and teach them how to do code reviews as well.
From there I was able to show them how looking at these changes could help them make improvements to their testing strategy. You should implement it, get yourself up to speed and then share your newfound skills. If your company does not do code reviews, then reach out to your development team and ask if there is a campaign on the side that you can look at to provide a review. They are usually very happy to accept the help and can work with you to do the review as well.
Q: How do you do a code review if you follow an agile methodology?
A: It’s built-in. Typically what happens in an agile workflow is there is a development in progress, they create a branch and then the developer is “done” working on it when a pull request comes up. Typically another developer will then do a code review and either approve or decline those changes. Once they’re approved, they move on to testing and then go into production. By definition, code reviews are built into an agile development process.
Q: Is it possible to see the pre and post-code changes, or compare two versions in the same branch?
A: Using Github, this is very easy to see. The pull request’s file change view shows what the code was like before the changes and after the changes. You can also use the command line or any other tool that helps with code reviews.
Q: How do you compare the same code in two different branches, in case the code works in one branch but doesn’t in another?
A: This is more of a debugging scenario. In those cases, I would check out the branches themselves and try to run the code in each. If you can run the code in both branches, you can start looking at the differences and compare them. This may not be a pull request, rather it may be something you can do on the command line. It’s something that’s fairly easy to visualize.
Q: In your opinion, does QA do initial code review or does this qualify as a peer review?
A: It depends on your organization, but I think they’re the same thing. Where I started, testers also handled all the releases, so it makes sense for us to do code reviews and merge them. It depends on where you fit in your organization to determine how much you will do.
Q: How would you approach the situation if If the sprint cycle is just 15 days and QA cannot do a code review?
A: At the organizations where I’ve worked, we’ve done multiple deploys every day. So if you have 15 days between deploys, you have time to do code reviews. I’m not suggesting that you break with your workflow where you expect someone to assign code reviews to you. That’s a bigger organizational change. Initially, if you see that there are pull requests open and in a code review state, and you know that you will eventually be testing the changes that happen from those requests, I would recommend you take a look at these pull requests. Whether or not you approve or merge those pull requests, that’s a larger decision that you can make with your organization.
As part of the testing process, you should see what changes coming down the pipeline, and you can find those changes in pull requests. Then you can take some time – around thirty minutes per day – to figure out what’s coming up for the rest of the week. Depending on complexity and volume, you can start with one code review per day, or even one code review every few days. Regardless of what you do, you should be able to look actively at where those pull requests are, determine if they will have an impact on you or your testing, and take some time to look at them and see what is happening.
There is always more to learn when it comes to QA and software testing. From staying on top of the latest trends to honing your skills, QA blogs can provide great advice and insights.
While there are many great resources out there, it can feel overwhelming to know which thought leaders to trust when navigating the world of testing experts. To help you out, here are the top seven blogs that we think you should keep on your radar.
A big believer in a full-stack approach to automation, Joe Colantonio’s blog delves into the different ways test automation can impact your QA team. He covers a wide range of topics, from DevOps to the way you set up and provision your environment. Through his writing, he focuses on how to incorporate automation into all areas of the software development lifecycle. In his podcast, TestTalks, you can also find his interviews with some of the biggest leaders in the testing industry. If you’re looking to invest in automation testing and performance testing tools, he writes product reviews on his blog as well.
Angie Jones’ blog is a must-read for anyone who is looking to incorporate test automation into their software testing strategy. She writes about everything from industry trends in test automation and building a top-notch QA team. She also focuses on best practices and techniques, like how to build an automation framework that is agile-friendly. For those of you who are more hands-on, you can find her free test automation courses on the blog as well.
If you like continuous testing, Eran Kinsbruner’s blog is a must-follow. He writes about everything related to this topic, with a focus on web and mobile applications. An industry veteran, his blog is also a great resource for dealing with each step of the application development cycle.
He’s a frequent speaker at global QA and testing conferences and guest writes for many different publications. Make sure to keep an eye out for his articles on the TestCraft blog as well!
If you’re looking QA blogs that offer a consistently updated list of software testing conferences and workshops, Chris Kenst’s blog is the place to look. An automation engineer for BloomNation, Kenst also writes about all things software development and testing. He’s passionate about teams achieving fast, “shippable quality” for their companies and offers many practical examples for doing just that.
Be sure to keep an eye out for his speaking engagements as well, including his upcoming webinar (with TestCraft). If you’re looking to learn how to conduct code reviews as a tester, this is the webinar for you.
The Friendly Tester
Richard Bradshaw’s blog, The Friendly Tester, discusses everything from automation testing to his own growth as a software tester. As the CEO (also known as “FriendlyBoss”) of Ministry of Testing, Bradshaw also offers unique insights into the software testing community at large. His mixture of testing tips and personal musings make this a great QA blog to follow. Make sure to also check out his “Whiteboard Testing” YouTube channel, which has videos on different software testing best practices.
Contrary to what you might think, Alan Richardson did not name his blog “Evil Tester” in response to the “Friendly Tester” blog mentioned above. Rather, he sees “evil testing” as a fundamental attitude shift where testers actually take responsibility for how they test. Instead of acting passively, Richardson believes that testers should own their processes and ways of thinking when doing software testing.
Unlike other QA blogs, Richardson focuses on this change in attitude, combined with skills overviews and other tips to approach testing in a more pragmatic way. This is the blog for you if you’re looking for professional advice coupled with some life lessons.
James Bach’s blog for his consulting business, Satisfice, focuses on different applications of his Rapid Software Testing methodology using the Context-Driven School (CDS) of software testing thought. Rather than focus purely on the rote formalization of technical work, this approach prioritizes skills and problem-solving. Coming from a programming background, Bach’s blog offers a unique outlook that appeals to both testing professionals and engineers.
Measuring test automation efficiency, its ROI (return on investment), and value to the overall quality of a product is one of the common topics in test automation. For years, managers, practitioners, and agile teams are trying to justify and measure test automation. This topic is still relevant when connecting it to continuous testing and DevOps, but a bit different than before.
In the past, teams were trying to measure test automation around the percentage of test cases that are automatable, and the cost savings associated with this task. When putting test automation in the context of DevOps, the driver for measuring test automation and testing, in general, should be the VALUE of the tests and their ability to mitigate and reduce quality risks.
Continuous Test Automation and Different Practitioners
Before covering ways to measure test automation effectiveness, let’s do a short recap of the personas that test automation is serving.
Test automation must be measured not just by automating functional testing by SDETs and/or business testers. Developers within feature teams also take an active part in test automation, hence, such activity needs to consider all practitioners.
This means that test automation should have proper metrics that cover different test tools that these personas use daily to achieve their quality as well as their personal goals.
The above is extremely important since if there is a wrong match between the persona and the tool they use to create test automation scripts, the measurements are in a sense irrelevant.
In addition, the usage of the tools by the practitioners need to also be measured in the context of the test pyramid according to the project’s complexity and needs. Testing UI/API, Functional, Integration, Non-Functional aspects (load, accessibility, etc.) and unit testing need to be measured accordingly.
When considering the ROI of testing, and test automation specifically, executive and managers need to consider the risks of lack of quality, escaped defects, late releases, etc. rather than just the cost of a software license. This does not imply that there is no need for proper due diligence, considerations, and matching between the test automation and the job to be done.
Lastly, the type of application in many cases will determine a lot of metrics, tools, and considerations teams will take: Web vs. Mobile vs. IoT vs. other or mix of apps has a great impact on many of the test automation metrics.
Measuring Test Automation ROI – Key Recommendations and Metrics
After establishing the ground for measuring automation by defining personas, application type, test types and the difference between test automation and test automation in DevOps, let’s focus on measuring and defining solid and high-value test automation.
Since test automation within continuous testing is something teams execute multiple times a day, per each code commit, within CI, and expect value, fast and actionable feedback from – therefore, putting extra emphasis on the creation of the tests is a strong recommendation.
As in the below sketch, three pillars of good test automation include the high value of the tests, high reliability, and fast execution and feedback. If these are part of the pillars, some of the metrics of how to measure test automation ROI must be around these points.
When determining the ROI of test automation the process stages also need to be part of the math:
Test authoring (time to create a test, ease of use, script reusability)
Test execution abilities (parallel testing, connectivity, and integration with standard tools and frameworks)
Test results analysis
Test maintenance simplicity
Software license cost per user/floating
Number of resources required for the above
Generic Continuous Testing in DevOps List of Metrics
How quickly are testing activities moving, and what is slowing down these activities?
Percentage of automated vs. manual tests.
Application quality measurements
Number of escaped defects and in which areas.
MTTD — mean time to detection of the defect.
Pipeline efficiency measurements
The number of user stories implemented per iteration.
Test automation as part of DoD across iterations.
Broken builds with categories.
CI length trending.
Lab availability and utilization.
Quality cost measurements
Operational costs, lab availability issues.
Cost of hardware/software.
Costs of defects by severity and stage
Finally, as an important reminder, there is the common formula of ROI measurement:
ROI = Gain – Investment/Investment
If to use the above formula and refer to an old but still relevant blog from TestingWhiz, and make some placements according to some key measures (feel free to configure based on your own project needs), we can see the following: The gain will be the objectives and goals you wish to receive from test automation – reduction in feedback, shorten cycles, release of manual engineers, converted into $$, and so on.
Measuring test automation ROI should focus on more than just licenses cost and reducing testing cycles. It is a mix of the value, reusability, analysis time, and matching to the project, persona, and technology teams are using.
The recommendation is to come up with a mix of the above, place some abstract “grades” per each of these items, and prioritize them to reach the answer.
From a great practice I received from my friends at USAA, I can also recommend the following formula to choose the right tool for the project.
The above recommends choosing the tools using a set of considerations from both test development practice (BDD, ATDD, etc.), type of testing E2E/Visual, and the ease of use that includes also documentation. Each consideration gets a weight and scoring key that when multiplying both, you get a result that can guide your selection.
Monthly Roundup of All Things Testing and QA: April 2019
April was an exciting month in the world of software testing. We hosted meetups in Berlin filled with learning and discussion, and are looking forward to traveling across more of Europe in May! We’ve also gathered our top live content, as well as other industry articles and events that caught our attention this month.
What was your best testing advice? Amanda Green, TestCraft’s test automation expert, asked our community about the best testing advice they have given or received throughout their careers. This blog post highlights some of our favorite responses.
TestCraft Week in Berlin, Germany. From March 28-April 2, TestCraft’s resident test automation expert Nimrod Arbel hosted two meetups in Berlin: “Connecting QA Professionals to DevOps Era” and “Machine Learning in the New Age of QA Automation Tools”. He also ran a hands-on workshop entitled, “Monitor Your Application in Under Ten Minutes.”
Having fun at our “Connecting QA Professionals to the DevOps Era” meetup in Berlin.
STPCON Spring 2019. From April 1-4, test professionals convened in San Francisco to hear from leaders in software testing, as well as attend a variety of sessions and workshops.
STAREast. One of the longest-running conferences on software testing and quality assurance, the STAREast conference happened in Orlando during the last week of April.
TestCraft Meetup: “How Machine Learning is Used in the New Age of QA Automation Tools”. Save your spot here or sign up for future meetups happening in your city!
Amsterdam Meetup – Monday, May 13
Paris Meetup – Tuesday, May 14
QUEST 2019: Quality Engineered Software and Testing Conference & Expo. Happening May 13-17 in Chicago, this conference is a leading source for new technologies and proven methods for Quality Engineered Software and testing.
TestCraft Webinar: Participating in Code Reviews as a Tester. We are excited to welcome our newest guest speaker, Chris Kenst, to our webinar series. On May 21st, he will give a hands-on demonstration of how testers can participate in code reviews without any prior coding knowledge. Make sure to sign up and save your spot.
TestBash Netherlands 2019. Now in its third year, TestBash offers a variety of hands-on software testing workshops on May 23-24 in Utrecht.
Nordic Testing Days. Happening May 29-31 in Tallinn, Estonia, this testing conference offers an assortment of tutorials, talks, and workshops.