Whether you are a tester, a developer, a Product Owner, a UX designer, or any other person involved with software development or in any other testing field, you will find interesting articles on Lyon Testing blog.
Today topics bring us back to school with basic software testing techniques:
A Far-West with “The regular, the blank and the error”
Always test these 5 conditions
Example for Domain testing and the ‘Counterstrings’ tip
Write better bug titles
With acquiring experience in companies we may be brought to conduct complex testing. Our brain then tends to forget about basics and tends to overcomplicate things. It is sometimes good to take a step back.
Lots of companies don’t have any Testing or QA Team, and in DevOps environment, most of them are thinking about adding a new role to start a new project for test automation. Maybe that you already have some unit tests, integration tests or service tests that are executed on your CI, and these have for sure been written by developers.
But what is the strategy behind that? Are these tests really relevant and useful? Are they passing all the time and not ignored? Do you need a new team to manage these test activities or is the actual team with no dedicated tester enough? And finally what are all these roles including “Developers in Test” in the name?
Automation is a project
First, I like to say one thing: Automation is a software development project. And I think I will write it twice, in bold is better: Automation is a software development project.
Let’s say you are working on a e-shop, at first only available with a browser desktop. If you want to provide your customers with a new Android and iOS application, will you add a ticket in Jira (or whatever) “Build the Android/iOS app” and assign it to one of the developers building the Angular desktop application? Probably not? At least you will create a new project, create a new team, and will look for someone with the skills to build a smartphone application.
Same for test automation, automation is a real new project and should be treated as such. Don’t expect a half-time intern to do it! Don’t expect your developers will do it during a workshop week! Don’t expect your developers to build it during 2 sprints and let them add tests during each sprint.
A project needs a strategy
I cannot give you THE strategy that fits your environment. I can only give you a list of questions that you need to answer then you will be able to start thinking about a coherent strategy:
Why do you need to automatize?
What is the goal?
Release the product faster without the bottleneck of manual testing?
Let other testers of the team remain concentrated on exploratory testing?
Do you really need it? Why not testing it “manually”?
Have you calculated the Return on Investment?
What levels can be automatically tested (API, Services, UI)?
How the tests will be ran? Locally, in the CI jobs of the product, or better in a separate CI job?
Can I be helped by AI?
What testing data is available and can be used?
What environment will I be able to use? Is it possible to have a real isolated test environment?
Automation needs a dedicated team. “So if it’s a real project with a strategy, I probably need a team and people in this team, right?”. Yes, you are right, you need resources. But who?
Developers know how to code, but aren’t the best testers. So they will just try to automate all the things, and will probably start with useless things. You could try to learn them the testing Mindset, but I’m not sure they will have enough interest in the topic.
Testers know how to test, but aren’t the best developers. You can train them to help them become a tester able to code testing. If you can wait for several weeks and let them spend full-time learning code, that’s a good idea. Do you really want to do this?
Of course, needless to say that managers, UX designers, Head of Customer success or agile coaches won’t be the perfect candidates too.
Are you lost?
Several options are available to you:
Hire an automation specialist, one that is a very good developer but also has the testing mindset and will be able to build a perfect strategy in order to only write relevant and useful tests at the correct level (Integration, API, UI). Sadly, they are not numerous and probably already very busy,
Use a codeless tool that will be used by your testers. Some of them are very easy to learn, and with a little help from developers for very specific cases, you should be able to automatize major testing features. These tools are very expensive, and will need significant maintenance efforts,
Build a team of developers and testers. Because a good strategy will probably be to write different kind of tests: unit tests, integrations tests, API/Service tests, UI tests with or without visual checking (remember the Cohn pyramid of tests). For this, testing and development skills are needed.
I personally think that, in most cases, if you cannot find an automation expert, the best solution will be the third one at least to start the project. Later, you will maybe have someone able to manage it alone, maybe a former developer or a tester.
Do I need a STE, STAE, SDET or a SET?
That’s a lot of acronyms. Are they used for the same role or not?
S stands for Software, D for Developer or Design, T for Testing and E for Engineer. Then you can create your own acronym. Microsoft was the first to introduce “Software Developer Engineer in Test” (SDET) in 2005. Google adopted “Software Engineer in Test” (SET) for the same role. But you can also find STAE or STE.
Another role you may find is “Software Design Engineer in Test” (that’s another SDET) which is a very technical one. Tasks involve auditing the architecture of the software, testing and suggesting improvements on components and interfaces. The SDET (with a D for Design) can also write test scripts to achieve these goals, then he is also a SDET (with a D for Developer).
Software Test Automation Engineers (STAE’s) are often less technical and need less programming skills because they mostly focus on automation using tools like QTP or TestComplete.
So according to your context, who do you need?
It’s up to you to find the right strategy and be helped by the good people. Don’t be ashamed to try things and iterate. Don’t be afraid to say that good automation is not easy. It can be easy, but the one that has lots of value, the one that fails only when there is a real issue, the one easy to debug, the one that is fast, the one that gives good reports is very hard to reach. And don’t forget you will still need non-automated and exploratory testing to fill the gaps.
Good luck, and don’t hesitate to talk about your context and give feedback.
Recruiting for a new hire, or finding a new job of your own, has never been easy. It becomes even more difficult when your profession is poorly known and often ill considered.
If you read blogs or magazines dedicated to software testing then you have probably encountered some of the common misconceptions about testing: that testing is mostly a repetitive and boring task, that testers are solely responsible for errors on production servers, that everything can be automated and therefore the job of the tester will disappear, that testers are unskilled developers, etc.
If the recruitment process is managed by people who do not understand testing, then they are unlikely to know how to evaluate testers. If they cannot see how a person could be a good fit, they might hire someone not suitable or miss that special someone.
As a candidate in front of people whose recruitment methods are the same for a tester as for any other role, then you have little chance to really understand how great the job could be. You may run away thinking that the company has a bad culture, or that testing is not valued by engineering teams, or that you will be the lone tester considered as a bottleneck.
We will see in this article how to define what might be the characteristics of the candidate you are looking for, then how to attract good applicants, and finally how to evaluate them with an interview or an audition.
What are the characteristics of a good candidate?
Obviously, there is no precise answer. It will greatly depend on your actual need, the team that the applicant will have to join, and what you imagine as the future of your environment.
If you’re looking for an automation engineer, then of course you’ll need to find someone that knows how to code. But if your framework uses Ruby, is it a good idea to dismiss the expert python candidate?
A good tester must have the ability to learn quickly. Whatever position you offer, it is almost certain that what you think your need is today will become obsolete within a couple of years, and this is not only true for automation but for all testing experts. A good candidate is one that will stay with your organisation as it evolves.
Even if almost everyone agrees that the requirement is useless, there is still a tendency to read “ISTQB required” on some job offers. I guess it’s because it’s very easy to sort people that are certified and people that are not, and then quickly select a few to enter the recruitment process. Do you really think that someone who has passed an exam is definitely a good candidate?
Certifications are useful if you can have the list of people who failed the exam, because they were not able to learn or study enough to be sure of their success. Sadly you can only know who succeeds, and success proves only one skill: the ability to pass an exam, which is not the job you want them to do. Certification isn’t an indicator of good, and you will meet a lot of good candidates who have no certification and bad candidates who are certified.
A good tester must be curious. Firstly because they cannot spend all of their testing career doing the same tasks. They need to constantly improve, learn new things, and without being very curious about everything around their craft they cannot be motivated in learning new stuff. Secondly because they must discover a wide variety of information about the business and the technology involved in the product they’re working on.
They also need to be a good communicator. A tester is the one who will have to communicate bad news – “Houston, there’s a huge problem here”. They are also the one who must communicate with people that don’t like to communicate, which includes some developers. They talk to management, product owners, and sometimes with customers, not only in face-to-face conversations but also using many tools: email, instant messaging, phone, bug tracker tools, etc. A good tester needs to communicate in a diplomatic way, without being too intrusive and without being ignored. That’s a difficult skill.
How to attract a good applicant, and how to retain them?
Certainly not in lying. Whether with an offer, an interview, or an audition, you will have to make them feel you have the best job to offer, but do not say things that you won’t be able to offer.
If you can offer flexibility in schedules and holidays, these are things that might be appreciated, maybe more than a good salary. Good testers place a great deal of importance on progressing in their field of activity. If you can help them to attend the conferences of their choice, do your best to make this happen, and don’t hesitate to sponsor those conferences: by doing this you will show how you value learning in your teams.
A good company culture is one that enables everyone to give their best. That’s why a good candidate will prefer a company that values mentorship over old-fashioned management without trust or flexibility for experimentation. Can you offer this and prove it?
In a previous edition of Testing Trapeze, Daniel Barrow shared how he was convinced that PushPay is a great place to work. I loved the idea of first having a coffee while talking in a relaxed and informal way about the company culture, and not only about your CV.
Resumes and cover letters are still requested, sometimes using the company template. Some companies also send a form containing many questions that are not related to the job that will have to be done. I’m sure that these practices are very damaging to the image of the company and are deterring a lot of candidates. Maybe the best one is among them and you’re about to lose them by asking for this.
How to efficiently evaluate an applicant?
The difficulty when you have candidates in front of you for a short period of time is to evaluate them correctly. To imagine them integrated in your team, while guarding against the cognitive biases that you might be the victim of.
Beware of confirmation bias. You could make the mistake of deciding the fate of the candidate in a few short seconds. You might quickly have a negative image of an introverted person, and we know that introverts can be excellent testers (poor sales people, but that’s not what we’re looking for, are we?!) . Or you might overvalue someone that knows how to introduce himself efficiently but is a poor candidate compared to other qualities expected.
Watch for Status Quo bias. When choosing between an average candidate with knowledge in a particular programming language and another candidate who does not know this language at all but may be able to learn in a few weeks, it might be better to prefer the latter.
It is now popular in IT to prefer auditions over interviews. This is a very efficient way to test an applicant. You can ask them to perform some testing on your application at home, or you can ask them to come into your organisation to work a few hours or days directly with the team. The candidate might do some pair testing, add some automatic checks, or fix a failing test if that’s part of the job you propose.
You and your team will gain a better understanding about the candidate when you will see them work in real conditions, directly using the tools you will ask them to use. The candidate can share work-related knowledge with the team members you will ask them to work with. If the audition is successful, then that’s a good way to test the fit between a potential employee and a company, on both sides.
You must be aware that not everyone will have time to engage in this process. Some have a job during the work day, a family to spend time with in the evening, and commitments during the weekend. Some candidates will have several opportunities to evaluate and won’t be willing to block one or several days of their calendar just for you, unless they really want to work for you, and only for you. Additionally you should pay the candidate for the time they spend working for you. You may not be able to do this with a lot of candidates.
If you cannot do auditions then you can evaluate the candidate by asking him to think about some quick testing tasks. For example:
ask to design a test. Prepare a user story (add bugs or incomplete information inside), ask him to write acceptance criteria and see what happens
ask him to talk about how he would test something easy to understand, a very small piece of software or anything else (a bulb, a coffee machine, the trackpad of your laptop…)
you can use puzzles like these ones and evaluate his ability to explore
prepare a dev environment and ask him to add a check in your automatic test framework if it will part of the job
or ask him how he would interview and evaluate a candidate for a testing job
As a company you are not alone in the market. You must have the best showcase to attract good candidates. Your hiring process should reflect how awesome the company is and evaluate the candidate according to the job they will have to do.
Testing skills are very specific and not easy to assess, a lots of biases are to be avoided on both sides. You can audition the applicant and make him work with the team or use some simple exercises to see how he manages them.
As an applicant, finding a good company where you will be allowed to do good testing is a challenge. Some will have very poor hiring processes for awesome jobs. Others will have very shiny and sexy hiring processes for boring jobs.
Hiring needs to be improved in order to achieve its goal to find the best fit for the best candidate for the best job. That’s not easy in software testing.
This article was originally published in Testing Trapeze in June 2017. Testing Trapeze sadly stopped being published and is no more accessible online.
“Soft skills require more courage than hard skills.” This sentence enlightens the following article from Dan Rockwell : 7 Powers of Vulnerability which is an interesting approach to help testers who are in charge of leading a team.
Welcome to our newly reading recommendations. This week, we suggest you to read the following articles about :
Code Coverage and Test Coverage
Shifting left, and shifting right
An introduction to generative Testing
In people minds, there is often a confusion between Code coverage and Test Coverage, which are, you guessed it, not the same at all. To explain that, Dan Ashby used an analogy, using one of his little boy toy. If you’re interested or still need to understand the difference between these two types of coverage, then I highly recommend you to read this first blog post: Code Coverage vs Test Coverage; Subjectivity and Usefulness
Finally, let’s talk about generative testing which is a method that generates (randomly or not) test cases for you. You just need to give, as an input, what a query can look like and what kind of parameters are possible then, as an output, you will have test cases. It can be thousand or millions, and then you can chose a specific set randomly or with other criteria if you don’t want to (or just can’t) use all combinations. The Medium post from Jon Normington is here: A smarter way to QA: introducing generative testing
And you? Did you read a very good blog post recently and would like to share it?
Today, let’s talk about Man in the Middle method that allows anyone to track any traffic sent and received by a smartphone or a browser on your computer. Sometimes, you cannot put the data you want in your application, but if you can modify content received by a browser or an application just before sending it to the smartphone, then you have a very convenient way to test with on-demand data.
This is what I will explain in this article with a practical example.
I will focus on Testing in a desktop browser, but also with Android and iOS applications. For this, I will use Charles Proxy. Later in another blog post, I will maybe write about mitmproxy which is free and open source, but a bit less easy to use. So I will detail how to use Charles Proxy with a MacBook, a desktop browser and an iPhone in this article.
Man In the Middle Proxy
What is Charles Proxy?
Charles Proxy is an interactive man-in-the-middle proxy for HTTP and HTTPS. It enables anyone to view all of the HTTP and SSL/HTTPS traffic between one of their machine and the Internet. This includes requests, responses and the HTTP headers (which contain the cookies and caching information). It is also possible to modify requests before they are sent to the server, or, and this is what I will describe here, the response received which opens lots of opportunities.
How to Install Charles
Charles is available for Windows, MacOS and Linux. Just download the one you need on Download page and follow instructions.
You will get a trial version, because Charles is not free (while mitmproxy is), but it’s worth the price. I have to admit that I first use mitmproxy, but Charles is way easier to use and if you really need a proxy, you may probably not regret the expense.
How to capture things…?
…that will be displayed in a Desktop browser
If we want to capture packets from a browser, we must configure it to use Charles. It is by default configured to use port 8888, but you can modify it for anything you want by opening Proxy Settings.
You can open Help then Local IP address to know what IP needs to be used in order to access the browser, but most of the time you will probably do it locally by using localhost (127.0.0.1).
Let’s configure our browser. Go to the proxy settings, then add 127.0.0.1 as the address of the proxy and 8888 as the port.
You will now capture all packets you request and responses but only for http servers. For example, if https://www.lyontesting.fr is captured by Charles you will see this response which is not quite very comfortable to read.
Just right click on the request, and select “Enable SSL proxying”.
Then refresh the page in browser and you will see that the result is not better. That’s because the page www.lyontesting.fr is redirected to lyontesting.fr. Go to Proxy/SSL Proxying Settings and adapt the location host in order to match all cases you want to catch.
The fun thing is that you can modify the response that will be sent to the browser. For this, just add breakpoints like this.
In Proxy/Breakpoint settings…, select “Enable breakpoints”, add a breakpoint that detects POST request with host=lyontesting.fr and select Response.
Now refresh again the page, the response will not be sent to the browser and is waiting for your approval.
In the “Edit response” tab, you can modify the html. Here I replace “Lyon Testing” with “World Testing”. And the result is displayed in the browser.
And now, you probably think that this could have been done with the dev tools of your browser quite simply, and you are right. But these easy steps were there to show you the basics of Charles and proxying. These things can also be used for smartphone applications and that’s what we will see now.
…that will be used in a smartphone App
Charles Proxy now also comes as an iOS application. With it, you can specify which hostnames to include or exclude in the recording (for example, only the requests to the server of your application), same for SSL Proxying. To install certificates, follow the instructions.
The Charles proxy app is quite a recent application, and it’s not possible to modify the responses received. Maybe that it will be added soon as a new feature. So if you want to edit responses, you’ll need to use the Desktop version of Charles proxy. Lucky you, you already installed it before. If not, better late than never.
Your smartphone must be on the same local Network (if using the same WiFi then it should be ok) in order to be able to capture the traffic from the smartphone. In Charles, go to Help then Local IP Addresses.
You will find the IP address that needs to be used as the manual proxy in your WiFi access settings on the smartphone. Here, with an iPhone, using default Port 8888 used by Charles.
That’s it, flows should be displayed in your Charles Proxy. You also need to install the Charles certificate on the Smartphone. Procedure is explained in Help/SSL Proxying menu. Let me copy/paste it for you: “Configure your device to use Charles as its HTTP proxy on 192.168.2.1:8888, then browse to chls.pro/ssl to download and install the certificate. Note that on iOS 10 and later you must then go into Settings > General > About > Certificate Trust Settings and enable the Charles certificate to be trusted.”
Just like at the beginning of this article, right click on a flow then “Enable SSL proxying” may be needed.
OK, so now you know how to display the http/s traffic. What could we do with this?
Filter and alter request or response
When you test, it’s not always easy to display what you want. Depending on the testability of your product, it can be hard to check all corner cases if the production back-end must be used and cannot be tricked.
With a proxy, you can change what will be displayed.
Let’s see what kind of data a weather application sends; for example, the one I may use almost every day before going out with my bike: rainToday. Hopefully one that doesn’t use certificate Spinning, otherwise our goal would be harder to reach.
Select a town when it’s raining….
…and see what is captured by Charles. Requests to weatherpro.consumer.meteogroup.com should appear in the application, then I suggest you to add a filter in order to not display other requests.
And when you select a town, you will see that 2 requests has been sent:
a GET /weatherpro/RainService.php?method=getRainChart&areaID=UWZFR4562
A GET /weatherpro/WeatherServiceFeed.php?format=xml&lid=1837351&mode=observation
Display the response of the first one, and you will see a json similar to this one:
I guess that rain has a value between 0 and 100, but because future cannot be 100% certain, rainRadar is giving approximate values for every minutes (interval: 60000, 60 values for avg, min and max).
What if we want to test extreme values like:
Or better, invalid values and see how the application behaves:
I just have to copy-paste the 3 lines, avg, min, max in a text editor, then replace first values with 50, 0, 100 and second values with 0, 100, 0 (repeated 3 times each in order to be more visible in the app). The result is:
But we need to intercept the flow, then need to add a breakpoint that detect a GET request and the following Path: /weatherpro/RainService.php* then capture response.
Select the town again, or another one and the breakpoint will stop the flow.
In order to edit the flow, select Edit Response and Json Text.
Then replace the 3 lines with the above-mentioned lines, then click on “Execute”. Alright, you were probably not quick enough if you read this at the same time. Practice and you will be fast enough in order to fake the application.
The new values will be interpreted by the application, which is not crashing or displaying values outside of the display, that’s a good point.
And you, what can you fake on the application you are testing? Are you able to make it crash when displaying unwanted data?
Under certain conditions you may have difficulties using Charles Proxy (or any other Man In The Middle tool) if the application forbids network proxies or use SSL pinning. Read for example this well explained article: How to make your iOS apps more secure with SSL pinning
For Android, beginning Android 7 (SDK v24), the SSL network is not directly visible unless you add a piece of code, but it’s probably a bad idea to activate the following trick on any production release. See this very clear article about this subject: Enable Android Nougat ‘Charles’ing SSL network
Charles proxy is a very powerful tool, you can modify what you want and test an application or a website even when some features are not yet available, alter data received, modify requests sent and so on.
I hope this article has been useful to you. Please comment with what you were able to do thanks to Charles Proxy, I’m really interested by your stories.
Welcome to our new weekly recommendations. This week features:
Testing your contracts
The need for testing strategy / test plan document
Breaking the Test Case Addiction
If your company is providing endpoints for one or many consumers and need to test it, where would you start? Steven Burton shares his overall vision in Testing your contracts – Part 1 of 5. Stay tuned for the upcoming posts which will probably be more detailed!
Finally, you read in the first article that “Testing practices and tools need to evolve to address the challenges of achieving “Quality at Speed” amid the increasing complexity of systems, environments, and data.”. But what about slowing down to go faster. Does it seem incongruous to you? Then you should read this article written by Lemi Orhan Ergin: How to Slow Down to Go Faster Than Ever in Software Development
Thanks for your faithfulness, and see you in few weeks for next reading recommendations!