Table of Contents
Introduction
How does testing an app on three platforms with a single test sound? That was the goal of our framework, we aimed to test the webapp and ios/android app with one single test on the User Interface side.
We as QA engineers always try to optimise execution and maintenance times and even testing strategies making it easy to write tests in an efficient way we even wanted to involve our POs or anyone who had interest in writing test for that matter in the creation of tests using the cucumber tool.
This Article is about the steps we took to make that possible, what challenges we encountered and what possible solutions were found and the ones we decided to adapt, we also explain the use of tools like browserstack cloud platform and how it makes the automation run smoother and faster.
Requirements Analysis
We’re starting off by our requirement analysis, in this chapter we’re going to talk about the chosen tools, every tool has to have a purpose and lets face it, some tools also have cons so when it comes down to it have to makes sure the pros outweigh the cons.
We also have to think about compatibility between tools, that means that every tool we chose has to be compatible with all others and vice versa, so a lot of research has goes into it.
One more important thing that one should keep in mind is that not all tools are free, some need a monthly fee, like a subscription an obvious example would be Browserstack our cloud platform.
As mentioned before, Browserstack is a provider of real remote devices and we had to pay a monthly fee for the five threads we had available. We chosen this particular cloud based platform because not only is easy to integrate but it also only offers ‘real devices’ and that is important when it comes to testing, we want to make sure we get as close to the real thing as possible, in my opinion a simulated or an emulated device will never be the same as a real device, of course it can come close but there will always be differences.
The next tool we also mentioned is cucumber, this is more of a way of testing than a tool, it supports Behaviour Driven Development (BDD), it helps explain the application flow aspect in Gherkin syntax. It is one of the most wildly used automation testing tools due to many reasons, first of all its open source and free and it supports multiple languages like Python, Java and JavaScript.
It can be integrated without any problem with Selenium, Appium, Ruby on rails and other testing tools.
A big plus for us was the little amount of technical skills required to write tests, as we wanted not only the technical members of the team writing test but any one interested in doing so to be able to formulate tests, this is possible with cucumber unlike many other automation tools and once the tests are written they’re clear and easy to understand, here’s an example:
Feature: Login
Scenario: Login verification
Given user is on login page
When user enters credentials and logs in
Then verify login is successful
Any of these phrases can be reused without the need to add or edit any code.
For mobile automation we chose Appium, an open-source framework that allows the QAs to conduct automated app testing on different platforms like Android, iOS, and Windows. As for web automation we chose the most used framework, Selenium, which is used for webapps and does not support system or mobile application hence the necessity of using both Appium and Selenium as automation frameworks.
The next tool on the list is GitLab CI/CD which is the part of GitLab that you use for all of the continuous methods (Continuous Integration, Delivery, and Deployment). With GitLab CI/CD, you can test, build, and publish your software with no third-party application or integration needed. Further down the article examples of Gitlab CI/CD scripts will be shown explaining how it works.
Every respectable QA framework has to have a report, for this particular one we chose Serenity Report as it works extremely well with cucumber making the report fast and easy to read, as a plus the serenity report is widely customisable making only the relevant information needed to appear in the report a long with important details like Emailable HTML reports, custom field reporting, and gradle setup and If you are using Gradle, you can use the report task to generate any configured extended reports, which brings us to our next tool, Gradle(more on reporting further in the article).
I think many will be questioning this choice versus the Maven tool, so let me just say both are able to cache dependencies locally and download them in parallel. As a library consumer, Maven allows one to override a dependency, but only by version meanwhile Gradle provides customisable dependency selection and substitution rules that can be declared once and handle unwanted dependencies project-wide and that’s why we chose it over Maven.
To finish this chapter, the is only thing left to mention is the development language, we chose Java because it has significant advantages over other languages and environments that make it suitable for just about any programming task.
Test Planning
What is a test plan?
A Test Plan is a detailed document that describes the test strategy, objectives, schedule, estimation, deliverables, and resources required to perform testing for a software product.
The test plan helps us determine the effort needed to assure the quality of the application under testing, it also serves as a blueprint to control software testing as a whole which is monitored by the QA team. As I see it, it is important to have a test plan to help the people outside the team understand the details of testing.
How did we ensure a correct TEST PLAN?
Test Strategy
- Scope
The scope of testing is very important, we should know what part of the app is assigned to our team, what should, what shouldn’t and what can’t be tested via automation testing, there are a lot of advantages to automated testing but like everything is has some limitation as well.
In this case our scope was basically the main part of the app, subscription, login, authentication and home page.
The framework we built was meant to automate this part for all three platforms.
2. Testing type
UI testing, graphical user interface testing is the process of testing a product’s graphical user interface to ensure it meets its specifications, we also had envisioned API testing but we didn’t get that far, it is safe to say that our framework was prepared to integrate API testing.
3. Documentation
The documentation is essential for the continuity of any project, reliable documentation is a must it helps keep track of all aspects of an application and it improves the quality of a software product, the main goal of documentation is development, maintenance and knowledge transfer to other developers in case it is needed.
We created a readme file with all the documentation and all the steps necessary to prepare a testing environment, automation tools, environment access, testing mobiles setup, accounts used for testing and even local machine setup for running test in local.
Test Objectives
Test planning is where all elements connect together, all the teams, ios, android, web including our POs, after they evaluate the customer needs and define our plan/strategy, each team develops new functionalities that needed testing, thats where we come in as QAs, our objective is to test each of theses functionalities before they get to production.
Resource Planning
We came together with the developers to define what tests should be prioritised, each member of the team had assigned a number of test cases, we each made sure the tests were completed before the end of the sprint, making sure the continuity of the cycle.
The number of threads our cloud platform (Browserstack) had was another resource to consider, we only had five threads, so between us members of the team (three) and the automation running in the background we had sometimes to calculate, adjust and adapt our time to avoid wasting time waiting for threads to clear.
Same goes for gitlab CI, sometimes we had to wait for ports to clear so that the automation process could pick one and launch the automation, we shared the repository with other teams so that part was difficult to adapt.
Test Case Development
A test case is a scenario measuring the functionality of many actions or conditions to verify an expected result, they can be for manual testing or automated testing, a very important thing to keep in mind is that test cases are intended to test basic behaviour such as a login action, many make the mistake of creating a long test case which becomes a nightmare to maintain specially in automation defying the purpose of automated testing, having a small controlled test case allows more flexibility in how to test features.
1. How to write a test case?
Step 1: Test Case Locators
Test cases should all bear unique IDs to represent them. In most cases, following a convention for this naming ID helps with organisation, clarity, and understanding of the test case, in the second part of the chapter I explain more about IDs and why they should be selected over xpaths.
Step 2: Test Description
This description should detail what unit, feature, or function is being tested or what is being verified,m this is done through the cucumber language as seen before, it is a step by step process explaining how the test will run and with the help of a feature name and a scenario name a simple look at the test case should be enough to understand the goal of said test.
Step 3: Assumptions and Pre-Conditions
This entails any conditions to be met before test case execution. One example would be requiring a valid account for a login, a cucumber word is used for preconditions, for example:
GIVEN the user already has an account
Step 4: Test Data
This relates to the variables and their values in the test case. In the example of an email login, it would be the username and password for the account.
Step 5: Steps to be Executed
These should be easily repeatable steps as executed from the end user’s perspective. For instance, a test case for logging into an app might include these steps:
- Open email app home page
- Enter username.
- Enter password.
- Click “Enter” or “Login” button.
An example using Gherkin/cucumber as seen before tool would be:
Feature: Login
Scenario: Login verification
Given user is on login page
When user enters credentials and logs in
Then verify login is successful
2. Some of the issues we faced and how we were able to solve them
- Missing ids:
For any successful UI automation locators or element ids are needed, it’s how the automation tool knows where to find a specific element on the view.
There are many ways to locate elements on the view, some are less reliable than others, using xpaths for example is unreliable and is considered bad practice to use them, simply because if the element location is changed or a new element is introduced into the view, the xpath locator will most likely change making it impossible to locate the element, and will result in an “element not found” error, on the other hand using IDs is way more reliable because first of all they are unique, so even if the element were to change place or if we had a new element introduced to the view the automation tool would find the element by id, another benefit of using IDs is the speed of the automation because most tools/browsers are optimised to find ids first.
The issue we encountered is having missing IDs for elements needed for the automation, it is a well known issue for QAs, as a temporary measure we used xpaths when possible for the sake of advancing with the automation but always keeping in mind that a locator refactor was going to be needed, we also made sure that the refactor was going to be minimal by preparing the framework in such a way where a single only a single change to the code was sgoing to be needed and so one of our team members coordinated multiple meets with the developers to solve this issue.
- Views unable to load:
Automating ios, we would be sometimes faced with the issue of the Appium automation tool unable to load certain views, this is because of the huge amount of information a certain view has, sometimes it would take up to five minutes to load a view which is not sustainable, the solution here is to limit the tool to load up to a certain depth in the view where the element we need is located, making it easier and faster for the tool.
The Appium capability we used for this is –> “SnapshotMaxDepth”:”10″ and this would limit your locator tree to ten layers, SnapshotMaxDepth is set to fifty by default.
- Execution times:
An issue all QA automation engineers face is execution times specially when it’s UI automation and have a regression test to run, regression testing is a software practice that ensures an application still runs as expected after any code changes in the app, the regression is responsible for the application stability overall.
The best practice here would be using a function that waits for the element to show on the view instead of using sleeps, which is definitely not a good practice, when the waitForVisible function runs, it checks for the element we want to find every 500ms, this ensures we go on with the test case once we find the element opposed to putting a five second sleep where even if the element is present, the test will still be on hold for five whole seconds.
Test Environment & Test Execution
First of all, what is a test environment?
It is usually hard to create an exact copy of the production environment, in this chapter I will explain the difference between test and staging environment and I will explain the role of these test environments and how we used them.
A test environment is an environment that allows you to run already defined test cases, it enables you to create identical environments every time you need to test a product, it’s very important for a testing engineer to ensure reliable results, a testing engineer uses test environments to test a very specific part of an application, and what makes this environment so different than production is the setup needed for testing different parts of the app.
A staging environment replicates the prod environment, it is important that this environment is an exact copy of production, a very detailed documentation is needed to achieve this.
While a test environment is focused on testing individual components, the staging environment is focused on testing the whole application. Basically, the staging environment is a safe playground in which you can test the whole application and this is what makes staging ideal for end-to-end testing or performance testing.
We managed these environments through automation by build a deployment automation CI, as mentioned before the tool we used is Gitlab CI, this tool helped us automate and run test suits through scripts in which we mentioned the targeted repository the tests we want to run, the chosen environment and obviously a platform of choice ios/android/web, which was launched on browserstack, our cloud provider.
Architecture
The framework was divided into three packages iOS, Android and web each containing:
The elements or locators.
The flows containing the actual flow of the test:
Implementation
- Other examples of BDD testing would be:
- HOOKS:
Cucumber Hooks use Before and After.
It works at the start and the end of the test scenario or feature. As the name suggests, @before hook gets executed before any other test scenario, and @after hook gets executed after executing the scenario, this helped us control some of the precondition.
Our Hooks had more uses one of which was having our tests named in BrowserStack, making easy to distinguish between tests, specially if multiple are running, this was done by calling a function in the before hook to send the name to browserstack as a capability or preset to the test.
Another feature of the framework is profiling, we can basically create a Json profile that has chosen capabilities like screen size and browser for web tests, exact mobiles and operating system version to execute our tests.
Reporting
After each execution we get a report generated by serenity where details of the run are shown, serenity is a tool used for reporting as well as other thing like writing actual tests using its framework, we had it implemented because of the rich reports it offers, serenity reports can be customised and adapted to the needs of the framework, in addition to standard HTML reports it provides emailable reports and even HTML reports as a single page react application.
This tool can be set up using Gradle to generate any configured extended reports, you can even customise it by overriding property values by passing in other properties to override the ones by default, all this is done in the serenity.properties file.
This is an example of a report:
And this would be the build info screen:
Conclusion
Quality Assurance is a very complex process that includes many steps to ensure the security and quality of the software under development, when the QA process is properly organised and executed it enables you to track and eliminate bugs, performance issues and even security issues before the release. The framework we created was meant to do all this for three platforms while keeping an optimal performance and needing as little maintenance as possible regarding the core architecture of the framework, we adapted it to both our needs and the clients needs.
Neglecting QA activities may create the illusion of faster feature delivery or reduced costs/resources for the project but in the long run not paying enough attention to the quality assurance process will lead to serious risks both technical and commercial and would even jeopardise the security of the product, in my humble opinion outsourcing QA is the key to building a stable secure product that users will find comfortable and pleasant to use.
