TestOps Katalon – Overcome 06 challenges of continuous testing

QA professionals have been facing multiple challenges implementing Continuous Testing daily to contribute to a productive development cycle. Within today’s article, we will dive deep into the most common bottlenecks of testing, what recommendations from experts are, and what TestOps Katalon has to offer Katalon Studio Enterprise users.

06 Challenges of Continuous Testing

From the Continuous Testing Report 2020 by Sogeti and Broadcom surveyed by 500 IT executives from the top North American and EU Companies, when being asked how challenging it was to implement continuous testing:

• Almost 62% of the respondents said they found it challenging to have skilled professionals to build their continuous testing strategy.
• 56% of the respondents mentioned that they could not achieve in-sprint testing.
• 63% of people surveyed admitted that they can not get enough visibility or information throughout the development lifecycle to act accordingly.
• Meanwhile, embedding non-functional testing into the pipeline or access the on-demand test environments and test data are aware issues of over 60% of the IT executives.

Finding skilled professionals to build a continuous testing strategy

DevOps is not an easy process, and embed tests into DevOps to achieve continuous testing is not within everyone’s skill set. Most business entities depend on skillful engineers to make it happen. However, the shortage of professional resources is becoming a big barrier to companies’ success within this field nowadays. Especially, the pressure to cut costs and maintain businesses are more real than ever during this hard time of the pandemic. This creates more challenges for companies to retain their talents. Therefore, business organizations commonly find it difficult to have skillful professionals to overcome the problem of having testing happened in their DevOps.  

Getting visibility throughout the development cycle

The ability to deliver new functionality with high speed and confidence is the key to competitive advantage for the business nowadays. However, not many organizations can keep up with this fast pace.

Every company operation relies on some sort of software and services. Failure to deliver quality in applications directly impacts the company workflow and the company’s brands. Let’s say if you can not access the Amazon website for just a few minutes, it will heavily hurt the Amazon Brand and create a significanttest envir revenue loss. Therefore, the visibility into performance, value, and return on investment on what we are building and delivering are vital to an organization’s success.

However, 78% of the respondents agreed that they were still far from making that full visibility happen. Furthermore, the results of this survey are very similar to last year’s report indicating that the well-known issues are persistent.

Accessing test environments and test data on-demand

From the test environment standpoint, the ability to spin up the environment dynamically is the key. Limitations within test environments and test data prevent businesses from achieving a high level of maturity in continuous testing. However, as per the report, the test environment is one of the greatest impediments to continuous testing. 36% of the professionals mentioned that they spent over 50% of their time managing just their environment, which is a lot of time.

In test data management, given the current Given the current compliance of the General Data Protection Regulatory (GDPR) in test data management, it is critical to have virtualized synthetic data generated. It needs to be sorted in a way that aligns with the customer regulation standard.

The report has showcased several issues regarding test data and highlighted that more than half of the IT engineers surveyed are in line with the dependencies on administrators to get the data they need. It means that they have to wait. 47% mentioned the challenge of having the right test data version. Having to manage different versions of test data for different versions of software is the real pain.

Having the right testing strategy, process, and methodology

In World Quality Report 20-21, Capgemini & micro Focus consolidates 1750 interviews with employees with various titles from multiple industries across the words. The interviewees got to answer how often their QA team achieves mentioned targets when testing key applications?

Only 45% of people thought they had the right testing strategy, process, and methodology. This means more than half of the interviewees faced difficulties figuring out the right ones when testing for key applications. This is one of the sequential consequences of the lack of skillful resources that were mentioned previously. It is also because of the nature of the testing process. This is a topic that companies need to continuously discussed to optimize their testing cycle.

Enough time available for testing

It is also not a surprise that having enough time is challenging for most of the QA team. You can frequently find yourself under pressure to ensure the quality of the product. Especially at companies that still only test at the end of the development cycle, where there was not much time left. The problem became worst during this time of Covid 19 compared to the previous year. It is not rare to see the QA admit to sacrificing their quality coverage to meet their delivery schedule and vice versa. Furthermore, the interviewees believed that this percentage should go lower if they can talk to each of the team members.

Have the right level of test automation within the organization.

Another challenge is to have the right level of test automation within the organization. Only 58% team interviewed thought they were going well with this. The challenges haven’t been resolved since the number is as same as in last year’s report. Many people are moving forward to reduce the redundant tasks, which explain the popularity of test automation tools in the last few years. It is importantly attached to the implementation of intelligent autonomous test solutions. The automated processes can run the software, identify needs, spin out test cases, run those tests and generate the run reports without interventions. This is the procedure that many organizations are heading to and also the goal of Katatalon as the automation testing solution provider.

Experts recommendations to overcome testing blockers

From the Continuous Testing Report 2020 and the World Quality Report 2021, experts suggested three key recommendations overcome the persistent blockers in testing:

1. Test Orchestration

Test Orchestration will fuels efficiency and transparency throughout your DevOps pipeline. Continued testing automation and orchestration go hand in hand to achieve continuous testing. While test automation address a single task, test orchestration embeds testing into the CICD pipeline to tie together both automated and manual tasks.

Test Orchestration allows each and every phase of the development lifecycle to be transparent. It provides comprehensive validity into each phase and keeps track of the corresponding testing activities – member in charge, any area of improvement, and track the success metric over time.

A sufficient Test Orchestration would help solve:

• Lack of skilled engigeers/ professtionals
• Lack of visibility throughout the SDLC
• Lack of test environment & test data on-demand.

2. Shift-left & Shift tight Testing

Shift-left & Shift-right Testing will make quality not only the responsibility of the QA team but shared ownership. This helps fasten feedback loops & deployment speed. The approaches sound contrast but in fact, can be applied at the same time.

Shift left testing is a well-known process that most professionals have adopted to identify areas of improvement as early as possible. Your team can defect, mitigates risks with speed and reliability across all disciplines such as unit, functional, API, performance, security integration, etc.

On the other hand, shift-right testing is still far from being widespread. A solid shift-right practice includes learning from a decisive view of production monitoring data to understand real user behavior. You can craft new iterations based on those insights and increase the accuracy of the validation process. When we know the end-users behavior, we see what happens significantly on production to focus on what matters to the users rather than putting all the effort to cover everything. That is the spirit of Shift-right Testing.

Shift-right testing relates to techniques such as features flags, crowd testing. It is also about the mechanism that allows companies to put code to production as fast as possible and have it verified from the end environment but still keep it safe. It provides you with the capability to have quick action in case anything goes wrong in the production.

Applying Shift-Left & Shift-Right testing would help solve:

• Lack of time for thorough testing: Testing will be happening throughout the whole process. From the time we receive requirements, build code to the time we have the new functionalities live in production. It is not one time work anymore. We test more in the life cycle to save time and spend less testing effort at the end.
• Lack of in-sprint testing: Testing happens early within the spin of development without waiting for another sprint to complete.
• Lack of a fitting testing strategy process and methodology. These are excel methodology that will give a decent plan of ways to go and we know what to do, when to app in those approaches.
• Find the right level of automation. We do not need to do everything, but we have the visibility to understand what matters the most to focus on.

3. Artificial Intelligence and Machine Learning

Artificial Intelligence and Machine Learning provide us with a smarter way of working to free up space for more value-adding tasks.

AI/ML is no longer a “good-to-have option” but a key criterion for most customers when choosing between new QA solutions.

AL/ML have replaced repetitive manual tasks, optimized the entire process from different aspects, and ease the need for scaling for organizations.

Applying Aritificial Intelligence and Machine learning would help:

• Resolve tradeoffs between Quality and Speed
• Enhance Efficiency and effectiveness of the whole process
• Eliminate testing bottlenecks in the SDLC

Enabling Quality at Speed with TestOps Katalon

Katalon offers various products that cover different aspects of the testing lifecycle.

• Katalon Studio Enterprise
• TestOps Katalon
• Katalon Recorder
• Katalium

Among those four products, Katatalon Studio and TestOps are the most well-known platform in the market.

Katalon Studio Enterprise

Katalon Studio is positioned as an All-in-One test generation solution for Web, API, Mobile, Desktop testing, and more to come in the future. The tool covers a wide range of testing environments such as Android and iOs, Firefox, Chrome, Safari, and the new Edge running on Chromium. More than that, Katalon Studio is available on container platforms such as Docker and Cloud Services like Amazon, Azure DevOps to deploy automation testing to Cloud DevOps. This helps enable remote working,which is a norm nowadays. Overall, Katalon mission is to deliver a simple, extensible, and scalable low-code platform.

2020 was the second consecutive year that Katalon was named the Best Automation Testing  Software reviewed by its’ users and customers on Gartner. The company has just announced a successful raise of $27 Million in series A funding which will be used to scale the go-to-market model following successful patterns from high-growth companies such as Atlassian, GitHub, GitLab, Postman, Snyk, and Auth0.

TestOps Katalon

On the other hand, TestOps is the new commercialized solution for on-demand environments. Once the test scripts is generated by Katalon Studio Enterprise, TestOps Katalon will take care of the rest of the testing life cycle.

Below is the more detailed architecture build of TestOps Katalon, which is the main focus of this article.

TestOps Katalon is built to be an Open – Complete – Smart Test Orchestration Solution. It consists of various components from Test Management – Test Planning – Configuration – Report & Analytics. The Idea of TestOps Katalon is not only to provide the whole solution for testing life cycle from planning to monitoring but also to integrate with any of your assistant tools such as ALM, CICD, environment provider to help you archive continuous testing.

TestOps Katalon – Overcome 06 challenges of continuous testing.

TestOps Katalon approach to overcome those challenges are very similar to the recommendations from experts. It enables quality at speed within by:

• Being an open, complete & smart test orchestration platform. The tool helps customers deal with test planning, configuration, execution, test environment, Katalon report, test analysis. Without professional skills. Users still get visibility throughout the testing cycle and embed automated into the DevOps pipeline.
• Making Shift Left & Right Testing adoption feasible for business. It seamlessly integrates with other platforms and methodology such as ALM platforms, production monitoring tools or BDD TDD approaches.
• Being powered by Artificial Intelligence and Machine Learning. Katalon is also gradually introducing Mac features into the product.

TestOps Katalon highlighted features.

Let’s dive into a real use case and take a closer look at the example workflow below. You can have an idea of how to utilize the combination of Katalon Studio Enterprise, Katalon Runtime Engine, and TestOps Katalon to:

• Optimize Test Execution Speed
• Boost productivity of test investigation
• Achieve comprehension and transparency to quality & coverage.

1. TestOps Katalon Integrates with Test Generation tools

Firstly, You can generate a Test Suites collections that consist of multiple Test Suites using Katalon Studio  

2. Configuration for the test run in TestOps Katalon

Then you can get into TestOps Katalon. At Configuration, you can:

•  Configure Script Repository from TestOps to use the projects from Katalon Studio directly or from Git.    

• See your test Environment used: TestOps currently support Local environment, Kubernetes, Circle CI, TestCloud

• Set up your Agent to be Local or Docker

3. Schedule Test Run

Then you can set up a Test Planning to schedule the run of the Test Collections above.

There are 3 ways for you to pick the desired Project to run:

• Test Suites Collection:

• Katalon Command

• Or you can pick Generic command in case you do not use Katalon Studio as the Test Generator and Test Execution.

You also can choose to run in parallel to save execution time.

4. Testops Katalon Live Report

When the test is executed, you can see the live execution at the TestOps Katalon Reports Dashboard,

For example, you select to execute the 4 Test Suites below in parallel with the 2 agents “ My Agent” and “My Agent 2”. Katalon automatically distributes the agents equally to the targeted Test Suites.

“My Agent” will take care of 2 Test Suites, and “ My Agent 2” will take care of the remaining ones.

Execution Auto distribution

Another beautiful feature of TestOps Katalon is auto distributing the executions to available Agents for an on-time schedule.

For example, You can see that “My Agent” took more than 13 mins to complete the first execution. There might be some problem here that resulted in a fail execution.

On the other hand, other executions take only 4 minutes.

When “My Agent 2” finished the Test Suites, it does not need to wait for the “My Agent” to run the existing ones. It will instead proactively pick the TestSuites previously assigned to “My Agent”

Once we have that capability, all of the Execution will be distributed equally to the capability of our execution. This helps us save a lot of time and boost the execution.

High Level Katalon Reports & Analytics

Finally, once the run is done, you will have beautiful Katalon Reports and Analytics from the high level to the detailed level.

As the stockholder, QA Manager you can easily monitor the test activities as well as it’s quality.

Detailed Report & Analytics

As the QA lead or the tester yourself, you can even look more into details on your test performance and see where things go wrong. With TestOps data-driven report, you can see detailed logs, pictures, errors.

Report that is helpful for debugging

You even compare the result with other runs using “Similar Failure” features which will help your debugging life a bit easier.

TestOps basically identify the same problem encountered in the previous run. If you have resolved that issue already, defect and linked it with a Jira task, and let’s say your co-worker has investigated the root cause. You can utilize it without going through the same process.

You can also group the failure and let the team sit down to investigate them all at one time. This helps increase the productivity of your debugging process while reducing significantly your investigation time.  

Release Readiness

The Main Dashboard of TestOps Katalon aims to provide you full visibility of the testing life cycle in just one place.

It will be easier for you to understand what is going on and what decision to make accordingly.

Linked to Jira release

Of course, these are not everything TestOps Katalon provides.

TestOps also connect with other CICD tools and other ALM to pull more information here for all the data.

But you can see that with the release readiness provided by TestOps Katalon today when you have the integration with Jira, it will pull Jira release into TestOps. The tool will also pull the requirement that we need to implement for that release into TestOps.

• At the Day to delivery: you can see how many days left until the release date of each release as planned.
• At the Development: the number indicates the percentage of requirements had been changed to “Done” status.
• At Test: you can have the overview of the passed test cases proportion

You can also go to the release to see more information on which Test Cases passed or failed along with its detail and its flakiness.

Platform Statistics

There is also “Platform Statistics”, where you can see if there is anything go wrong with a specific type of environment including OS, browsers,

Builds information

Besides, you will also see Builds information. This is very helpful for a team that needs to make decisions on which build can be picked and put into production.

Requirement Coverage

Finally, Requirement Coverage capability is a cherry on top. This is where you can go to have an overall idea of where things stand in terms of the requirements within the happening release.

For example, there are 4 Test Cases scheduled for this release, and 1 of them had failed.

The majority of the requirements in green have been fully executed successfully.

This map provides you with the overview quality of the requirement and helps you connect requirements to Test Cases and to Test Runs results.

Traceability Matrix

Another view you may find very useful is called “Traceability Matrix”

With this view, we have the link throughout the requirement to Test Case, Test Result, and Defect. We have been on calls with several customers and apparently, many of them were doing this manually. This is why Traceability Matric was built to give you an automatic view of test quality in just one place.

TestOps Katalon Live Demo

Interested in seeing a live demo with TestOps Katalon? Book a call here with a Katalon Insider or leave a message below: