QAOps refers to maintaining software quality by approaching it with a DevOps mindset. You must have come across the term “DevOps”. DevOps refers to Software Development (Dev) and IT Operations (Ops) and establishes a relationship between development and IT Operations. The purpose of introducing DevOps as a business practice is to improve the collaboration between the two business units.
Quality Assurance (QA) plays a major role in the delivery of a quality software product. If it is possible to bring software development and software operations together, wouldn’t it be possible for organizations to bring software QA and software operations together as a business practice? This brings a new flavor to DevOps known as QAOps framework.
Introduction to QAOps Framework
In simple terms, QAOps aims at improving the software delivery process, making it faster and stable without compromising on the quality of your website or web application. Technically speaking, a QAOps framework integrates the QA processes into the software operation to make a more seamless and integrated software operational model. QAOps framework integrates the QA processes, automation and QA Reporting dashboard with the Software Development Lifecycle (SDLC) process. In a nutshell, QAOps takes the core ideas from continuous testing in DevOps, such as CI/CD and bringing together the siloed teams together to work on the pipeline and applies the same to QA process.
Definition of QAOps
While there is no official definition for QAOps, we can define the technique based on these two principles.
- The main idea of introducing the QAOps framework is to integrate the continuous testing in DevOps with the Continuous Integration (CI)/Continuous Deployment(CD) pipeline rather than performing software testing at indefinite intervals.
- QAOps framework increases the collaboration between the QA engineers and developers. Therefore, QA engineers must work closely with software developers, operations personnel and everyone involved in the CI/CD pipeline.
Basic Practices For QAOps Frameworks
QAOps solves the most common disputes that persist between the testing team and Quality Assurance teams. Problems where testers “expect” the issue to have been solved by the QA team and vice versa are completely eliminated through the QAOps practice. QAOps brings in cooperation among the different departments of the organization. QAOps makes use of the following high-level testing practices to put the concept of QAOps into action –
1. Automated Testing
This is one of the main pillars of the QAOps framework. Automation testing means performing tests with the help of technology and tools, and very minimal human effort. QA engineers must study the product in detail and understand the specifications before building the automation framework. Depending on the product and the actual stage of development, QA engineers can decide which tests can be automated successfully to help save time and test the functionalities in a more effective way.
17 Key Benefits Of Automation Testing For A Successful Release
The most obvious types of testing to automate are going to be your regression test cases. They consume the quality time of the testers which can be effectively used to build automation test cases. Similarly, frequently used functionality in the product should also be automated. Why? As your product grows in functionality and features, you don’t want your most-used functions to fail and create a bad experience for the users!
Let us consider a user story to understand this better. Andrew, a web tester who is primarily tasked to ensure browser compatibility of the web-application. Andrew being in the testing domain realizes how time-consuming manual cross browser testing can turn out to be. Testing a web-application or a webpage over hundreds of browsers + OS combinations one by one can consume a significant amount of his time and his bandwidth in a release window.
https://cdn.lambdatest.com/assets_black_theme/images/videos/realtime-video.webm
Not to mention, testing the same dozens or hundreds of browsers + OS combinations is bound to get monotonous and dull for Andrew after a while. Andrew also realizes that if he devotes all of his time in performing manual browser compatibility testing over the same test script and combinations, every week, then finding unique test cases would be a little too much to expect. So what can Andrew do?
This is exactly where automated cross browser testing can come to his aid. With open-source frameworks like Selenium, automation testing can become a lifeline for Andrew to complete his test cycles on-schedule. However, Andrew would still need to figure out which test cases to automate as he cannot automate everything.
Automation testing from scratch will require a thorough phase of planning and documentation. However, once that phase gets over and Andrew has got the right test suites at his disposal, along with the right automated cross browser testing tool then the road ahead is going to be very pleasant.
2. Parallel Testing
As part of the QAOps framework, testing should run quickly (in parallel with the delivery pipeline). Slowing down the testing efforts will directly affect the delivery process. Running automated tests will definitely speed up the testing process, but not when they are being executed run in a serial manner. To overcome this issue, it’s important for testing engineers to run multiple tests at once (together) rather than running them one after the other.
Let’s continue with the story of Andrew that we considered in the example of automated cross browser testing.
Realizing the perks of automation testing over manual cross browser testing, Andrew came up with multiple automation test scripts to cover different modules of his web application. However, there came another issue that he wasn’t ready for! Andrew used Selenium WebDriver to automate his test scenarios for browser compatibility. Selenium WebDriver would only execute one test at a time and will queue the others. As a consequence, Andrew noticed that the test cycles are still going to get delayed due to serial execution of test scripts.
Selenium WebDriver Tutorial for Cross Browser Testing
Now, he implemented his test suite over a Selenium Grid to leverage parallel testing with Selenium. Using Selenium Grid capabilities Andrew was able to run multiple test cases, simultaneously. This reduced his overall test execution by multiple folds allowing him to complete his test requirements on time.
Read More: How Parallel Testing Instantly Improves Your Workflow
Parallel testing requires larger hardware and infrastructure with more CPUs to run the tests simultaneously. But, the benefits of getting results quickly without disturbing the delivery pipeline is definitely worth the investment in a few CPUs when compared to revenue losses due to delayed product delivery. In modern-day technology, businesses can take advantage of cloud-based cross browser testing tools such as LambdaTest which provides a scalable infrastructure to perform parallel testing.
3. Test Scalability
Once you go-live with your web-application, you collect feedback from your customers, work on suggestions, consider incorporating new features in your upcoming sprints. After every release cycle, your web-application continues to scale, and with it scales the testing requirements! As for every new migration that is committed from CI/CD pipeline, the ripple effect of new code changes over an already running code has to be calculated and validated. So when it comes to infrastructure investment, you need to keep the Scalability testing in your checklist.
That is not all that Scalability is about! Scalability testing also helps to determine the application performance at various conditions by modifying the testing loads. The results of the test show the application’s response to differential loads. The testing routine should be scalable with the CI/CD pipeline. At times, the CI/CD pipeline scales up and down depending on the project requirements. During these times, the testing should also be in sync with the CI/CD pipeline. Scalability testing helps QA engineers to reveal the performance-related challenges of web applications.
As a standard QAOps practice, QA teams must have the scalable infrastructure to perform testing and increase the speed of tests when needed.
Let us wander upon what Andrew did to overcome the issues around scalability?
As Andrew’s web application added more and more features to itself. The cross browser testing checklist also expanded for him as there were now few features which were configured using CSS Subgrid or other newly introduced CSS properties which were not widely accepted by the majority of browsers.
As a workaround, developers had set fallbacks for these problems and Andrew’s responsibility was to make sure that these new applied workarounds are working well with the existing application.
He had to consider scaling up his Selenium Grid to test on more legacy and latest browsers, browser versions, operating systems for both Windows and macOS, also for Android and iOS. Turns out if he keeps on adding these devices to his in-house Selenium Grid infrastructure, he may end up having a huge bill on his manager’s desk. Working in a small scale enterprise he cannot go for cloud-based device labs or infrastructure providers such as RackSpace, or AWS device farm. So what can he do?
He looked out for more options and landed on LambdaTest, which offered him an online Selenium Grid that can scale as per his requirements. He was able to eliminate the hassle of maintaining his in-house infrastructure now, as LambdaTest provided him with machines hosted on-cloud, ready to fire up with a click of his mouse. The best part about LambdaTest Selenium Grid was that not only was it offering the legacy browser versions, it was also updated with the latest browser versions. This saved him time, effort, and the headache that comes with maintaining a Selenium Grid of his own.
Integrating Dev & Ops with QA
The final practice to achieve a QAOps framework is to make the QA activities a part of the CI/CD pipeline. One of the easiest ways to integrate Development and IT operations with QA is to make developers write test cases and IT operations engineers to identify potential UI/UX problems with the web application through QA team assistance. This gives the perfect opportunity for developers and QA engineers to collaborate and get a clear picture of the complete QA process. It also does not mean that QA engineers can take their hands off these processes and stay idle. This collaboration between the different stakeholders only makes the whole process of development and testing more efficient.
Back to Andrew, our test automation specialist, who decided to give QAOps ago and shared his perspective with his manager and stakeholders. He realized that the sooner test cases are written, the faster is the go-to-market launch. He suggested implementing shift-left testing to incorporate the QAOps framework in his enterprise. Ever since the number of anomalies caught after CI/CD pipeline migration was notably reduced.
Life Cycle Of QAOps Framework
QAOps is about setting the right platform with the tools on the CI/CD pipeline to make sure the newly built code is tested and validated. The process of setting up the testing platform involves three major steps (as shown in the above diagram). The processes under QA are familiar to us all as they are similar to the major steps involved in an automation testing life cycle to ensure a stable release of the application.
The QAOps process contains three unique steps – Trigger, Execute, and Report.
1. Trigger
One of the important aspects of QA testing is to trigger the right tests whenever an application functionality change is made in the CI/CD pipeline. The tests should be triggered only based on the changes that are made on the functionality. Otherwise, precious time will be lost in testing the unwanted areas of the application (where there are no changes) or critical areas might be missed out. In short, the more the number of tests, the greater the time to run the tests and share the results. To balance this situation, it’s important for the business to map the tests to the features that are getting built.
Therefore, the triggering step should be well planned and mapped with the automation testing life cycle. This is the first step in the QAOps process and if all goes well, the whole team can be confident of the product release.
2. Execute
The next step in the QAOps process is Execution. After the triggering step, different tests will execute on the functionality. As explained previously, it’s important to make sure the tests run in parallel to save time and generate faster results. To achieve parallel testing, make sure you have the infrastructure that can scale and distribute the loads according to the need. Also, make sure the continuous testing in DevOps environment is highly available to avoid any testing hiccups during the QAOps life cycle.
3. Reporting
Once the tests are triggered and executed, the reporting process kicks in. The reporting module showcases the results of the tests. It’s important to design the reporting module correctly to make the QAOps process an efficient one. A perfect reporting module design should provide quick summary information (in a snapshot) as well as provide detailed information. This will be helpful for anyone taking a look at the reports. Additionally, the reporting module should also be able to store the history of previously run tests so that individual stakeholders can compare the results. Reports should be made easily available and on-demand, when in need.
It’s important that you have the QAOps process implemented properly within your business to make sure you get benefited not just for the project that you are working on, but also on other projects within the organization.
Where can QAOps be used?
As pointed out earlier, though the QAOps framework is more powerful with the help of automated browser testing, it does not rule out the possibility of performing manual testing on the application. Manual cross browser testing can be extremely useful to provide detailed information about the web-application, not to forget, you need unique test cases that can come only by manual testing.
With that said, let us look into particular types of testing, apart from cross browser testing, where the QAOps framework can be a handful.
Functional Exploratory Testing
This testing makes sure if the application is running as expected when met with unexpected scenarios on the fly. No test cases are created in advance, as this type of testing is mostly based on the “thinking” of the testing personnel. Experienced test engineers reproduce the potential application crash scenarios and uncover the bugs using this technique.
Here are a few advantages of functional exploratory testing.
- It is more an ad-hoc kind of testing approach with the intention of finding bugs in the web application.
- Cognitive thinking from the tester when compared to scripted code.
- The next steps of action are decided based on what is currently performed by the user.
- Helps the tester to drill down to the smallest areas of the application and cover different edge cases.
Functional Exploratory testing can be used in organizations when a critical application has to be tested and there are experienced testers in the team. It can also be a great learning curve for new testers as they can look at the application from a completely different perspective.
Using QAOps, the collaboration between the QA & Operations team can be well established. With a better understanding of the operation, a QA can deep dive further for exploratory testing in the wild.
Regression Testing
Regression testing plays an important role when you have already developed the software and you want to release an update with a new feature or an enhancement to the existing feature(s). At this time, QAOps will be helpful for test engineers to see if the newly added information has introduced any faults to the existing product.
In an agile software development model, regression testing is often viewed as a process that adds additional overhead to the existing process. QAOps can solve this problem as the process is very easy and affordable by the business.
Geolocation Testing
Every product development company or even for the simple fact, if you are developing a web application, you must know whether it will fit into a particular location. For example, you are developing a B2C web-application for Spain then you have to test over different browsers from various GeoIPs belonging to Spain. Why? It may sound a little shocking to you, but your web application may render differently on different GeoIPs due to internet standards and policies of a country.
With the help of QAOps, your QA team can collaborate with the Operations team to be more familiar with these internet standards. That way they will be able to get the test cycle driven in the right direction from the very start, rather than wandering in the dark and figuring it out by themselves later on.
How QAOps can be used within an organization?
QAOps might seem a similar term like DevOps, but it does not require a complete cultural change in the workplace. This framework aids developers and testers to mutually collaborate and work in getting faster results and/or testing new pieces of the application thoroughly before release.
However, to make this happen, it’s very important to communicate the value of QAOps to key decision-makers and stakeholders (especially the technical team). They must understand the value behind implementing a QAOps framework. By doing this, it helps the rest of the organization (developers, testing engineers, IT operations managers, etc.,) to understand the requirements and work accordingly. It will take time to convince the stakeholders about this new framework; however, once in place, the company will start realizing the importance of having the QAOps framework in practice.
QAOps Framework – Different School of Thoughts??
There are different school of thoughts that revolve around the concept of QAOps. While it is relatively a new term, some people even question if the term is a valid one or not. A thread on Reddit says that “QAOps ain’t a thing, and it can never be a thing”. QA is a part of the DevOps process and there is no point simply adding an “Ops” term after the word QA. DevOps includes every aspect of development as well as operations and if QA is a part of the development pipeline, it will be included as part of the development cycle. On the other hand, there are social media tags being used for QAOps by various software testing communities.
QAOps was also a proposal raised in the Selenium conference 2018.
Even though the debate exists whether the term QAOps should exist or not, from a business perspective, it’s important for the company to understand the importance of QA Testing. In this modern age of DevOps, QAOps helps to highlight the challenges faced by the QA engineers and showcase the importance of keeping QA testing in line with the DevOps process. QAOps has immensely helped the testing engineers to be on a par with the development (CI/CD) pipeline.
As discussed earlier, QAOps will take its time to get into organizations and become a part of the DevOps process. The stakeholders must explain the benefits and the value of the framework to the key decision-makers and make them understand the importance of it for the organization. Once the framework is introduced, the framework will emphasize the importance of automated software testing and QA to deliver quality applications.
Can QAOps Be Referred As Continuous Testing in DevOps?
In a common DevOps approach, both developers and testers play a vital role. It’s the role of the QA that brings together both the development team and the IT operations together. When we say QA, it’s not just plain testing but continuous testing in DevOps! An organization with a strong strategy for continuous testing in DevOps makes them collaborate to deliver the best web application in the quickest of time. As a result, it’s fair enough to refer to QAOps as Continuous Testing in the DevOps approach.
Let’s see how organizations can embrace continuous testing in DevOps across their different business functions.
- A Continuous Integration environment (with both CI/CD) should be configured in order for the developers and testers to perform the tests on the code.
- Dev teams MUST perform continuous testing by testing every new functionality that is built in the application. They must test the application for its functionality, performance, ease of use of the interface. One of the best tools for this purpose is Selenium. They must build aggressive test cases to ensure every edge case scenario for the functionality that is built. It is the developer who is the best person to know the entire functionality in detail since he built it.
- In parallel, QA teams must also start testing the new functionality to stay on course with the development pipeline.
- After the testing is completed by the QA team and sign off is given, the IT Operations team must perform a deployment of the web application and provide sign off. Only after the signoff, the release process should begin.
- Similarly, post the release activities, the IT Operations team should keep an eye on the web application to make sure everything is working as expected. They must monitor every aspect of the application to ensure efficiency.
By integrating the Continuous testing process into the SDLC process, businesses can achieve a successful product delivery cycle.
Best Practices
Based on previous experiences and hearing from a lot of people about the DevOps practices across different organizations, here is a set of best practices to embrace Continuous Testing as a part of the DevOps process.
- The Continuous Testing process should be integrated into the software development lifecycle. This helps businesses to reduce the risks and ensure faster product release and time to market.
- All the teams in the organization (dev, IT operations, QA) should be part of the entire release cycle. They must ensure to collaborate and communicate effectively during the entire course of the release.
- CI/CD should be done on a regular basis, preferably daily basis. Everyone’s work must be merged into the release pipeline. This helps in identifying the defects at the early stages and rectifying them before it’s too late.
- Quality teams should be a part of the release planning, requirements gathering process. It will be more effective if the quality teams work alongside the development teams to provide vital inputs during development.
- Every team’s metrics should be defined and measured at defined intervals during the release process.
- QA test engineers should make use of automation tools and scripts to automate as much of the testing possible. Deep regression testing should be performed on the existing features when new functionality is developed to make sure existing features are not broken.
- Developers must start thinking as testers, and testers must do the vice versa, (i.e.) fix the code. This makes everyone in the organization takes responsibility for the overall quality of the product.
Wrapping Up
QAOps, Continuous Testing in DevOps as we can also call it, when correctly designed and implemented, definitely paves the road to faster software delivery. This gives the development teams the confidence for a faster time to market without any quality compromises. Reiterating the same point from above, putting QAOps into operation requires a lot of convincing to the stakeholders of the business. Once you are past this stage, it should pretty much be a smooth sail forward.
Published on Java Code Geeks with permission by Praveen Mishra, partner at our JCG program. See the original article here: What Is QAOps? And Why It Matters For Your Web Application? Opinions expressed by Java Code Geeks contributors are their own. |
