Automated software testing is one of the mandatory activities that any software organization should perform in order to be able to guarantee quality of their products. However, very often this process turns to be a pretty complicated thing, especially when it comes to automated testing of modern complex systems consisting of a number of different parts. All those parts are based on different technologies and, apparently, different technologies should be used to test them. In reality QA engineers operate with various testing tools in order to perform testing of each part of the system, to test the whole system, to perform different test types such as functional, performance, integration, etc. Each testing tool produces its own test results and QAs should gather all that data from different sources and analyze them in order to be able to make any decisions about quality of the system. It would be nice to put all that stuff to one place and to completely automate this complicated process. That would increase quality of the test automation itself and allow to make automated testing as a step of the Continuous Delivery process.
I have posted already about a new DevOps and CD tool for Fusion Middleware applications FlexDeploy. I was pleased with the opportunity to have a look at the beta version of the system (3.0) and was able to kick the tires. Among many other cool features the new version delivers Test Automation functionality. FlexDeploy 3.0 becomes a powerful instrument that puts Test Automation process under the control and makes it much easier by implementing the following concepts:
- Central management panel to perform, control and monitor all automated testing activities in the organization
- Hub for gathering test results from the all used testing tools
- Central tool for building QA dashboard reports across the organization for making “Go-Live/Not-Go-Live” decisions
- Support Automated Testing as a mandatory step of the CD process
In this post I am going to highlight the key FlexDeploy features related to Test Automation which may help to understand how the concepts above are implemented and how it actually works.
Integration with 3rd party testing tools
Actually FlexDeploy doesn’t perform any automated tests itself. This job is to be done by a great number of various testing tools. FlexDeploy just controls the process and, obviously, it is able to interact with the testing tools. This interaction ends up with the following tasks:
- Execute tests
- Import test results
The integration with 3rd party testing tools is implemented by the use of plugins. At this point FlexDeploy 3.0 provides plugins for JUnit and TestNG tools only. However, the range of plugins is going to get much wider in the upcoming versions. As far as I know the tools like Oracle Application Testing Suite (OATS), Selenium, Test Complete, etc. are on the roadmap. Actually, if FlexDeploy doesn’t provide a plugin for some testing tool out-of-the-box we can always use Shell plugin which allows to execute batch scripts in order to interact with the testing tool.
Basically, the plugins act as adaptors for various testing tools. They command testing tools to run tests and they import test results into unified test result storage. This allows to analyze test results and build dashboard reports abstracting away from the exact testing tool.
Test Definition
FlexDeploy introduces a notion of Test Definition. This abstraction represents one or a bunch of test cases related to some business use case. For example, for a banking system we can define test definitions such as “Loan arrangement”, “Loan repayment”, “Loan overdue”, etc.
Basically, when FlexDeploy is running automated tests it is, actually, running test definitions one-by-one. And each test definition, in its turn, is running the actual test cases with a corresponding testing tool. It knows what testing tool is going to be used, how to interact with it, what bunch of test cases (defined at the testing tool) should be used, how to import test results and how to qualify them. In order to interact with a testing tool a test definition uses a workflow. Workflows is a powerful mechanism which is commonly used in FlexDeploy to perform build and deploy activities. Now it is used for automated testing as well.
Actually a test workflow invokes a plugin to interact with a testing tool. Besides just running test cases and importing test results, a test workflow can also perform any other activities as well. For example some preparation work before running tests (like warming up phase) or some finalization activities after running tests.
Test Qualifiers
A test definition is able to analyze test results and come up with the conclusion whether the test run succeeded or failed. This feature is based on test qualifiers. A test definition can contain a number of test qualifiers such as “Number of Test Cases PASSED greater than X”, “Percentage of Test Cases FAILED less than Y”, etc.
If all test qualifiers, defined at the test definition, return true, then the test run is considered successful. FlexDeploy provides a set of some predefined test qualifiers, but they plan to allow users to define their own custom qualifiers basing on Groovy expressions.
Test Set
A test set is just a group of test definitions. For example for a banking system we could define test sets such as “Loans”, “Deposits”, “Forex”, etc. A test set can contain its own qualifiers in order to check whether the entire set of test definitions passed or failed.
Test Execution
At this point automated tests can be executed either manually:
or automatically as a step of a deployment process:
I guess the upcoming versions of FlexDeploy will allow to automatically execute tests on a schedule basis independently from the deployment process.
Each test execution (either manual or within deployment) comes up with corresponding test results:
Dashboard
FlexDeploy 3.0 provides a new dashboard tab “Test Automation”. It represents the passed/failed ratio of test executions across environments, the duration time of test runs and response times:
This information is also available on “Test Automation” section of the project definition. In this case all the charts are based on test results related to this project:
By implementing test automation functionality FlexDeploy 3.0 has made a giant step along the path of getting continuous delivery process fully automated. Of course, there are still lots of things to be done, but they have built a good basis which is going to be just enriched with new test automation features in upcoming FlexDeploy versions.
That’s it!
| Reference: | Automated Software Testing of Fusion Middleware applications with FlexDeploy from our JCG partner Eugene Fedorenko at the ADF Practice blog. |
