Regression Testing - Test Suites
Regression testing is supported by Real Load using “Test Job Templates” and “Test Suites”.
Features of Real Load Test Suites:
- Test suites are the grouping of multiple test job templates and are executed as a single “run” with different test scenarios.
- Due to the universal architecture, test jobs of any kind can be part of a test suite (HTTP Test Wizard Jobs, JUnit Jobs, Selenium Jobs ..).
- The results of the test suite “runs” are archived and can be compared with each other.
- In addition, the result of a particular test job of a test suite run can be compared with the result of the same test job executed by other test suite runs.
- All test jobs of a test suite support the powerful ability to be executed with hundreds or thousands of concurrent users and can even executed on a cluster of measuring agents.
- The test jobs of a test suite are grouped into “execution groups”, where the test jobs of an execution group can be executed in sequential or parallel order.
- The execution groups can be nested, whereby a separate measuring agent or measuring agent cluster can be defined for each execution group.
- A “run” of a test suite can be triggered either interactively or via the Remote User API.
This seems a bit complicated at first glance. However, as you will see, creating and running test suites is fairly easy.
Test Job Templates
Test job templates can be created by clicking ‘Save as Template’ in the dropdown menu of any load test job:
You can freely choose the name of the template, but it must be unique.
The template is then created. For verification purposes, the test job properties are displayed which you can also change here.
The list of all templates will then be displayed. Here you can delete, clone and modify individual templates and you can create also new load test jobs from a template.
Adding a Test Suite
As soon as one or more test job templates are created, you can add a new test suite.
You can freely choose the name of the test suite, but it must be unique.
After that the test suite is created and the “Test Suite Editor” is automatically invoked.
Editing a Test Suite
It is recommended that you configure a measuring agent or cluster on which the test suite runs will be executed so that you are not asked about it every time you start a test suite run.
Only “Execution Groups” can be defined at the main level of a test suite. This means that you have first to add one or more execution groups to the test sequence.
At the top right of the editor area you can configure the “Main Execution Order”, which controls whether the execution groups at the main level are executed one after the other (= sequential ) or concurrently (= parallel).
Adding an Execution Group
Here you can specify on which measuring agent or cluster the execution group will be executed, whereby the default option “inherit from parent” is recommended. By setting the “Execution Order” you can determine whether the elements of the execution group are executed sequentially or in parallel.
Note: Execution groups can also be nested, which means that an execution group can contain several sub-execution groups, which can then contain sub-sub execution groups. The advantage of nesting execution groups is, among other things, that the measured data of the executed the test jobs are presented in aggregated form at the level of each execution group.
Adding Test Job Templates to an Execution Group
Finally, click the icon at the top left corner to return to the Test Suites Dashboard.
Executing a Test Suite “Run”
A test suite “run” can be triggered either interactively or via the Remote User API.
To start a “run” click on the green icon of the test suite - which means that all test jobs of the test suite are executed.
In the upper area of the green window, summarized/aggregated measurement results over all executed test jobs are displayed.
In the left area of the green window, the progress of the of the test suite execution is displayed in real time, whereby the measured values of the currently executed (load) test jobs are updated every 4 seconds.
Multiple test suites can be executed concurrently, but the same test suite cannot be executed twice at the same time. Note that closing the green window will not abort the execution of the test suite “run”.
Tip: If a load test job of a test suite takes several minutes to execute, you can also view the detailed statistics of the load test job in a second Web Browser window in the Real Time View. All you have to do is select the Real Time View in the second browser window.
After a test suite run is completed you can click on “Display Result”.
Test Suite “Run” Results
Test Suite Run Results are always historical data, whereby the execution groups and the test jobs are painted as they were at the time of execution.
In the upper area summarized/aggregated measurement results over all executed test jobs are displayed.
Clicking on the chart icon of a test job displays the “Test Result Details”. By clicking the file icon, you can view the contents of the test job output files.
Comparing Test Suite Run Results
The summarized/aggregated measurement results of the test suite runs can be compared across all or a selection of runs.
Furthermore, the measurement results of the individual test jobs can be compared across all or a selection of runs.