System Monitoring

Architecture Overview

RealLoad System Monitoring is a separate subsystem whose functions can be controlled from the RealLoad Portal, but runs independently of the Portal Server. This means that System Monitoring remains functional even when the Portal Server is down.

The System Monitoring Architecture extends the RealLoad Product by 3 Components:

1. Alert Processor: A stand-alone, independent process that collects Operating System Metrics form all OS systems where OSHI Daemons are installed and executes Monitoring Rules, whose results are instantly visible at real-time in the RealLoad Portal. Every RealLoad customer who wants to perform System Monitoring needs their own Alert Processor.
2. OSHI Daemon: A process that collects local Operating System Metrics and reports them to the Alert Processor. Any Linux and Windows systems are supported.
3. JavaScript Processor: A generic component (OS process) without any context that receive and execute Java Scripts and return the execution result to the caller. Any Linux and Windows systems are supported.

Monitoring Rules are usually executed on the Alert Processor by calling the JavaScript Processor, as this is resource-optimal. Alternatively, Monitoring Rules can also be executed locally on OSHI Daemons called via the Alert Processor (if a JavaScript Processor is installed on the OSHI Daemon), which allows starting or restarting local OS Processes as well as executing HTTP requests directly from the OSHI Daemon.

RealLoad: System Monitoring Features

After installing the Alert Processor, the OSHI Daemons and the JavaScript Processors, there is no further configuration required for such components - all System Monitoring settings can be managed directly from the RealLoad Portal.

• The Unique ID of newly installed OSHI Daemons (= OSHI Systems) can be entered in the RealLoad Portal, whereby such an entry authorizes the OSHI Daemon to establish a network connection to the Alert Processor.

• The Metrics measured by the OSHI Systems are instantly visible on the RealLoad Portal in real-time (+5 seconds).

• All current Alert States of Monitoring Rules executed of any OSHI System are also visible in real-time on the RealLoad Portal.

• Predefined Templates of the most frequently required Monitoring Rules are already available.

• Own Monitoring Rules (written in JavaScript) can be developed online in the RealLoad Portal within a very short time, without the need of any additional tools. A powerful wizard supports you to develop, debug and test Monitoring Rules.

• Multiple Monitoring Rules can be assigned to a Monitoring Group. The Monitoring Group settings determine which OSHI Systems are monitored and at what time interval the Monitoring Rules are executed. Multiple Alert Devices (Email, “Mobile Companion” App installed on iOS or Android, SMS) can be grouped to an Alert Group and assigned to one or multiple Monitoring Groups. The Alert Group settings determine what type of alerts are sent to the Alert Devices.

• Alert notifications can also be received when an OSHI Daemon loses connection to the Alert Processor, and when the connection is restored (for example if the monitored operating system or server is down or restarted).

RealLoad: OSHI Daemon Features

OSHI Daemons are installed locally on OS Systems to be monitored and report Operating System Metrics to the Alert Processor in real time.

The following Operating System Metric Types are continuously measured and reported:

• OS Info: e.g. OS type, version and system boot time
• CPU Info: e.g. CPU usage in %, interrupts and context switches per second
• Memory Info: e.g. used physical memory in %, used swap memory in %, hard page faults per second
• Network Interfaces Info: e.g. current throughput per network interface, network interface error rate
• Network Connections Info: all current network connections with their TCP und UDP states
• Local Disks Info: e.g. disk capacity, disk manufacturer, R/W queue length
• File Systems Info: e.g. file system mount point, total size, used size in %
• OS Processes INFO: all current processes of the OS, e.g. process PID and name, process start arguments

In total, +/- 200 Operating System Metrics are measured.

Additional, own metrics can be measured using shell scripts called directly by Monitoring Rules (see next subsection).

RealLoad OSHI Daemon + Local JavaScript Processor

Installing a local JavaScript Processor on an OSHI Daemon is optional, but has additional advantages.

By installing a local JavaScript Processor on a OSHI Daemon the Daemon can:

• Execute HTTP Requests periodically as Monitoring Rule, which results can also trigger a monitoring alert.
• Execute any local OS Process periodically as Monitoring Rule (for example a bash script), which process-output can also trigger a monitoring alert.
• Periodically Monitor OS Processes, and restart dead processes as daemon ¹ (which can also be reported as a monitoring alert – in case of the absence of an OS Process and when restarting an OS process).

¹ = Restart dead processes as daemon: Depends with which OS account the JavaScript Processor is started - or requires to set a sticky bit for the restart script.

Implementing System Monitoring

Prerequisites

To implement System Monitoring, you need your own “Core Monitoring” system with an Alert Processor and a local JavaScript Processor running both on the same (dedicated) OS system.

You can either install and operate such a “Core-Monitoring” system yourself – or commission us to operate a corresponding instance in the cloud.

After your “Core-Monitoring” system is up and running, it must be registered in the RealLoad Portal by our Support Team and linked to your RealLoad (main-user-)account.

You can then install OSHI Daemons (+ optionally their JavaScript Processors) yourself on any of your systems and also register them yourself in the RealLoad Portal.

Registering OSHI Systems in the RealLoad Portal

If you look at the log file contents of a newly installed OSHI Daemon, you will notice that it cannot yet connect to the Alert Processor because it does not yet have the necessary authorization.

However, in the first lines of the log file you will see the Unique Daemon ID of the newly installed OSHI Daemon. Copy this value into a text file or into an editor.

Direct Visible Operating System Metrics

The most important operating system metrics of the OSHI Systems are directly visible in the RealLoad Portal:

• CPU & Memory (real-time view)
• Network (real-time view)
• Network Connections (snapshot)
• Disks (snapshot)
• Filesystems (snapshot)
• OS Processes (snapshot)

Import Monitoring Rules from Templates

Ready-made Monitoring Rules Templates are available for monitoring the most important system metrics.

You can import such templates and, if necessary, adapt them to your needs (which is usually not necessary). Proceed as follows:

Monitoring Groups

To execute Monitoring Rules, you need at least one “Monitoring Group”. The Monitoring Group determines which OSHI Systems will be monitored and at what time interval the monitoring will be performed.

Input Fields:

• Monitoring Group Title : You can enter any text, but it must be unique across all Monitoring Groups.
• Description : Optional input field.
• Execution Interval : Select the time interval in which the (later added) Monitoring Rules are executed.
• Group Execution Enabled : Set this switch to on.

Assigning Monitoring Rules to a Monitoring Group

Generic Input Fields:

• Rule Enabled : Let this switch to on.
• Execute Rule (“On Alert Processor” or “On OSHI Systems”) : For “normal” rules always select “On Alert Processor”. The option “On OSHI Systems” should only be selected for rules that execute HTTP requests or start OS processes, which also requires that all OSHI systems of the Monitoring Group have a locally installed JavaScript Processor.

Rule Configuration Input Fields:

• Depending on the Monitoring Rule you select, none, one, or more Configuration Input Fields will be displayed. The input fields are pre-filled with default values ​​that you can overwrite and change according to your needs.

Alert Devices & Alert Groups

You can register several different Alert Devices (such as Email recipients, SMS recipients and a Mobile Phones with installed RealLoad “Mobile Companion App”) and assign them to one or more “System Monitoring” Alert Group.

Please note that all Alert Devices are shared between Synthetic Monitoring and System Monitoring. This means that adding/changing/deleting an Alert Devices will affect both Synthetic Monitoring and System Monitoring.

Adding an Alert Group

Input Fields:

• Alert Group Title : You can enter any text, but it must be unique across all Alert Groups.
• Alert Group Description : Optional input field.
• Alert Group Enabled : Let this switch to on.
• Report Measured Errors : This switch controls whether notifications are sent for errors detected by monitoring rules.
• Report Measured Warnings : This switch controls whether notifications are sent for warnings detected by monitoring rules.
• Repeat Alerts : If this switch is off, notifications will only be sent when an alert state changes (new/modified/cancelled alert).
• Report Alerts (“Instantly” or “After N Repeat”) : This allows to control whether a notification is sent immediately or with a delay when an error or warning is detected.
• On active OSHI System disconnects (“No/Info/Warning/Error Notification”) : This allows to control whether a notification is sent when the network connection of an OSHI Daemon to the Alert Processor is interrupted or reestablished.

Adding an Alert Device

Adding Alert Devices to Alert Groups

Assigning Alert Groups to Monitoring Groups and/or to Monitoring Rules

Developing own Monitoring Rules

In the following we will show how you can develop your own Monitoring Rules in JavaScript.

Example

This example monitors whether a specific operating system process exists. To do this we need two pieces of information:

• The list of all OS processes
• A text fragment of an OS process command line to search for.

In the Add Monitoring Rule form enter the following data:

• Rule Title : “Check OS Process exist”
• Add two Configuration Input Fields (Label = “OS Process Title” and “Command Line Text Fragment”, Data Type = String) : Theoretically, the configuration of the rule can be hard coded. However, in order to be able to use the rule as flexibly as possible, “Configuration Input Fields” are used. The “Configuration Input Field” with the label “OS Process Title” is not required for the logic of detecting the OS process, but is part of the alert notification.
• JavaScript Function Name : “checkOsProcessExist”
• Function Parameter #3 : Select osProcessesSnapshot

Some tips:

• The “Save Code” button saves the source code of the rule permanently. Alternatively, you can also enter <Ctrl>-S in the editor.
• The “Update Code and Test Again” button saves the source code in memory only and triggers a new test of the rule.
• By clicking on the “Down Arrow” icon you can enlarge the editor area.
• You can copy the JSON data displayed in the JavaScript console into the JSON editor of a second browser window and format it there.

The source code of the complete rule looks like this:

/* Monitoring Rule.
 * Description: Check if an OS process exists.
 */
function checkOsProcessExist(ruleConfig, ruleContext, osProcessesSnapshot) {
	// console.log(`ruleConfig = ${JSON.stringify(ruleConfig)}`);
	// console.log(`ruleContext = ${JSON.stringify(ruleContext)}`);
	// console.log(`osProcessesSnapshot = ${JSON.stringify(osProcessesSnapshot)}`);
	
	const vProcessTitle = ruleConfig.vProcessTitle;
	const vCommandLineFragment = ruleConfig.vCommandLineFragment;
	
	const systemDescription = ruleContext.oshiSystem.systemDescription;
	const oshiDaemonUniqueId = ruleContext.oshiSystem.oshiDaemonUniqueId;
	
	// count the number of processes that contain the command line fragment
	let osProcessCounter = 0;
	osProcessesSnapshot.osProcessesArray.forEach(osProcess => {
	  const commandLine = osProcess.commandLine;
	  if (commandLine.includes(vCommandLineFragment) === true) {
	    osProcessCounter++;
	  }
	})

    // create the alert context
	const alertContextKey = oshiDaemonUniqueId + '|checkOsProcess_' + vProcessTitle;   // must be unique overall OSHI Systems and Monitoring Rules
	const alertContextDescription = `'${systemDescription}': check if OS process '${vProcessTitle}' exists`;
	let alertMessage = '';
	let alertStatus = 'ok'; // possible values are : 'ok', 'warning' or 'error'
	switch(osProcessCounter) {
    case 0:
      alertMessage = `'${systemDescription}': the OS process '${vProcessTitle}' not exists`;
      alertStatus = 'error';
      break;
    case 1:
      alertMessage = `'${systemDescription}': the OS process '${vProcessTitle}' exists`;
      alertStatus = 'ok';
      break;
    default:
      alertMessage = `'${systemDescription}': the command line fragment '${vCommandLineFragment}' is ambiguous for OS process '${vProcessTitle}'`;
      alertStatus = 'warning';
      break;
	}
	const alertContextArray = [];
	alertContextArray.push({alertContextKey, alertContextDescription, alertMessage, alertStatus});

	// return rule result
	const ruleResult = {
		alertContextArray: alertContextArray
	};
    // console.log(`alertContextArray = ${JSON.stringify(alertContextArray)}`);
	return ruleResult;
}

1. The variables vProcessTitle and vCommandLineFragment are extracted from the function parameter #1 (ruleConfig) and contain the values of the Configuration Input Fields.

2. The variables systemDescription and oshiDaemonUniqueId are extracted from the function parameter #2 (ruleContext) and are initialized by the System Monitoring Alert Processor.

3. Then the number of processes containing the command line text fragment is counted by evaluating the function parameter #3 (osProcessesSnapshot in this example).

4. Finally, the Alert Context is created. As you can see, an array of alert contexts is returned as the function result, which means that a rule can also create multiple alert contexts. In this example, however, only one alert context is created.

Each Alert Context must contain:

• an alertContextKey : This key must be unique overall OSHI systems and monitoring rules. It is best to combine the oshiDaemonUniqueId with another unique property of the monitoring rule, as shown in this example.
• the alertContextDescription : A short description of the alert context. This is only displayed in the RealLoad Portal.
• the alertMessage : This text will be sent as a part of the Alert Notification to the Alert Devices.
• the alertStatus : Which must be ‘ok’, ‘warning’ or ’error’. Any other values are not supported.

Executing OS Processes from Rules

By executing Operating System Processes based on Rules, you can enhance System Monitoring with any custom metrics. For example, you can run Bash scripts to collect additional data and process it in Monitoring Rules. You can also restart dead OS Processes.

Executing Operating System Processes from Monitoring Rules is only supported on OSHI Systems that have a local JavaScript Processor installed. This also means that all OSHI Systems of the Monitoring Group to which the rule is assigned must support a local JavaScript Processor.

In Addition the Monitoring Rules must configured to be executed on “OSHI Systems” (instead to be executed on the “Alert Processor”).

The javascript-processor.properties (on the OSHI Systems) controls if and how many OS Processes can be started per rule execution, and if OS processes can be started as a daemon:

# The JavaScript processor features
JavaScriptStartOsProcessesEnabled=true
JavaScriptStartOsProcessesMaxInstances=3
JavaScriptStartOsProcessesAsDaemonEnabled=true

The JavaScript Engine for executing the Monitoring Rules has been extended by the OSProcess object.

OS processes can either be started as a normal process, with their output being evaluated by the monitoring rule. Such normal processes are killed if they still exists after the monitoring rule is executed. Monitoring Rules that start a normal OS process should always call the JavaScript method waitFor() after start() - see below.

Alternatively, OS processes can also be started as a daemon, with such OS processes surviving the end of the rule’s execution. This can therefore be used to restart dead operating system processes.

JavaScript Constructor : OsProcess.newInstance(<Array of String> OS command line arguments)

Example: The following monitoring rule checks whether the operating system parameter DefaultTasksMax is configured and large enough:

/* Monitoring Rule.
 * Description: Check if the OS parameter DefaultTasksMax is configured and large enough.
 */
function checkOsParamDefaultTasksMax(ruleConfig, ruleContext) {
	// console.log(`ruleConfig = ${JSON.stringify(ruleConfig)}`);
	// console.log(`ruleContext = ${JSON.stringify(ruleContext)}`);
	
	// get Configuration Input Field: DefaultTasksMax Min. Value [Number]
	const vDefaultTasksMaxMinValue = ruleConfig.vDefaultTasksMaxMinValue;
	
	const systemDescription = ruleContext.oshiSystem.systemDescription;
	const oshiDaemonUniqueId = ruleContext.oshiSystem.oshiDaemonUniqueId;
	
	// execute bash script
	const osProcess = OsProcess.newInstance(['bash', '-c', 'systemctl show | grep DefaultTasksMax']);
	osProcess.setCollectStdout();
	osProcess.start();
	osProcess.waitFor();
	const processStdoutLines = osProcess.getStdoutLines();
	// console.log(`processStdoutLines = ${JSON.stringify(processStdoutLines)}`);
	
	// extract systemctl value and create te alert context 
	const alertContextKey = oshiDaemonUniqueId + '|checkOsParamDefaultTasksMax';
	const alertContextDescription = `Check if '${systemDescription}' OS param DefaultTasksMax >= ${vDefaultTasksMaxMinValue}`;
	let alertStatus = 'ok';
	let alertMessage = '';
	
	if ((processStdoutLines.length === 0) || (!processStdoutLines[0].includes('DefaultTasksMax'))) {
	    alertStatus = 'error';
	    alertMessage = 'OS param DefaultTasksMax not configured';
	} else {
	    const currentDefaultTasksMax = Number(processStdoutLines[0].split('=')[1]);
	    // console.log(`currentDefaultTasksMax = ${currentDefaultTasksMax}`)
	    alertMessage = 'OS param DefaultTasksMax = ' + currentDefaultTasksMax;
	    if (currentDefaultTasksMax < vDefaultTasksMaxMinValue) {
	        alertStatus = 'error';
	        alertMessage = alertMessage + ' : value too small, should be at least ' + vDefaultTasksMaxMinValue;
	    }
	}
	
	const alertContextArray = [];
	alertContextArray.push({alertContextKey, alertContextDescription, alertMessage, alertStatus});
	
	// return rule result
	const ruleResult = {
		alertContextArray: alertContextArray
	};
	return ruleResult;
}

Executing HTTP Requests from Rules

Executing HTTP Requests from Monitoring Rules requires - analogous to executing OS Processes from Monitoring Rules - that a local JavaScript Processor is available on the OSHI Systems.

Theoretically (and also possible) HTTP Requests can also be executed on the Alert Processor, but this usually makes little sense since you usually want to check the runtime behavior from a specific location - or because you want to check whether a certain service is accessible from the localhost. For this reason, Monitoring Rules that execute HTTP Requests should be configured to be executed on “OSHI Systems” (instead of on the “Alert Processor”).

The javascript-processor.properties (on the OSHI Systems) controls if and how many HTTP Clients can be started per rule execution, and the maximum size of a received HTTP Response Content (in bytes). Each individual HTTP client can execute multiple requests one after the other or simultaneously, so that in a monitoring rule you usually only need one instance of an HTTP client.

# The JavaScript processor features
JavaScriptHTTPClientEnabled=true
JavaScriptHTTPClientMaxConcurrentInstances=4
JavaScriptHTTPClientMaxStoredResponseContentSize=10000000

The JavaScript Engine for executing the Monitoring Rules has been extended by the HTTPClient object which is directly linked with the Java class com.dkfqs.tools.http.HTTPClient. Detailed documentation about the HTTPClient is available at https://download.realload.com/   - com.dkfqs.tools.jar - JavaDoc .

JavaScript Constructor 1 : HTTPClient.newInstance() // create a new instance with a default processing timeout per URL of 30 seconds, and with 2 processing threads.

JavaScript Constructor 2 : HTTPClient.newInstance(<Number> processingTimeout, <Number> numProcessingThreads) // create a new instance with a processing timeout per URL in milliseconds, and with the number of processing threads (valid range is 1..64).

Example 1 of 2 : HTTP GET Request

/* Monitoring Rule.
 * Description: .. add description here ..
 */
function executeHttpGetRequest(ruleConfig, ruleContext) {
	// console.log(`ruleConfig = ${JSON.stringify(ruleConfig)}`);
	// console.log(`ruleContext = ${JSON.stringify(ruleContext)}`);
	
	const systemDescription = ruleContext.oshiSystem.systemDescription;
	const oshiDaemonUniqueId = ruleContext.oshiSystem.oshiDaemonUniqueId;
	
	const httpRequestMethod = 'GET';
	const url = 'https://www.realload.com';
	const warnExecutionTimeMillis = 3000;
	
	const alertContextKey = oshiDaemonUniqueId + '|' + httpRequestMethod + ' ' + url;
	const alertContextDescription = `Check if ${url} is reachable`;
	let alertMessage = '';
	let alertStatus = 'ok';
	
	 // execute the HTTP GET
    const httpClient = HTTPClient.newInstance(); // create a new HTTPClient instance with a default processing timeout of 30 seconds 
    const httpRequest = httpClient.newRequest(httpRequestMethod, url);
    // const httpRequestHeader = httpRequest.getHttpRequestHeader();
    // httpRequestHeader.addOrReplaceHeaderField('uuid', '123456789');
    // console.log(`httpRequestHeader = ${httpRequestHeader.toJsonObject()}`)
    const httpResponse = httpClient.sendSyncRequest(httpRequest); // execute the HTTP request
    // process the HTTP response
	if (httpRequest.hasErrorException() === true) {
	    alertMessage = `'${systemDescription}': URL ${url} not reachable : ${httpRequest.getErrorException()}`;
	    alertStatus = 'error';
	} else {
	    const httpStatusCode = httpResponse.getHttpStatusCode();
	    const responseContentType = '' + httpResponse.getHttpResponseHeader().getContentMIMEType();
	    const performanceData = httpResponse.getHttpPerformanceData().toJsonObject();
	    const responseContent = httpResponse.getHttpResponseContent().getContentAsString();
	    
	    // console.log(`HTTP status code = ${httpStatusCode}`);
	    // console.log(`Response content type = '${responseContentType}'`);
	    // console.log(`HTTP performance data = ${performanceData}`);
	    // console.log(`HTTP response content = ${responseContent}`);
	    
	    alertMessage = `'${systemDescription}': URL ${url} : `;
	    if (httpStatusCode !== 200) {
	        alertMessage = alertMessage + `invalid HTTP status code ${httpStatusCode}`;
	        alertStatus = 'error';
	    } else {
	        if (responseContentType !== 'text/html') {
	            alertMessage = alertMessage + `invalid response content type '${responseContentType}'`;
	            alertStatus = 'error';
	        } else {
	            const executionTime = JSON.parse(performanceData).executionTime;
	            alertMessage = alertMessage + `exec time ${executionTime} ms`;
	            if (executionTime > warnExecutionTimeMillis) {
	                alertStatus = 'warning';
	            }
	        }
	    }
	}
	httpClient.closeAbort(); // free all internal resources and close all connections of the HTTPClient
	
	const alertContextArray = [];
	alertContextArray.push({alertContextKey, alertContextDescription, alertMessage, alertStatus});

	// return rule result
	const ruleResult = {
		alertContextArray: alertContextArray
	};
	return ruleResult;
}

Example 2 of 2 : HTTP POST Request

/* Monitoring Rule.
 * Description: .. add description here ..
 */
function executeHttpPostRequest(ruleConfig, ruleContext) {
	// console.log(`ruleConfig = ${JSON.stringify(ruleConfig)}`);
	// console.log(`ruleContext = ${JSON.stringify(ruleContext)}`);
	
	const systemDescription = ruleContext.oshiSystem.systemDescription;
	const oshiDaemonUniqueId = ruleContext.oshiSystem.oshiDaemonUniqueId;
	
	const httpRequestMethod = 'POST';
	const url = 'https://portal.realload.com/PublicAPI';
	const warnExecutionTimeMillis = 3000;
	
	const alertContextKey = oshiDaemonUniqueId + '|' + httpRequestMethod + ' ' + url;
	const alertContextDescription = `Check if ${url} is reachable`;
	let alertMessage = '';
	let alertStatus = 'ok';
	
	 // execute the HTTP POST
    const httpClient = HTTPClient.newInstance(); // create a new HTTPClient instance with a default processing timeout of 30 seconds 
    const httpRequest = httpClient.newRequest(httpRequestMethod, url);
    // const httpRequestHeader = httpRequest.getHttpRequestHeader();
    // httpRequestHeader.addOrReplaceHeaderField('uuid', '123456789');
    // console.log(`httpRequestHeader = ${httpRequestHeader.toJsonObject()}`)
    const httpRequestContent = HTTPClient.newHTTPRequestContent('{"action":"getPortalServerInfo"}');    // set JSON data as request content
    httpRequest.setHttpRequestContent(httpRequestContent, "application/json");
    const httpResponse = httpClient.sendSyncRequest(httpRequest); // execute the HTTP request
    // process the HTTP response
	if (httpRequest.hasErrorException() === true) {
	    alertMessage = `'${systemDescription}': URL ${url} not reachable : ${httpRequest.getErrorException()}`;
	    alertStatus = 'error';
	} else {
	    const httpStatusCode = httpResponse.getHttpStatusCode();
	    const responseContentType = '' + httpResponse.getHttpResponseHeader().getContentMIMEType();
	    const performanceData = httpResponse.getHttpPerformanceData().toJsonObject();
	    const responseContent = httpResponse.getHttpResponseContent().getContentAsString();
	    
	    // console.log(`HTTP status code = ${httpStatusCode}`);
	    // console.log(`Response content type = '${responseContentType}'`);
	    // console.log(`HTTP performance data = ${performanceData}`);
	    // console.log(`HTTP response content = ${responseContent}`);
	    
	    alertMessage = `'${systemDescription}': URL ${url} : `;
	    if (httpStatusCode !== 200) {
	        alertMessage = alertMessage + `invalid HTTP status code ${httpStatusCode}`;
	        alertStatus = 'error';
	    } else {
	        if (responseContentType !== 'application/json') {
	            alertMessage = alertMessage + `invalid response content type '${responseContentType}'`;
	            alertStatus = 'error';
	        } else {
	            const executionTime = JSON.parse(performanceData).executionTime;
	            alertMessage = alertMessage + `exec time ${executionTime} ms`;
	            if (executionTime > warnExecutionTimeMillis) {
	                alertStatus = 'warning';
	            }
	        }
	    }
	}
	httpClient.closeAbort(); // free all internal resources and close all connections of the HTTPClient
	
	const alertContextArray = [];
	alertContextArray.push({alertContextKey, alertContextDescription, alertMessage, alertStatus});

	// return rule result
	const ruleResult = {
		alertContextArray: alertContextArray
	};
	// console.log(`ruleResult = ${JSON.stringify(ruleResult)}`);
	return ruleResult;
}