Ubuntu 16/18/20 Remote Proxy Recorder manual install

Ubuntu 16/18/20 Remote Proxy Recorder Install Instructions

Prerequisites

Supported Hardware

  • Amazon EC2 Cloud instance, or
  • Own hosted server with any Intel or AMD CPU

Minimum Requirements

  • Minimum required CPU Cores of Processor: 2
  • Minimum required Memory: 8 GB
  • Minimum required Disk: 32 GB
  • Minimum required Network Speed: 100 Mbps (1000 Mbps or faster strongly recommended)

Environment and Location

The Remote Proxy Recorder can be placed at any network location, but the control port must be reachable from the Portal Server.

Network & System Tuning

In /etc/sysctl.conf add:

# TCP/IP Tuning
# =============
fs.file-max = 524288
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_synack_retries = 3
net.ipv4.tcp_max_orphans = 65536
net.ipv4.tcp_fin_timeout = 30
net.ipv4.ip_local_port_range = 16384 60999
net.core.somaxconn = 256
net.core.rmem_max = 1048576
net.core.wmem_max = 1048576

in /etc/security/limits.conf add:

# TCP/IP Tuning
# =============
* soft     nproc          262140
* hard     nproc          262140
* soft     nofile         262140
* hard     nofile         262140
root soft     nproc          262140
root hard     nproc          262140
root soft     nofile         262140
root hard     nofile         262140

Enter: systemctl show -p TasksMax user-0

output: TasksMax=8966

if you get a value less than 262140 then add in /etc/systemd/system.conf

# Ubuntu Tuning
# =============
DefaultTasksMax=262140

Reboot the system and verify the settings. Enter: ulimit -n

output: 262140

Enter: systemctl show -p TasksMax user-0

output: TasksMax=262140

Install Dependencies

Install haveged

sudo apt-get update
sudo apt-get install haveged

Install OpenJDK 11

Get the Java Installation Kit

wget https://download.java.net/java/GA/jdk11/13/GPL/openjdk-11.0.1_linux-x64_bin.tar.gz

Install OpenJDK Java 11

gunzip openjdk-11.0.1_linux-x64_bin.tar.gz
tar -xvf openjdk-11.0.1_linux-x64_bin.tar
rm openjdk-11.0.1_linux-x64_bin.tar
sudo bash
mv jdk-11.0.1 /opt/OpenJDK
cd /opt/OpenJDK
ls -al
chown root -R jdk-11.0.1
chgrp root -R jdk-11.0.1

Execute the following commands (still as sudo bash):

update-alternatives --install "/usr/bin/java" "java" "/opt/OpenJDK/jdk-11.0.1/bin/java" 1
update-alternatives --install "/usr/bin/javac" "javac" "/opt/OpenJDK/jdk-11.0.1/bin/javac" 1
update-alternatives --install "/usr/bin/keytool" "keytool" "/opt/OpenJDK/jdk-11.0.1/bin/keytool" 1
update-alternatives --install "/usr/bin/jar" "jar" "/opt/OpenJDK/jdk-11.0.1/bin/jar" 1
update-alternatives --set "java" "/opt/OpenJDK/jdk-11.0.1/bin/java"
update-alternatives --set "javac" "/opt/OpenJDK/jdk-11.0.1/bin/javac"
update-alternatives --set "keytool" "/opt/OpenJDK/jdk-11.0.1/bin/keytool"
update-alternatives --set "jar" "/opt/OpenJDK/jdk-11.0.1/bin/jar"
exit # end sudo bash

Verify the Java 11 installation.

java -version

openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment 18.9 (build 11.0.1+13)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.1+13, mixed mode)

Generate the Certificate Authority (CA) Root Certificate

For technical reasons, the Remote Proxy Recorder generates “fake” web server certificates during operation in order to break the encryption between the web browser and the web servers and to record the data exchanged.

In order for this to work, the Remote Proxy Recorder needs its own CA root certificate, which you then have to import into your browser.

Thus for security reasons, never use a CA root certificate from us or someone else for the Remote Proxy Recorder root certificate. Always create your own CA root certificate.

Example:

C:\Scratch2>openssl genrsa -des3 -out myCAPrivate.key 2048
Generating RSA private key, 2048 bit long modulus
.......................+++
.............................+++
unable to write 'random state'
e is 65537 (0x10001)
Enter pass phrase for myCAPrivate.key:
Verifying - Enter pass phrase for myCAPrivate.key:

C:\Scratch2>openssl pkcs8 -topk8 -inform PEM -outform PEM -in myCAPrivate.key -out myCAPrivateKey.pem -nocrypt
Enter pass phrase for myCAPrivate.key:

C:\Scratch2>openssl req -x509 -new -nodes -key myCAPrivate.key -sha256 -days 3700 -out myCARootCert.pem
Enter pass phrase for myCAPrivate.key:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CH
State or Province Name (full name) [Some-State]:Bern
Locality Name (eg, city) []:Bern
Organization Name (eg, company) [Internet Widgits Pty Ltd]:My Pty Ltd
Organizational Unit Name (eg, section) []:QA
Common Name (e.g. server FQDN or YOUR name) []:DKFQS Proxy Recorder Root
Email Address []:

C:\Scratch2>dir
 Volume in drive C is OS
 Volume Serial Number is AEF7-CFB1

 Directory of C:\Scratch2

06 Feb 2022  20:44    <DIR>          .
06 Feb 2022  20:44    <DIR>          ..
06 Feb 2022  20:40             1.743 myCAPrivate.key
06 Feb 2022  20:41             1.704 myCAPrivateKey.pem
06 Feb 2022  20:44             1.350 myCARootCert.pem
               3 File(s)          4.797 bytes
               2 Dir(s)  310.772.580.352 bytes free

Install the Remote Proxy Recorder

Create the DKFQS account which is running the Remote Proxy Recorder

sudo adduser dkfqs    # follow the questions, remember or write down the password

Install the Remote Proxy Recorder

Login with the dkfqs account (SSH) - or - Enter: sudo -u dkfqs bash | OR: Install Samba to get convenient access to /home/dkfqs as Samba dkfqs user

Create the directory /home/dkfqs/proxy (as dkfqs user):

cd /home/dkfqs
mkdir proxy

Create the following sub-directories at /home/dkfqs/proxy (as dkfqs user):

  • bin
  • config
  • log
cd /home/dkfqs/controller
mkdir bin config log

Copy the following files to the bin directory /home/dkfqs/proxy/bin

  • bcmail-jdk15on-168.jar
  • bcpg-jdk15on-168.jar
  • bcpkix-jdk15on-168.jar
  • bcprov-jdk15on-168.jar
  • bctls-jdk15on-168.jar
  • com.dkfqs.remoteproxyrecorder.jar

Copy the following files to the config directory /home/dkfqs/proxy/config

  • config.properties
  • myCAPrivateKey.pem (the private key of your self generated CA root certificate)
  • myCARootCert.pem (your self generated CA root certificate)

Modify the config.properties file. Set (modify) the following properties:

  • ControlServerHttpsCertificateCN (set the public DNS name or the IP address of the Remote Proxy Recorder)
  • ControlServerHttpsCertificateIP (set the public IP address of the Remote Proxy Recorder)
  • ControlServerAuthToken
  • ProxyServerDefaultCaRootCertFilePath (set the CA root certificate of the proxy)
  • ProxyServerDefaultCaRootPrivateKeyFilePath (set the private key of the CA root certificate)

Example: config.properties

ControlServerLogLevel=info
ControlServerHttpsPort=8081
ControlServerHttpsCertificateCN=proxy2.realload.com
ControlServerHttpsCertificateIP=83.150.39.45
#Note: the control server authentication token is required to connect to the control server
ControlServerAuthToken=krungthep

ProxyServerLogLevel=warn
ProxyServerPort=8082
ProxyServerDefaultCaRootCertFilePath=/home/dkfqs/proxy/config/myCARootCert.pem
ProxyServerDefaultCaRootPrivateKeyFilePath=/home/dkfqs/proxy/config/myCAPrivateKey.pem
#Note: the proxy authentication credentials are replaced on the fly when the portal user connects via the control interface to the control server
ProxyServerDefaultAuthenticationUsername=max
ProxyServerDefaultAuthenticationPassword=meier

First Test - Start the Remote Proxy Recorder manually (as dkfqs user)

cd /home/dkfqs/proxy/bin
export CLASSPATH=bcmail-jdk15on-168.jar:bcpg-jdk15on-168.jar:bcpkix-jdk15on-168.jar:bcprov-jdk15on-168.jar:bctls-jdk15on-168.jar:com.dkfqs.remoteproxyrecorder.jar
java -Xmx2048m -DconfigProperties=../config/config.properties -Dnashorn.args="--no-deprecation-warning" com.dkfqs.remoteproxyrecorder.main.StartRemoteProxyRecorder

> Remote Proxy Recorder V0.2.0
> Max. Memory = 2048 MB
> Internal RSA 2048 bit keypair generated in 85 ms
> 2021-06-05 23:24:37.710 | QAHTTPd | WARN | QAHTTPd V1.3-V started
> 2021-06-05 23:24:37.710 | QAHTTPd | INFO | HTTPS server starting at port 8081
> 2021-06-05 23:24:37.726 | QAHTTPd | INFO | HTTPS server ready at port 8081
> 2021-06-05 23:24:38.722 | Proxy | WARN | ProxyRecorder V1.1.0 started at port 8082

Create the Remote Proxy Recorder Startup Script (as root)

sudo bash # become root
cd /etc/init.d
vi RemoteProxyRecorder

Edit - create /etc/init.d/RemoteProxyRecorder

#!/bin/sh
# /etc/init.d/RemoteProxyRecorder
# install with: update-rc.d RemoteProxyRecorder defaults

### BEGIN INIT INFO
# Provides:          RemoteProxyRecorder
# Required-Start:    $local_fs $network $time $syslog
# Required-Stop:     $local_fs $network
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start RemoteProxyRecorder daemon at boot time
# Description:       RemoteProxyRecorder daemon
### END INIT INFO

case "$1" in
  start)
    if [ -f /home/dkfqs/proxy/log/RemoteProxyRecorder.log ]; then
       mv /home/dkfqs/proxy/log/RemoteProxyRecorder.log /home/dkfqs/proxy/log/RemoteProxyRecorder.log_$(date +"%Y_%m_%d_%H_%M")
    fi
    sudo -H -u dkfqs bash -c 'CLASSPATH=/home/dkfqs/proxy/bin/bcmail-jdk15on-168.jar:/home/dkfqs/proxy/bin/bcpg-jdk15on-168.jar:/home/dkfqs/proxy/bin/bcpkix-jdk15on-168.jar:/home/dkfqs/proxy/bin/bcprov-jdk15on-168.jar:/home/dkfqs/proxy/bin/bctls-jdk15on-168.jar:/home/dkfqs/proxy/bin/com.dkfqs.remoteproxyrecorder.jar;export CLASSPATH;nohup java -Xmx4096m -DconfigProperties=/home/dkfqs/proxy/config/config.properties -Dnashorn.args="--no-deprecation-warning" com.dkfqs.remoteproxyrecorder.main.StartRemoteProxyRecorder 1>/home/dkfqs/proxy/log/RemoteProxyRecorder.log 2>&1 &'
    ;;
  stop)
       PID=`ps -o pid,args -e | grep "StartRemoteProxyRecorder" | egrep -v grep | awk '{print $1}'`
       if [ ! -z "$PID" ] ; then
          echo "RemoteProxyRecorder stopped with pid(s) : $PID"
          kill -9 ${PID} 1> /dev/null 2>&1
       fi
    ;;
  status)
       PID=`ps -o pid,args -e | grep "StartRemoteProxyRecorder" | egrep -v grep | awk '{print $1}'`
       if [ ! -z "$PID" ] ; then
          echo "RemoteProxyRecorder running with pid(s) : $PID"
       else
          echo "No RemoteProxyRecorder running"
       fi
    ;;
  *)
    echo "Usage: /etc/init.d/RemoteProxyRecorder {start|stop|status}"
    exit 1
    ;;
esac

exit 0

Change owner and file protection of /etc/init.d/RemoteProxyRecorder (root at /etc/init.d):

chown root RemoteProxyRecorder
chgrp root RemoteProxyRecorder
chmod 755 RemoteProxyRecorder

Register /etc/init.d/RemoteProxyRecorder to be started at system boot (root at /etc/init.d):

update-rc.d RemoteProxyRecorder defaults

Reboot the system. Login as dkfqs and check /home/dkfqs/proxy/log/RemoteProxyRecorder.log

Register and Verify the Remote Proxy Recorder

  • Sign-in at the ‘Portal Server’
  • Follow the instructions at User Guide