Clustering in AWS

Out of the box the Connect2id server is configured for simple UDP/IP multicast based clustering. This is the most efficient transport method due to its minimal networking overhead.

AWS and most other cloud providers however do not support multicast because servers are typically not provisioned on the same network switch and that would be difficult to implement anyway.

The Connect2id server employs an Infinispan datagrid from JBoss / RedHat to cluster its nodes. Infinispan in turn uses JGroups to handle the actual messaging down the network stack.

In EC2 the multicast node auto-discovery will have to be replaced with another method, e.g. S3_PING (that seems to be the recommended method). Then, the nodes will have to use direct TCP links instead of UDP multicast to replicate the data among themselves.

JGroups provides two modules for implementing the initial node discovery on AWS:

1. S3_PING

This module is included in JGroups and uses a shared S3 bucket to let the nodes discover one another and exchange their IP addresses.

http://www.jgroups.org/manual/index.html#_s3_ping

2. AWS_PING

This module is provided by a third-party developer, and uses an AWS API for discovery.

http://www.jgroups.org/manual/index.html#_aws_ping

https://github.com/meltmedia/jgroups-aws

Configuring the Connect2id server for S3_PING

The location of the JGroups settings file is given in the Infinispan configuration file:

WEB-INF/infinispan.xml

Replace the original multicast configuration with the following XML (also included in the infinispan JAR):

<stack-file name="jgroups-config" path="default-configs/jgroups-ec2.xml"/>

Set the following system properties at JVM startup, or alternatively, use the above file as a template and plug in your own custom values. In both cases the result should be the same.

<tr>
    <td>jgroups.tcp.address</td>
    <td>IP address to use for the TCP transport.</td>
    <td>127.0.0.1</td>
    <td>No</td>
</tr>
<tr>
    <td>jgroups.tcp.port</td>
    <td>Port to use for TCP socket</td>
    <td>7800</td>
    <td>No</td>
</tr>
<tr>
    <td>jgroups.s3.access_key</td>
    <td>The Amazon S3 access key used to access an S3 bucket</td>
    <td></td>
    <td>Yes</td>
</tr>
<tr>
    <td>jgroups.s3.secret_access_key</td>
    <td>The Amazon S3 secret key used to access an S3 bucket</td>
    <td></td>
    <td>Yes</td>
</tr>
<tr>
    <td>jgroups.s3.bucket</td>
    <td>Name of the Amazon S3 bucket to use. Must be unique and must already exist.</td>
    <td></td>
    <td>Yes</td>
</tr>
<tr>
    <td>jgroups.s3.pre_signed_delete_url</td>
    <td>The pre-signed URL to be used for the DELETE operation.</td>
    <td></td>
    <td>Yes</td>
</tr>
<tr>
    <td>jgroups.s3.pre_signed_put_url</td>
    <td>The pre-signed URL to be used for the PUT operation.</td>
    <td></td>
    <td>Yes</td>
</tr>
<tr>
    <td>jgroups.s3.prefix</td>
    <td>If set, S3_PING searches for a bucket with a name that starts with the prefix value.</td>
    <td></td>
    <td>No</td>
</tr>

System property	Description	Default	Required?

Configuring the Connect2id server for AWS_PING

TBD