Clustering in AWS

Starting with Connect2id server 6.15 cluster setup in AWS no longer requires a special configuration based on S3_PING. Just follow the regular database and cluster setup instructions.

The clustering stack of the Connect2id server is based on Infinispan and JGroups, and out of the box it’s configured for UDP/IP multicast node discovery and message passing. Multicast is the most efficient way to pass messages around a cluster, especially with many nodes, thanks to its minimal networking overhead. In order for multicast to work, the Connect2id server nodes must be on the same subnet, connected to the same (physical) switch / router.

Unfortunately AWS and other popular cloud provider don’t support IP multicast, due to limitations of their virtual environments. Therefore, in order to deploy a Connectid server cluster in the AWS cloud, the Infinispan / JGroups stack must be configured for different networking:

• S3_PING for initial node auto-discovery, where a shared S3 bucket is used to let the Connect2id server nodes discover one another when a new one goes up.

• TCP for passing messages between the Connect2id server nodes.

Setup instructions

Step 1. Create an AWS user

We first need to create a dedicated AWS user for Connect2id server nodes to access the S3 bucket.

Sign in to the AWS console and click on the Identity and Access Management (IAM) tab.

Under Users click on Create New Users and enter the desired username, e.g. c2id. Make sure Generate an access key for each user is selected. Then click on Create.

Record the generated user credentials:

• Access key ID, e.g. AKIAIHGHNQXQKYJEPEMA
• Secure access key, e.g. GjE5InROhZXvysBLyrRL0mJkjlrUJpMxLX7YUwhF

Also, record the ARN (Amazon Resource Identifier) for the new user. This can be viewed when you choose the c2id user in the IAM users lists.

• User ARN, e.g. arn:aws:iam::673931619296:user/c2id

No other credentials are required for the c2id user. Note that you must be the AWS account administrator or have the required permission to be able to create a new user.

Step 2. Create S3 bucket

Click on the S3 management tab and then on Create Bucket.

Enter a bucket name, e.g. s3ping-infinispan-c2id.

The name will be used to construct the bucket URL, i.e. https://s3ping-infinispan-c2id.s3.amazonaws.com and should therefore contain letters, numbers and dashes only. Do not include dots as this will cause SSL certificate validation problems later on!

Step 3. Assign permission policy to S3 bucket

Left-click on the newly created S3 bucket and select Properties, then Permissions. This will open up a panel where you can specify the access permissions for the bucket.

Click on Edit bucket policy and paste the following JSON in the provided text area:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Principal": {
				"AWS": "[USER-ARN]"
			},
			"Action": [
				"s3:GetObjectVersion",
				"s3:DeleteObject",
				"s3:DeleteObjectVersion",
				"s3:GetObject",
				"s3:PutObject"
			],
			"Resource": "arn:aws:s3:::[BUCKET]/*"
		},
		{
			"Effect": "Allow",
			"Principal": {
				"AWS": "[USER-ARN]"
			},
			"Action": [
				"s3:ListBucketVersions",
				"s3:ListBucket"
			],
			"Resource": "arn:aws:s3:::[BUCKET]"
		}
	]
}

where [USER-ARN] should be replaced with the ARN of the registered user (step 1), e.g. arn:aws:iam::673931619296:user/c2id and [BUCKET] with the bucket name (step 2), e.g. s3ping-infinispan-c2id.

Click on Save to apply the entered policy to the S3 bucket. The registered jgroups user will then have permission to list the bucket’s content and view, create and delete objects in it.

Step 4. Configure Infinispan / JGroups for S3_PING discovery

The Connect2id server WAR package comes with a set of ready JGroups configurations. For AWS with S3_PING use default-jgroups-s3ping.xml. To apply it, open your current Infinispan configuration file, e.g. /WEB-INF/infinispan-mysql.xml, and update the jgroups file path like this:

<jgroups>
   <stack-file name="jgroups-config" path="default-jgroups-s3ping.xml"/>
</jgroups>

Then pass the following Java system properties to each Connect2id server node:

These Java system properties can be passed on JVM startup with the -D command line parameter, e.g. by editing tomcat/bin/setenv.sh (if you’re using a Tomcat web server).

export CATALINA_OPTS="$CATALINA_OPTS -Djgroups.s3.bucket=s3ping-infinispan-c2id -Djgroups.s3.access_key=AKIAIHGHNQXQKYJEPEMA -Djgroups.s3.secret_access_key=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"

Alternatively, if you’re using Amazon’s Elastic Beanstalk you can set the properties from the EB console by selecting the application, clicking on Default configuration, then Configuration and finally Software Configuration. This will present you with a panel named Environment Properties where you can provide arbitrary properties to your application, including the the S3 bucket settings.

Step 5. Deploy your service / application to EC2

Deploy your Connect2id server nodes and check your server logs. If JGroups encounters a faulty S3_PING configuration it will throw a fatal exception and Infinispan will abort.

Some common exceptions and how to resolve them:

The S3 bucket name, access key or secret key may be invalid:

java.io.IOException: bucket ‘s3ping-infinispan-c2id’ could not be accessed (rsp=403 (Forbidden). Maybe the bucket is owned by somebody else or the authentication failed

or

org.infinispan.commons.CacheException: java.io.IOException: bucket ‘/tmp/jgroups’ could not be accessed (rsp=301 (Moved Permanently). Maybe the bucket is owned by somebody else or the authentication failed

You registered an S3 bucket name with dots, which caused the wildcard SSL certificate of AWS to be rejected by JGroups:

org.infinispan.commons.CacheException: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching s3ping.cluster.c2id.s3.amazonaws.com found.

Automation

Note that the steps above can be fully automated via the AWS API:

• JGroups user creation
• Bucket creation and policy setting
• Deployment to EC2 / Elastic Beanstalk and System Properties setting.