How to configure Infinispan for AWS S3_PING discovery

This is a detailed guide how to configure an Infinispan cluster to run in the EC2 cloud of Amazon Web Services (AWS). You can use to deploy a Connect2id server or any other app / service that uses Infinispan in the Amazon cloud.

JGroups

Clustering in Infinispan is handled by the popular open-source JGroups toolkit. It deals with all aspects of node discovery and providing reliable messaging between the nodes. These two functions are implemented by distinct protocols / modules:

1. Node discovery – This can be based on a predefined list of node IP addresses, or protocols such as UDP PING.

2. Node-to-node messaging – Based on UDP multicast, TCP or tunnel transport.

Due to limitations of the AWS cloud and its virtual networking, native UDP is not supported there. JGroups therefore has to be configured with TCP for node-to-node messaging. As for node discovery, several alternatives exist. In this guide we’ll describe setup of the S3_PING protocol which uses an S3 bucket (shared cloud storage) to let the nodes communicate their IP addresses.

Instructions

Step 1. Create an AWS user

We first need to create a dedicated AWS user for JGroups to access the S3 bucket.

Sign in to the AWS console and click on the Identity and Access Management (IAM) tab.

Under Users click on Create New Users and enter the desired username, e.g. jgroups. Make sure Generate an access key for each user is selected. Then click on Create.

Record the generated user credentials:

• Access key ID, e.g. AKIAIHGHNQXQKYJEPEMA
• Secure access key, e.g. GjE5InROhZXvysBLyrRL0mJkjlrUJpMxLX7YUwhF

Also, record the ARN (Amazon Resource Identifier) for the new user. This can be viewed when you choose the jgroups user in the IAM users lists.

• User ARN, e.g. arn:aws:iam::673931619296:user/jgroups

No other credentials are required for the jgroups user. Note that you must be the AWS account administrator or have the required permission to be able to create a new user.

Step 2. Create S3 bucket

Click on the S3 management tab and then on Create Bucket.

Enter a bucket name, e.g. s3ping-infinispan-c2id.

The name will be used to construct the bucket URL, i.e. https://s3ping-infinispan-c2id.s3.amazonaws.com and should therefore contain letters, numbers and dashes only. Do not include dots as this will cause SSL certificate validation problems later on!

Step 3. Assign permission policy to S3 bucket

Left-click on the newly created S3 bucket and select Properties, then Permissions. This will open up a panel where you can specify the access permissions for the bucket.

Click on Edit bucket policy and paste the following JSON in the provided text area:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Principal": {
				"AWS": "[USER-ARN]"
			},
			"Action": [
				"s3:GetObjectVersion",
				"s3:DeleteObject",
				"s3:DeleteObjectVersion",
				"s3:GetObject",
				"s3:PutObject"
			],
			"Resource": "arn:aws:s3:::[BUCKET]/*"
		},
		{
			"Effect": "Allow",
			"Principal": {
				"AWS": "[USER-ARN]"
			},
			"Action": [
				"s3:ListBucketVersions",
				"s3:ListBucket"
			],
			"Resource": "arn:aws:s3:::[BUCKET]"
		}
	]
}

where [USER-ARN] should be replaced with the ARN of the registered user (step 1), e.g. arn:aws:iam::673931619296:user/jgroups and [BUCKET] with the bucket name (step 2), e.g. s3ping-infinispan-c2id.

Click on Save to apply the entered policy to the S3 bucket. The registered jgroups user will then have permission to list the bucket’s content and view, create and delete objects in it.

Step 4. Configure Infinispan / JGroups for S3_PING discovery

The Infinispan JAR comes with a set of default JGroups configurations. For AWS use default-configs/default-jgroups-ec2.xml.

Make sure the following jgroups element is present in your Infinispan XML configuration (implies Infinispan version 7 or later, earlier versions have a different configuration format):

<jgroups>
   <stack-file name="jgroups-config" path="default-configs/default-jgroups-ec2.xml"/>
</jgroups>

Then supply the following Java system properties to your application:

If you’re using Amazon’s Elastic Beanstalk you can set these properties from the EB console by selecting your application, clicking on Default configuration, then Configuration and finally Software Configuration. This will present you with a panel named Environment Properties where you can provide arbitrary properties to your application, including the the S3 bucket settings.

Why use system properties to set the credentials? We recommend this approach because it allows you to package your application’s WAR without any sensitive credentials in it.

To hard-wire the bucket credentials into your WAR make a copy of the default-jgroups-ec2.xml found in the Infinispan JAR, replace the appropriate ${jgroups.*} placeholder, save it your WEB-INF folder and make a reference to it from the Infinispan configuration XML file.

Step 5. Deploy your service / application to EC2

Deploy your application and check your server logs. If JGroups encounters a faulty S3_PING configuration it will throw a fatal exception and Infinispan will abort.

Common exceptions and how to resolve them:

The S3 bucket name, access key or secret key may be invalid:

java.io.IOException: bucket ‘s3ping-infinispan-c2id’ could not be accessed (rsp=403 (Forbidden). Maybe the bucket is owned by somebody else or the authentication failed

or

org.infinispan.commons.CacheException: java.io.IOException: bucket ‘/tmp/jgroups’ could not be accessed (rsp=301 (Moved Permanently). Maybe the bucket is owned by somebody else or the authentication failed

You registered an S3 bucket name with dots, which caused the wildcard SSL certificate of AWS to be rejected by JGroups:

org.infinispan.commons.CacheException: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching s3ping.cluster.c2id.s3.amazonaws.com found.

Automation

Note that the deployment procedure can be fully automated via the AWS API:

• JGroups user creation
• Bucket creation and policy setting
• Deployment to EC2 / Elastic Beanstalk and System Properties setting.

Alternative node discovery protocols for use in AWS

JGroups developers have contributed alternative discovery protocols for use in AWS:

• NATIVE_S3_PING uses the same protocol as S3_PING, but via the AWS API client library for Java.

• AWS_PING – utilises a feature called EC tags via the AWS API.

We hope to review them and provide instructions for them in some future article. Stay tuned.