How to configure Infinispan for AWS S3_PING discovery
This is a detailed guide how to configure an Infinispan cluster to run in the EC2 cloud of Amazon Web Services (AWS). You can use to deploy a Connect2id server or any other app / service that uses Infinispan in the Amazon cloud.
JGroups
Clustering in Infinispan is handled by the popular open-source JGroups toolkit. It deals with all aspects of node discovery and providing reliable messaging between the nodes. These two functions are implemented by distinct protocols / modules:
Node discovery -- This can be based on a predefined list of node IP addresses, or protocols such as UDP PING.
Node-to-node messaging -- Based on UDP multicast, TCP or tunnel transport.
Due to limitations of the AWS cloud and its virtual networking, native UDP is not supported there. JGroups therefore has to be configured with TCP for node-to-node messaging. As for node discovery, several alternatives exist. In this guide we'll describe setup of the S3_PING protocol which uses an S3 bucket (shared cloud storage) to let the nodes communicate their IP addresses.
Instructions
Step 1. Create an AWS user
We first need to create a dedicated AWS user for JGroups to access the S3 bucket.
Sign in to the AWS console and click on the Identity and Access Management (IAM) tab.
Under Users click on Create New Users and enter the desired username,
e.g. jgroups. Make sure Generate an access key for each user is selected.
Then click on Create.
Record the generated user credentials:
- Access key ID, e.g.
AKIAIHGHNQXQKYJEPEMA - Secure access key, e.g.
GjE5InROhZXvysBLyrRL0mJkjlrUJpMxLX7YUwhF
Also, record the ARN (Amazon Resource Identifier) for the new user. This can be
viewed when you choose the jgroups user in the IAM users lists.
- User ARN, e.g.
arn:aws:iam::673931619296:user/jgroups
No other credentials are required for the jgroups user. Note that you must be
the AWS account administrator or have the required permission to be able to
create a new user.
Step 2. Create S3 bucket
Click on the S3 management tab and then on Create Bucket.
Enter a bucket name, e.g. s3ping-infinispan-c2id.
The name will be used to construct the bucket URL, i.e.
https://s3ping-infinispan-c2id.s3.amazonaws.com and should therefore contain
letters, numbers and dashes only. Do not include dots as this will cause SSL
certificate validation problems later on!
Step 3. Assign permission policy to S3 bucket
Left-click on the newly created S3 bucket and select Properties, then Permissions. This will open up a panel where you can specify the access permissions for the bucket.
Click on Edit bucket policy and paste the following JSON in the provided text area:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "[USER-ARN]"
},
"Action": [
"s3:GetObjectVersion",
"s3:DeleteObject",
"s3:DeleteObjectVersion",
"s3:GetObject",
"s3:PutObject"
],
"Resource": "arn:aws:s3:::[BUCKET]/*"
},
{
"Effect": "Allow",
"Principal": {
"AWS": "[USER-ARN]"
},
"Action": [
"s3:ListBucketVersions",
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::[BUCKET]"
}
]
}
where [USER-ARN] should be replaced with the ARN of the registered user (step
1), e.g. arn:aws:iam::673931619296:user/jgroups and [BUCKET] with the
bucket name (step 2), e.g. s3ping-infinispan-c2id.
Click on Save to apply the entered policy to the S3 bucket. The registered
jgroups user will then have permission to list the bucket's content and view,
create and delete objects in it.
Step 4. Configure Infinispan / JGroups for S3_PING discovery
The Infinispan JAR comes with a set of default JGroups configurations. For AWS
use default-configs/default-jgroups-ec2.xml.
Make sure the following jgroups element is present in your Infinispan XML
configuration (implies Infinispan version 7 or later, earlier versions have a
different configuration format):
<jgroups>
<stack-file name="jgroups-config" path="default-configs/default-jgroups-ec2.xml"/>
</jgroups>
Then supply the following Java system properties to your application:
| Property name | Description |
|---|---|
| jgroups.s3.bucket | The name of the S3 bucket |
| jgroups.s3.access_key | The access key for the AWS user |
| jgroups.s3.secret_access_key | The secure access key for the AWS user |
If you're using Amazon's Elastic Beanstalk you can set these properties from the EB console by selecting your application, clicking on Default configuration, then Configuration and finally Software Configuration. This will present you with a panel named Environment Properties where you can provide arbitrary properties to your application, including the the S3 bucket settings.
Why use system properties to set the credentials? We recommend this approach because it allows you to package your application's WAR without any sensitive credentials in it.
To hard-wire the bucket credentials into your WAR make a copy of the
default-jgroups-ec2.xml found in the Infinispan JAR, replace the appropriate
${jgroups.*} placeholder, save it your WEB-INF folder and make a reference
to it from the Infinispan configuration XML file.
Step 5. Deploy your service / application to EC2
Deploy your application and check your server logs. If JGroups encounters a faulty S3_PING configuration it will throw a fatal exception and Infinispan will abort.
Common exceptions and how to resolve them:
The S3 bucket name, access key or secret key may be invalid:
java.io.IOException: bucket 's3ping-infinispan-c2id' could not be accessed (rsp=403 (Forbidden). Maybe the bucket is owned by somebody else or the authentication failed
or
org.infinispan.commons.CacheException: java.io.IOException: bucket '/tmp/jgroups' could not be accessed (rsp=301 (Moved Permanently). Maybe the bucket is owned by somebody else or the authentication failed
You registered an S3 bucket name with dots, which caused the wildcard SSL certificate of AWS to be rejected by JGroups:
org.infinispan.commons.CacheException: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching s3ping.cluster.c2id.s3.amazonaws.com found.
Automation
Note that the deployment procedure can be fully automated via the AWS API:
- JGroups user creation
- Bucket creation and policy setting
- Deployment to EC2 / Elastic Beanstalk and System Properties setting.
Alternative node discovery protocols for use in AWS
JGroups developers have contributed alternative discovery protocols for use in AWS:
NATIVE_S3_PING uses the same protocol as S3_PING, but via the AWS API client library for Java.
AWS_PING -- utilises a feature called EC tags via the AWS API.
We hope to review them and provide instructions for them in some future article. Stay tuned.