Database, caching and clustering

The Connect2id server utilises Infinispan to manage its data:

To persist durable data, such as OAuth 2.0 client registrations and authorisations, to a database;
Store transient data, such as sessions, in-memory;
Cache frequently used data, for snappy service with maximum throughput;
Cluster multiple Connect2id server nodes together, replicating session and other in-memory data between the nodes.

1. Supported databases

Choose a database where the Connect2id server will persist its data:

	MySQL	PostgreSQL	DynamoDB	LDAP
Version	5.7.8+	9.5+	n/a	v3
Type	Relational	Relational	NoSQL	directory
Deployment	On premise Cloud	On premise Cloud	AWS cloud	On premise

Notes:

The lightweight relational H2 database (1.4+) is also supported, but should only be used for Connect2id server testing and development purposes. Out of the box the server is configured with an embedded H2 database instance.
You can also use an API compatible database, such as MariaDB 10.2.7+ or AWS AuroraDB.
Any LDAP v3 compatible directory is technically supported. We recommend OpenLDAP 2.4+ or OpenDJ 3.0+.
To ensure the high-availability and durability of your Connect2id server data, use a database cluster set up for replication and fail-over.

2. Supported cluster modes

Choose a suitable cluster mode for your scaling needs:

	Replication Mode	Stateless Mode
Number of server nodes	1 - 4	1 - ∞
In-memory / cache data storage location	Local: JVM heap	External: Redis
Suitable for	small to medium loads predictable loads	small to large loads quick up/down scaling

Notes:

To ensure high-availability of your OpenID and OAuth 2.0 service you should run a Connect2id server cluster with at least two nodes.
Server nodes can be dynamically added and removed.
A lightweight “local” mode (clustering disabled, data cached on the JVM heap) is also supported, but should only be used for Connect2id server testing and development purposes. This is the out-of-the-box configuration (with H2 as backend database).
In “replication” cluster mode the JVM on each node must be provisioned with sufficient heap memory for the expected maximum number of sessions and other transient objects.
In “stateless” cluster mode the Redis store be provisioned with sufficient memory for the expected maximum number of sessions and other transient objects.
A good way to estimate your memory requirements is to load the Connect2id server with a number of simulated sessions containing typical data via the session store API.

3. Configuration

Once you’ve chosen a backend database and cluster mode the actual set up is as simple as:

Linking to the appropriate Infinispan XML configuration file
Setting up a database and the required connection parameters
If the “replication” cluster mode is selected, allowing inbound TCP access for the sync messages

The three steps are explained below.

3.1 Link to the appropriate Infinispan XML configuration file

The Connect2id server comes with a set of ready Infinispan configuration files for each supported cluster mode and database combination in the WEB-INF directory of its web application package:

/WEB-INF/infinispan-replication-mysql.xml
/WEB-INF/infinispan-replication-postgres95.xml
/WEB-INF/infinispan-replication-dynamodb.xml
/WEB-INF/infinispan-replication-ldap.xml
/WEB-INF/infinispan-stateless-redis-mysql.xml
/WEB-INF/infinispan-stateless-redis-postgres95.xml
/WEB-INF/infinispan-stateless-dynamodb.xml
/WEB-INF/infinispan-stateless-redis-ldap.xml
/WEB-INF/infinispan-local-h2.xml

Out of the box the Connect2id server is configured in “local” mode with an embedded H2 database:

/WEB-INF/infinispan-local-h2.xml

To switch to the configuration of your choice, e.g. “replication” cluster mode with a PostgreSQL database, you have two possibilities:

Edit the infinispan.configurationFile context parameter in /WEB-INF/web.xml:

<context-param>
    <description>
        The location of the Infinispan configuration file. Must be
        relative to the web application root directory.
    </description>
    <param-name>infinispan.configurationFile</param-name>
    <param-value>/WEB-INF/infinispan-replication-postgres95.xml</param-value>
</context-param>

Or, set the Java system property infinispan.configurationFile:

-Dinfinispan.configurationFile=/WEB-INF/infinispan-replication-postgres95.xml

3.2 Set up the database and connection parameters

Here are the setup instructions for each supported database type:

3.2.1 MySQL

To persist data to an MySQL 5.7.8+ database:

Create а new database, the character set must be UTF8:
```
CREATE DATABASE c2id CHARACTER SET utf8;
```
Set the JDBC properties in the Infinispan XML configuration, or better still, override them with system properties:
- dataSource.url – the JDBC URL
- dataSource.user – the database user
- dataSource.password – the database password
The Connect2id server will automatically create the tables it requires when it accesses the database for the first time.

3.2.2 PostgreSQL

To persist data to a PostgreSQL 9.5+ database:

Create а new database, the character set must be UTF8:
```
CREATE DATABASE c2id CHARACTER SET utf8;
```
Set the JDBC properties in the Infinispan XML configuration, or better still, override them with system properties:
- dataSource.host – the database host name / IP address
- dataSource.databaseName – the database name (schema)
- dataSource.user – the database user
- dataSource.password – the database password
The Connect2id server will automatically create the tables it requires when it accesses the database for the first time.

3.2.3 DynamoDB

To persist data to DynamoDB in the AWS cloud:

Set the connection properties in the Infinispan XML configuration, or better still, override them with system properties:
- dynamodb.region – the preferred AWS region, defaults to us-east-1
Make sure the AWS credentials for accessing the DynamoDB tables are configured in way that the default AWS credentials provider chain can look them up, ideally via IAM instance profile roles. The Connect2id server requires access to the following DynamoDB actions:
- “dynamodb:BatchGetItem”
- “dynamodb:BatchPutItem”
- “dynamodb:CreateTable”
- “dynamodb:DeleteItem”
- “dynamodb:DescribeTable”
- “dynamodb:GetItem”
- “dynamodb:PutItem”
- “dynamodb:Query”
- “dynamodb:Scan”
- “dynamodb:UpdateItem”
The Connect2id server will automatically create the tables it requires when it accesses DynamoDB for the first time.
After the DynamoDB tables are created enable auto-scaling for each table.

3.2.4 LDAP

To persist data to an LDAP v3 compatible directory server, such as OpenLDAP 2.4+ or OpenDJ 2.6+:

Setup the LDAP directory with the required schema, indices and branches for data.
Let Infinispan use the configuration file WEB-INF/infinispan-ldap.xml
Set the connection properties in the Infinispan XML configuration, or better still, override them with system properties:
- ldapUser.dn – the DN of Connect2id server account for connecting to the LDAP directory
- ldapUser.password – the password
- ldapServer.url – the LDAP server host name / IP address and port
- [map-name].ldapDirectory.baseDN – to override the default base DN for the entries for a given data structure

3.2.5 H2

To persist data to an H2 1.4+ database:

Set the JDBC properties in the Infinispan XML configuration, or better still, override them with system properties:
- dataSource.url – the JDBC URL
- dataSource.user – the database user
- dataSource.password – the database password
The Connect2id server will automatically create the tables it requires when it accesses the database for the first time.

3.2.6 Redis

If the Connect2id server is used in “stateless” cluster mode where in-memory and cached data will be stored in Redis:

We recommend that you set up two separate Redis servers (or clusters):
- One Redis server for transient data intended to be stored in-memory, such as sessions. This Redis server should be provisioned with enough memory to hold all current sessions and other objects that will be stored in memory. Make sure the eviction policy of this Redis server is set to volatile-lru:
```
maxmemory-policy volatile-lru
```
- Another Redis server for caching data only. Configure it with an allkeys-lru policy, to let Redis evict the least recently keys when the maxmemory limit is reached. There is no particular recommendation how much memory to allocate to the caching Redis server. See what works best for you in terms of service latency and budget.
```
maxmemory-policy allkeys-lru
```
Set the Redis connection properties in the Infinispan XML configuration, or better still, override them with system properties:
- redisMapHost, redisMapPort – to point to the host and port of the Redis server for storing transient data, such as sessions.
- redisCacheHost, redisCachePort – to point to the host and port of the Redis server for cached data.

3.3 Open inbound ports (replication cluster mode)

If you’re going to run the Connect2id server in a “replication” cluster mode allow inbound TCP traffic to the following ports:

port 7800
port 7900 through 7905

These ports are required by the clustering layer (JGroups) to receive sync messages (7800) and detect node failures (7900-7905).

The Connect2id server will pick a “site local” (non-routable) IP address on the host machine to attach the TCP listening sockets, e.g. from the 192.168.0.0 or 10.0.0.0 address range. To specify the address otherwise set the jgroups.tcp.address Java system property, e.g. like this:

-Djgroups.tcp.address=match-interface:eth0

Check out the bind_addr section in the JGroups docs for information.

If for some reason IPv6 networking gets used by mistake, you can force the JVM to use IPv4:

-Djava.net.preferIPv4Stack=true

4. How to override an Infinispan XML configuration setting with a Java system property

Any ${property-name:default-value} placeholder found in the Infinispan XML configuration file can be overridden with a Java system property:

In this example, if a Java system property named dataSource.password is found, it will be used to set the database password, otherwise the default value (“secret”) will apply:

<property name="dataSource.password">${dataSource.password:secret}</property>

The external configuration guide has tips for setting system properties from environment variables, local files and other locations.

5. FAQ

5.1 How to add a new node to a Connect2id server cluster?

Simply start the new node and let the load balancer direct HTTP requests to it once the health check endpoint becomes available and returns status 200.

If your cluster is in replication mode, the joining node will automatically discover the coordinator (via the configured discovery protocol, e.g. JDBC_PING) and start communicating with it and the other nodes.

If your cluster is in stateless mode, the nodes don’t sync data between them, and hence there is no need for them to discover one another.

5.2 How to remove a node from a Connect2id server cluster?

Make the load balancer stop directing HTTP requests to it, then shut down the node graciously. On Tomcat you can do this by calling its bin/shutdown.sh script.

5.3 How to track the cluster size?

Check out the infinispan.numClusterMembers metric.

The gauge will get incremented when a new node joins, decremented when a node leaves.

5.4 Where are the cluster events and errors logged?

The Infinispan and JGroups messages are appended to the log of the Connect2id server.

Check out our guide how to interpret Infinispan and JGroups messages written to the log.

5.5 Why is the invalidation cluster mode discontinued?

The lightweight invalidation cluster mode was discontinued in v6.15 due to reliability issues of its implementation in Infinispan 8.2. Users are advised to switch to a matching “stateless” Infinispan configuration.

The invalidation mode will be made available again once the underlying issues in Infinispan have been fixed.

5.6 What JGroups protocols are used?

The JGroups clustering layer is configured to use the following protocols:

For message transport:
- TCP – for Connect2id server setups with MySQL, PostgreSQL, or DynamoDB backend database.
- UDP/IP multicast – for setups with an LDAP v3 backend database.
For initial node discovery:
- JDBC_PING – for setups with MySQL or PosgreSQL backend database.
- DYNAMODB_PING – for setups with DynamoDB backend database.
- PING (UDP/IP multicast) – for setups with LDAP v3 backend database.

5.7 Is WAN / cross-datacentre replication possible?

Yes. Full cross-datacentre replication can be implemented in the “stateless” mode when configured with a DynamoDB backend database and global tables.

Partial replication is possible in the “stateless” mode with an SQL backend, for the persisted data, such as client registrations and long-lived authorisations. An OAuth 2.0 or OpenID Connect client application would not be able to switch seamlessly between regions, e.g. in the middle of an OAuth 2.0 flow. Its credentials and authorisations (refresh tokens), however, will work. The subject sessions established with the IdP can be replicated, but the Connect2id server must be configured to persist them to the SQL store. Normally the subject sessions are kept in memory (Redis) only, for quick access.

Contact support to receive further assistance.

6. Description of the Infinispan maps and caches

Here is a description of the individual Infinispan key / value maps and caches used by the Connect2id server. This information can be useful in order to fine tune the storage parameters.

Key / value maps that must be persisted:

clients.registrationsMap – The details of each registered OAuth 2.0 / OpenID Connect client. Aim to have all client entries cached in memory, as this is critical to the overall Connect2id server performance.
authzStore.longLivedAuthzMap – The long-lived (sticky) OAuth 2.0 authorisations (consent), including the associated refresh tokens. Caching of these entries speeds up processing of login requests as well as refresh token lookup.

Key / value maps where persistence is recommended, but not critical:

authzStore.accessTokenMap – The active identifier-based OAuth 2.0 access tokens. If only self-contained (JWT-encoded) access tokens are issued, this map will not be utilised. If full persistence is not configured, the map should use passivation to move entries evicted from memory to the backend database.
authzStore.revocationJournalMap – The token revocation events. If full persistence is not configured, the map should use passivation to overflow entries from memory to the backend database.
sessionStore.sessionMap – The end-user sessions with the Connect2id server. If full persistence is not configured, the map should use passivation to the backend database.
sessionStore.subjectMap – An index of the sessions for each logged in end-user. If full persistence is not configured, the map should use passivation to the backend database.

Key / value maps and caches where persistence generally isn’t needed:

authzStore.codeMap – The pending OAuth 2.0 authorisation codes before they get exchanged for a token. The allocated map capacity should be large enough to accommodate the expected number of pending OAuth 2.0 codes at any one time, else passivation must be configured.
op.authSessionMap – The authentication session state for each end-user who has a login page currently opened. The allocated map capacity should be large enough to accommodate the expected number of open login pages at any one time, else passivation must be configured.
op.consentSessionMap – The consent session state for each end-user who has an login page currently opened. The allocated map capacity should be large enough to accommodate the expected number of open login pages at any one time, else passivation must be configured.
clients.remoteJWKSetCache – Caches remote client JWK sets referenced by URL.
clients.remoteRequestJWTClaimsCache – Caches the claims of validated OpenID request objects referenced by URL.
op.clientRegTokenMap – Caches expended initial client registration tokens that may be used only once.