Database, caching and clustering

The Connect2id server utilises Infinispan to:

Store its data, such as client registrations, supporting a variety of database technologies, including low-latency in-memory storage for short-lived and cached objects on the JVM heap or in Redis.
Cluster two or more Connect2id server nodes together.

1. Supported databases

Choose a database for persisting Connect2id server data:

Database	Version	Type	Deployment
MySQL	5.7.8+ (except 8.x)	SQL	On premise Cloud
PostgreSQL	9.5+	SQL	On premise Cloud
MS SQL Server	2016+	SQL	On premise Cloud
DynamoDB	n/a	NoSQL	AWS cloud
LDAP	v3	directory	On premise Cloud
H2	1.4+	SQL	Embedded

Notes:

MySQL: You can also use a compatible database, such as MariaDB 10.2.7+ or AWS Aurora.
PostgreSQL: You can also use a compatible database, such CockroachDB.
Microsoft SQL server is supported with Connect2id server v7.18+.
Any LDAP v3 compatible directory is supported. We recommend OpenLDAP 2.4+ or OpenDJ 3.0+.
The lightweight relational H2 database is intended for testing and development purposes. Out of the box the Connect2id server is configured with an embedded H2 database instance.
To ensure the high-availability and durability of your Connect2id server data, use a database cluster set up for replication and fail-over.
The multitenant Connect2id server edition supports MySQL, PostgreSQL, SQL Server and H2.

2. Supported cluster modes

Choose a cluster mode for your scaling needs:

	Replication Mode	Stateless Mode
Server nodes	1 - 2	1 - ∞
Short-lived / cache data storage	Local: JVM heap	External: Redis or the main database
Ideal for	small to medium loads predictable loads	small to large loads quick up/down scaling

To make your OpenID and OAuth 2.0 service highly-available deploy a Connect2id server cluster with at least two nodes. A single node handles in the order of 100 to 300 OpenID Connect requests (in the code flow) per CPU core. Server nodes can be added and removed dynamically to / from the cluster.

Replication mode – Ideal for medium, predictable loads. For improved performance sessions and other short-lived objects are stored in Infinispan on the JVM heap. Persisted server data that is frequently used is also cached there. The in-memory data is continuously replicated between the nodes. The JVM on each node should be provisioned with sufficient heap capacity for the expected maximum number of sessions and other objects stored in memory.

Stateless mode – Ideal for large loads, allows quick up / down cluster scaling. The Connect2id server state is stored entirely in the underlying database(s).

Basic – Using the configured SQL or DynamoDB database to store all short and long-lived server data.
Redis – Improves performance and latency by using Redis as an additional in-memory database to cache frequently used server data and store sessions and other short-lived objects. The Redis instance(s) should be provisioned with sufficient memory to handle peak usage.

Standalone – Intended primarily for Connect2id server testing and development. Clustering is disabled, data is persisted to an embedded relational H2 database, caching is on the JVM heap. This is the out-of-the-box configuration.

One way to estimate your memory requirements is to load the Connect2id server with a number of simulated sessions containing typical data via the session store API.

3. Configuration

When you’ve chosen a backend database and cluster mode the actual set up is as simple as:

Link to the appropriate Infinispan XML configuration file.
Set up a database and its connection parameters.
If the replication cluster mode is selected, allow inbound port access for the sync messages.
Configure a health check URL.

The four steps are explained below.

3.1 Link to the Infinispan XML configuration file

The Connect2id server comes with a set of Infinispan configuration files for each cluster mode and database combination in the WEB-INF directory of its c2id.war package.

Cluster in replication mode:

/WEB-INF/infinispan-replication-mysql.xml
/WEB-INF/infinispan-replication-postgres95.xml
/WEB-INF/infinispan-replication-sqlserver.xml
/WEB-INF/infinispan-replication-dynamodb.xml
/WEB-INF/infinispan-replication-ldap.xml

Cluster in stateless mode:

/WEB-INF/infinispan-stateless-mysql.xml
/WEB-INF/infinispan-stateless-postgres95.xml
/WEB-INF/infinispan-stateless-sqlserver.xml
/WEB-INF/infinispan-stateless-dynamodb.xml

Cluster in stateless mode, with Redis for caching and storing short-lived objects:

/WEB-INF/infinispan-stateless-redis-mysql.xml
/WEB-INF/infinispan-stateless-redis-postgres95.xml
/WEB-INF/infinispan-stateless-redis-sqlserver.xml
/WEB-INF/infinispan-stateless-redis-ldap.xml

Local (no clustering) mode:
```
/WEB-INF/infinispan-local-h2.xml
```

Out of the box the Connect2id server is configured in local mode with an embedded H2 database:

/WEB-INF/infinispan-local-h2.xml

For the multi-tenant Connect2id server edition:

Cluster in stateless mode:

/WEB-INF/infinispan-multitenant-stateless-mysql.xml
/WEB-INF/infinispan-multitenant-stateless-postgres95.xml
/WEB-INF/infinispan-multitenant-stateless-sqlserver.xml

Cluster in stateless mode, with Redis for caching and storing short-lived objects:

/WEB-INF/infinispan-multitenant-stateless-redis-mysql.xml
/WEB-INF/infinispan-multitenant-stateless-redis-postgres95.xml
/WEB-INF/infinispan-multitenant-stateless-redis-sqlserver.xml

Local (no clustering) mode:

/WEB-INF/infinispan-multitenant-local-h2.xml

To switch to the configuration of your choice, e.g. stateless cluster mode with a PostgreSQL database, you have two possibilities:

Set the infinispan.configurationFile Java system property:

-Dinfinispan.configurationFile=/WEB-INF/infinispan-stateless-postgres95.xml

Or, edit the infinispan.configurationFile context parameter in /WEB-INF/web.xml:

<context-param>
    <description>
        The location of the Infinispan configuration file. Must be
        relative to the web application root directory.
    </description>
    <param-name>infinispan.configurationFile</param-name>
    <param-value>/WEB-INF/infinispan-stateless-postgres95.xml</param-value>
</context-param>

3.2 Set up the database and connection parameters

The setup instructions for each supported database type:

3.2.1 MySQL

To persist data to an MySQL 5.7.8+ database:

Create а new database, the character set must be UTF8:
```
CREATE DATABASE c2id CHARACTER SET utf8
```
Set the JDBC properties in the Infinispan XML configuration, or better still, override them with system properties:
- dataSource.url – the JDBC URL. The provided MariaDB JDBC driver supports MySQL, MariaDB and AWS Aurora specific configurations, with a set of fail-over and load-balancing modes.
- dataSource.user – the database user
- dataSource.password – the database password
The Connect2id server uses the default settings of its Hikari JDBC connection pool. They can be tuned by adding an appropriate property to the SQL store parameters in the Infinispan XML configuration.
The Connect2id server will automatically create the tables it requires when it accesses the database for the first time.

3.2.2 PostgreSQL

To persist data to a PostgreSQL 9.5+ database:

Create а new database, the character set must be UTF8:
```
CREATE DATABASE c2id ENCODING UTF8
```
Set the JDBC properties in the Infinispan XML configuration, or better still, override them with system properties:
- dataSource.host – the database host name / IP address
- dataSource.databaseName – the database name (schema)
- dataSource.user – the database user
- dataSource.password – the database password
The Connect2id server uses the default settings of its Hikari JDBC connection pool. They can be tuned by adding an appropriate property to the SQL store parameters in the Infinispan XML configuration.
The Connect2id server will automatically create the tables it requires when it accesses the database for the first time.

3.2.3 Microsoft SQL Server

To persist data to an SQL Server 2016+ database:

Create а new database, the collation is irrelevant, the character should be the default UCS-2 / UTF-16 supporting i18n or may be set to UTF8 (supported from SQL Server version 2019 on):
```
CREATE DATABASE c2id
```
Set the JDBC properties in the Infinispan XML configuration, or better still, override them with system properties:
- dataSource.url – the JDBC URL
- dataSource.user – the database user
- dataSource.password – the database password
The Connect2id server uses the default settings of its Hikari JDBC connection pool. They can be tuned by adding an appropriate property to the SQL store parameters in the Infinispan XML configuration.
The Connect2id server will automatically create the tables it requires when it accesses the database for the first time.

3.2.4 DynamoDB

To persist data to DynamoDB in the AWS cloud:

Set the connection properties in the Infinispan XML configuration, or better still, override them with system properties:
- dynamodb.region – the preferred AWS region, defaults to us-east-1.
- dynamodb.httpProxyHost, dynamodb.httpProxyPort – optional proxy for accessing the DynamoDB endpoint. Since v9.3.
- dynamodb.encryptionAtRest – enables optional server-side encryption, defaults to false.
- dynamodb.enableStream – enables optional streaming for global tables.
- dynamodb.tablePrefix – optional prefix for all tables created by the Connect2id server, defaults to none.
- dynamodb.enableTTL.sessionStore.sessionMap – set to true to enable automatic expiration of subject sessions by DynamoDB (via a “ttl” attribute). Also set the sessionStore.sessionMap.expirationInterval Java system property to zero (0) to disable the Infinispan expiration thread. Expiration may be moved to DynamoDB if there are no relying parties registered for OpenID Connect back-channel logout notifications, as session expiration notifications can no longer be delivered to them.
Make sure the AWS credentials for accessing the DynamoDB tables are configured in way that the default AWS credentials provider chain can look them up, ideally via IAM instance profile roles. The Connect2id server requires access to the following DynamoDB actions:
- “dynamodb:BatchGetItem”
- “dynamodb:BatchPutItem”
- “dynamodb:CreateTable”
- “dynamodb:DeleteItem”
- “dynamodb:DescribeTable”
- “dynamodb:GetItem”
- “dynamodb:PutItem”
- “dynamodb:Query”
- “dynamodb:Scan”
- “dynamodb:UpdateItem”
The Connect2id server will automatically create the tables it requires when it accesses DynamoDB for the first time.
After the DynamoDB tables are created enable auto-scaling for each table.
If you want to run Connect2id server instances in two or more AWS regions, replicating data between them asynchronously, set up global DynamoDB tables.

3.2.5 LDAP

To persist data to an LDAP v3 compatible directory server, such as OpenLDAP 2.4+ or OpenDJ 2.6+:

Setup the LDAP directory with the required schema, indices and branches for data.
Let Infinispan use the configuration file WEB-INF/infinispan-replication-ldap.xml
Set the connection properties in the Infinispan XML configuration, or better still, override them with system properties:
- ldapUser.dn – the DN of Connect2id server account for connecting to the LDAP directory
- ldapUser.password – the password
- ldapServer.url – the LDAP server host name / IP address and port
- [map-name].ldapDirectory.baseDN – to override the default base DN for the entries for a given data structure

3.2.6 H2

To persist data to an H2 1.4+ database:

Set the JDBC properties in the Infinispan XML configuration, or better still, override them with system properties:
- dataSource.url – the JDBC URL
- dataSource.user – the database user
- dataSource.password – the database password
The Connect2id server uses the default settings of its Hikari JDBC connection pool. They can be tuned by adding an appropriate property to the SQL store parameters in the Infinispan XML configuration.
The Connect2id server will automatically create the tables it requires when it accesses the database for the first time.

3.2.7 Redis

If the Connect2id server is deployed in stateless cluster mode where short-lived and cached data will be stored in Redis:

We recommend that you set up two separate Redis servers (or clusters):
- One Redis server for transient data intended to be stored in-memory, such as sessions. This Redis server should be provisioned with enough memory to hold all current sessions and other objects that will be stored in memory. Make sure the eviction policy of this Redis server is set to volatile-lru:
```
maxmemory-policy volatile-lru
```
- Another Redis server for caching data only. Configure it with an allkeys-lru policy, to let Redis evict the least recently keys when the maxmemory limit is reached. There is no particular recommendation how much memory to allocate to the caching Redis server. See what works best for you in terms of service latency and budget.
```
maxmemory-policy allkeys-lru
```
Set the Redis connection properties in the Infinispan XML configuration, or better still, override them with system properties:
- redisMapHost, redisMapPort – to point to the host and port of the Redis server for storing transient data, such as sessions.
- redisCacheHost, redisCachePort – to point to the host and port of the Redis server for cached data.

3.3 Open inbound ports (replication cluster mode)

TCP

If you’re going to run the Connect2id server in a replication cluster mode with an SQL database or DynamoDB allow inbound TCP traffic to the following ports:

port 7800
port 7900 through 7905

These ports are required by the clustering layer (JGroups) to receive sync messages (7800) and detect node failures (7900-7905).

The Connect2id server will pick a “site local” (non-routable) IP address on the host machine to attach the TCP listening sockets, e.g. from the 192.168.0.0 or 10.0.0.0 address range. To specify the address otherwise set the jgroups.tcp.address Java system property, e.g. like this:

-Djgroups.tcp.address=match-interface:eth0

Check out the bind_addr section in the JGroups docs for information.

If for some reason IPv6 networking gets used by mistake, you can force the JVM to use IPv4:

-Djava.net.preferIPv4Stack=true

UDP / multicast

If you’re deploying the Connect2id server in replication cluster mode with an LDAP backend database the discovery and replication will be facilitated via UDP / multicast, at port 45588. To override it:

-Djgroups.udp.mcast_port=45588

3.4 Health check URL

Configure your load balancer / reverse proxy to use the health-check endpoint for determining a node’s status. The health-check guide has helpful information.

4. How to override an Infinispan XML configuration setting with a Java system property

Any ${property-name:default-value} placeholder found in the Infinispan XML configuration file can be overridden with a Java system property:

In this example, if a Java system property named dataSource.password is found, it will be used to set the database password, otherwise the default value (“secret”) will apply:

<property name="dataSource.password">${dataSource.password:secret}</property>

The external configuration guide has tips for setting system properties from environment variables, local files and other locations.

5. FAQ

5.1 How to add a new node to a Connect2id server cluster?

Simply start the new node and let the load balancer direct HTTP requests to it once the health check endpoint becomes available and returns status 200.

If your cluster is in replication mode, the joining node will automatically discover the coordinator (via the configured discovery protocol, e.g. JDBC_PING) and start communicating with it and the other nodes.

If your cluster is in stateless mode, the nodes don’t sync data between them, and hence there is no need for them to discover one another.

5.2 How to remove a node from a Connect2id server cluster?

Make the load balancer stop directing HTTP requests to it, then shut down the node graciously. On Tomcat you can do this by calling its bin/shutdown.sh script.

5.3 How to track the cluster size?

For a cluster in replication mode check out the infinispan.numClusterMembers metric.

The gauge will get incremented when a new node joins, decremented when a node leaves.

5.4 Where are the cluster events and errors logged?

The Infinispan and JGroups messages are appended to the log of the Connect2id server.

Check out our guide how to interpret Infinispan and JGroups messages written to the log.

5.5 What JGroups protocols are used?

In a replication mode the JGroups clustering layer is configured to use the following protocols:

For message transport:
- TCP – for Connect2id server setups with MySQL, PostgreSQL, SQL Server or DynamoDB backend database.
- UDP/IP multicast – for setups with an LDAP v3 backend database.
For initial node discovery:
- JDBC_PING – for setups with MySQL, PosgreSQL or SQL Server backend database.
- DYNAMODB_PING – for setups with DynamoDB backend database.
- PING (UDP/IP multicast) – for setups with LDAP v3 backend database.

5.7 Is WAN / cross-datacentre replication supported?

Yes. Full cross-datacentre replication can be implemented in the “stateless” mode when configured with a DynamoDB backend database and global tables.

Partial replication is possible in the “stateless” mode with an SQL backend, for the persisted data, such as client registrations and long-lived authorisations. An OAuth 2.0 or OpenID Connect client application would not be able to switch seamlessly between regions, e.g. in the middle of an OAuth 2.0 flow. Its credentials and authorisations (refresh tokens), however, will work. The subject sessions established with the IdP can be replicated, but the Connect2id server must be configured to persist them to the SQL store. Normally the subject sessions are kept in memory (Redis) only, for quick access.

Contact support to receive further assistance.

6. Description of the Infinispan maps and caches

Here is a description of the individual Infinispan key / value maps and caches used by the Connect2id server. This information can be useful in order to fine tune the storage parameters.

Key / value maps that must be persisted:

clients.registrationsMap – The registered OAuth 2.0 / OpenID Connect clients. For optimal performance aim to have all client entries cached in memory.
authzStore.longLivedAuthzMap – The long-lived (sticky) OAuth 2.0 authorisations (consent), including the associated refresh tokens. Caching of these entries speeds up processing of login requests as well as refresh token lookup.
authzStore.revocationJournalMap – The authorisation revocation journal.
tenantRegistry.tenants – The Connect2id server tenants. Applies to the multitenant edition of the server only.

Key / value maps where persistence is recommended, but not critical:

authzStore.accessTokenMap – The active identifier-based OAuth 2.0 access tokens. If only self-contained (JWT-encoded) access tokens are issued, this map will not be utilised. If full persistence is not configured, the map should use passivation to move entries evicted from memory to the backend database.
sessionStore.sessionMap – The end-user sessions with the Connect2id server. If full persistence is not configured, the map should use passivation to the backend database.
sessionStore.subjectMap – An index of the sessions for each logged in end-user. If full persistence is not configured, the map should use passivation to the backend database.

Key / value maps and caches where persistence generally isn’t needed:

authzStore.codeMap – The pending OAuth 2.0 authorisation codes awaiting exchange for a token. If stored in memory passivation is recommended.
op.authSessionMap – The authentication session state for each end-user who has a login page currently opened. If stored in memory enough capacity should be allocated to accommodate the expected number of open login pages at any one time, else passivation must be configured.
op.consentSessionMap – The consent session state for each end-user who has an login page currently opened. If stored in memory enough capacity should be allocated to accommodate the expected number of open login pages at any one time, else passivation must be configured.
clients.remoteJWKSetCache – Caches remote client JWK sets referenced by URL.
clients.remoteRequestJWTClaimsCache – Caches the claims of validated OAuth 2.0 request objects referenced by URL.
op.clientRegTokenMap – Caches expended initial client registration tokens that may be used only once.