Database, caching and clustering configuration

Caching, in-memory storage, persistence and clustering of Connect2id server’s own data is handled by an embedded Infinispan 8.2 instance:

  • Frequently used data is cached locally on the JVM heap, maximising server performance and throughput.

  • Enables set up of a cluster of Connect2id servers to handle large user bases while remaining highly-available:

  • Nodes can be added or removed dynamically to match demand.

  • Durable data such as client registrations and long-lived authorisations can be persisted to an LDAP v3 directory, PostgreSQL 9.5+, MySQL 5.7.8+ or H2 1.4+ database.

1. Infinispan configuration file

The backend database, caching and clustering are configured in a single Infinspan XML file.

The Connect2id server comes with sample Infinispan configurations for an LDAP, PostgreSQL, MySQL and H2 backend database. There is also a special configuration where Redis assumes the role of the primary cache / in-memory store, and Infinispan as the secondary (in lightweight invalidation mode).

The sample Infinispan configurations are located in the WEB-INF directory of the web application:

WEB-INF/infinispan-{ldap|postgres95|mysql|h2|-redis}.xml

Out of the box the Connect2id server is configured to persist its data to an embedded H2 database.

To switch to a different backend database, e.g. PostgreSQL, edit the infinispan.configurationFile context parameter in web.xml to point to the matching Infinispan configuration XML file for the database, then restart the Connect2id server (nodes):

<context-param>
    <description>
        The location of the Infinispan configuration file. Must be
        relative to the web application root directory.
    </description>
    <param-name>infinispan.configurationFile</param-name>
    <param-value>/WEB-INF/infinispan-postgres95.xml</param-value>
</context-param>

1.1 LDAP backend

This configuration lets the Connect2id server persist its data to an LDAP v3 compatible directory server, such as OpenLDAP 2.4+ or OpenDJ 2.6+:

  • Setup the LDAP directory with the required schema, indices and branches for data.

  • Let Infinispan use the configuration file WEB-INF/infinispan-ldap.xml

  • Set the ldapUser, ldapServer and ldapDirectory properties, or better still, override them with system properties.

1.2 PostgreSQL backend

To persist data to a PostgreSQL 9.5+ database:

  • Create а new database, the character set must be UTF8:

    CREATE DATABASE c2id CHARACTER SET utf8;
  • Use WEB-INF/infinispan-postgres95.xml

  • Set the JDBC dataSource properties, or override them with system properties.

  • The Connect2id server will automatically create the tables it requires when it accesses the database for the first time.

1.3 MySQL backend

To persist data to an MySQL 5.7.8+ database:

  • Create а new database, the character set must be UTF8:

    CREATE DATABASE c2id CHARACTER SET utf8;
  • Use WEB-INF/infinispan-mysql.xml

  • Set the JDBC dataSource properties, or override them with system properties.

  • The Connect2id server will automatically create the tables it requires when
    it accesses the database for the first time.

1.4 H2 backend

To persist data to an H2 1.4+ database:

  • Use WEB-INF/infinispan-h2.xml

  • Set the JDBC dataSource properties, or override them with system properties.

  • The Connect2id server will automatically create the tables it requires when
    it accesses the database for the first time.

1.5 Two-tier cache / in-memory store with Infinispan and Redis

The Connect2id server also supports setups with Redis as a primary in-memory store for objects, while Infinispan is used as secondary cache, in lightweight invalidation mode. This enables quick scaling (up and down) of the Connect2id server cluster, especially in deployments that must support many users (10 million and more).

  • Use WEB-INF/infinispan-{ldap|postgres95|mysql|}-redis.xml

  • Set or override the properties for the underlying database (see LDAP, MySQL or PostgreSQL instructions above).

  • Set or override the redisMapHost / redisCacheHost and redisMapPort / redisCachePort properties.

There is a detailed guide for setting up Infinispan with Redis.

2. Infinispan maps and caches

The Connect2id server utilises 12 named Infinispan key / value maps and caches. Here is a list of their names, usage and recommended settings.

Key / value maps that must be persisted:

  • clients.registrationsMap — Stores the details of each registered OAuth 2.0 / OpenID Connect client. The size setting should allow all client entries to be cached in memory, as this is critical to the overall Connect2id server performance.

  • authzStore.longLivedAuthzMap — Stores the long-lived (sticky) OAuth 2.0 authorisations (consent) and any associated refresh tokens. Caching of long-lived authorisations speeds up returning logins and refresh token lookup.

Key / value maps where persistence is recommended, but not critical:

  • authzStore.accessTokenMap — Stores the active identifier-based OAuth 2.0 access tokens. If only self-contained (JWT-encoded) access tokens are issued, this map will not be utilised. If full persistence is not configured, the map should use passivation to move entries evicted from memory to the backend database.

  • authzStore.revocationJournalMap — Stores the token revocation events. If full persistence is not configured, the map should use passivation to overflow entries from memory to the backend database.

  • sessionStore.sessionMap — Stores the user sessions with the Connect2id server. If full persistence is not configured, the map should use passivation to the backend database.

  • sessionStore.subjectMap — An index of the sessions for each logged in user. If full persistence is not configured, the map should use passivation to the backend database.

Key / value maps and caches where persistence generally isn’t needed:

  • authzStore.codeMap — Stores the pending OAuth 2.0 authorisation codes before they get exchanged for a token. The size setting should be large enough to accommodate the pending OAuth 2.0 codes at any one time, else passivation must be configured.

  • op.authSessionMap — Stores the authentication session state for each user who has an OpenID Connect login page currently opened. The size setting should be large enough for the expected number of open login pages at any one time, else passivation must be configured.

  • op.consentSessionMap — Stores the consent session state for each user who has an OpenID Connect login page currently opened. If there’s not enough memory to fit all consent sessions in memory passivation must be configured.

  • clients.remoteJWKSetCache — Caches remote client JWK sets referenced by URL.

  • clients.remoteRequestJWTClaimsCache — Caches the claims of validated OpenID request objects referenced by URL.

  • op.clientRegTokenMap — Caches expended initial client registration tokens that may be used only once.

3. How to override an Infinispan XML setting with a Java system property

By putting a ${property-name:default-value} placeholder in the Infinispan configuration XML file a setting can be overridden with a Java system property:

Example:

<property name="dataSource.password">${dataSource.password:secret}</property>

If a Java system property named dataSource.password is found, it will be used to set the password, otherwise the default value ("secret") will apply.

Java system properties can be fed from the JVM command line (e.g. via the bin/setenv.sh script in Apache Tomcat), or via a dedicated ServletContextListener launched before Infinispan.

Example bin/setenv.sh snippet to set (override) the data source URL for a MySQL / MariaDB database:

export CATALINA_OPTS="$CATALINA_OPTS -DdataSource.url=jdbc:mariadb://localhost/c2id?useUnicode=yes\&amp\;characterEncoding=UTF-8"

Important caveat: Characters that have a special meaning in the shell, such as & and ;, must be escaped.

4. Configuring cluster networking

Infinispan uses the popular JGroups library to provide reliable messaging between the Connect2id server nodes and features such as automatic cluster formation, node discovery and health checks.

Supported transport protocols:

4.1 How to setup a replicated Connect2id server cluster on your local network

All you need to do is put one or more additional Connect2id servers on your local network, which should be enabled for IP multicast. The nodes will automatically detect one another and form a replication cluster, using UDP as the message transport. The detection and negotiation process can be observed in the log of each server node (at DEBUG level), or by using a JMX console to Infinispan.

4.2 JGroups configuration

Out of the box the Connect2id server is configured to point to the default JGroups UDP settings in the default-jgroups-udp.xml file (for a local network with IP multicast). This XML file is packaged in Infinispan’s JAR:

WEB-INF/libs/infinispan-core-[version].jar

To use a different JGroups configuration edit the global transport configurationFile property in your current WEB-INF/infinispan-*.xml:

<jgroups>
    <stack-file name="jgroups-config" path="default-configs/default-jgroups-udp.xml"/>
</jgroups>

Also, check out our guides on:

You can read more about configuration of JGroups in the Infinispan manual, or on the JGroups website:

4.3 WAN replication

The Connect2id server can also be operated across data centres by means of WAN replication. The following Infinispan and JGroups documents offer further pointers on how to set that up:

5. Resources