Skip to content
Connect2id

Two tier caching with Redis

Support for two-tier caching was discontinued in Connect2id server 6.15 due to reliability issues with Infinispan in invalidation mode.

The case for two tier caching

Version 5.0.3 of the Connect2id server introduced support for two tiered in-memory storage and caching of server data. This is achieved by employing Redis as primary in-memory / cache store. The Infinispan instances embedded in each Connect2id server node still play a role, but only to cache data secondarily, in super lightweight invalidation mode.

When to consider two tier caching with Redis:

  • You want to be able to scale processing power (Connect2id server nodes) and in-memory storage / caching independently from one another.
  • You want to take advantage of AWS ElastiCache or similar readily available Redis-based service of your chosen cloud provider.
  • You want to keep Connect2id server nodes relatively lightweight in terms of RAM, for GC or other reasons.
  • You serve a large number of identities (10M+ end-users). If your end-users are fewer than that a regular single tier deployment with Infinispan in replication or distribution mode is sufficient.
Two tier caching with Redis

Two tier in-memory / cache store with Infinispan and Redis

How to set up Redis

We recommend that you set up two separate Redis instances (servers or server clusters) for the second tier in-memory / cache store:

  1. One Redis instance for data intended to be stored in-memory only (with no other configured persistence), such as sessions and identifier-based access tokens. This instance should be provisioned with enough memory to hold all current sessions and other objects that will be stored in memory. Make sure the eviction policy of this Redis instance is set to volatile-lru:

    maxmemory-policy volatile-lru

  2. Another Redis instance for caching purposes only. Configure it with an allkeys-lru policy, to let Redis evict the least recently keys when the maxmemory limit is reached. There is no particular recommendation how much memory to allocate to the caching Redis instance. See what works best for you in terms of IdP / AS latency and budget.

    maxmemory-policy allkeys-lru

How to configure your Connect2id server

Step 1.

Edit the infinispan.configurationFile context parameter in the web.xml descriptor (found in the c2id.war) to point to the provided sample infinispan-redis.xml file:

<context-param>
    <description>
        The location of the Infinispan configuration file. Must be
        relative to the web application root directory.
    </description>
    <param-name>infinispan.configurationFile</param-name>
    <param-value>/WEB-INF/infinispan-redis.xml</param-value>
</context-param>

Step 2.

The sample infinispan-redis.xml file will persist identifier-based tokens, long-lived authorisations, the revocation journal and client registrations to an LDAP directory. If you intend to persist that data to a different type of database, e.g. PostgreSQL, swap the LDAP store declarations accordingly (you can use a suitable sample Infinispan configuration as template).

Step 3.

Resize the Infinispan invalidation caches in infinispan-redis.xml if needed. You can do this by editing the size attribute of the eviction element, where size determines the number of objects to cache in the JVM memory.

Ensure your Connect2id server nodes are provisioned with enough extra JVM heap memory to accommodate the content of all invalidation caches at full size. If you need assistance with estimating that contact Connect2id support.

Step 4.

Finally, let Infinispan know where the two Redis instances are located. You can do this by directly editing the Redis host and port attributes in infinispan-redis.xml, or by overriding the default localhost:6379 with Java system properties:

A. Edit Redis host and port directly

For each of the following Infinispan invalidation caches set the Redis host and port to point to the Redis instance for in-memory storage of data:

  1. sessionStore.sessionMap
  2. sessionStore.subjectMap
  3. authzStore.codeMap
  4. op.authSessionMap
  5. op.consentSessionMap
  6. op.clientRegTokenMap

For example:

<redis-server host="redis-maps.c2id.com" port="6379"/>

For each of the following Infinispan invalidation caches set the Redis host and port to point to the Redis instance for caching data:

  1. authzStore.accessTokenMap
  2. authzStore.longLivedAuthzMap
  3. authzStore.revocationJournalMap
  4. clients.registrationsMap
  5. clients.remoteJWKSetCache
  6. clients.remoteRequestJWTClaimsCache

For example:

<redis-server host="redis-caches.c2id.com" port="6379"/>

B. Provide Redis host and port via Java system properties

This the recommended method. Override the default host and port attributes with Java system properties like this:

  1. Let redisMapHost and redisMapPort point to the Redis instance for regular in-memory storage of Connect2id server data.

  2. Let redisCacheHost and redisCachePort point to the Redis instance for caching Connect2id server data.

This can for instance be done at JVM startup or via the bin/setev.sh script of Apache Tomcat. Example:

-DredisMapHost=redis-maps.c2id.com -DredisCacheHost=redis-caches.c2id.com

Step 5.

Finally, restart or redeploy the Connect2id server node(s) for the updated configuration to take effect.

If the Redis configuration is invalid or has conflicts the Connect2id server will abort startup and log a detailed error message.