How to setup two tier caching with Infinispan and Redis
The case for two tier caching
Version 5.0.3 of the Connect2id server introduced support for two tiered in-memory storage and caching of server data. This is achieved by employing Redis as primary in-memory / cache store. The Infinispan instances embedded in each Connect2id server node still play a role, but only to cache data secondarily, in super lightweight invalidation mode.
When to consider two tier caching with Redis:
- You want to be able to scale processing power (Connect2id server nodes) and in-memory storage / caching independently from one another.
- You want to take advantage of AWS ElastiCache or similar readily available Redis-based service of your chosen cloud provider.
- You want to keep Connect2id server nodes relatively lightweight in terms of RAM, for GC or other reasons.
- You serve a large number of identities (10M+ end-users). If your end-users are fewer than that a regular single tier deployment with Infinispan in replication or distribution mode is sufficient.
Two tier in-memory / cache store with Infinispan and Redis
How to set up Redis
We recommend that you set up two separate Redis instances (servers or server clusters) for the second tier in-memory / cache store:
One Redis instance for data intended to be stored in-memory only (with no other configured persistence), such as sessions and identifier-based access tokens. This instance should be provisioned with enough memory to hold all current sessions and other objects that will be stored in memory. Make sure the eviction policy of this Redis instance is set to volatile-lru:
Another Redis instance for caching purposes only. Configure it with an allkeys-lru policy, to let Redis evict the least recently keys when the maxmemory limit is reached. There is no particular recommendation how much memory to allocate to the caching Redis instance. See what works best for you in terms of IdP / AS latency and budget.
How to configure your Connect2id server
Edit the infinispan.configurationFile context parameter in the
descriptor (found in the
c2id.war) to point to the provided sample
<context-param> <description> The location of the Infinispan configuration file. Must be relative to the web application root directory. </description> <param-name>infinispan.configurationFile</param-name> <param-value>/WEB-INF/infinispan-redis.xml</param-value> </context-param>
infinispan-redis.xml file will persist identifier-based tokens,
long-lived authorisations, the revocation journal and client registrations to
an LDAP directory. If you intend to persist that data to a different type of
database, e.g. PostgreSQL, swap the LDAP store declarations accordingly
(you can use a suitable sample Infinispan configuration as template).
Resize the Infinispan invalidation caches in
infinispan-redis.xml if needed.
You can do this by editing the size attribute of the eviction element,
where size determines the number of objects to cache in the JVM memory.
Ensure your Connect2id server nodes are provisioned with enough extra JVM heap memory to accommodate the content of all invalidation caches at full size. If you need assistance with estimating that contact Connect2id support.
Finally, let Infinispan know where the two Redis instances are located. You can
do this by directly editing the Redis host and port attributes in
infinispan-redis.xml, or by overriding the default
localhost:6379 with Java
A. Edit Redis host and port directly
For each of the following Infinispan invalidation caches set the Redis host and port to point to the Redis instance for in-memory storage of data:
<redis-server host="redis-maps.c2id.com" port="6379"/>
For each of the following Infinispan invalidation caches set the Redis host and port to point to the Redis instance for caching data:
<redis-server host="redis-caches.c2id.com" port="6379"/>
B. Provide Redis host and port via Java system properties
This the recommended method. Override the default host and port attributes with Java system properties like this:
Let redisMapHost and redisMapPort point to the Redis instance for regular in-memory storage of Connect2id server data.
Let redisCacheHost and redisCachePort point to the Redis instance for caching Connect2id server data.
This can for instance be done at JVM startup or via the
of Apache Tomcat. Example:
Finally, restart or redeploy the Connect2id server node(s) for the updated configuration to take effect.
If the Redis configuration is invalid or has conflicts the Connect2id server will abort startup and log a detailed error message.