Monitoring

1. Overview

The Connect2id server provides over 100 useful metrics and backend health checks. These are implemented with the Dropwizard Metrics library.

Access to the monitoring endpoint is protected by means of a long-lived token. The token must be passed with each HTTP request in the Authorization header:

Authorization: Bearer ztucZS1ZyFKgh0tUEruUtiSTXhnexmd6

The token can be alternatively passed as an access_token URL query parameter, for example, to enable polling of the healthcheck endpoint from a load balancer / reverse proxy:

https://demo.c2id.com/monitor/v1/healthcheck?access_token=ztucZS1ZyFKgh0tUEruUtiSTXhnexmd6

Security note: Avoid passing the token as a query parameter to prevent the credential from appearing in web server logs.

Note that the available metrics can also be reported to a remote data collection / dashboard server via JMX or the Graphite protocol.

More information can be found in the monitoring configuration reference.

2. Web API overview

3. Resources

3.1 /monitor/v1/metrics

Connect2id server metrics.

3.1.1 GET

Retrieves a set of metrics collected by the Connect2id server.

Header parameters:

• Authorization Must specify the configured bearer access token for this web API. The token can be alternatively passed as a query parameter (not recommended), see below.

Query parameters:

• [ pretty = false ] {true|false} Optional query parameter, causes pretty JSON formatting to be applied to the response entity body. Defaults to false.

• [ access_token ] {string} Alternative method to pass the configured bearer access token for this web API.

Success:

• Code: 200

• Content-Type: application/json

• Body: {object} A JSON object representing the Connect2id server metrics. Depending on the configured backend(s) also including LDAP store metrics, Redis store metrics and / or SQL store metrics.

Errors:

• 401 Unauthorized
• 500 Internal Server Error

Example request, with pretty JSON printing:

GET /monitor/v1/metrics?pretty=true HTTP/1.1
Host: c2id.com
Authorization: Bearer ztucZS1ZyFKgh0tUEruUtiSTXhnexmd6

Example response:

HTTP/1.1 200 Success
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: application/json

{
  "version" : "3.0.0",
  "gauges"  : {
    "authzSessionStore.numSessions"                    : { "value" : 0 },
    "authzStore.ldapConnector.maxAvailableConnections" : { "value" : 5 },
    "authzStore.ldapConnector.numAvailableConnections" : { "value" : 1 },
  ...
  }
  ...
}

3.2 /monitor/v1/metrics/prometheus

Connect2id server metrics, in Prometheus format.

3.2.1 GET

Retrieves a set of metrics collected by the Connect2id server, converted to Prometheus format.

Header parameters:

• Authorization Must specify the configured bearer access token for this web API. The token can be alternatively passed as a query parameter (not recommended), see below.

Query parameters:

• [ access_token ] {string} Alternative method to pass the configured bearer access token for this web API.

Success:

• Code: 200

• Content-Type: text/plain

• Body: {object} A JSON object representing the Connect2id server metrics, converted to Prometheus format. Depending on the configured backend(s) also including LDAP store metrics, Redis store metrics and / or SQL store metrics.

Errors:

• 401 Unauthorized
• 500 Internal Server Error

Example request:

GET /monitor/v1/metrics/prometheus HTTP/1.1
Host: c2id.com
Authorization: Bearer ztucZS1ZyFKgh0tUEruUtiSTXhnexmd6

Example response:

HTTP/1.1 200 Success
Content-Type: text/plain; version=0.0.4;charset=utf-8

# HELP authzSessionStore_numSessions Generated from Dropwizard metric import (metric=authzSessionStore.numSessions, type=com.nimbusds.openid.connect.provider.authz.session.InfinispanAuthorizationSessionStore$$Lambda$295/197373476)
# TYPE authzSessionStore_numSessions gauge
authzSessionStore_numSessions 0.0
# HELP authzStore_accessTokenMap_sqlStore_pool_ActiveConnections Generated from Dropwizard metric import (metric=authzStore.accessTokenMap.sqlStore.pool.ActiveConnections, type=com.zaxxer.hikari.metrics.dropwizard.CodaHaleMetricsTracker$3)
# TYPE authzStore_accessTokenMap_sqlStore_pool_ActiveConnections gauge
authzStore_accessTokenMap_sqlStore_pool_ActiveConnections 0.0
# HELP authzStore_accessTokenMap_sqlStore_pool_IdleConnections Generated from Dropwizard metric import (metric=authzStore.accessTokenMap.sqlStore.pool.IdleConnections, type=com.zaxxer.hikari.metrics.dropwizard.CodaHaleMetricsTracker$2)
# TYPE authzStore_accessTokenMap_sqlStore_pool_IdleConnections gauge
authzStore_accessTokenMap_sqlStore_pool_IdleConnections 5.0
...

3.2 /monitor/v1/ping

Endpoint for HTTP ping requests.

3.2.1 GET

Performs an HTTP ping request to the Connect2id server.

Header parameters:

• Authorization Must specify the configured bearer access token for this web API. The token can be alternatively passed as a query parameter (not recommended), see below.

Query parameters:

• [ access_token ] {string} Alternative method to pass the configured bearer access token for this web API.

Success:

• Code: 200

• Content-Type: text/plain

• Body: {string} The string “pong”.

Errors:

• 401 Unauthorized
• 500 Internal Server Error

Example ping request:

GET /monitor/v1/ping HTTP/1.1
Host: c2id.com
Authorization: Bearer ztucZS1ZyFKgh0tUEruUtiSTXhnexmd6

Example response:

HTTP/1.1 200 Success
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/plain

pong

3.3 /monitor/v1/threads

Current thread dump of the Connect2id server.

3.3.1 GET

Gets a thread dump of the Connect2id server.

Header parameters:

• Authorization Must specify the configured bearer access token for this web API. The token can be alternatively passed as a query parameter (not recommended), see below.

Query parameters:

• [ pretty = false ] {true|false} Optional query parameter, causes pretty JSON formatting to be applied to the response entity body. Defaults to false.

• [ access_token ] {string} Alternative method to pass the configured bearer access token for this web API.

Success:

• Code: 200

• Content-Type: text/plain

• Body: {text} The obtained Java thread dump.

Errors:

• 401 Unauthorized
• 500 Internal Server Error

Example request:

GET /monitor/v1/threads HTTP/1.1
Host: c2id.com
Authorization: Bearer ztucZS1ZyFKgh0tUEruUtiSTXhnexmd6

Example response:

HTTP/1.1 200 Success
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/plain

main id=1 state=RUNNABLE (running in native)
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
    at java.net.ServerSocket.implAccept(ServerSocket.java:530)
    at java.net.ServerSocket.accept(ServerSocket.java:498)
    at org.apache.catalina.core.StandardServer.await(StandardServer.java:446)
    at org.apache.catalina.startup.Catalina.await(Catalina.java:713)
    at org.apache.catalina.startup.Catalina.start(Catalina.java:659)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:351)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:485)

Reference Handler id=2 state=WAITING
    - waiting on <0x41febc07> (a java.lang.ref.Reference$Lock)
    - locked <0x41febc07> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:503)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)

...

3.4 /monitor/v1/healthcheck

Health checks built into the Connect2id server.

3.4.1 GET

Runs built-in health checks and returns the result.

• Authorization Must specify the configured bearer access token for this web API. The token can be alternatively passed as a query parameter (not recommended), see below.

Query parameters:

• [ pretty = false ] {true|false} Optional query parameter, causes pretty JSON formatting to be applied to the response entity body. Defaults to false.

• [ access_token ] {string} Alternative method to pass the configured bearer access token for this web API.

Success:

• Code: 200

• Content-Type: application/json

• Body: {object} A JSON object representing a successful health check result.

Errors:

• 401 Unauthorized
• 500 Internal Server Error – on a failed health check, with the body representing the result.

Example request, with pretty JSON printing:

GET /monitor/v1/healthcheck?pretty=true HTTP/1.1
Host: c2id.com
Authorization: Bearer ztucZS1ZyFKgh0tUEruUtiSTXhnexmd6

Example response indicating good health:

HTTP/1.1 200 Success
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: application/json

{
	"authzStore.codeMap.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"authzStore.expendedTokenMap.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"authzStore.idAccessTokenMap.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"authzStore.longLivedAuthzMap.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"authzStore.revocationJournalMap.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"clients.registrationsMap.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"clients.remoteJWKSetCache.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"clients.remoteRequestJWTClaimsCache.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"federation.registrationsMap.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"op.authSessionMap.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"op.consentSessionMap.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"op.dPoPJTICache.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"sessionStore.sessionMap.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"sessionStore.sessionMap.sqlStore.pool.ConnectivityCheck": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"sessionStore.subjectMap.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	}
}

Example response indicating the backend LDAP server is down or unavailable:

HTTP/1.1 500 Internal Server Error
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: application/json

{
	"sessionStore.sessionMap.sqlStore.pool.ConnectivityCheck": {
		"healthy": false,
		"message": "sessionStore.sessionMap.sqlStore - Connection is not available, request timed out after 30000ms (total=0, active=0, idle=0, waiting=2)",
		"error": {
			"type": "java.sql.SQLTransientConnectionException",
			"message": "sessionStore.sessionMap.sqlStore - Connection is not available, request timed out after 30000ms (total=0, active=0, idle=0, waiting=2)",
			"stack": [
				"com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:686)",
				"com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:179)",
                ...
				"java.base/java.lang.Thread.run(Thread.java:829)"
			],
			"cause": { "..." }
		},
		"duration": 30000,
		"timestamp": "2023-11-29T10:19:53.511+02:00"
	}
}

4. Representations

4.1 Core metrics set

Core Connect2id server metrics consisting of gauges, meters and timers:

• version {string} Identifies the Dropwizard Metrics API version, set to “3.0.0”.

• gauges {object} Gauges for tracking the count of current sessions and other objects:

  • server.version The Connect2id server version.
  • authzSessionStore.numSessions The number of authorisation sessions. Indicates how many end-users are currently in the process of logging in. The value may be cached.
  • sessionStore.numSessions The number of subject (end-user) sessions. Note that a user may have more than one active session. The value may be cached.
  • sessionStore.queuedNotifications The number of queued notifications (session start, session end) to subscribers. This feature is not used by the Connect2id server at present.
  • clientStore.numRegistrations The number of OpenID Connect / OAuth 2.0 client registrations. The value may be cached.
  • clientStore.numCachedRemoteJWKSets The number of cached remote (URL referenced) client JWK sets. The value may be cached.
  • numCachedRemoteRequestObjects The number of cached remote (URL referenced) OpenID request objects. The value may be cached.
  • clientStore.numCachedExpendedTokens The number of cached expended initial access tokens for OpenID Connect / OAuth 2.0 client registration. Removed in v14.0.
  • authzStore.numAuthzCodes The number of OAuth 2.0 authorisation codes waiting to be exchanged for an access token. The value may be cached.
  • authzStore.numIdAccessTokens The number of currently active identifier- based access tokens. The value may be cached.
  • authzStore.numLongLivedAuthorizations The number of (cached) long-lived authorisations. The value may be cached.
  • authzStore.numRevocationJournalEntries The number of (cached) revocation journal entries. The value may be cached.
  • authzStore.numExpendedTokenEntries The number of expended token entries. The value may be cached. Since v14.0.
  • dPoP.numCachedJTIs The number of locally cached DPoP proof JWT JTI (“jti”) entries intended to ensure the single use of received DPoP proofs at the token and UserInfo endpoints.
  • infinispan.numClusterMembers The number of Connect2id cluster members, as seen from the queried member. If the cluster has degraded into “split brain” state, for example due to networking issues, the reported number will not be consistent across the entire cluster. Check the server logs for more information. Zero (0) if clustering is disabled or “stateless”.
  • infinispan.isCoordinator Indicates if the queried cluster member is the current coordinator (true or false). false if clustering is disabled or “stateless”.

• meters {object} Event meters:

  • op.idTokenIssues Meters ID token issues.

  • authzEndpoint.successfulRequests Meters successful requests at the OAuth 2.0 authorisation endpoint.

  • authzEndpoint.invalidRequests Meters invalid OAuth 2.0 client requests at the OAuth 2.0 authorisation endpoint. Covers authorisation error responses with the “invalid_request” and other codes (save for “access_denied” metered by “authzEndpoint.failedSubjectAuthentications” and “authzEndpoint.consentDenials”) as well as non-redirecting errors.

  • authzEndpoint.failedSubjectAuthentications Meters failed subject (end-user) authentications at the OAuth 2.0 authorisation endpoint.

  • authzEndpoint.consentDenials Meters consent denials at the OAuth 2.0 authorisation endpoint.

  • directAuthzEndpoint.successfulRequests Meters successful direct authorisation requests.

  • clientStore.newRegistrations Meters new client registrations.

  • clientStore.reads Meters client registration reads.

  • clientStore.updates Meters client registration updates.

  • clientStore.deletions Meters client registration deletions.

  • authzStore.codeIssues Meters authorisation code issues.

  • authzStore.codeExchanges Meters authorisation code exchanges for an access / refresh / ID token.

  • authzStore.accessTokenIssues Meters access token issues.

  • authzStore.refreshTokenIssues Meters refresh token issues.

  • authzStore.tokenRefreshes Meters token refreshes.

  • authzStore.longLivedAuthz.newAdditions Meters addition of new long-lived (persisted) authorisations.

  • authzStore.longLivedAuthz.updates Meters updates of long-lived (persisted) authorisations.

  • authzStore.longLivedAuthz.revocations Meters revocations of long-lived (persisted) authorisations.

  • authzStore.shortLivedAuthz.newAdditions Meters addition of new short-lived (transient) authorisations.

  • authzStore.rotatedRefreshTokenReplayRevocations – Meters authorisation revocations due to rotated refresh token replay. Since v14.2.

  • authzStore.rotatedRefreshTokenAllowedReuses – Meters allowed reuses of rotated refresh tokens within the configured grace period. Since v14.2.

  • sessionStore.newSessions Meters creation of new subject (end-user) sessions.

  • sessionStore.sessionRetrievals Meters retrieval of subject (end-user) sessions.

  • sessionStore.sessionAuthUpdates Meters updates of the authentication details of subject (end-user) sessions.

  • sessionStore.sessionDataUpdates Meters updates of the data attribute of subject (end-user) sessions.

  • sessionStore.sessionClosures Meters closures (or log-outs) of subject (end-user) sessions.

  • sessionStore.sessionExpirations Meters expirations of subject (end-user) sessions.

  • sessionStore.sessionExpirationsWithoutData Meters the number of sessions expired by Infinispan where the session data was missing, resulting in a orphaned subject index entry that will need to be explicitly purged if not automatically expired by the store. See sessionStore.internal.subjectIndexPurgeInterval.

  • parEndpoint.successfulRequests Meters successful pushed authorisation request submissions.

  • parEndpoint.invalidRequests Meters failed pushed authorisation request submissions that produced an invalid_request error.

  • parEndpoint.invalidClientErrors Meters failed pushed authorisation request submissions that produced an invalid_client error.

  • parEndpoint.serverErrors Meters failed pushed authorisation request submissions that produced an HTTP 500 error.

  • tokenEndpoint.code.successfulRequests Meters successful token requests with an OAuth 2.0 authorisation code grant.

  • tokenEndpoint.code.invalidClientErrors Meters failed token requests with an OAuth 2.0 authorisation code grant that produced an invalid_client error.

  • tokenEndpoint.code.unauthorizedClientErrors Meters failed token requests with an OAuth 2.0 authorisation code grant that produced an unauthorized_client error.

  • tokenEndpoint.code.invalidGrantErrors Meters failed token requests with an OAuth 2.0 authorisation code grant that produced an invalid_grant error.

  • tokenEndpoint.code.invalidScopeErrors Meters failed token requests with an OAuth 2.0 authorisation code grant that produced an invalid_scope error.

  • tokenEndpoint.code.serverErrors Meters failed token requests with an OAuth 2.0 authorisation code grant that produced an HTTP 500 error.

  • tokenEndpoint.refreshToken.successfulRequests Meters successful token requests with an OAuth 2.0 refresh token grant.

  • tokenEndpoint.refreshToken.invalidClientErrors Meters failed token requests with an OAuth 2.0 refresh token grant that produced an invalid_client error.

  • tokenEndpoint.refreshToken.unauthorizedClientErrors Meters failed token requests with an OAuth 2.0 refresh token grant that produced an unauthorized_client error.

  • tokenEndpoint.refreshToken.invalidGrantErrors Meters failed token requests with an OAuth 2.0 refresh token grant that produced an invalid_grant error.

  • tokenEndpoint.refreshToken.invalidScopeErrors Meters failed token requests with an OAuth 2.0 refresh token grant that produced an invalid_scope error.

  • tokenEndpoint.refreshToken.serverErrors Meters failed token requests with an OAuth 2.0 refresh token grant that produced an HTTP 500 error.

  • tokenEndpoint.password.successfulRequests Meters successful token requests with an OAuth 2.0 password grant.

  • tokenEndpoint.password.invalidClientErrors Meters failed token requests with an OAuth 2.0 password grant that produced an invalid_client error.

  • tokenEndpoint.password.unauthorizedClientErrors Meters failed token requests with an OAuth 2.0 password grant that produced an unauthorized_client error.

  • tokenEndpoint.password.invalidGrantErrors Meters failed token requests with an OAuth 2.0 password grant that produced an invalid_grant error.

  • tokenEndpoint.password.invalidScopeErrors Meters failed token requests with an OAuth 2.0 password grant that produced an invalid_scope error.

  • tokenEndpoint.password.serverErrors Meters failed token requests with an OAuth 2.0 password grant that produced an HTTP 500 error.

  • tokenEndpoint.clientCredentials.successfulRequests Meters successful token requests with an OAuth 2.0 client credentials grant.

  • tokenEndpoint.clientCredentials.invalidClientErrors Meters failed token requests with an OAuth 2.0 client credentials grant that produced an invalid_client error.

  • tokenEndpoint.clientCredentials.unauthorizedClientErrors Meters failed token requests with an OAuth 2.0 client credentials grant that produced an unauthorized_client error.

  • tokenEndpoint.clientCredentials.invalidGrantErrors Meters failed token requests with an OAuth 2.0 client credentials grant that produced an invalid_grant error.

  • tokenEndpoint.clientCredentials.invalidScopeErrors Meters failed token requests with an OAuth 2.0 client credentials grant that produced an invalid_scope error.

  • tokenEndpoint.clientCredentials.serverErrors Meters failed token requests with an OAuth 2.0 client credentials grant that produced an HTTP 500 error.

  • tokenEndpoint.jwtBearer.successfulRequests Meters successful token requests with an OAuth 2.0 JWT bearer assertion grant.

  • tokenEndpoint.jwtBearer.invalidClientErrors Meters failed token requests with an OAuth 2.0 JWT bearer assertion grant that produced an invalid_client error.

  • tokenEndpoint.jwtBearer.unauthorizedClientErrors Meters failed token requests with an OAuth 2.0 JWT bearer assertion grant that produced an unauthorized_client error.

  • tokenEndpoint.jwtBearer.invalidGrantErrors Meters failed token requests with an OAuth 2.0 JWT bearer assertion that produced an invalid_grant error.

  • tokenEndpoint.jwtBearer.invalidScopeErrors Meters failed token requests with an OAuth 2.0 JWT bearer assertion that produced an invalid_scope error.

  • tokenEndpoint.jwtBearer.serverErrors Meters failed token requests with an OAuth 2.0 JWT bearer assertion that produced an HTTP 500 error.

  • tokenEndpoint.saml2Bearer.successfulRequests Meters successful token requests with a SAML 2.0 bearer assertion grant.

  • tokenEndpoint.saml2Bearer.invalidClientErrors Meters failed token requests with a 2.0 SAML 2.0 bearer assertion grant that produced an invalid_client error.

  • tokenEndpoint.saml2Bearer.unauthorizedClientErrors Meters failed token requests with a SAML 2.0 bearer assertion grant that produced an unauthorized_client error.

  • tokenEndpoint.saml2Bearer.invalidGrantErrors Meters failed token requests with a SAML 2.0 bearer assertion grant that produced an invalid_grant error.

  • tokenEndpoint.saml2Bearer.invalidScopeErrors Meters failed token requests with a SAML 2.0 bearer assertion grant that produced an invalid_scope error.

  • tokenEndpoint.saml2Bearer.serverErrors Meters failed token requests with a SAML 2.0 bearer assertion grant that produced an HTTP 500 error.

  • tokenEndpoint.tokenExchange.successfulRequests Meters successful token requests with a token exchange grant.

  • tokenEndpoint.tokenExchange.invalidClientErrors Meters failed token requests with a 2.0 token exchange grant that produced an invalid_client error.

  • tokenEndpoint.tokenExchange.unauthorizedClientErrors Meters failed token requests with a token exchange grant that produced an unauthorized_client error.

  • tokenEndpoint.tokenExchange.invalidGrantErrors Meters failed token requests with a token exchange grant that produced an invalid_grant error.

  • tokenEndpoint.tokenExchange.invalidScopeErrors Meters failed token requests with a token exchange grant that produced an invalid_scope error.

  • tokenEndpoint.tokenExchange.serverErrors Meters failed token requests with a token exchange grant that produced an HTTP 500 error.

  • tokenEndpoint.invalidRequests Meters failed token requests that produced an invalid_request error.

  • tokenEndpoint.unsupportedGrantRequests Meters failed token requests that produced an unsupportedGrant error.

  • tokenIntrospectionEndpoint.activeTokens Meters successful token introspection requests that returned a response for an active token.

  • tokenIntrospectionEndpoint.inactiveTokens Meters successful token introspection requests that returned a response for an inactive token (invalid, expired, revoked or not matching the expected audience).

  • tokenIntrospectionEndpoint.invalidRequests Meters failed token introspection requests that produced an HTTP 400 error.

  • tokenIntrospectionEndpoint.invalidClientErrors Meters failed token introspection requests that produced an HTTP 401 error.

  • tokenIntrospectionEndpoint.insufficientScopeErrors Meters failed token introspection requests that produced an HTTP 403 error.

  • tokenIntrospectionEndpoint.serverErrors Meters failed token introspection requests that produced an HTTP 500 error.

  • tokenRevocationEndpoint.successfulRequests Meters successful token revocation requests.

  • tokenRevocationEndpoint.invalidRequests Meters failed token revocation requests that produced an HTTP 400 error.

  • tokenRevocationEndpoint.invalidClientErrors Meters failed token revocation requests that produced an HTTP 401 error.

  • tokenRevocationEndpoint.invalidTokens Meters silently failed token revocation requests due to an invalid or expired token.

  • tokenRevocationEndpoint.serverErrors Meters failed token revocation requests that produced an HTTP 500 error.

  • userInfoEndpoint.successfulRequests Meters successful UserInfo requests.

  • userInfoEndpoint.invalidRequestErrors Meters failed UserInfo requests that produced an HTTP 400 error.

  • userInfoEndpoint.invalidTokenErrors Meters failed UserInfo requests that produced an HTTP 401 error.

  • userInfoEndpoint.insufficientScopeErrors Meters failed UserInfo requests that produced an HTTP 403 error.

  • userInfoEndpoint.serverErrors Meters failed UserInfo requests that produced an HTTP 500 error.

• timers {object} Timers:

  • tokenEndpoint.code.handlerTimer Times processing of OAuth 2.0 authorisation code grants at the token endpoint.
  • tokenEndpoint.refreshToken.handlerTimer Times processing of OAuth 2.0 refresh token grants at the token endpoint.
  • tokenEndpoint.password.handlerTimer Times processing of OAuth 2.0 password grants at the token endpoint.
  • tokenEndpoint.clientCredentials.handlerTimer Times processing of OAuth 2.0 client credentials grants at the token endpoint.
  • tokenEndpoint.jwtBearer.handlerTimer Times processing of OAuth 2.0 JWT bearer assertion grants at the token endpoint.
  • sessionStore.subjectIndexPurgeTask Times the execution duration of the periodic task for purging orphaned subject index entries. If the purge thread is disabled the metric will appear with no data.
  • claimsSource.retrievalTimer Times retrieval of UserInfo and other claims from the claims source backend.

Example metrics set:

{
  "version" : "3.0.0",
  "gauges"  : { "authzStore.numAuthzCodes"               : { "value": 0 },
                "authzStore.numIdAccessTokens"           : { "value": 0 },
                "authzStore.numLongLivedAuthorizations"  : { "value": 1 },
                "authzStore.numRevocationJournalEntries" : { "value": 0 },
  ...
  }
  ...
}

4.2 SQL store metrics

Additional metrics for each Infinispan map / cache configured with a persisting SQL store. Metrics from the underlying Hikari connection pool are also included. More information is available in the guide.

• gauges {object} Gauges:

  • sqlStore.pool.TotalConnections A cached gauge, refreshed on demand at 1 second resolution, indicating the total number of SQL connections in the pool. Since v15.0.

  • sqlStore.pool.IdleConnections A cached gauge, refreshed on demand at 1 second resolution, indicating the number of idle SQL connections in the pool. Since v15.0.

  • sqlStore.pool.ActiveConnections A cached gauge, refreshed on demand at 1 second resolution, indicating the number of active (in-use) SQL connections in the pool. Since v15.0.

  • sqlStore.pool.PendingConnections A cached gauge, refreshed on demand at 1 second resolution, indicating the number of threads awaiting an SQL connection from the pool. Since v15.0.

  • sqlStore.pool.MinConnections The minimum number of SQL connections in the pool. Since v15.0.

  • sqlStore.pool.MaxConnections The maximum number of SQL connections in the pool. Since v15.0.

  • [infinispan-cache-name].sqlStore.pool.TotalConnections A cached gauge, refreshed on demand at 1 second resolution, indicating the total number of SQL connections in the pool. Removed in v15.0.

  • [infinispan-cache-name].sqlStore.pool.IdleConnections A cached gauge, refreshed on demand at 1 second resolution, indicating the number of idle SQL connections in the pool. Removed in v15.0.

  • [infinispan-cache-name].sqlStore.pool.ActiveConnections A cached gauge, refreshed on demand at 1 second resolution, indicating the number of active (in-use) SQL connections in the pool. Removed in v15.0.

  • [infinispan-cache-name].sqlStore.pool.PendingConnections A cached gauge, refreshed on demand at 1 second resolution, indicating the number of threads awaiting an SQL connection from the pool. Removed in v15.0.

  • [infinispan-cache-name].sqlStore.pool.MinConnections The minimum number of SQL connections in the pool. Removed in v15.0.

  • [infinispan-cache-name].sqlStore.pool.MaxConnections The maximum number of SQL connections in the pool. Removed in v15.0.

• meters {object} Meters:

  • sqlStore.pool.ConnectionTimeoutRate Meters the rate of timeouts waiting to obtain an SQL connection from the pool. Since v15.0.

  • [infinispan-cache-name].sqlStore.pool.ConnectionTimeoutRate Meters the rate of timeouts waiting to obtain an SQL connection from the pool. Removed in v15.0.

• timers {object} Timers:

  • sqlStore.pool.Wait Times the wait time to obtain an SQL connection from the pool. Since v15.0.

  • [infinispan-cache-name].sqlStore.pool.Wait Times the wait time to obtain an SQL connection from the pool. Removed in v15.0.

  • [infinispan-cache-name].sqlStore.loadTimer Times loading of individual records from the SQL store.

  • [infinispan-cache-name].sqlStore.writeTimer Times writing of individual records to the SQL store.

  • [infinispan-cache-name].sqlStore.deleteTimer Times deleting of individual records from the SQL store.

  • [infinispan-cache-name].sqlStore.processTimer Times processing of records from the SQL store for stream operations.

  • [infinispan-cache-name].sqlStore.purgeTimer Times purging of expired records from the SQL store.

• histograms {object} Histograms:

  • sqlStore.pool.ConnectionCreation Histogram of SQL connection creation time, in milliseconds. Since v15.0.

  • sqlStore.pool.Usage Histogram of SQL connection usage time, in milliseconds. Since v15.0.

  • [infinispan-cache-name].sqlStore.pool.ConnectionCreation Histogram of SQL connection creation time, in milliseconds. Removed in v15.0.

  • [infinispan-cache-name].sqlStore.pool.Usage Histogram of SQL connection usage time, in milliseconds. Removed in v15.0.

4.3 Redis store metrics

Additional metrics for each Infinispan map / cache configured with a Redis store. The metrics are sourced from the underlying Jedis connection pool:

• gauges {object} Gauges:

  • [infinispan-cache-name].redisStore.numActiveConnections The number of active Redis client connections in the pool.

  • [infinispan-cache-name].redisStore.numIdleConnections The number of idle Redis client connections in the pool.

  • [infinispan-cache-name].redisStore.numWaitingForConnection The number of threads waiting for a Redis client connection.

  • [infinispan-cache-name].redisStore.meanWaitingTimeForConnectionMs The mean time waiting to borrow a Redis client connection from the pool, in milliseconds.

  • [infinispan-cache-name].redisStore.maxWaitingTimeForConnectionMs The maximum time waiting to borrow a Redis client connection from the pool, in milliseconds.

4.4 DynamoDB store metrics

Additional metrics for each Infinispan map / cache configured with a DynamoDB store.

• timers {object} Timers:

  • [infinispan-cache-name].dynamoDB.getTimer Times DynamoDB get operations.

  • [infinispan-cache-name].dynamoDB.putTimer Times DynamoDB put operations.

  • [infinispan-cache-name].dynamoDB.deleteTimer Times DynamoDB delete operations.

  • [infinispan-cache-name].dynamoDB.processTimer Times DynamoDB process (scan) operations.

  • [infinispan-cache-name].dynamoDB.purgeTimer Times DynamoDB purge expired entries operations.

  • [infinispan-cache-name].dynamoDB.invalidItemHmacCounter Counts the number of retrieved DynamoDB items which failed the HMAC SHA-256 check (if enabled).

4.5 Gauge

A gauge from a metrics set.

• value {integer} The current gauge value.

Example gauge:

{
  "value" : 950140456
}

4.6 Meter

A meter from a metrics set.

• count {integer} The number of metered events.
• m1_rate {float} The one-minute exponentially-weighted moving average rate at which events have occurred since start up. This rate has the same exponential decay factor as the one-minute load average in the top Unix command.
• m5_rate {float} The five-minute exponentially-weighted moving average rate at which events have occurred since start up. This rate has the same exponential decay factor as the five-minute load average in the top Unix command.
• m15_rate {float} The fifteen-minute exponentially-weighted moving average rate at which events have occurred since start up. This rate has the same exponential decay factor as the fifteen-minute load average in the top Unix command.
• mean_rate {float} The mean rate at which events have occurred since start up.
• units = “events/second” The unit measure.

Example meter:

{
  "count"     : 2,
  "m15_rate"  : 0.00220381749348163,
  "m1_rate"   : 0.029527305437977176,
  "m5_rate"   : 0.006503044431934881,
  "mean_rate" : 4.2031348626575445E-4,
  "units"     : "events/second"
}

4.7 Timer

A timer from a metrics set.

• count {integer} The number of timed events.
• min {float} The lowest duration value.
• mean {float} The arithmetic mean duration value.
• max {float} The highest duration values.
• p50 {float} The value at the 50th percentile in the distribution.
• p75 {float} The value at the 75th percentile in the distribution.
• p95 {float} The value at the 95th percentile in the distribution.
• p98 {float} The value at the 98th percentile in the distribution.
• p99 {float} The value at the 99th percentile in the distribution.
• p999 {float} The value at the 99.9th percentile in the distribution.
• stddev {float} The standard deviation of the sampled duration values.
• m1_rate {float} The one-minute exponentially-weighted moving average rate at which events have occurred since start up. This rate has the same exponential decay factor as the one-minute load average in the top Unix command.
• m5_rate {float} The five-minute exponentially-weighted moving average rate at which events have occurred since start up. This rate has the same exponential decay factor as the five-minute load average in the top Unix command.
• m15_rate {float} The fifteen-minute exponentially-weighted moving average rate at which events have occurred since start up. This rate has the same exponential decay factor as the fifteen-minute load average in the top Unix command.
• mean_rate {float} The mean rate at which events have occurred since start up.
• duration_units = “seconds” The duration unit measure.
• rate_units = “calls/second” The rate unit measure.

Example timer:

{
  "count"          : 6,
  "max"            : 0.01028953,
  "mean"           : 6.647419832286938E-4,
  "min"            : 6.56503E-4,
  "p50"            : 6.647430000000001E-4,
  "p75"            : 6.647430000000001E-4,
  "p95"            : 6.647430000000001E-4,
  "p98"            : 6.647430000000001E-4,
  "p99"            : 6.647430000000001E-4,
  "p999"           : 6.647430000000001E-4,
  "stddev"         : 9.152683616619665E-8,
  "m15_rate"       : 0.0016652947966130432,
  "m1_rate"        : 1.9309212342695557E-4,
  "m5_rate"        : 0.0015804743529140488,
  "mean_rate"      : 0.0015484305097827385,
  "duration_units" : "seconds",
  "rate_units"     : "calls/second"
}

4.8 Histograms

A histogram from a metrics set.

• count {integer} The number of timed events.
• min {float} The lowest duration value.
• mean {float} The arithmetic mean duration value.
• max {float} The highest duration values.
• p50 {float} The value at the 50th percentile in the distribution.
• p75 {float} The value at the 75th percentile in the distribution.
• p95 {float} The value at the 95th percentile in the distribution.
• p98 {float} The value at the 98th percentile in the distribution.
• p99 {float} The value at the 99th percentile in the distribution.
• p999 {float} The value at the 99.9th percentile in the distribution.
• stddev {float} The standard deviation of the sampled duration values.

Example histogram:

{
  "count": 4,
  "max": 4,
  "mean": 1.4949055507513855,
  "min": 0,
  "p50": 1,
  "p75": 4,
  "p95": 4,
  "p98": 4,
  "p99": 4,
  "p999": 4,
  "stddev": 1.5881400253003708
}

4.9 Health check results

Connect2id server health check results.

• [infinispan-cache-name].availability {object} A JSON object representing the result of checking the availability of the given Infinispan map / cache. Reports degraded map / cache states due to a “spit brain” condition (in a replicated or distributed cluster).

  • healthy {true|false} Indicates the health.

  • [ message ] {string} Diagnostic message if healthy is false.

  • duration {number} Time it took to perform the check, in milliseconds.

  • timestamp {string} Time when the check was completed.

• [infinispan-cache-name].sqlStore.pool.ConnectivityCheck {object} A JSON object representing the health of the connection pool when an SQL store is configured. The check obtains a JDBC connection from the pool and immediately returns it.

  • healthy {true|false} Indicates the health.

  • [ message ] {string} Diagnostic message if healthy is false.

  • duration {number} Time it took to perform the check, in milliseconds.

  • timestamp {string} Time when the check was completed.

Example results indicating good health:

{
	"authzStore.codeMap.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"authzStore.expendedTokenMap.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"authzStore.idAccessTokenMap.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"authzStore.longLivedAuthzMap.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"authzStore.revocationJournalMap.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"clients.registrationsMap.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"clients.remoteJWKSetCache.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"clients.remoteRequestJWTClaimsCache.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"federation.registrationsMap.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"op.authSessionMap.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"op.consentSessionMap.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"op.dPoPJTICache.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"sessionStore.sessionMap.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"sessionStore.sessionMap.sqlStore.pool.ConnectivityCheck": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	},
	"sessionStore.subjectMap.availability": {
		"healthy": true,
		"duration": 0,
		"timestamp": "2023-11-29T09:54:09.693+02:00"
	}
}

Example result indicating a failed Hikari connection pool health check:

{
	"sessionStore.sessionMap.sqlStore.pool.ConnectivityCheck": {
		"healthy": false,
		"message": "sessionStore.sessionMap.sqlStore - Connection is not available, request timed out after 30000ms (total=0, active=0, idle=0, waiting=2)",
		"error": {
			"type": "java.sql.SQLTransientConnectionException",
			"message": "sessionStore.sessionMap.sqlStore - Connection is not available, request timed out after 30000ms (total=0, active=0, idle=0, waiting=2)",
			"stack": [
				"com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:686)",
				"com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:179)",
                ...
				"java.base/java.lang.Thread.run(Thread.java:829)"
			],
			"cause": { "..." }
		},
		"duration": 30000,
		"timestamp": "2023-11-29T10:19:53.511+02:00"
	}
}

5. Errors

401 Unauthorized

The request was denied due to an invalid or missing bearer access token.

Example:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer
Content-Type: application/json

{
  "error"             : "missing_token",
  "error_description" : "Unauthorized: Missing Bearer access token"
}

500 Internal Server Error

An internal server error has occurred. Check the Connect2id server logs for details.

Example:

HTTP/1.1 500 Internal Server Error