Logging

The Connect2id server uses the popular Log4j 2 framework for logging. The messages can be output to a variety of destinations, such as a local text file, the console, a database or a service. Reconfiguration on the fly is possible.

The Connect2id server instructs Log4j to gets its configuration from the following location, by setting the log4jConfiguration context parameter in web.xml:

WEB-INF/log4j.xml

Shipped configuration

The Connect2id server ships with a Log4j configuration for writing the messages to tomcat/logs/c2id-server.log. The output is rolled to a new file each day or when the current log file size reaches 250 megabytes in size.

To switch logging to the standard output set the log4j.loggers.root.appender Java system property to console.

The severity level is info. To switch to a different level set the log4j.level Java system property.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="error" name="C2idServer">
    <Appenders>
        <!-- Log file location uses Tomcat system variable, change for other web servers -->
        <RollingFile name="rolling-file"
                     fileName="${sys:catalina.home}/logs/c2id-server.log"
                     filePattern="${sys:catalina.home}/logs/c2id-server-%d{yyyy-MM-dd}-%i.log.gz">
            <PatternLayout pattern="%d{ISO8601} %p %t %c{1} - %m%n"/>
            <Policies>
                <TimeBasedTriggeringPolicy />
                <SizeBasedTriggeringPolicy size="250 MB"/>
            </Policies>
        </RollingFile>
        <Console name="console" target="SYSTEM_OUT" follow="true">
            <PatternLayout pattern="%d{ISO8601} %p %t %c{1} - %m%n"/>
        </Console>
    </Appenders>
    <Loggers>
        <Root level="${sys:log4j.level:-info}">
            <AppenderRef ref="${sys:log4j.loggers.root.appender:-rolling-file}"/>
        </Root>
    </Loggers>
</Configuration>

Modifying the logging configuration

Refer to the Log4j manual for instructions how to edit the configuration file.

The supported message formats, such as text pattern or JSON, are described in the layouts section.

The supported output destinations, such as file or database, are described in the appenders section.

To write the logs to Logstash

Logstash is a popular server for central aggregation and analysis of logs.

There are two ways to pipe Connect2id server logs to Logstash:

1. Via the included log4j2-jsonevent-layout plugin (available since Connect2id server v6.18). Its primary use is to produce JSON formatted log entries, but it also can be configured to pipe output to Logstash. Check out the plugin documentation for how to set this up.

2. Install a Logstash plugin to convert incoming log messages from Log4j to the expected format. One such plugin is logstash-log4j2 from Jurriaan Mous.

We looked at both approaches and the second one seems the best with the currently available plugins. For that we recommend the logstash-log4j2 plugin from Jurriaan Mous.

Setting up Log4j itself is just a matter of specifying a remote appender with the Logstash server host name / IP address and port number:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="error" name="C2idServer">
    <Appenders>
        <Socket name="Logstash" host="my.logstash.server.net" port="9201">
              <JSONLog4j2Layout singleLine="true"/>
        </Socket>
    </Appenders>
    <Loggers>
        <Root level="info">
            <AppenderRef ref="Logstash"/>
        </Root>
    </Loggers>
</Configuration>

How to debug the Log4j initialisation in Apache Tomcat

Add the log4j2.debug=true Java system property to the Tomcat startup script:

export CATALINA_OPTS="$CATALINA_OPTS -Dlog4j2.debug=true"

Logger names

The messages are written under topics, using separate loggers, to ease scanning of events per server endpoint as well as system-wide events.

• MAIN – The main logger. Records general configuration, startup, shutdown and system messages. Note that Infinispan, JGroups and other subsystems typically output messages under their own loggers.

• DISCOVERY – Discovery and OpenID provider / OAuth2.0 authoristion server metadata endpoint messages.

• CLIENT-REG – Client registration endpoint messages.

• FED-REG – Federation Client registration endpoint messages.

• PAR – Pushed authorisation request endpoint messages.

• AUTHZ-SESSION – Authorisation session endpoint messages (for the login and consent handler).

• DIRECT-AUTHZ – Direct authorisation endpoint messages.

• AUTHZ-STORE – Authorisation store messages.

• SESSION-STORE – Subject session store messages.

• TOKEN – Token endpoint messages.

• TOKEN-INTROSPECT – OAuth 2.0 token introspection endpoint messages.

• TOKEN-REVOKE – Token revocation endpoint messages.

• USERINFO – UserInfo endpoint messages.

• LOGOUT – Logout endpoint messages.

• FC-LOGOUT – Front-channel logout messages.

• BC-LOGOUT – Back-channel logout messages.

• CHECK-SESSION – Check session messages.

Don’t forget to watch the console and the catch-all log

Don’t forget to watch the console output of your web server. If you’re using Apache Tomcat its default configuration writes console messages to logs/catalina.out.

The console output may contain the following messages:

• Uncaught Java exceptions.
• Certain messages from service threads launched by Connect2id server components such as Infinispan cluster events and JGroups networking events.
• Thread dumps, if requested via system signal.

Sensitive credentials

Sensitive credentials, such as tokens, secrets and API keys, don’t get logged at INFO or coarser levels by the Connect2id server.

When you log at DEBUG or finer levels, take all necessary measures to secure the logs from unauthorised access.

Note that the servlet container (Apache Tomcat) at startup may log credentials which are passed via command line arguments and the CATALINA_OPTS environment variable. To prevent that edit tomcat/conf/server.xml and update the VersionLoggerListener configuration like this:

<Listener className="org.apache.catalina.startup.VersionLoggerListener" logArgs="false"/>