Connect2id server 10.3 introduces new plugin interface for handling software statements

2020-11-25

This new release of the Connect2id server ships two new features and one bug fix.

SPI for handling software statements

One way of managing and securing client registration is with software statements, a signed JWT embedded into the client metadata which locks selected parameters, for example the client name, logo and other configurations, and further allows the request to be authenticated. Software statements appear in OpenBanking with dynamic client registration, where client application parameters are kept and attested by a central directory.

Previously, to handle such statements or requests secured in some other way, a custom web service had to be layered on top of the clients API.

This new release introduces a plugin interface (SPI) for intercepting incoming HTTP requests, and potentially validating and rewriting them to implement software statements or some other security profile for client registration.

Sample client registration request with an embedded software statement:

POST /clients HTTP/1.1
Content-Type: application/json
Accept: application/json

{
  "redirect_uris"      : [ "https://client.example.org/cb" ],
  "software_statement" : "eyJhbGciOiJSUzI1NiJ9.
eyJzb2Z0d2FyZV9pZCI6IjROUkIxLTBYWkFCWkk5RTYtNVNNM1IiLCJjbGll
bnRfbmFtZSI6IkV4YW1wbGUgU3RhdGVtZW50LWJhc2VkIENsaWVudCIsImNs
aWVudF91cmkiOiJodHRwczovL2NsaWVudC5leGFtcGxlLm5ldC8ifQ.
GHfL4QNIrQwL18BSRdE595T9jbzqa06R9BT8w409x9oIcKaZo_mt15riEXHa
zdISUvDIZhtiyNrSHQ8K4TvqWxH6uJgcmoodZdPwmWRIEYbQDLqPNxREtYn0
5X3AR7ia4FRjQ2ojZjk5fJqJdQ-JcfxyhK-P8BAWBd6I2LLA77IG32xtbhxY
fHX7VhuU5ProJO8uvu3Ayv4XRhLZJY4yKfmyjiiKiPNe-Ia4SMy_d_QSWxsk
U5XIQl5Sa2YRPMbDRXttm2TfnZM1xx70DoYi8g6czz-CPGRi4SW_S2RKHIJf
IjoI3zTJ0Y2oe0_EJAiXbL6OyF9S5tKxDXV8JIndSA"
}

Enabled PKCE methods configuration

An op.authz.allowedPKCE configuration property is added setting the enabled PKCE methods.

To limit the allowed code challenge methods to S256 only:

op.authz.allowedPKCE=S256

The allowed PKCE methods will be advertised in the server metadata under code_challenge_methods_supported.

Resolved issues

Fixes a DB schema schema bug affecting 10.x deployments with MySQL 5.7.x. See the release notes for details.

Download

Standard Connect2id server edition

Apache Tomcat package with Connect2id server 10.3: Connect2id-server.zip

SHA-256: a740bc2558becb304e51f171a5a579fa46b6ced54b5b192bd108eed289fd3a4f

Connect2id server 10.3 WAR package: c2id.war

SHA-256: bc3bb2b6424d90d13d342e6a5a77601b62220197ecc93ffb8058bceb75fdf79e

Multi-tenant edition

Apache Tomcat package with Connect2id server 10.3: Connect2id-server-mt.zip

SHA-256: 0c2fe2f57d3cd56ae7f96883f99085adcd3842b21dbae8f05b6e9e8a8de559c6

Connect2id server 10.3 WAR package: c2id-multi-tenant.war

SHA-256: fa8c7a638aa24ad5f0ca1366148de3b40703a33364860b7de29525122ceded96

Questions?

Contact Connect2id support.

Release notes

10.3 (2020-11-25)

Configuration

  • /WEB-INF/oidcProvider.properties

    • op.authz.allowedPKCE -- New optional configuration property specifying the allowed PKCE (RFC 7636) code challenge methods which OAuth 2.0 clients may use at the authorisation endpoint, as comma and / or space separated list. The default allowed code challenge methods are "plain" and "S256" (all RFC 7636 methods).

      Authorisation requests which use a code challenge method that isn't allowed by the configuration will be rejected with an invalid_request error.

      The allowed code challenge methods will be advertised in the OpenID provider / OAuth 2.0 authorisation server "code_challenge_methods_supported" metadata field.

SPI

  • Upgrades the Connect2id server SDK to com.nimbusds:c2id-server-sdk:4.23

  • com.nimbusds.openid.connect.provider.spi.reg.RegistrationInterceptor

    • New SPI for intercepting and optionally modifying HTTP POST, GET, PUT and DELETE requests at the client registration endpoint. Can be used to process software statements (RFC 7591, section 2.3) and signed (JWT) registration requests (such as those in Open Banking Dynamic Client Registration).

Resolved issues

  • Fixes issue in the MySQL schema for the federation_clients table where MySQL 5.7.x doesn't accept a second TIMESTAMP column with NON NULL declaration. MySQL 8.x is not affected (issue server/614).

Dependency changes

  • Upgrades to com.nimbusds:c2id-server-sdk:4.23

  • Upgrades to com.nimbusds:oauth2-oidc-sdk:8.27

  • Upgrades to com.thetransactioncompany:java-property-utils:1.16

Connect2id server 10.2

2020-11-13

The 10.2 release of the Connect2id server brings two security updates and an update to the logout API.

Preventing mix-up attacks on plain OAuth 2.0 clients

The OAuth 2.0 security topics describes the mix-up attack. An OAuth 2.0 client which interacts with two or-more authorisation servers, where the client has used dynamic client registration or one of the servers has been compromised, can become susceptible to the mix-up attack.

The universal countermeasure is to let the authorisation response include the issuer identifier, as an extra iss parameter:

HTTP/1.1 302 Found
Location: https://client.example/cb?
 code=x1848ZT64p4IirMPT0R-X3141MFPTuBX-VFL_cvaplMH58
 &state=ZWVlNDBlYzA1NjdkMDNhYjg3ZjUxZjAyNGQzMTM2NzI
 &iss=https%3A%2F%2Fc2id.com

The OAuth client, when receiving an authorisation success or error response, needs to check that the returned iss parameter matches the expected issuer URL of the authorisation server when the request was started. This measure effectively enables clients to detect mix-up attacks and prevent impersonation.

Note that because OpenID relying parties (RPs) receive an ID token, which states the issuer URL of the server, and RPs are required to validate the token iss (along with other claims in the ID token), they are not susceptible to the mix-up attack.

The iss parameter is being standardised in a new spec at the OAuth WG. The spec also defines a new server metadata parameter, authorization_response_iss_parameter_supported, to let clients find out if the iss response parameter is provided and hence they can start checking for it.

{
  "issuer"                                         : "https://c2id.com",
  "jwks_uri"                                       : "https://c2id.com/jwks.json",
  ...
  "authorization_response_iss_parameter_supported" : true
}

Starting with this release the Connect2id server will include the iss in all authorisation responses.

Tightened URI scheme security

In order to allow callbacks into native clients OAuth 2.0 and OpenID Connect allow clients to be registered with redirection URIs with schemes other than the https scheme.

An article published this week by the security researcher Lauritz Holtmann discusses a class of XSS attacks which may get launched by dynamically registered clients which have set the redirect_uri scheme to javascript, data or vbscript. Firefox was one of the first browsers that started blocking top-level data URIs in 2017, but other and older browsers can still be susceptible to this attack.

To close this loophole the Connect2id server will now prohibit registration of authorisation and post-logout redirection URIs with the schemes javascript, data and vbscript.

The Connect2id server will also block registration of client metadata parameters intended to render links in the browser with URI schemes other than https and http, such as the optional client_uri (client homepage), policy_uri (client privacy policy) and tos_uri (client terms and conditions).

Connect2id server deployments which allow open / dynamic registration are advised to upgrade.

Logout web API update

The logout session web API of the Connect2id server receives an update which fixes a shortcoming in the original design of the API.

A relying party which makes a logout request has the option to specify a post_logout_redirect_uri where the browser should be sent after the logout request is processed. This post-logout redirection URI must be registered in advance, similar to the possible redirect_uris in authorisation requests.

https://c2id.com/logout?
 id_token_hint=eyJhbGciOiJSUzI1NiIsImtpZCI6IjFlOWdkazcifQ.ewogImlzcyI6...
 &post_logout_redirect_uri=https%3A%2F%2Fclient.example.com%2Flogout-page
 &state=UVaefaeroth3dah1

When the end-user is not logged in or their session has expired the original logout API of the Connect2id server will return an invalid_session error to the UI. In cases when the RP has requested a post-logout redirection the UI would not be able to trigger the post-logout redirection, leaving the end-user at the IdP logout page.

The logout API is now updated to no longer return invalid_session errors to the UI, thus allowing a post-logout redirection also in those cases when there is no active session. Note that the post-logout redirection requires an ID token hint to be included in the request, and the Connect2id server will check it to ensure the URI is registered for the RP which received it.

Download

Standard Connect2id server edition

Apache Tomcat package with Connect2id server 10.2: Connect2id-server.zip

SHA-256: 13197322db606ec1a030b7d2a792901fbea0f33987b2503332603440b4089d86

Connect2id server 10.2 WAR package: c2id.war

SHA-256: ed2a5c28636a6d56029529bd77bff2cdef30c0464e3c344f59d03343a5549720

Multi-tenant edition

Apache Tomcat package with Connect2id server 10.2: Connect2id-server-mt.zip

SHA-256: b0c98a1094c1d3b829afa39f2375ce66ef10e37183d9e05d82c30471d94e37da

Connect2id server 10.2 WAR package: c2id-multi-tenant.war

SHA-256: 74e338dc23c3a4f74794f8edd34c98c1855181fd8ecdbb5079ca967ba733914d

Questions?

Contact Connect2id support.

Release notes

10.2 (2020-11-13)

Summary

  • Includes the Connect2id server issuer URL as "iss" parameter in authorisation responses. Intended for OAuth 2.0 clients using more than one authorisation server to prevent a class of attacks called "mix-up" attacks. Clients interacting with a single OAuth 2.0 server and OpenID relying parties which receive an ID token (and accordingly validate the "iss" claim) are not affected. See OAuth 2.0 Authorization Server Issuer Identifier in Authorization Response (draft-meyerzuselhausen-oauth-iss-auth-resp-01) for further details.

  • Tightens the URI scheme policy for OAuth 2.0 client metadata to prevent potential attacks using malicious "javascript", "data" and "vbscript" schemes in old and insufficiently protected web browsers. Deployments which utilise open client registration are advised to upgrade. See the article at https://security.lauritz-holtmann.de/post/sso-security-redirect-uri/ for further details.

  • Updates the logout web API to facilitate post-logout redirections in cases when the end-user session has expired.

Configuration

  • /WEB-INF/oidcProvider.properties

    • op.policy, op.tos, op.serviceDocs -- If the optional policy, terms of service and service documentation links are specified by absolute URL, the scheme must be either "https" or "http".

Web API

  • /.well-known/openid-configuration, /.well-known/oauth-authorization-server

    • Adds the "authorization_response_iss_parameter_supported" metadata parameter (the value is set to true), as specified in OAuth 2.0 Authorization Server Issuer Identifier in Authorization Response (draft-meyerzuselhausen-oauth-iss-auth-resp-01).

  • /authz-sessions/rest/v3/

    • Includes an "iss" (issuer) parameter in OAuth 2.0 authorisation success and error responses, as specified in OAuth 2.0 Authorization Server Issuer Identifier in Authorization Response (draft-meyerzuselhausen-oauth-iss-auth-resp-01). Intended for OAuth 2.0 clients using more than one authorisation server to prevent a class of attacks called "mix-up" attacks. Clients interacting with a single OAuth 2.0 server and OpenID relying parties which receive an ID token (and accordingly validate the "iss" claim) are not affected.

  • /clients

    • Registration of OAuth 2.0 clients and OpenID relying parties with "redirect_uris" or "post_logout_redirect_uris" containing the custom URI schemes "javascript", "data" and "vbscript" is prohibited for security reasons. Native applications can continue registering redirection URIs with custom URI schemes which don't match the prohibited.

    • Registration of OAuth 2.0 clients with "client_uri", "policy_uri" or "tos_uri" with URI schemes other than "https" and "http" is prohibited for security reasons.

    • Registration of OpenID relying parties will no longer permit custom URI schemes for "frontchannel_logout_uri" and "backchannel_logout_uri". The only permitted URI scheme is "https".

  • /logout-sessions/rest/v1/

    • Updates the logout session web API to not return an "invalid_session" error when the submitted end-user session SID (typically stored in a browser cookie at the IdP domain) is invalid or expired. OpenID relying parties (RP) making a logout request with a valid "post_logout_redirect_uri" will thus be able to complete the redirection, regardless of the end-user session state at the IdP.

      Following this change "invalid_session" errors will no longer be returned by the logout session web API.

Resolved issues

  • Prohibits registration of OAuth 2.0 clients and OpenID relying parties with "redirect_uris" or "post_logout_redirect_uris" with the custom URI schemes "javascript", "data" and "vbscript" for security reasons. Also requires browser rendered URIs derived from client and server metadata, such as "client_uri", "policy_uri" and "tos_uri" to have an "https" or "http" URI. See https://security.lauritz-holtmann.de/post/sso-security-redirect-uri/ for more information (issue server/611).

  • Prohibits registration of OpenID relying parties with custom "frontchannel_logout_uri" and "frontchannel_logout_uri" URI schemes, thus making "https" the only allowed URI scheme (issue server/612).

  • Improves INFO logging at the logout session endpoint (issue server/610).

Dependency changes

  • Upgrades to com.nimbusds:oauth2-oidc-sdk:8.26

Connect2id server 10.1 with OpenID Connect federation and Identity Assurance / eKYC updates

2020-10-28

The second OpenID Connect Federation 1.0 interop, which took place in September, tested automatic federation clients, which are now fully supported by the Connect2id server. To find out what this new OpenID Connect extension is about and how it enables identity federations to be operated at virtually any scale, in hierarchical and mesh topologies of trust, revisit the Connect2id server 10.0 release announcement.

The second topic of this release is the Identity Assurance / eKYC extension. The Connect2id server will now accept a special case of requests for verified claims, those that involve multiple or different verifications. To simplify the processing of such requests we've been contemplating some changes to the authZ session API, in one of the next Connect2id server releases (feel free to chime in!). We'll continue updating the guide for IdA / eKYC providers with new examples and tips.

The underlying OAuth 2.0 / OpenID Connect SDK (v8.23) was also revised, for a nicer, immutable OO development when dealing with regular as well as verified claims.

For a full description of the new features, changes and bug fixes in Connect2id server 10.1 check the release notes below.

Download

Standard Connect2id server edition

Apache Tomcat package with Connect2id server 10.1: Connect2id-server.zip

SHA-256: cf8f58e3c1525a2aebd399090d09ded28f4f76f48586f3c84f70aa1bdaa7716d

Connect2id server 10.1 WAR package: c2id.war

SHA-256: d2a440e956959d1695dc3009c3b6b024f9291c773657d48a025ebba15e971a43

Multi-tenant edition

Apache Tomcat package with Connect2id server 10.1: Connect2id-server-mt.zip

SHA-256: 333cd88a1b96d49b0c1b651fd43181a35fadec38a72e27bb178234fdc1ff4aa3

Connect2id server 10.1 WAR package: c2id-multi-tenant.war

SHA-256: bf64cfdc26e040737017298faab5b6c3ebaf36253ba87a8bcdff12265cceed0e

Questions?

Contact Connect2id support.

Release notes

10.1 (2020-10-29)

Summary

  • Adds support for OpenID Connect Federation 1.0 (draft 12) automatic clients. Automatic clients don't register explicitly with the Connect2id server, instead they proceed directly to the authorisation endpoint, where they submit a signed request object (JWT) containing additional federation- specific claims, thus letting the request object also serve as an implicit authenticated registration request. The Connect2id server will perform the regular federation trust chain resolution and retrieve the client's metadata from its federation entity statement published at a well-known endpoint. The client registration will expire according to a configured policy - a set lifetime or the trust chain lifetime.

    See https://openid.net/specs/openid-connect-federation-1_0-12.html.

  • Updates OpenID Connect for Identity Assurance 1.0 (draft 11) support to accept "claims" authentication request parameters where the "verified_claims" element is a JSON array. This is intended to enable requests for claims sets with different verification requirements (e.g. different trust frameworks). Previously such requests will result in a invalid_request error at the authorisation endpoint.

    Example "claims" request for two verifications, using the "eidas_ial_substantial" and the "de_aml" trust frameworks:

    {
"userinfo": {
  "verified_claims": [
    {
      "verification": {
        "trust_framework": {
          "value": "eidas_ial_substantial"
        }
      },
      "claims": {
        "given_name": null,
        "family_name": null
      }
    },
    {
      "verification": {
        "trust_framework": {
          "value": "de_aml"
        }
      },
      "claims": {
        "birthdate": null
      }
    }
  ]
}
}

    The consent prompt in the authorisation session API will list the requested verified claims across all elements in the "verified_claims" JSON array.

    Note that "claims" -> "verification" will only present the first requested "verification" element if multiple are found. Use the "claims" -> "raw_request" to obtain all verifications. A future version of the Connect2id server will update the consent prompt to simplify handling of verified claims requests with multiple verifications.

    See https://openid.net/specs/openid-connect-4-identity-assurance-1_0-11.html, section 6.3.3.

  • Connect2id server deployments with an AWS DynamoDB backend can be configured to include a SHA-256 based message authentication code (HMAC) in each stored item to guarantee its integrity and authenticity while stored in the database.

Configuration

  • /WEB-INF/oidcProvider.properties

    • op.federation.clientRegistrationTypes -- The configuration property for the supported OpenID Connect Federation 1.0 client registration types now allows for automatic as well as explicit registration.

    • op.federation.autoClientAuthMethods.ar -- New configuration property specifying the supported methods for authenticating implicit registration requests of OpenID Connect Federation 1.0 automatic clients at the OAuth 2.0 authorisation endpoint. The only currently supported method is "request_object".

    • op.federation.autoClientLifetime -- New configuration property for the lifetime of registered OpenID Connect Federation 1.0 automatic clients, in seconds. If zero or negative the lifetime will be determined by the trust chain expiration time. When explicitly set must not be shorter than 5 minutes (300 seconds) to allow sufficient time for the completion of a single OAuth 2.0 flow with an authorisation, token and UserInfo request by the registered relying party. The default lifetime is one hour (3600 seconds).

  • /WEB-INF/infinispan-*-dynamodb.xml

    • Updates the DynamoDB store schema to v1.8 and adds the option to secure the integrity and authenticity of the stored DynamoDB items with a SHA-256 based Message Authentication Code (HMAC), appended to each item in a "_hmac#256" binary attribute. If a retrieved DynamoDB item is missing the "_hmac#256" attribute or its verification fails the Connect2id server will produce an HTTP 500 Internal Server Error, log the error with a "DS0131" code and increment the ".dynamoDB.invalidItemHmacCounter" metric.

      To enable this HMAC protection the Connect2id server must be configured with a secret 256 bit key (as BASE-64 encoded string) in the "dynamoDB.hmacSHA256Key" Java system property.

      The HMAC protection must be enabled for newly provisioned or empty DynamoDB tables. Enabling the protection for DynamoDB tables with prior existing items will cause HMAC validation errors on their retrieval.

  • /WEB-INF/infinispan-stateless-dynamodb.xml

    • DynamoDB consistent reads for "sessionStore.sessionMap" can be enabled by setting the "dynamodb.consistentReads.sessionStore.sessionMap" Java system property to true.

Web API

  • /authz-sessions/rest/v3/

    • The consent prompt "claims" parameter adds support for verified claims requests (OpenID Connect Identity Assurance 1.0) with multiple verifications.

      The "claims" -> "new" -> "essential", "voluntary" arrays will list the requested verified claims (prefixed with "verified:") across all "verified_claims" elements (if multiple are found).

      The "claims" -> "verification" -> "id_token", "userinfo" objects will include the first found verification element (if multiple are found), the full list of verification objects can be obtained by parsing the "claims" -> "raw_request" parameter (requires op.authz.includeRawClaimsRequestInPrompt=false).

  • /monitor/v1/metrics

    • Adds new ".dynamoDB.invalidItemHmacCounter" metric of type counter for Connect2id server deployments with a DynamoDB database. Counts the number of retrieved DynamoDB items which failed the HMAC SHA-256 check (if enabled).

SPI

  • PasswordGrantHandler, JWTGrantHandler, SAML2GrantHandler

    • Consented claims which name is prefixed with "id_token:" will be included in the ID token instead of via the default method, the UserInfo endpoint. This feature is adopted from the authorisation session web API.

  • Included OpenID Connect HTTP claims source

    • Updates the included AdvancedClaimsSource SPI implementation for sourcing OpenID claims from an HTTP endpoint to version 2.1 (com.nimbusds:oidc-claims-source-http:2.1). The JSON object representing the request will now include a "claims_transport" parameter hinting how the requested claims are going to be transported to the relying party: "userinfo" for the UserInfo endpoint, "id_token" for the ID token, omitted if not applicable (for claims requested by an internal plugin, such as an access token codec).

Resolved issues

  • OpenID claims requests with JSON array of verified_claims must not fail (issue server/605).

  • Fixes NPE in OpenID Connect Federation 1.0 client registration error processing logic (issue server/585).

  • Front-channel logout requests must not include the issuer identifier (iss) as query parameter when a session identifier (sid) isn't included (issue server/595).

  • Updates the OAuth 2.0 authorisation endpoint to reject signed (JWS) request objects (JARs) where the optional JWT "sub" (subject) claim is set to the client_id value, see the latest cross-JWT confusion security recommendation in https://tools.ietf.org/html/draft-ietf-oauth-jwsreq-29#section-10.8 (issue server/588).

  • Fixes the reported "request" (JAR) parameter name in illegal_request error messages from the OAuth 2.0 authorisation endpoint (issue server/589).

  • Ignore unknown key types in a JWK set, when parsing a client JWK set and in other cases (issue nimbus-jose-jwt/377).

  • Logs at INFO level the enabled / disabled status of every loaded claims source at Connect2id server startup under OP7107 (issue server/600).

  • Logs at TRACE level the decoded self-contained access token authorisation under AS0544 (issue authz-server/178).

  • HTTP POST and PUT requests to the client registration endpoint where the entity body exceeds the hard-wired character limit must return a HTTP 400 Bad Request, not HTTP 500 Internal Server Error. Raises the limit from 15 thousand to 20 thousand characters (issue server/579).

Dependency changes

  • Upgrades to com.nimbusds:oauth2-oidc-sdk:8.23.1

  • Updates to com.nimbusds:nimbus-jose-jwt:8.20.1

  • Updates to com.nimbusds:oauth2-authz-store:16.3

  • Updates to com.nimbusds:oidc-claims-source-http:2.1

  • Updates to BouncyCastle 1.66

  • Updates to Infinispan 9.4.20

  • Upgrades to com.nimbusds:infinispan-cachestore-dynamodb:3.7.2

  • Updates to com.unboundid:unboundid-ldapsdk:5.1.1

  • Updates to com.amazonaws:aws-java-sdk-bundle:1.11.880

Nimbus JOSE+JWT 9.0

2020-09-10

There is a new major release of the Nimbus JOSE+JWT library. No new features are being added, but there is a breaking change - the JSON Smart dependency is now safely shaded, to prevent potential conflicts in projects with other dependencies on JSON Smart. The credits for this long awaited change go to Toma Velev.

The shading is declared like this in the pom.xml:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.2.4</version>
    <configuration>
        <relocations>
            <relocation>
                <pattern>net.minidev.json</pattern>
                <shadedPattern>com.nimbusds.jose.shaded.json</shadedPattern>
            </relocation>
        </relocations>
        <artifactSet>
            <includes>
                <include>net.minidev:json-smart</include>
            </includes>
        </artifactSet>
    </configuration>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
        </execution>
    </executions>
</plugin>

Upgrading

After the shading classes from the JSON Smart library will no longer appear in the public APIs. If you rely on calls to such classes see the release notes below for how to update your code. Some users may not need any changes at all.

Discussion on the release and back-porting process post v9.0

Josh Cummings from Spring Security started a discussion on how to proceed with releases and back-ports in v8.x and v9.x, you can comment in the thread.

Upgrading the OAuth 2.0 / OpenID Connect SDK

A volunteer opened a ticket to upgrade the OAuth 2.0 SDK. There will likely also be a discussion about shading JSON Smart in the SDK as well.

Release notes

version 9.0 (2020-09-06)

  • Replaces or removes all JSON Smart interfaces and classes from all visible classes and methods in the library. Replaces all net.minidev.json.JSONObject method arguments and return types with java.util.Map<String,Object>. Replaces all net.minidev.json.JSONArray method arguments and return types with java.util.List<Object>. Removes all net.minidev.json.JSONAware interfaces of classes, leaving their toJSONString methods intact.
  • Adds new JSONObjectUtils.toJSONString static methods for serialising a java.util.Map<String,Object> to a JSON object entity.
  • Adds new JSONObjectUtils.newJSONObject static helper method.
  • Adds new JSONArrayUtils.newJSONArray static helper method.
  • Shades the net.minidev:json-smart:1.3.1 dependency.
  • JWK.parse with missing "kty" member must throw ParseException (iss #363).

The JSON Canonicalisation Scheme (RFC 8785) in action and how to secure JSON objects with HMAC

2020-09-07

We recently had the task to add HMAC security to DynamoDB items stored by the Connect2id server in the AWS cloud. DynamoDB is a key-value database which works well with JSON objects. Hash-based Message Authentication Codes (HMAC) is a common cryptographic method for ensuring the integrity and authenticity of data, which employs a secret key and a hashing algorithm, such as SHA-2.

The principle of HMAC

The code in the HMAC algorithms is computed over the binary representation of the data and typically stored alongside the data, for easy inspection. To check that the data has not changed the computation must be repeated by inputting the same key and the data in the exact same order of its binary representation. The stored and the recomputed codes are then compared. If the two codes don't match this is a sign that the data integrity was affected, i.e. it was modified in some way. The secret key ensures that only its holder can compute and verify the code. The code thus also serve to authenticate the data.

Example HMAC with SHA-256 of the "Hello, world!" string, which is converted to its UTF-8 / ASCII bytes representation to be input into the computation:

Input text (UTF-8): Hello, world!
Secret key (BASE64 encoded): lHG3evumVZLuM2nluoxI8hVxnSL3V8r7U+8mqUp2NR4=
Code (BASE64 encoded): UtTLGRell8x8up6b8sF44o18telWQITfOHQSxUGSNPA=

Let's see the resulting code if the comma is removed from the "Hello, world!" string:

Input text (UTF-8): Hello world!
Secret key (BASE64 encoded): lHG3evumVZLuM2nluoxI8hVxnSL3V8r7U+8mqUp2NR4=
Code (BASE64 encoded): wATR9kyMBXiog8ENqMqsP68ZDQSuvUtxh4ArvL+sgx4=

Notice how the computed code changed entirely!

The issue with JSON serialisation

JSON can represent identical data in different ways and this is part of what makes it so useful.

Consider the following identical JSON data, formatted in three different ways:

JSON formatting for compactness:

{"key":"9cea8d2d","name":"Alice Adams","age":21}

As above, but with a different member ordering:

{"key":"9cea8d2d","age":21,"name":"Alice Adams"}

"Pretty" JSON formatting for better human readability:

{
  "key" : "9cea8d2d",
  "name" : "Alice Adams",
  "age" : 21
}

If we take the above JSON strings (as ASCII or UTF-8 bytes) this will produce three different binary representations. And if we input them into a HMAC SHA-256 computation we will get three different codes. This means that for HMAC to work reliably and prevent false errors (code mismatches) the JSON data must always be presented in some deterministic, also called normalised or canonical, order.

The JSON Canonicalisation Scheme (JCS) comes to rescue

The JSON Canonicalization Scheme (JCS) is a standard algorithm for putting arbitrary JSON data in a deterministic format. It was originally motivated by the need to perform reliable cryptographic operations on JSON data, such as applying digital signatures and HMACs.

The JCS employs methods such as stripping redundant whitespace and sorting the members in JSON objects.

When JCS is applied to the sample JSON object we get:

{"age":21,"key":"9cea8d2d","name":"Alice Adams"}

Now that we got our JSON data in a canonical format we can safely take the string bytes to be input into the HMAC computation.

Input text (UTF-8): {"age":21,"key":"9cea8d2d","name":"Alice Adams"}
Secret key (BASE64 encoded): lHG3evumVZLuM2nluoxI8hVxnSL3V8r7U+8mqUp2NR4=
Code (BASE64 encoded): MLE+O33O8Rv2fdSajlaK6h3wT8yKQkbNuPCoWDRWcz4=

Storing the HMAC as part of the JSON data

The computed message authentication code can be naturally included in the JSON object that it is meant to secure:

{
  "key" : "9cea8d2d",
  "name" : "Alice Adams",
  "age" : 21,
  "_hmac#s256" : "MLE+O33O8Rv2fdSajlaK6h3wT8yKQkbNuPCoWDRWcz4="
}

To recompute the code simply remove the _hmac#s256 member and input the bytes of the JSON string (in JCS form!) to the HMAC SHA-256 algorithm.

JCS in Java

Samuel Erdtman, co-author of the JCS standard, has provided a Java open source implementation. It is only 28 KBytes of Java bytecode!

https://github.com/erdtman/java-json-canonicalization

Example Java code to turn a JSON string into canonical form and gets its bytes for HMAC or digital signature computation:

import java.nio.charset.StandardCharsets;
import org.erdtman.jcs.JsonCanonicalizer;

String json = "{ /* some JSON */ }";
JsonCanonicalizer jc = new JsonCanonicalizer(json);
byte[] hmacInput = jc.getEncodedString().getBytes(StandardCharsets.UTF_8);

How to secure DynamoDB items with HMAC

Let's now apply our JSON canonicalisation strategy to secure DynamoDB items with HMAC SHA-256:

import java.nio.charset.StandardCharsets;
import javax.crypto.Mac;
import javax.crypto.SecretKey;
import javax.crypto.spec.SecretKeySpec;
import com.amazonaws.services.dynamodbv2.document.Item;
import org.erdtman.jcs.JsonCanonicalizer;

// Generate the secret key, must have the same size as the hash
// The key must be stored securely!
byte[] secretKeyBytes = new byte[256/8];
new SecureRandom().nextBytes(secretKeyBytes);
SecretKey secretKey = new SecretKeySpec(secretKeyBytes, "HmacSHA256")

// Some DynamoDB item to secure with HMAC
Item item;

// Get the item JSON in canonical format
JsonCanonicalizer jc = new JsonCanonicalizer(item.toJSON());
byte[] hmacInput = jc.getEncodedString().getBytes(StandardCharsets.UTF_8);

// Compute the HMAC
Mac hmacSHA256 = Mac.getInstance("HmacSHA256");
hmacSHA256.init(secretKey);
byte[] hmac = hmacSHA256.doFinal(hmacInput);

// Include the HMAC in the DynamoDB item, the item can be saved now
item = item.withBinary("_hmac#s256", hmac);

To verify the HMAC of a DynamoDB item:

import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import javax.crypto.Mac;
import com.amazonaws.services.dynamodbv2.document.Item;
import org.erdtman.jcs.JsonCanonicalizer;

// Make sure the HMAC attribute isn't missing
if (! item.hasAttribute("_hmac#s256")) {
    throw new Exception("Missing item HMAC attribute");
}

// Extract the HMAC
byte[] storedHMAC = item.getBinary("_hmac#s256");

Item baseItem = item.removeAttribute("_hmac#s256");

// Get the item JSON in canonical format
JsonCanonicalizer jc = new JsonCanonicalizer(item.toJSON());
byte[] hmacInput = jc.getEncodedString().getBytes(StandardCharsets.UTF_8);

// Recompute the HMAC
Mac hmacSHA256 = Mac.getInstance("HmacSHA256");
hmacSHA256.init(secretKey);
byte[] recomputedHMAC = hmacSHA256.doFinal(hmacInput);

// Compare the two HMACs
if (! MessageDigest.isEqual(storedHMAC, recomputedHMAC)) {
    throw new Exception("Invalid item HMAC");
}