Uploaded image for project: 'OpenAM'
  1. OpenAM
  2. OPENAM-14771

Multi Server deployment documentation is unclear if using the default keystore.jceks

    Details

    • Sprint:
      AM 2019.6 - Lathe
    • Story Points:
      2
    • Needs backport:
      No
    • Support Ticket IDs:
    • Needs QA verification:
      No
    • Functional tests:
      No
    • Are the reproduction steps defined?:
      No (add reasons in the comment)

      Description

      Bug description

      The install guide section for Installing Multiple Servers -> Configuring Sites -> To Add a Server to a Site (https://backstage.forgerock.com/docs/am/6.5/install-guide/#add-servers-to-site) is unclear on how to deploy a default installation for development purposes where a customer may not have keys in dedicated stores, but is using the keystore.jceks. 

      The documentation states: 

      You do not need to copy the keystore.jceks file and its password files across servers unless you had created your new keys in this keystore.
      

      Which is confusing. In testing adding the 2nd server to the site and completing configuration (copying /secrets) but not the keystore.jceks, .storepass and .keypass, leaves the 2nd server seemingly functional but will fail in generating or validating stateless tokens.

      The issue is that the /secrets/encrypted/storepass and /secrets/encrypted/entrypass is not automatically created on the 2nd server, and the storepass created on the 1st server is specific to they keystore on server1 as it contains the encrypted value of the randomly generated password. Copying this to server 2 means that server2 won't be able to access the keystore for signing (only booting - when using the .storepass)

      The following error is seen in the logs:

      OAuth2Provider:04/10/2019 06:06:26:896 PM CEST: Thread[http-nio-8080-exec-6,5,main]: TransactionId[3cdf1e62-0576-49e1-a1cd-760418ad8b83-31634]OAuth2Provider:04/10/2019 06:06:26:896 PM CEST: Thread[http-nio-8080-exec-6,5,main]: TransactionId[3cdf1e62-0576-49e1-a1cd-760418ad8b83-31634]ERROR: Unhandled exception:org.restlet.resource.ResourceException: Internal Server Error (500) - The server encountered an unexpected condition which prevented it from fulfilling the request at org.restlet.resource.ServerResource.doHandle(ServerResource.java:527) at org.restlet.resource.ServerResource.post(ServerResource.java:1341) at org.restlet.resource.ServerResource.doHandle(ServerResource.java:606) at org.restlet.resource.ServerResource.doNegotiatedHandle(ServerResource.java:662) at org.restlet.resource.ServerResource.doConditionalHandle(ServerResource.java:348) at org.restlet.resource.ServerResource.handle(ServerResource.java:1020) at org.restlet.resource.Finder.handle(Finder.java:236) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.routing.Router.doHandle(Router.java:422) at org.forgerock.openam.rest.service.RestletRealmRouter.doHandle(RestletRealmRouter.java:69) at org.restlet.routing.Router.handle(Router.java:641) at org.forgerock.openam.rest.RealmRoutingFactory$RestletRealmRouter$Delegate.handle(RealmRoutingFactory.java:279) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.routing.Router.doHandle(Router.java:422) at org.forgerock.openam.rest.RealmRoutingFactory$RestletRealmRouter.doHandle(RealmRoutingFactory.java:257) at org.restlet.routing.Router.handle(Router.java:641) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.routing.Router.doHandle(Router.java:422) at org.forgerock.openam.rest.service.RestletRealmRouter.doHandle(RestletRealmRouter.java:69) at org.restlet.routing.Router.handle(Router.java:641) at org.forgerock.openam.rest.RealmRoutingFactory$RestletRealmRouter$Delegate.handle(RealmRoutingFactory.java:279) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.routing.Router.doHandle(Router.java:422) at org.forgerock.openam.rest.RealmRoutingFactory$RestletRealmRouter.doHandle(RealmRoutingFactory.java:257) at org.restlet.routing.Router.handle(Router.java:641) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.routing.Router.doHandle(Router.java:422) at org.forgerock.openam.rest.service.RestletRealmRouter.doHandle(RestletRealmRouter.java:94) at org.restlet.routing.Router.handle(Router.java:641) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.engine.application.StatusFilter.doHandle(StatusFilter.java:140) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.engine.CompositeHelper.handle(CompositeHelper.java:202) at org.restlet.engine.application.ApplicationHelper.handle(ApplicationHelper.java:77) at org.restlet.Application.handle(Application.java:385) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.routing.Router.doHandle(Router.java:422) at org.restlet.routing.Router.handle(Router.java:641) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.routing.Router.doHandle(Router.java:422) at org.restlet.routing.Router.handle(Router.java:641) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.engine.CompositeHelper.handle(CompositeHelper.java:202) at org.restlet.Component.handle(Component.java:408) at org.restlet.Server.handle(Server.java:507) at org.restlet.engine.connector.ServerHelper.handle(ServerHelper.java:63) at org.restlet.engine.adapter.HttpServerHelper.handle(HttpServerHelper.java:143) at org.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:1117) at org.forgerock.openam.rest.RestEndpointServlet$RestletHandler.handle(RestEndpointServlet.java:183) at org.forgerock.http.handler.Handlers$UndescribedAsDescribableHandler.handle(Handlers.java:179) at org.forgerock.openam.dpro.session.ProofOfPossessionTokenFilter.filter(ProofOfPossessionTokenFilter.java:87) at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:53) at org.forgerock.http.filter.TransactionIdInboundFilter.filter(TransactionIdInboundFilter.java:86) at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:53) at org.forgerock.http.servlet.HttpFrameworkServlet.service(HttpFrameworkServlet.java:264) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.forgerock.openam.rest.RestEndpointServlet$HttpServletWrapper.service(RestEndpointServlet.java:254) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.forgerock.openam.rest.RestEndpointServlet.service(RestEndpointServlet.java:132) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.forgerock.openam.validation.ResponseValidationFilter.doFilter(ResponseValidationFilter.java:59) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.forgerock.openam.headers.SetHeadersFilter.doFilter(SetHeadersFilter.java:80) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.sun.identity.setup.AMSetupFilter.doFilter(AMSetupFilter.java:115) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.forgerock.openam.audit.context.AuditContextFilter.doFilter(AuditContextFilter.java:46) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:200) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)Caused by: com.google.common.util.concurrent.UncheckedExecutionException: org.forgerock.openam.secrets.SecretInitialisationException: Could not load some secret stores at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050) at com.google.common.cache.LocalCache.get(LocalCache.java:3952) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958) at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4964) at org.forgerock.openam.secrets.Secrets.getRealmSecrets(Secrets.java:139) at org.forgerock.oauth2.core.RealmOAuth2ProviderSettings.getProviderKeys(RealmOAuth2ProviderSettings.java:1199) at org.forgerock.oauth2.core.RealmOAuth2ProviderSettings.getStatelessTokenVerificationKeys(RealmOAuth2ProviderSettings.java:305) at org.forgerock.openam.oauth2.token.OAuth2JwtTokenHelper.hasValidSignature(OAuth2JwtTokenHelper.java:218) at org.forgerock.openam.oauth2.token.stateless.StatelessTokenHelper.verifySignature(StatelessTokenHelper.java:477) at org.forgerock.openam.oauth2.token.stateless.StatelessTokenHelper.readAccessToken(StatelessTokenHelper.java:182) at org.forgerock.openam.oauth2.token.stateless.StatelessTokenStore.readAccessToken(StatelessTokenStore.java:288) at org.forgerock.openam.oauth2.token.OpenAMTokenStore.readAccessToken(OpenAMTokenStore.java:151) at org.forgerock.oauth2.core.OAuth2TokenIntrospectionHandler.getIntrospectableToken(OAuth2TokenIntrospectionHandler.java:184) at org.forgerock.oauth2.core.OAuth2TokenIntrospectionHandler.introspect(OAuth2TokenIntrospectionHandler.java:64) at org.forgerock.oauth2.core.TokenIntrospectionService.introspect(TokenIntrospectionService.java:80) at org.forgerock.oauth2.restlet.TokenIntrospectionResource.introspect(TokenIntrospectionResource.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.restlet.resource.ServerResource.doHandle(ServerResource.java:508) ... 109 moreCaused by: org.forgerock.openam.secrets.SecretInitialisationException: Could not load some secret stores at org.forgerock.openam.secrets.Secrets.resolveSecretStores(Secrets.java:258) at org.forgerock.openam.secrets.Secrets.loadSecretStores(Secrets.java:227) at org.forgerock.openam.secrets.Secrets.loadGlobalSecretStores(Secrets.java:192) at io.vavr.Lazy.computeValue(Lazy.java:162) at io.vavr.Lazy.get(Lazy.java:156) at org.forgerock.openam.secrets.Secrets.loadRealmSecrets(Secrets.java:196) at com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:165) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) ... 130 moreCaused by: org.forgerock.openam.secrets.SecretInitialisationException: Unable to load keystore at org.forgerock.openam.secrets.config.KeyStoreSecretStore.lambda$createStore$4(KeyStoreSecretStore.java:152) at org.forgerock.secrets.GenericSecret.lambda$revealAsText$0(GenericSecret.java:120) at org.forgerock.secrets.GenericSecret.reveal(GenericSecret.java:82) at org.forgerock.secrets.GenericSecret.revealAsText(GenericSecret.java:115) at org.forgerock.secrets.GenericSecret.revealAsUtf8(GenericSecret.java:161) at org.forgerock.secrets.keystore.KeyStoreSecretStore.<init>(KeyStoreSecretStore.java:102) at org.forgerock.openam.secrets.config.KeyStoreSecretStore.createStore(KeyStoreSecretStore.java:154) at org.forgerock.openam.secrets.config.KeyStoreBasedSecretStoreProvider.getStore(KeyStoreBasedSecretStoreProvider.java:50) at org.forgerock.openam.secrets.config.KeyStoreBasedSecretStoreProvider.getStore(KeyStoreBasedSecretStoreProvider.java:38) at org.forgerock.openam.secrets.Secrets.resolveSecretStores(Secrets.java:245) ... 140 moreCaused by: java.lang.IllegalStateException: Unable to load keystore at org.forgerock.openam.shared.security.crypto.KeyStoreBuilder.build(KeyStoreBuilder.java:144) at org.forgerock.openam.secrets.config.KeyStoreSecretStore.lambda$createStore$4(KeyStoreSecretStore.java:150) ... 149 more
      

      Accessing the jwk_uri on the second server results in:

      {"error_description": "Internal Server Error (500) - The server encountered an unexpected condition which prevented it from fulfilling the request","error": "server_error"}
      

      and the OAuth2Provider debug:

      Caused by: org.forgerock.openam.secrets.SecretInitialisationException: Could not load some secretstores        at org.forgerock.openam.secrets.Secrets.resolveSecretStores(Secrets.java:258)        at org.forgerock.openam.secrets.Secrets.loadSecretStores(Secrets.java:227)        at org.forgerock.openam.secrets.Secrets.loadGlobalSecretStores(Secrets.java:192)        at io.vavr.Lazy.computeValue(Lazy.java:162)        at io.vavr.Lazy.get(Lazy.java:156)        at org.forgerock.openam.secrets.Secrets.getGlobalSecrets(Secrets.java:127)        at org.forgerock.oauth2.core.AgentOAuth2ProviderSettings.getProviderKeys(AgentOAuth2ProviderSettings.java:141)        at org.forgerock.oauth2.core.AgentOAuth2ProviderSettings.getJWKSet(AgentOAuth2ProviderSettings.java:373)        at org.forgerock.openidconnect.restlet.OpenIDConnectJWKEndpoint.getJWKSet(OpenIDConnectJWKEndpoint.java:73)        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)        at java.lang.reflect.Method.invoke(Method.java:498)        at org.restlet.resource.ServerResource.doHandle(ServerResource.java:511)        ... 93 moreCaused by: org.forgerock.openam.secrets.SecretInitialisationException: Unable to load keystore        at org.forgerock.openam.secrets.config.KeyStoreSecretStore.lambda$createStore$4(KeyStoreSecretStore.java:150)        at org.forgerock.secrets.GenericSecret.lambda$revealAsText$0(GenericSecret.java:120)        at org.forgerock.secrets.GenericSecret.reveal(GenericSecret.java:82)        at org.forgerock.secrets.GenericSecret.revealAsText(GenericSecret.java:115)        at org.forgerock.secrets.GenericSecret.revealAsUtf8(GenericSecret.java:161)        at org.forgerock.secrets.keystore.KeyStoreSecretStore.<init>(KeyStoreSecretStore.java:102)        at org.forgerock.openam.secrets.config.KeyStoreSecretStore.createStore(KeyStoreSecretStore.java:152)        at org.forgerock.openam.secrets.config.KeyStoreBasedSecretStoreProvider.getStore(KeyStoreBasedSecretStoreProvider.java:50)        at org.forgerock.openam.secrets.config.KeyStoreBasedSecretStoreProvider.getStore(KeyStoreBasedSecretStoreProvider.java:38)        at org.forgerock.openam.secrets.Secrets.resolveSecretStores(Secrets.java:245)        ... 106 moreCaused by: java.lang.IllegalStateException: Unable to load keystore        at org.forgerock.openam.shared.security.crypto.KeyStoreBuilder.build(KeyStoreBuilder.java:144)        at org.forgerock.openam.secrets.config.KeyStoreSecretStore.lambda$createStore$4(KeyStoreSecretStore.java:148)        ... 115 more
      

      To resolve these errors either copy the following files to the 2nd (and subsequent) servers:

      <config_dir>/<context>/keystore.jceks

      <config_dir>/<context>/.storepass

      <config_dir>/<context>/.keypass

      <config_dir>/secrets/encrypted/storepass

      <config_dir>/secrets/encrypted/entrypass

      OR 

      Copy the /secrets/encrypted folder from server1 to server2 and use encode.jsp to encrypt the value contained in .storepass and replace the value in /secrets/encrypted/storepass with this value.

       

      It is very common practice customers to use the default keys for evaluation or development environments, only replacing them in higher environments. With the documentation as is, assuming keystore.jceks will only be used for boot purposes and not signing keys, I would expect customers to run across this issue frequently.

      Expected behaviour
      Documentation is confusing and should clearly state that the outlined configuration is aligned with a productionised approach using separate keystores for signing keys where keystore.jceks is only used for booting. To deploy for development/evaluation using the default keystore.jceks the files must be copied to the 2nd and subsequent servers.
      
      Current behaviour
      2nd server in the deployment will be unable to validate or generate stateless tokens (including serving the JWK_URI).

      Work around

      Copy the above files and restart server

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                cristina.herraz Cristina Herraz
                Reporter:
                bradley.tarisznyas Brad Tarisznyas
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: