[OPENAM-14771] Multi Server deployment documentation is unclear if using the default keystore.jceks Created: 12/Apr/19  Updated: 15/Nov/19  Resolved: 30/Apr/19

Status: Resolved
Project: OpenAM
Component/s: documentation, install
Affects Version/s: 6.5.0, 7.0.0
Fix Version/s: 6.5.2, 7.0.0

Type: Bug Priority: Minor
Reporter: Brad Tarisznyas Assignee: Cristina Herraz
Resolution: Fixed Votes: 0
Labels: AME, SHAKESPEARE
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates
relates to OPENAM-12598 Replacing the keystore.jceks file fro... Open
Sprint: AM 2019.6 - Lathe
Story Points: 2
Needs backport:
No
Support Ticket IDs:
Needs QA verification:
No
Functional tests:
No
Are the reproduction steps defined?:
No (add reasons in the comment)

 Description   

Bug description

The install guide section for Installing Multiple Servers -> Configuring Sites -> To Add a Server to a Site (https://backstage.forgerock.com/docs/am/6.5/install-guide/#add-servers-to-site) is unclear on how to deploy a default installation for development purposes where a customer may not have keys in dedicated stores, but is using the keystore.jceks. 

The documentation states: 

You do not need to copy the keystore.jceks file and its password files across servers unless you had created your new keys in this keystore.

Which is confusing. In testing adding the 2nd server to the site and completing configuration (copying /secrets) but not the keystore.jceks, .storepass and .keypass, leaves the 2nd server seemingly functional but will fail in generating or validating stateless tokens.

The issue is that the /secrets/encrypted/storepass and /secrets/encrypted/entrypass is not automatically created on the 2nd server, and the storepass created on the 1st server is specific to they keystore on server1 as it contains the encrypted value of the randomly generated password. Copying this to server 2 means that server2 won't be able to access the keystore for signing (only booting - when using the .storepass)

The following error is seen in the logs:

OAuth2Provider:04/10/2019 06:06:26:896 PM CEST: Thread[http-nio-8080-exec-6,5,main]: TransactionId[3cdf1e62-0576-49e1-a1cd-760418ad8b83-31634]OAuth2Provider:04/10/2019 06:06:26:896 PM CEST: Thread[http-nio-8080-exec-6,5,main]: TransactionId[3cdf1e62-0576-49e1-a1cd-760418ad8b83-31634]ERROR: Unhandled exception:org.restlet.resource.ResourceException: Internal Server Error (500) - The server encountered an unexpected condition which prevented it from fulfilling the request at org.restlet.resource.ServerResource.doHandle(ServerResource.java:527) at org.restlet.resource.ServerResource.post(ServerResource.java:1341) at org.restlet.resource.ServerResource.doHandle(ServerResource.java:606) at org.restlet.resource.ServerResource.doNegotiatedHandle(ServerResource.java:662) at org.restlet.resource.ServerResource.doConditionalHandle(ServerResource.java:348) at org.restlet.resource.ServerResource.handle(ServerResource.java:1020) at org.restlet.resource.Finder.handle(Finder.java:236) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.routing.Router.doHandle(Router.java:422) at org.forgerock.openam.rest.service.RestletRealmRouter.doHandle(RestletRealmRouter.java:69) at org.restlet.routing.Router.handle(Router.java:641) at org.forgerock.openam.rest.RealmRoutingFactory$RestletRealmRouter$Delegate.handle(RealmRoutingFactory.java:279) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.routing.Router.doHandle(Router.java:422) at org.forgerock.openam.rest.RealmRoutingFactory$RestletRealmRouter.doHandle(RealmRoutingFactory.java:257) at org.restlet.routing.Router.handle(Router.java:641) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.routing.Router.doHandle(Router.java:422) at org.forgerock.openam.rest.service.RestletRealmRouter.doHandle(RestletRealmRouter.java:69) at org.restlet.routing.Router.handle(Router.java:641) at org.forgerock.openam.rest.RealmRoutingFactory$RestletRealmRouter$Delegate.handle(RealmRoutingFactory.java:279) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.routing.Router.doHandle(Router.java:422) at org.forgerock.openam.rest.RealmRoutingFactory$RestletRealmRouter.doHandle(RealmRoutingFactory.java:257) at org.restlet.routing.Router.handle(Router.java:641) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.routing.Router.doHandle(Router.java:422) at org.forgerock.openam.rest.service.RestletRealmRouter.doHandle(RestletRealmRouter.java:94) at org.restlet.routing.Router.handle(Router.java:641) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.engine.application.StatusFilter.doHandle(StatusFilter.java:140) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.engine.CompositeHelper.handle(CompositeHelper.java:202) at org.restlet.engine.application.ApplicationHelper.handle(ApplicationHelper.java:77) at org.restlet.Application.handle(Application.java:385) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.routing.Router.doHandle(Router.java:422) at org.restlet.routing.Router.handle(Router.java:641) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.routing.Router.doHandle(Router.java:422) at org.restlet.routing.Router.handle(Router.java:641) at org.restlet.routing.Filter.doHandle(Filter.java:150) at org.restlet.routing.Filter.handle(Filter.java:197) at org.restlet.engine.CompositeHelper.handle(CompositeHelper.java:202) at org.restlet.Component.handle(Component.java:408) at org.restlet.Server.handle(Server.java:507) at org.restlet.engine.connector.ServerHelper.handle(ServerHelper.java:63) at org.restlet.engine.adapter.HttpServerHelper.handle(HttpServerHelper.java:143) at org.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:1117) at org.forgerock.openam.rest.RestEndpointServlet$RestletHandler.handle(RestEndpointServlet.java:183) at org.forgerock.http.handler.Handlers$UndescribedAsDescribableHandler.handle(Handlers.java:179) at org.forgerock.openam.dpro.session.ProofOfPossessionTokenFilter.filter(ProofOfPossessionTokenFilter.java:87) at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:53) at org.forgerock.http.filter.TransactionIdInboundFilter.filter(TransactionIdInboundFilter.java:86) at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:53) at org.forgerock.http.servlet.HttpFrameworkServlet.service(HttpFrameworkServlet.java:264) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.forgerock.openam.rest.RestEndpointServlet$HttpServletWrapper.service(RestEndpointServlet.java:254) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.forgerock.openam.rest.RestEndpointServlet.service(RestEndpointServlet.java:132) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.forgerock.openam.validation.ResponseValidationFilter.doFilter(ResponseValidationFilter.java:59) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.forgerock.openam.headers.SetHeadersFilter.doFilter(SetHeadersFilter.java:80) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.sun.identity.setup.AMSetupFilter.doFilter(AMSetupFilter.java:115) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.forgerock.openam.audit.context.AuditContextFilter.doFilter(AuditContextFilter.java:46) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:200) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)Caused by: com.google.common.util.concurrent.UncheckedExecutionException: org.forgerock.openam.secrets.SecretInitialisationException: Could not load some secret stores at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050) at com.google.common.cache.LocalCache.get(LocalCache.java:3952) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958) at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4964) at org.forgerock.openam.secrets.Secrets.getRealmSecrets(Secrets.java:139) at org.forgerock.oauth2.core.RealmOAuth2ProviderSettings.getProviderKeys(RealmOAuth2ProviderSettings.java:1199) at org.forgerock.oauth2.core.RealmOAuth2ProviderSettings.getStatelessTokenVerificationKeys(RealmOAuth2ProviderSettings.java:305) at org.forgerock.openam.oauth2.token.OAuth2JwtTokenHelper.hasValidSignature(OAuth2JwtTokenHelper.java:218) at org.forgerock.openam.oauth2.token.stateless.StatelessTokenHelper.verifySignature(StatelessTokenHelper.java:477) at org.forgerock.openam.oauth2.token.stateless.StatelessTokenHelper.readAccessToken(StatelessTokenHelper.java:182) at org.forgerock.openam.oauth2.token.stateless.StatelessTokenStore.readAccessToken(StatelessTokenStore.java:288) at org.forgerock.openam.oauth2.token.OpenAMTokenStore.readAccessToken(OpenAMTokenStore.java:151) at org.forgerock.oauth2.core.OAuth2TokenIntrospectionHandler.getIntrospectableToken(OAuth2TokenIntrospectionHandler.java:184) at org.forgerock.oauth2.core.OAuth2TokenIntrospectionHandler.introspect(OAuth2TokenIntrospectionHandler.java:64) at org.forgerock.oauth2.core.TokenIntrospectionService.introspect(TokenIntrospectionService.java:80) at org.forgerock.oauth2.restlet.TokenIntrospectionResource.introspect(TokenIntrospectionResource.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.restlet.resource.ServerResource.doHandle(ServerResource.java:508) ... 109 moreCaused by: org.forgerock.openam.secrets.SecretInitialisationException: Could not load some secret stores at org.forgerock.openam.secrets.Secrets.resolveSecretStores(Secrets.java:258) at org.forgerock.openam.secrets.Secrets.loadSecretStores(Secrets.java:227) at org.forgerock.openam.secrets.Secrets.loadGlobalSecretStores(Secrets.java:192) at io.vavr.Lazy.computeValue(Lazy.java:162) at io.vavr.Lazy.get(Lazy.java:156) at org.forgerock.openam.secrets.Secrets.loadRealmSecrets(Secrets.java:196) at com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:165) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) ... 130 moreCaused by: org.forgerock.openam.secrets.SecretInitialisationException: Unable to load keystore at org.forgerock.openam.secrets.config.KeyStoreSecretStore.lambda$createStore$4(KeyStoreSecretStore.java:152) at org.forgerock.secrets.GenericSecret.lambda$revealAsText$0(GenericSecret.java:120) at org.forgerock.secrets.GenericSecret.reveal(GenericSecret.java:82) at org.forgerock.secrets.GenericSecret.revealAsText(GenericSecret.java:115) at org.forgerock.secrets.GenericSecret.revealAsUtf8(GenericSecret.java:161) at org.forgerock.secrets.keystore.KeyStoreSecretStore.<init>(KeyStoreSecretStore.java:102) at org.forgerock.openam.secrets.config.KeyStoreSecretStore.createStore(KeyStoreSecretStore.java:154) at org.forgerock.openam.secrets.config.KeyStoreBasedSecretStoreProvider.getStore(KeyStoreBasedSecretStoreProvider.java:50) at org.forgerock.openam.secrets.config.KeyStoreBasedSecretStoreProvider.getStore(KeyStoreBasedSecretStoreProvider.java:38) at org.forgerock.openam.secrets.Secrets.resolveSecretStores(Secrets.java:245) ... 140 moreCaused by: java.lang.IllegalStateException: Unable to load keystore at org.forgerock.openam.shared.security.crypto.KeyStoreBuilder.build(KeyStoreBuilder.java:144) at org.forgerock.openam.secrets.config.KeyStoreSecretStore.lambda$createStore$4(KeyStoreSecretStore.java:150) ... 149 more

Accessing the jwk_uri on the second server results in:

{"error_description": "Internal Server Error (500) - The server encountered an unexpected condition which prevented it from fulfilling the request","error": "server_error"}

and the OAuth2Provider debug:

Caused by: org.forgerock.openam.secrets.SecretInitialisationException: Could not load some secretstores        at org.forgerock.openam.secrets.Secrets.resolveSecretStores(Secrets.java:258)        at org.forgerock.openam.secrets.Secrets.loadSecretStores(Secrets.java:227)        at org.forgerock.openam.secrets.Secrets.loadGlobalSecretStores(Secrets.java:192)        at io.vavr.Lazy.computeValue(Lazy.java:162)        at io.vavr.Lazy.get(Lazy.java:156)        at org.forgerock.openam.secrets.Secrets.getGlobalSecrets(Secrets.java:127)        at org.forgerock.oauth2.core.AgentOAuth2ProviderSettings.getProviderKeys(AgentOAuth2ProviderSettings.java:141)        at org.forgerock.oauth2.core.AgentOAuth2ProviderSettings.getJWKSet(AgentOAuth2ProviderSettings.java:373)        at org.forgerock.openidconnect.restlet.OpenIDConnectJWKEndpoint.getJWKSet(OpenIDConnectJWKEndpoint.java:73)        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)        at java.lang.reflect.Method.invoke(Method.java:498)        at org.restlet.resource.ServerResource.doHandle(ServerResource.java:511)        ... 93 moreCaused by: org.forgerock.openam.secrets.SecretInitialisationException: Unable to load keystore        at org.forgerock.openam.secrets.config.KeyStoreSecretStore.lambda$createStore$4(KeyStoreSecretStore.java:150)        at org.forgerock.secrets.GenericSecret.lambda$revealAsText$0(GenericSecret.java:120)        at org.forgerock.secrets.GenericSecret.reveal(GenericSecret.java:82)        at org.forgerock.secrets.GenericSecret.revealAsText(GenericSecret.java:115)        at org.forgerock.secrets.GenericSecret.revealAsUtf8(GenericSecret.java:161)        at org.forgerock.secrets.keystore.KeyStoreSecretStore.<init>(KeyStoreSecretStore.java:102)        at org.forgerock.openam.secrets.config.KeyStoreSecretStore.createStore(KeyStoreSecretStore.java:152)        at org.forgerock.openam.secrets.config.KeyStoreBasedSecretStoreProvider.getStore(KeyStoreBasedSecretStoreProvider.java:50)        at org.forgerock.openam.secrets.config.KeyStoreBasedSecretStoreProvider.getStore(KeyStoreBasedSecretStoreProvider.java:38)        at org.forgerock.openam.secrets.Secrets.resolveSecretStores(Secrets.java:245)        ... 106 moreCaused by: java.lang.IllegalStateException: Unable to load keystore        at org.forgerock.openam.shared.security.crypto.KeyStoreBuilder.build(KeyStoreBuilder.java:144)        at org.forgerock.openam.secrets.config.KeyStoreSecretStore.lambda$createStore$4(KeyStoreSecretStore.java:148)        ... 115 more

To resolve these errors either copy the following files to the 2nd (and subsequent) servers:

<config_dir>/<context>/keystore.jceks

<config_dir>/<context>/.storepass

<config_dir>/<context>/.keypass

<config_dir>/secrets/encrypted/storepass

<config_dir>/secrets/encrypted/entrypass

OR 

Copy the /secrets/encrypted folder from server1 to server2 and use encode.jsp to encrypt the value contained in .storepass and replace the value in /secrets/encrypted/storepass with this value.

 

It is very common practice customers to use the default keys for evaluation or development environments, only replacing them in higher environments. With the documentation as is, assuming keystore.jceks will only be used for boot purposes and not signing keys, I would expect customers to run across this issue frequently.

Expected behaviour
Documentation is confusing and should clearly state that the outlined configuration is aligned with a productionised approach using separate keystores for signing keys where keystore.jceks is only used for booting. To deploy for development/evaluation using the default keystore.jceks the files must be copied to the 2nd and subsequent servers.
Current behaviour
2nd server in the deployment will be unable to validate or generate stateless tokens (including serving the JWK_URI).

Work around

Copy the above files and restart server



 Comments   
Comment by Cristina Herraz [ 30/Apr/19 ]

Fixed in master and backported to 6.5.x

Generated at Tue Sep 22 10:40:04 UTC 2020 using Jira 7.13.12#713012-sha1:6e07c38070d5191bbf7353952ed38f111754533a.