[OPENIDM-15254] no more possible to change the default Encryption Keys Created: 03/Aug/20  Updated: 10/Aug/20  Resolved: 04/Aug/20

Status: Closed
Project: OpenIDM
Component/s: documentation, Module - OSGi Container / Framework integration
Affects Version/s: 7.0.0
Fix Version/s: 7.0.0

Type: Bug Priority: Critical
Reporter: Cyril Grosjean Assignee: Lana Frost
Resolution: Fixed Votes: 0
Labels: documentation
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

CentOS Linux release 7.8.2003 (Core)
openjdk version "11.0.7" 2020-04-14 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.7+10-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.7+10-LTS, mixed mode, sharing)
OpenIDM version "7.0.0-SNAPSHOT" (build: 20200724153319, revision: 5b9163d) jenkins-OpenIDM-build-master-2054

Attachments: File idp.p12     File keystore.jceks     File nohup.out     File nohup.out.debug     File openidm0.log.0     File secrets.json    
Target Version/s:
Verified Version/s:
QA Assignee: Son Nguyen
Story Points: 0.5
Sprint: 2020.10 - IDM


When I try to change the default self-signed SSL certificate (which has the openidm-localhost alias) by following https://qa-backstage.forgerock.com/docs/idm/7/security-guide/import-signed-cert.html, restarting IDM gives decryption errors and IDM can not be accessed anymore over HTTPS.


Steps to reproduce:

  • Start a default IDM instance
  • Stop the instance
  • Add a CA signed certificate to the keystore
  • update the secrets.json accordingly
  • Restart IDMopenidm0.log.0


The nohup.out will contain the following stack trace:


Executing ./startup.sh...

      2 Using OPENIDM_HOME:   /opt/forgerock/products/IDM/openidm

      3 Using PROJECT_HOME:   /opt/forgerock/products/IDM/projects/StarCas

      4 Using OPENIDM_OPTS:   -Xmx2048m -Xms2048m

      5 Using LOGGING_CONFIG: -Djava.util.logging.config.file=/opt/forgerock/products/IDM/projects/StarCas/conf/logging.properties

      6 WARNING: An illegal reflective access operation has occurred

      7 WARNING: Illegal reflective access by org.apache.felix.framework.ext.ClassPathExtenderFactory$DefaultClassLoaderExtender (file:/opt/forgerock/products/IDM/openidm/bin/felix.ja        r) to method java.net.URLClassLoader.addURL(java.net.URL)

      8 WARNING: Please consider reporting this to the maintainers of org.apache.felix.framework.ext.ClassPathExtenderFactory$DefaultClassLoaderExtender

      9 WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

     10 WARNING: All illegal access operations will be denied in a future release

     11 -> ShellTUI: Unable to read from stdin...exiting.

     12 [17] août 03, 2020 6:06:26.543 PM org.forgerock.openidm.config.logging.LogServiceTracker logEntry

     13 GRAVE: Bundle: org.forgerock.openidm.external-rest [250] bundle org.forgerock.openidm.external-rest:7.0.0.SNAPSHOT (250)[org.forgerock.openidm.external.rest(106)] : The activa        te method has thrown an exception

     14 org.apache.felix.log.LogException: org.osgi.service.component.ComponentException: Failed to initialize External REST service

     15         at org.forgerock.openidm.external.rest.RestService.activate(RestService.java:228)

     16         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

     17         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

     18         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

     19         at java.base/java.lang.reflect.Method.invoke(Method.java:566)

     20         at org.apache.felix.scr.impl.inject.methods.BaseMethod.invokeMethod(BaseMethod.java:228)



Caused by: java.security.UnrecoverableKeyException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.

    123         at java.base/com.sun.crypto.provider.KeyProtector.recover(KeyProtector.java:221)

    124         at java.base/com.sun.crypto.provider.JceKeyStore.engineGetKey(JceKeyStore.java:141)

    125         at java.base/java.security.KeyStore.getKey(KeyStore.java:1057)

    126         at java.base/sun.security.ssl.SunX509KeyManagerImpl.<init>(SunX509KeyManagerImpl.java:145)

    127         at java.base/sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:70)

    128         at java.base/javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:271)

    129         at org.forgerock.opendj.security.KeyManagers.getX509KeyManager(KeyManagers.java:311)

    130         at org.forgerock.opendj.security.KeyManagers.useJvmDefaultKeyManager(KeyManagers.java:366)

    131         at org.forgerock.openidm.external.rest.RestService.activate(RestService.java:213)

    132         ... 106 more


In spite the new keystore file can be read (default keystore.jcekskeystore type or keystore password unchanged) with keytool, 



Comment by Dirk Hogan [ 03/Aug/20 ]

Cyril Grosjean could it be that the cert you imported is corrupt? One of the exceptions occur when the RestService is initialized. This occurs when the KeyManager is initialized. Part of this initialization goes through the keystore and does some initialization of each of the keys. This initialization fails for the frdpclout.com alias - exactly the one you added. Part of this initialization involves encrypting/decrypting some canned data, just to see if it works. This is the part which blows up with the BadPaddingException. Can you retry the generation and import of the cert? - I suspect that the cert is somehow corrupt.

Comment by Dirk Hogan [ 03/Aug/20 ]

Cyril Grosjean I agree that it is very unlikely that IT gave you a corrupted cert. Googling a bit more, it seems that the error is generated when the cert's private key is being accessed with the wrong keystore/keystore-entry password. This is what the error message - 'Such issues can arise if a bad key is used during decryption' - is trying to communicate. I noticed that when the private key corresponding to your cert is being 'recovered' (which is a normal part of keystore initialization - I am understanding 'recovered' as a necessary part of initializing a cert - i.e. for a cert to be usable, the corresponding private key has to be accessible, and this is termed 'recovery', and it involves accessing the private key with its password). But the bottom line is that the password being used to recover the cert's private key is 'changeit', but from your keytool command above, could it be that the private key entry inherits the password of the store - i.e. PKI-FOrgeROck? (From the keytool documentation: "keypass is a password used to protect the private key of the generated key pair. If no password is provided, the user is prompted for it. If you press RETURN at the prompt, the key password is set to the same password as that used for the keystore.") Could it be that you didn't specify the password for a keystore entry, and thus it inherited the password of the keystore itself - thus the password  of the private key of the imported cert will be PKI-F0rgeR0ck, not changeit? This is a much more likely cause of the error - not a corrupted cert. Note that in the IDM doc example you cited in your defect filing, both the source and target keystore passwords were changeit. So I'm wondering if you could import this cert from a keystore with password changeit? Or specify the keypass for the cert as changeit prior to import?  

Comment by Cyril Grosjean [ 04/Aug/20 ]

Dirk Hogan You were correct Dirk, I was able to start IDM without errors by changing the private key entry password to be the same as the keystore password. Yet, I'd like to mention that:

  • the "keytool -importkeystore ... " command mentioned at https://qa-backstage.forgerock.com/docs/idm/7/security-guide/import-signed-cert.html#import-signed-cert looks wrong in my environment: the "-alias" option doesn't exist, I only have "-srcalias" and "-destalias", however, no error is reported when using "-alias" (??..). Also, the man pages for keytool confirm that "-alias" is not supported for the -importkeystore option".
  • I thus used the following command to import the certificate and private key in the keystore: "keytool -importkeystore -srckeystore /opt/forgerock/pki/keystores/idp.p12 -srcstoretype PKCS12 -srcstorepass PKI-F0rgeR0ck -srcalias 1 -destkeystore  security/keystore.jceks -deststoretype JCEKS -destalias frdpcloud.com -deststorepass changeit -destkeypass changeit"
  • When using the previous command, it works, yet keytool reports that the alias named "1" already exists, and asks me whether I want to replace it or not. I answer "yes", but the previous prompt is wrong, it should asks me whether I want to replace the target alias, not the source one. Anyway, of course, the source alias in the source keystore is left unchanged, it's just another proof that the keytool command has some issues..
  • Since it seems to be a Java security framework requirement (rather than an IDM requirement) that the certificate entry in the keystore should have the same password as the keystore itself, I can't understand why keytool lets people create certificate entries with a password different from the keystore password. At least, there should be some warning when doing so ..
  • Also, for information, it seems like that in a PKCS 12 file, it's mandatory to have the same keystore and certificate entry password: when testing and running the "keytool -importkeystore" command, using different values for "-srcstorepass" and "-srckeypass"  issues a warning saying the PKCS 12 format doesn't support different keystore and certificate entry passwords, and thus, that the "-srckeypass" value would be ignored.

With that, I think it could be worth adding at least 2 important details/fixes to the documentation (Lana Frost ?), because we can't expect to rely on the keytool developers to improve the tool:

  • replacing "-alias" with "-srcalias" and "-destalias", unless it's JDK version specific, but I don't think so.
  • when the source certificate entry password is different from the target keystore password, the "-destkeypass" option has to be specified with the same value as for the "-deststorepass" option. Failing to do so will not generate any issue when importing the certificate or later reading the certificate entry in the destination keystore, however, IDM will fail to start with the exception I mentioned initially when filing this defect.
Comment by Lana Frost [ 04/Aug/20 ]

That's fine - I will make those changes to the doc.

Comment by Dirk Hogan [ 04/Aug/20 ]

Very cool Cyril Grosjean and Lana Frost! Thanks for the follow-up. I'm relieved it's not a bug. 

Comment by Lana Frost [ 04/Aug/20 ]

https://stash.forgerock.org/projects/OPENIDM/repos/openidm-docs/pull-requests/2906/overview for the doc change

Comment by Son Nguyen [ 10/Aug/20 ]

Verified successfully on master

Generated at Thu Dec 03 19:16:08 UTC 2020 using Jira 7.13.12#713012-sha1:6e07c38070d5191bbf7353952ed38f111754533a.