[OPENIDM-7947] With DJ as a repo, OpenIDM fails to start when using HSM Created: 22/Mar/17  Updated: 29/Jan/18  Resolved: 24/Jan/18

Status: Closed
Project: OpenIDM
Component/s: Module - Repository DS
Affects Version/s: OpenIDM 5.5.0
Fix Version/s: OpenIDM 6.0.0

Type: Bug Priority: Major
Reporter: Laurent Bristiel [X] (Inactive) Assignee: Travis Haagen
Resolution: Fixed Votes: 0
Labels: DIXON, release-notes
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OpenIDM version "5.5.0-SNAPSHOT" (revision: d192b02)


Issue Links:
Depends
is required by OPENIDM-8826 Update DJ repo docs for HSM Closed
Relates
relates to OPENIDM-8853 We need to configure the embedded dj ... Closed
Target Version/s:
Verified Version/s:
QA Assignee: Ladislav Folta
Story Points: 3
Sprint: OpenIDM Sprint 80, OpenIDM DJ External Repo
Epic Link: External DJaaRepo

 Description   

When we configure OpenIDM with:

then on startup, we have errors and never reach "OpenIDM Ready".
Here is what we get in the log:

INFO: Started HttpServiceContext{httpContext=org.ops4j.pax.web.service.internal.DefaultSharedWebContainerContext@281f9e79}
Mar 22, 2017 1:56:15 AM org.forgerock.openidm.servlet.internal.ErrorServletComponent activate
INFO: Registered servlet at /error
Mar 22, 2017 1:56:30 AM org.forgerock.openidm.info.impl.HealthService$4 run
SEVERE: OpenIDM failure during startup, ACTIVE_NOT_READY: Required services not all started [org.forgerock.openidm.api-servlet, org.forgerock.openidm.audit, org.forgerock.openidm.authentication, org.forgerock.openidm.cluster, org.forgerock.openidm.config.manage, org.forgerock.openidm.managed, org.forgerock.openidm.policy, org.forgerock.openidm.repo.(orientdb|jdbc|opendj), org.forgerock.openidm.scheduler, org.forgerock.openidm.script, org.forgerock.openidm.security]
...
Mar 22, 2017 1:56:12 AM org.forgerock.openidm.config.installer.JSONConfigInstaller setConfig
INFO: Loaded changed configuration for org.forgerock.openidm.endpoint getavailableuserstoassign from endpoint-getavailableuserstoassign.json
Mar 22, 2017 1:56:12 AM org.forgerock.openidm.config.installer.JSONConfigInstaller setConfig
WARNING: Loading configuration file /home/testuser/jenkins/workspace/OpenIDM-Tests-Master-Core-Linux-OpenDJ/results/20170321-215030/security/hsm/IDM/openidm/conf/endpoint-getavailableuserstoassign.json failed 
java.io.IOException: Failed to store configuration in repository: No Such Entry: Entry ou=config,dc=openidm,dc=forgerock,dc=com cannot be added because its parent entry dc=openidm,dc=forgerock,dc=com does not exist in the server
        at org.forgerock.openidm.config.persistence.RepoPersistenceManager.store(RepoPersistenceManager.java:393)
        at org.apache.felix.cm.impl.CachingPersistenceManagerProxy.store(CachingPersistenceManagerProxy.java:246)
        at org.apache.felix.cm.impl.ConfigurationImpl.update(ConfigurationImpl.java:381)
        at org.apache.felix.cm.impl.ConfigurationAdapter.update(ConfigurationAdapter.java:137)


 Comments   
Comment by Jason Lemay [ 18/Jul/17 ]

The actual exception that happens when trying to run HSM with opendj as the repo is this:

java.net.SocketException: Socket Closed
	at java.net.AbstractPlainSocketImpl.setOption(AbstractPlainSocketImpl.java:212)
	at java.net.Socket.setSoTimeout(Socket.java:1141)
	at sun.security.ssl.BaseSSLSocketImpl.setSoTimeout(BaseSSLSocketImpl.java:631)
	at sun.security.ssl.SSLSocketImpl.setSoTimeout(SSLSocketImpl.java:2526)
	at org.opends.server.tools.LDAPConnection.connectToHost(LDAPConnection.java:320)
	at org.opends.server.util.cli.LDAPConnectionArgumentParser.connect(LDAPConnectionArgumentParser.java:260)
	at org.opends.server.util.cli.LDAPConnectionArgumentParser.connect(LDAPConnectionArgumentParser.java:185)
	at org.opends.server.util.cli.LDAPConnectionArgumentParser.connect(LDAPConnectionArgumentParser.java:89)
	at org.opends.server.tools.tasks.TaskTool.process(TaskTool.java:229)
	at org.opends.server.tools.ImportLdif.process(ImportLdif.java:238)
	at org.opends.server.tools.ImportLdif.mainImportLdif(ImportLdif.java:117)
	at org.forgerock.opendj.server.embedded.EmbeddedDirectoryServer.importLDIF(EmbeddedDirectoryServer.java:315)
	at org.forgerock.openidm.repo.opendj.impl.Activator.setupEmbeddedServer(Activator.java:356)
	at org.forgerock.openidm.repo.opendj.impl.Activator.initializeEmbeddedServer(Activator.java:280)
	at org.forgerock.openidm.repo.opendj.impl.Activator.start(Activator.java:105)
	at org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:697)
	at org.apache.felix.framework.Felix.activateBundle(Felix.java:2226)
	at org.apache.felix.framework.Felix.startBundle(Felix.java:2144)
	at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1371)
	at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
	at java.lang.Thread.run(Thread.java:748)
Jul 18, 2017 2:16:54 PM org.forgerock.openidm.logging.LogServiceTracker logEntry
SEVERE: Bundle: org.forgerock.openidm.repo-opendj [11] FrameworkEvent ERROR
org.apache.felix.log.LogException: org.osgi.framework.BundleException: Activator start error in bundle org.forgerock.openidm.repo-opendj [11].
	at org.apache.felix.framework.Felix.activateBundle(Felix.java:2276)
	at org.apache.felix.framework.Felix.startBundle(Felix.java:2144)
	at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1371)
	at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.felix.log.LogException: org.forgerock.opendj.server.embedded.EmbeddedDirectoryServerException: An error occurred while attempting to import LDIF file '/Users/jason/Desktop/openidm/db/opendj/scripts/populate_users.ldif' into embedded server with server root '/Users/jason/Desktop/openidm/db/openidm/opendj'. Error code is: 1
	at org.forgerock.opendj.server.embedded.EmbeddedDirectoryServer.importLDIF(EmbeddedDirectoryServer.java:320)
	at org.forgerock.openidm.repo.opendj.impl.Activator.setupEmbeddedServer(Activator.java:356)
	at org.forgerock.openidm.repo.opendj.impl.Activator.initializeEmbeddedServer(Activator.java:280)
	at org.forgerock.openidm.repo.opendj.impl.Activator.start(Activator.java:105)
	at org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:697)
	at org.apache.felix.framework.Felix.activateBundle(Felix.java:2226)
	... 4 more

The keystore fails to initialize due to SSL handshake errors

Comment by Jason Lemay [ 18/Jul/17 ]

There are a few approaches I can take to solving this jira.

  1. Always have the embedded DJ use the openidm keystore/truststore
    1. Generate a dj key and cert as part of the openidm keystore startup
    2. Change the configuration using the EmbeddedServer.getConfiguration() method.
  2. I only configure embedded dj to use the openidm keystore/truststore when HSM is enabled.
    1. Change the configuration using the EmbeddedServer.getConfiguration() method.
    2. Doc that you must generate the dj key/cert in your hsm provider
Comment by Jason Lemay [ 01/Aug/17 ]

This has been resolved. Embedded OpenDJ will now use the openidm keystore and truststore. Currently OpenDJ has no support for pkcs11 truststore, so in the case that the truststore is configured as pkcs11 embedded opendj uses the opendj blind trust management provider, which trusts all certificates. This will be fixed once opendj supports a pkcs11 truststore.

Since OpenDJ is using the openidm keystore/truststore we needed to generate a self-signed cert for embedded dj. This cert is now generated on started in the openidm keystore and truststore. The cert alias is by default server-cert, but can be configured with the property "openidm.config.crypto.opendj.localhost.cert"

Note: When configuring pkcs11 you MUST declare the keystore/truststore location as NONE in the boot.properties. This setting is case sensitive and must be all caps.

Comment by Laurent Bristiel [X] (Inactive) [ 13/Sep/17 ]

Ladislav Folta could you update our "HSM" suite following the advice from Jason (see last comment) and remove the known issue tag to make sure it is now working OK.

Comment by Ladislav Folta [ 26/Sep/17 ]

Reopening as the problem is still valid

Comment by Laurent Bristiel [X] (Inactive) [ 26/Sep/17 ]

Brendan Miller I think we should not try to fix this in 5.5 as this is too late.
Are you OK that we change the Target to 6.0?

Comment by Brendan Miller [ 26/Sep/17 ]

Yes this is fine, if we document it. As DJ is not supported in production, I expect not having HSM in test is okay.

Comment by Jason Lemay [ 28/Sep/17 ]

I was able to replicate this in the pyforge test suite only. I have not been able to replicate this outside of the test suite. This error occurs because the ssl handshake with dj when importing the ldif is returning an error stating that there is no cipher suites in common.

To get the pyforge tool to build softhsm on osx you need to specify the following properties in SoftHSM.py

    --with-openssl=/usr/local/opt/openssl 
    --with-sqlite3=/usr/local/opt/sqlite

The code for configure in SoftHSM.py should look like this:

    def configure(self, **configure_params):
        product_path = os.path.join(self.install_path, 'softhsm-product')
        args = '--with-openssl=/usr/local/opt/openssl --with-sqlite3=/usr/local/opt/sqlite --disable-gost --prefix %s' % product_path
        cmd = './configure'
Comment by Travis Haagen [ 23/Jan/18 ]

Ladislav Folta: I am having trouble running this test. Is this the correct command?

./run-pybot.py --category functional --suite security.hsm.hsm OpenIDM

Had to remove exclude_from_opendj from hsm.robot to get it to work at all. Now it seems to timeout looking for "OpenIDM ready".

Comment by Laurent Bristiel [X] (Inactive) [ 24/Jan/18 ]

yes Travis Haagen this is the correct way to launch it.
And indeed, the test is disabled in Master (tag exclude_from_opendj) so you have to remove the tag to launch it.
And then if it times out looking for "OpenIDM Ready", then you are hitting the bug we raised.

Comment by Ladislav Folta [ 24/Jan/18 ]

I managed to get the automated tests to work. I recalled seeing some jira about adding some steps into the docs, so I checked the test setup against the documentation and it was missing one step. I think we can close the ticket now.

Comment by Travis Haagen [ 24/Jan/18 ]

Resolving so that it can be tested by QA. No additional code changes made.

Please reopen ticket if we still think there is an issue.

NOTE: Version 2.3.0 of SoftHSM seems to assign a specific "slot" which you might need to use in hsm.conf. I noticed that the PyForge test always uses slot=0, which surprised me, but Jason Lemay thought it might be because PyForge is using a slightly older version of the SoftHSM library.

Comment by Ladislav Folta [ 29/Jan/18 ]

Updated our automated tests to support SoftHSM 2.3.0 and the tests passed OK.
Tested on: OpenIDM: 6.0.0-SNAPSHOT 986ce9b

Generated at Tue Sep 22 11:12:32 UTC 2020 using Jira 7.13.12#713012-sha1:6e07c38070d5191bbf7353952ed38f111754533a.