Uploaded image for project: 'OpenAM'
  1. OpenAM
  2. OPENAM-6734

Shutdown race condition between embedded OpenDJ and OpenAM persistent search restart

    Details

    • Support Ticket IDs:

      Description

      During a container initiated shutdown where an embedded OpenDJ is in use, at least one potential race condition exists between the OpenDJ shutdown and OpenAM shutdown processes.

      If the OpenDJ shutdown leads to a persistent search socket being closed at the server side, the client (OpenAM) can receive a notification of this before the DJLDAPv3PersistentSearch class knows it is shutting down - this filters down and causes it to attempt to immediately restart the connection.

      Restarting the psearch involves getting a SystemTimerPool instance, but if the OpenAM shutdown manager already knows it is shutting down it won't allow a shutdown listener to be added - leading to a rogue TimerPool instance with no shutdown hook and then throwing an uncaught runtime exception:

      java.lang.IllegalMonitorStateException: Failed to acquire lock registering the ShutdownListener
          at com.sun.identity.common.ShutdownManager.addShutdownListener(ShutdownManager.java:146)
          at com.sun.identity.common.ShutdownManager.addShutdownListener(ShutdownManager.java:126)
          at com.sun.identity.common.SystemTimerPool.getTimerPool(SystemTimerPool.java:78)
          at org.forgerock.openam.idrepo.ldap.psearch.DJLDAPv3PersistentSearch.restartPSearch(DJLDAPv3PersistentSearch.java:250)
          at org.forgerock.openam.idrepo.ldap.psearch.DJLDAPv3PersistentSearch.access$700(DJLDAPv3PersistentSearch.java:65)
          at org.forgerock.openam.idrepo.ldap.psearch.DJLDAPv3PersistentSearch$PSearchResultHandler.handleErrorResult(DJLDAPv3PersistentSearch.java:357)
          at org.forgerock.opendj.ldap.HeartBeatConnectionFactory$ConnectionImpl$AbstractWrappedResultHandler.handleErrorResult(HeartBeatConnectionFactory.java:291)
          at com.forgerock.opendj.util.AsynchronousFutureResult$Sync.innerSetErrorResult(AsynchronousFutureResult.java:173)
          at com.forgerock.opendj.util.AsynchronousFutureResult.handleErrorResult(AsynchronousFutureResult.java:311)
          at com.forgerock.opendj.ldap.AbstractLDAPFutureResultImpl.setResultOrError(AbstractLDAPFutureResultImpl.java:138)
          at com.forgerock.opendj.ldap.AbstractLDAPFutureResultImpl.adaptErrorResult(AbstractLDAPFutureResultImpl.java:127)
          at com.forgerock.opendj.ldap.LDAPConnection.close(LDAPConnection.java:690)
          at com.forgerock.opendj.ldap.LDAPClientFilter.handleClose(LDAPClientFilter.java:480)
      

      If the system is shutting down then getTimerPool should probably fail more gracefully, but this also means there are a lot of places where getTimerPool usage needs fixing because it is usually expected/assumed to be 'guaranteed'.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ian.packer Ian Packer [X] (Inactive)
                Reporter:
                ian.packer Ian Packer [X] (Inactive)
              • Votes:
                1 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: