Uploaded image for project: 'OpenDJ'
  1. OpenDJ
  2. OPENDJ-1783

Unfair write lock policy may cause unpredictable response times

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.6.0
    • Fix Version/s: None
    • Component/s: core server
    • Labels:
    • Support Ticket IDs:

      Description

      Under a very heavy write load, jstack revealed that all the servers worker threads were waiting for a write lock on a psearch connection:

      "Worker Thread 2" prio=10 tid=0x00007f422ceec000 nid=0x17e2 waiting on condition [0x00007f36e4a89000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00007f3c1d889bb8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
      	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
      	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
      	at org.opends.server.protocols.ldap.LDAPClientConnection$TimeoutWriteByteChannel.write(LDAPClientConnection.java:191)
      	at org.opends.server.extensions.RedirectingByteChannel.write(RedirectingByteChannel.java:163)
      	at org.opends.server.extensions.RedirectingByteChannel.write(RedirectingByteChannel.java:163)
      	at org.opends.server.types.ByteStringBuilder.copyTo(ByteStringBuilder.java:1001)
      	at org.opends.server.protocols.ldap.LDAPClientConnection.sendLDAPMessage(LDAPClientConnection.java:989)
      	at org.opends.server.protocols.ldap.LDAPClientConnection.sendSearchEntry(LDAPClientConnection.java:899)
      	at org.opends.server.core.SearchOperationBasis.sendSearchEntry(SearchOperationBasis.java:1363)
      	at org.opends.server.core.SearchOperationBasis.returnEntry(SearchOperationBasis.java:871)
      	at org.opends.server.core.SearchOperationBasis.returnEntry(SearchOperationBasis.java:622)
      	at org.opends.server.core.SearchOperationWrapper.returnEntry(SearchOperationWrapper.java:63)
      	at org.opends.server.core.PersistentSearch.processModify(PersistentSearch.java:587)
      	at org.opends.server.workflowelement.localbackend.LocalBackendModifyOperation$1.run(LocalBackendModifyOperation.java:678)
      	at org.opends.server.types.AbstractOperation.invokePostResponseCallbacks(AbstractOperation.java:1132)
      	at org.opends.server.core.ModifyOperationBasis.run(ModifyOperationBasis.java:554)
      	at org.opends.server.extensions.TraditionalWorkerThread.run(TraditionalWorkerThread.java:167)
      

      The problem here is that when the write lock is released, it is undefined which thread will acquire the lock. It is possible that a given thread will never acquire it.

      The use of a fair locking policy in the TimeoutWriteByteChannel should be investigated. Code-wise, simply creating the lock with a 'true' argument should suffice. However the performance impact will need measuring.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                cjr Chris Ridd
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: