Uploaded image for project: 'OpenDJ'
  1. OpenDJ
  2. OPENDJ-1354

replication threads BLOCKED in pendingChanges queue

    Details

    • Support Ticket IDs:
    • Sprint:
      Sprint 28, Sprint 29, Sprint 30

      Description

      Performing modify operations against 2 DS are blocked and returning no response.

      Topology is 4 servers (2 DS m1/m4 and 2 RS m2/m3).

      In access log on m4 we can see no response for modify, same for m1.

      tail access
      [21/Feb/2014:16:05:11 +0100] CONNECT conn=61 from=172.16.203.10:44534 to=172.16.203.15:4389 protocol=LDAP
      [21/Feb/2014:16:05:11 +0100] BIND REQ conn=61 op=0 msgID=1 version=3 type=SIMPLE dn="cn=directory manager"
      [21/Feb/2014:16:05:11 +0100] BIND RES conn=61 op=0 msgID=1 result=0 authDN="cn=Directory Manager,cn=Root DNs,cn=config" etime=1
      [21/Feb/2014:16:05:11 +0100] MODIFY REQ conn=61 op=1 msgID=2 dn="uid=user_11350,dc=europe,dc=com"

      Looking at jstack on DS m1 and m4, these threads are BLOCKED.

      grep BLOCKED /tmp/j1.txt
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)
      java.lang.Thread.State: BLOCKED (on object monitor)

      It seems that the threads BLOCKED are doing either:

      "Worker Thread 42" prio=10 tid=0x00007f879cdf0000 nid=0x31e4 waiting for monitor entry [0x00007f8528ccb000]
      java.lang.Thread.State: BLOCKED (on object monitor)
      at org.opends.server.replication.plugin.PendingChanges.commitAndPushCommittedChanges(PendingChanges.java:256)

      • waiting to lock <0x000000079c47f118> (a org.opends.server.replication.plugin.PendingChanges)
        at org.opends.server.replication.plugin.LDAPReplicationDomain.synchronize(LDAPReplicationDomain.java:2311)

      or

      "Worker Thread 40" prio=10 tid=0x00007f879d0ee000 nid=0x31e2 waiting for monitor entry [0x00007f8528ecd000]
      java.lang.Thread.State: BLOCKED (on object monitor)
      at org.opends.server.replication.plugin.PendingChanges.putLocalOperation(PendingChanges.java:165)

      • waiting to lock <0x000000079c47f118> (a org.opends.server.replication.plugin.PendingChanges)
        at org.opends.server.replication.plugin.LDAPReplicationDomain.generateChangeNumber(LDAPReplicationDomain.java:2821)
        at org.opends.server.replication.plugin.LDAPReplicationDomain.handleConflictResolution(LDAPReplicationDomain.java:2178)

      I've included the jstack for all 4 servers.

        Attachments

        1. j2-m1.txt
          119 kB
        2. j2-m2.txt
          125 kB
        3. j2-m3.txt
          185 kB
        4. j2-m4.txt
          200 kB

          Issue Links

            Activity

              People

              • Assignee:
                matthew Matthew Swift
                Reporter:
                gary.williams Gary Williams
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: