Uploaded image for project: 'OpenAM'
  1. OpenAM
  2. OPENAM-9947

CTS Reaper hangs forever if CTS server is stopped during deletion phase

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 12.0.2
    • Fix Version/s: 12.0.5, 13.5.1, 14.0.0
    • Component/s: CTS
    • Labels:
    • Sprint:
      AM Sustaining Sprint 30, AM Sustaining Sprint 31
    • Story Points:
      1
    • Support Ticket IDs:

      Description

      To reproduce:

      1) Setup 1xOpenAM with external CTS
      2) Add lot of expired tokens to the CTS so that reaper is constantly active and easier to catch
      3) While reaper is processing deletes, stop or firewall the CTS server (iptables -I INPUT -p tcp --dport 1389 -j DROP).
      4) Restart OpenDJ
      5) Unfirewall OpenDJ (if it was firewalled)
      6) OpenAM will never recover

      This is due to the latch mechanism used in the reaper, the thread will get stuck at:

      "pool-4-thread-1" prio=10 tid=0x00007f0931406000 nid=0x2b94 waiting on condition [0x00007f0924dcc000]
         java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00000000c4bb6dd0> (a java.util.concurrent.CountDownLatch$Sync)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
              at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
              at org.forgerock.openam.cts.reaper.CTSReaper.run(CTSReaper.java:124)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:745)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                peter.major Peter Major [X] (Inactive)
                Reporter:
                ian.packer Ian Packer [X] (Inactive)
                QA Assignee:
                Filip Kubáň [X] (Inactive)
              • Votes:
                2 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: