Uploaded image for project: 'OpenDJ'
  1. OpenDJ
  2. OPENDJ-7319

Addrate can run out of memory when --deleteMode off and --noPurge are set

    Details

    • Story Points:
      0.5
    • Support Ticket IDs:

      Description

      If JVM heap size is set to 512MB for addrate, then the following addrate command runs into out of memory error condition after about 50 minutes.

      addrate --hostname localhost --port 6389 --bindDN "cn=directory manager" --bindPassword welcome1 --noRebind --numConnections 16 --numConcurrentRequests 1 --deleteMode off --noPurge --maxDuration 28800 ~/ldif/addrate.template
      
        501 78609 78591   0  9:08AM ttys006    0:06.98 /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/bin/java -server -d64 -Xms512m -Xmx512m -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:MaxTenuringThreshold=1 -Dorg.opends.server.scriptName=addrate com.forgerock.opendj.ldap.tools.AddRate --hostname localhost --port 6389 --bindDN cn=directory manager --bindPassword welcome1 --noRebind --numConnections 16 --numConcurrentRequests 1 --deleteMode off --noPurge --maxDuration 28800 /Users/vincent.tran/ldif/addrate.template
      

      With the same JVM heap size of 512MB, if the --deleteMode off and --noPurge options are dropped (so that --deleteMode is set to default of fifo) from the addrate commmand, then addrate can basically run for hours without hitting an out of memory error condition.

      addrate --hostname localhost --port 6389 --bindDN "cn=directory manager" --bindPassword welcome1 --noRebind --numConnections 16 --numConcurrentRequests 1 --maxDuration 28800 ~/ldif/addrate.template
      
        501 79465 79447   0 10:03AM ttys006    0:13.50 /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/bin/java -server -d64 -Xms512m -Xmx512m -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:MaxTenuringThreshold=1 -Dorg.opends.server.scriptName=addrate com.forgerock.opendj.ldap.tools.AddRate --hostname localhost --port 6389 --bindDN cn=directory manager --bindPassword welcome1 --noRebind --numConnections 16 --numConcurrentRequests 1 --maxDuration 28800 /Users/vincent.tran/ldif/addrate.template
      

      Customer provided a heap dump where the JVM heap size was set to 2GB. The heap dump report generated by MAT (Eclipse Memory Analyzer) reveals that all the memory is being consumed by java.util.concurrent.ConcurrentSkipListMap:

      Problem Suspect 1
       One instance of "com.forgerock.opendj.ldap.tools.AddRate$AddPerformanceRunner" loaded by "sun.misc.Launcher$AppClassLoader @ 0x94cc0000" occupies 2,106,864,816 (99.75%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentSkipListMap$HeadIndex" loaded by "<system class loader>".
      Keywords
       java.util.concurrent.ConcurrentSkipListMap$HeadIndex
       com.forgerock.opendj.ldap.tools.AddRate$AddPerformanceRunner
       sun.misc.Launcher$AppClassLoader @ 0x94cc0000
       Details » (file:///var/folders/h5/06c0tvjd6832p2pwmgt871_c0000gn/T/report7475125734352277235/pages/18.html)
      Class Name | Shallow Heap | Retained Heap | Percentage
       ----------------------------------------------------------------------------------------------------------------------------
       com.forgerock.opendj.ldap.tools.AddRate$AddPerformanceRunner @ 0x94cd9e70 | 184 | 2,106,864,816 | 99.75%
      
      java.util.concurrent.ConcurrentSkipListMap @ 0x952ac210   48 2,106,863,664 99.75%   '- java.util.concurrent.ConcurrentSkipListMap$HeadIndex @ 0x8f46f4b0 32 2,106,863,616 99.75%       java.util.concurrent.ConcurrentSkipListMap$HeadIndex @ 0x953396d8   32 94,676,984 4.48%         java.util.concurrent.ConcurrentSkipListMap$HeadIndex @ 0x95303758   32 48,416,256 2.29%           java.util.concurrent.ConcurrentSkipListMap$HeadIndex @ 0x9519e570   32 7,040,512 0.33%           java.util.concurrent.ConcurrentSkipListMap$Index @ 0xc0a7d920   24 456 0.00%           java.util.concurrent.ConcurrentSkipListMap$Index @ 0xcc89f370   24 456 0.00%           java.util.concurrent.ConcurrentSkipListMap$Index @ 0xb2634750   24 456 0.00%           java.util.concurrent.ConcurrentSkipListMap$Index @ 0xa3205c00   24 432 0.00%           java.util.concurrent.ConcurrentSkipListMap$Index @ 0xb1614000   24 432 0.00%           java.util.concurrent.ConcurrentSkipListMap$Index @ 0xcff98f28   24 432 0.00%           java.util.concurrent.ConcurrentSkipListMap$Index @ 0xd0f20468   24 432 0.00%           java.util.concurrent.ConcurrentSkipListMap$Index @ 0xa57ab6e8   24 432 0.00%           java.util.concurrent.ConcurrentSkipListMap$Index @ 0xc04f9700   24 408 0.00%           java.util.concurrent.ConcurrentSkipListMap$Index @ 0xc1b33590   24 408 0.00%           java.util.concurrent.ConcurrentSkipListMap$Index @ 0xc4be4bc0   24 408 0.00%           java.util.concurrent.ConcurrentSkipListMap$Index @ 0xacf75ef8   24 408 0.00%           java.util.concurrent.ConcurrentSkipListMap$Index @ 0xca828cd0   24 408 0.00%           java.util.concurrent.ConcurrentSkipListMap$Index @ 0xb05697c0   24 408 0.00%           java.util.concurrent.ConcurrentSkipListMap$Index @ 0xb8dc23d0   24 408 0.00%           java.util.concurrent.ConcurrentSkipListMap$Index @ 0xaa3ac7a0   24 408 0.00%           java.util.concurrent.ConcurrentSkipListMap$Index @ 0xaa93b868   24 384 0.00%           java.util.concurrent.ConcurrentSkipListMap$Index @ 0xaaf0dab8   24 384 0.00%           java.util.concurrent.ConcurrentSkipListMap$Index @ 0xab918218   24 384 0.00%           java.util.concurrent.ConcurrentSkipListMap$Index @ 0xb98348d0   24 384 0.00%           java.util.concurrent.ConcurrentSkipListMap$Index @ 0xca2bb130   24 384 0.00%           java.util.concurrent.ConcurrentSkipListMap$Index @ 0xb056e188   24 384 0.00%           java.util.concurrent.ConcurrentSkipListMap$Index @ 0xcd3592e8   24 384 0.00%           java.util.concurrent.ConcurrentSkipListMap$Index @ 0xb10cbd80   24 384 0.00%       '- Total: 25 of 862,801 entries; 862,776 more    
      ----------------------------------------------------------------------------------------------------------------------------
      

      Some relevant source code snippets from AddRate.java:

      import java.util.concurrent.ConcurrentSkipListMap;
      
      private final ConcurrentSkipListMap<Long, String> dnEntriesAdded = new ConcurrentSkipListMap<>();
      
      void updateAdditionalStatsOnResult() {
        switch (delStrategy) {
        case RANDOM:
          long newKey;
          do {
            newKey = randomSeq.get().nextInt();
          } while (dnEntriesAdded.putIfAbsent(newKey, entryDN) != null);
          break;
        case FIFO:
        case OFF:
          long uniqueTime = operationStartTimeNs;
          while (dnEntriesAdded.putIfAbsent(uniqueTime, entryDN) != null) {
            uniqueTime++;​
          }
          break;
        default:
          throw new IllegalStateException("Unexpected deletion strategy: " + delStrategy);
        }
        entryCount.inc();
      }
      

      Even if addrate is invoked with the --deleteMode off and --noPurge options, it appears that the ConcurrentSkipListMap, dnEntriesAdded, is still maintained and added to. So, it seems like dnEntriesAdded could grow without limit if deleteMode is not set to fifo (default) or random.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                miroslav.meca Miroslav Meca
                Reporter:
                vincent.tran Vincent Tran
                Dev Assignee:
                Matthew Swift
                QA Assignee:
                Miroslav Meca
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: