Uploaded image for project: 'OpenDJ'
  1. OpenDJ
  2. OPENDJ-2631

OOME error while importing 100M entries (online-import) causes the server to crash

    Details

    • Sprint:
      OpenDJ Sprint 75, OpenDJ Sprint 80

      Description

      Trying to import 100M entries into DJ (used as a ID Store), the servers crashed after importing 98M entries with:

      head -60 hs_err_pid112970.log
      #
      # There is insufficient memory for the Java Runtime Environment to continue.
      # Native memory allocation (mmap) failed to map 65536 bytes for committing reserved memory.
      # Possible reasons:
      #   The system is out of physical RAM or swap space
      #   In 32 bit mode, the process size limit was hit
      # Possible solutions:
      #   Reduce memory load on the system
      #   Increase physical memory or swap space
      #   Check if swap backing store is full
      #   Use 64 bit Java on a 64 bit OS
      #   Decrease Java heap size (-Xmx/-Xms)
      #   Decrease number of Java threads
      #   Decrease Java thread stack sizes (-Xss)
      #   Set larger code cache with -XX:ReservedCodeCacheSize=
      # This output file may be truncated or incomplete.
      #
      #  Out of Memory Error (os_linux.cpp:2671), pid=112970, tid=139797551077120
      #
      # JRE version: Java(TM) SE Runtime Environment (8.0_25-b17) (build 1.8.0_25-b17)
      # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.25-b02 mixed mode linux-amd64 compressed oops)
      # Core dump written. Default location: /external/pyforge/core or core.112970
      #
      
      ---------------  T H R E A D  ---------------
      
      Current thread (0x00007f258c17f800):  JavaThread "C1 CompilerThread9" daemon [_thread_in_vm, id=113037, stack(0x00007f25274f5000,0x00007f25275f6000)]
      
      Stack: [0x00007f25274f5000,0x00007f25275f6000],  sp=0x00007f25275f3cf0,  free space=1019k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      V  [libjvm.so+0xa7923a]  VMError::report_and_die()+0x2ca
      V  [libjvm.so+0x4e552b]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x8b
      V  [libjvm.so+0x8e5613]  os::Linux::commit_memory_impl(char*, unsigned long, bool)+0x103
      V  [libjvm.so+0x8e5b69]  os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x29
      V  [libjvm.so+0x8df83a]  os::commit_memory(char*, unsigned long, unsigned long, bool)+0x2a
      V  [libjvm.so+0xa75169]  VirtualSpace::expand_by(unsigned long, bool)+0x1c9
      V  [libjvm.so+0x5f1aec]  CodeHeap::expand_by(unsigned long)+0x8c
      V  [libjvm.so+0x46f0b0]  CodeCache::allocate(int, bool)+0x60
      V  [libjvm.so+0x8b1ebc]  nmethod::new_nmethod(methodHandle, int, int, CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int)+0x1fc
      V  [libjvm.so+0x409204]  ciEnv::register_method(ciMethod*, int, CodeOffsets*, int, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int, bool, bool, RTMState)+0x6e4
      V  [libjvm.so+0x34def9]  Compilation::install_code(int)+0xc9
      V  [libjvm.so+0x34f3ec]  Compilation::compile_method()+0xec
      V  [libjvm.so+0x34f6bb]  Compilation::Compilation(AbstractCompiler*, ciEnv*, ciMethod*, int, BufferBlob*)+0x25b
      V  [libjvm.so+0x350469]  Compiler::compile_method(ciEnv*, ciMethod*, int)+0xa9
      V  [libjvm.so+0x49bc6a]  CompileBroker::invoke_compiler_on_method(CompileTask*)+0xc8a
      V  [libjvm.so+0x49e650]  CompileBroker::compiler_thread_loop()+0x620
      V  [libjvm.so+0xa29e8f]  JavaThread::thread_main_inner()+0xdf
      V  [libjvm.so+0xa29fbc]  JavaThread::run()+0x11c
      V  [libjvm.so+0x8e7d48]  java_start(Thread*)+0x108
      C  [libpthread.so.0+0x7a51]  start_thread+0xd1
      
      
      Current CompileTask:
      C1:4709010 7958       3       org.opends.server.api.ClientConnection::getPrivileges (215 bytes)
      
      
      ---------------  P R O C E S S  ---------------
      
      Java Threads: ( => current thread )
        0x00007f246c012000 JavaThread "JEEvictor" daemon [_thread_blocked, id=118183, stack(0x00007f1e92a3f000,0x00007f1e92b40000)]
      

      Note that:

      • We reproduced the problem several times, even after trying to increase heap size from 10GB to 20GB
      • There was more than enough memory on the host (80GB free) when the server crashes
      • We've run the exact same import with DJ2.6.3 without any problem (just after, on same host).

        Attachments

        1. hs_err_pid105747.log.zip
          844 kB
        2. opendj.zip
          1.94 MB
        3. replay_pid105747.log
          236 kB

          Issue Links

            Activity

              People

              • Assignee:
                ylecaillez Yannick Lecaillez
                Reporter:
                sberthol Sebastien Bertholet [X] (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: