Uploaded image for project: 'OpenAM'
  1. OpenAM
  2. OPENAM-17117

Service config XML dump consumes a lot of memory (whole config is read to memory)

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 5.5.1, 6.0.0, 6.0.0.1, 6.5.0, 6.0.0.6, 6.5.1, 6.5.2, 6.5.2.1, 6.5.2.2, 6.5.2.3, 5.5.2, 7.0.0, 6.5.3, 7.0.1
    • 5.5.3, 6.0.1, 6.5.4, 7.1.0, 7.0.2
    • sms
    • Rank:
      1|i02xa7:
    • AM Sustaining Sprint 81, AM Sustaining Sprint 82, AM Sustaining Sprint 83
    • 3
    • No
    • Yes
    • No
    • Yes and I used the same an in the description
    • 0
    • Future
    • None

    Description

      Bug description

      Scenario 1:
      When performing a recording where the service config may be dump, one may end up using a lot of memory while this is being done during production gathering. As memory demands will conflate GC activity, we should be more careful with memory use: Example:

      Caused by: java.lang.OutOfMemoryError: Java heap space
      	at java.util.Arrays.copyOf(Arrays.java:3332)
      	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
      	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
      	at java.lang.StringBuilder.append(StringBuilder.java:136)
      	at com.sun.identity.sm.ServiceManager.toXML(ServiceManager.java:1348)
      	at org.forgerock.openam.core.rest.record.DefaultDebugRecorder.exportConfigExport(DefaultDebugRecorder.java:294)
      

      Scenario 2:
      When doing upgrade on a large service, the UpgradeServices.writeBackup will serialize the whole AM configuration (taking 2X the string representation of the LDAP in memory ie: basically saying like needing 2-4 times the AM config in memory) and this itself may cause upgrade to be slow or need extremely large JVM size (compared to normal runtime) to get the AM upgraded.

      Caused by: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
      	at java.lang.StringCoding.encode(StringCoding.java:350)
      	at java.lang.String.getBytes(String.java:941)
      	at org.forgerock.openam.upgrade.UpgradeServices.writeBackup(UpgradeServices.java:287)
      	at org.forgerock.openam.upgrade.UpgradeServices.upgrade(UpgradeServices.java:146)
      	at com.sun.identity.config.upgrade.Upgrade.doUpgrade(Upgrade.java:70)
      

      Similar issue may happen on SSOADM (probably another tracking bug but same for exporting service config)

      As can be seen in those thread dump, the service config dump reads the whole services branch into a String (1st storage ~ 1.5 bytes per string string *representation of the AM config ) and then later does a toBytes(UTF) to dump to file. Note also that converter to byte[] cannot work for large string too. Definitely not good when performing AM recording for diagnostic collection.

      How to reproduce the issue

      1. A dozen realm with huge number Agents, SAML (huge metadata) and many stuff.

      Example:

      1. Install AM (default with dc=openam, dc=forgerock, dc=org)
      2. Create a realm tmp, tmp001, tmp002, tmp003
      3. Download tmp.ldif.bz2
      4. Load to the Directory on each realm (below is for tmp realm)
      5. bzcat tmp.ldif.bz2 | sed 's/tmp/tmp/g' | ldapmodify -p 10389 -D"cn=Directory Manager" -w dirmanagerpwd -c
      6. Repeat for tmp001
      7. bzcat tmp.ldif.bz2 | sed 's/tmp/tmp001/g' | ldapmodify -p 10389 -D"cn=Directory Manager" -w dirmanagerpwd -c
      8. Repeat until step 7 for realm tmp002 to tmp003.
      9. The above create 1000 OAuth2 agent each with 32K size.
      10. You may deactivate these realms. So that they do not use memory.
      11. Now do any config config or dump. You will not be able to do this with JVM memory < 2GB

      config export may  pass with smaller JVM.

      Expected behaviour
      Less memory usage 
      
      Current behaviour
      More memory use for more complicated AM config store.
      

      Work around

      Larger VM.

      Code analysis

      ServiceManager.java
      ... Have a interface to dump smaller chunks ...
      

      Attachments

        1. tmp.ldif.bz2
          994 kB
        2. tmp-org.ldif
          2 kB

        Issue Links

          Activity

            People

              chee-weng.chea C-Weng C
              chee-weng.chea C-Weng C
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: