Uploaded image for project: 'OpenAM'
  1. OpenAM
  2. OPENAM-11770

amster import-config overloads AM's JVM

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Duplicate
    • Affects Version/s: 14.0.0
    • Fix Version/s: None
    • Component/s: Amster, rest, REST SDK, sms
    • Labels:
    • Environment:
      Tomcat 8.5
      Java 8
      OpenAM 14.0.0
    • Target Version/s:
    • Sprint:
      AM Sustaining Sprint 43, AM Sustaining Sprint 44, AM Sustaining Sprint 45, AM Sustaining Sprint 46, AM Sustaining Sprint 47
    • Story Points:
      5
    • Needs backport:
      No
    • Support Ticket IDs:
    • Needs QA verification:
      No
    • Functional tests:
      No
    • Are the reproduction steps defined?:
      No (add reasons in the comment)

      Description

      Bug description

      trying to configure 700 realms with amster import-config leads to a 502 gateway error

      How to reproduce the issue

      Details steps outlining how to recreate the issue (remove this text)

      1. Setup OpenDJ 4.0.0 as external configuration data store
      2. Configure AM 5.0.0 with external configuation data store
      3. create an amster import structure with 700 realms or even more
      4. run amster config-import --path PATH_TO_AMSTER_IMPORT_DIRECTORY
      5. Expected behaviour
      amster import-config should succeed an AM should be configured accordingly
      
      Current behaviour
      amster throws 502 gateway error
      
      [main] DEBUG org.forgerock.openam.amster.loadster.importer.Importer - Exception occurred when reading file. 
      org.forgerock.openam.sdk.http.ServerErrorException: 502 Bad Gateway
       at org.forgerock.openam.sdk.http.DefaultErrorHandler.onServerError(DefaultErrorHandler.java:62)
       at org.forgerock.openam.sdk.http.HttpSessionImpl.handleUnsuccessfulResponse(HttpSessionImpl.java:273)
       at org.forgerock.openam.sdk.http.HttpSessionImpl.send(HttpSessionImpl.java:169)
       at org.forgerock.openam.sdk.http.RequestBuilder.put(RequestBuilder.java:226)
       at org.forgerock.openam.sdk.crest.CrestResourceProviderAsync.updateInstance(CrestResourceProviderAsync.java:185)
       at org.forgerock.openam.sdk.crest.CrestResourceProviderAsync.update(CrestResourceProviderAsync.java:155)
       at org.forgerock.openam.sdk.crest.HttpCrestResourceProvider.update(HttpCrestResourceProvider.java:158)
       at org.forgerock.openam.sdk.operations.CrestOperations.update(CrestOperations.java:119)
       at org.forgerock.openam.amster.loadster.importer.GenericImporter.importEntity(GenericImporter.groovy:55)
       at org.forgerock.openam.amster.loadster.importer.GenericImporter.importFile(GenericImporter.groovy:33)
       at org.forgerock.openam.amster.loadster.importer.Importer.importFile(Importer.groovy:178)
      

      On server side JVM GC can not catch up

      Exceprt from GC log
      5637.851: [Full GC (Ergonomics) [PSYoungGen: 2110976K->2055175K(2378752K)] [ParOldGen: 5592119K->5592119K(5592576K)] 7703095K->7647294K(7971328K), [Metaspace: 106611K->106611K(1144832K)], 7.1870721 secs] [Times: user=28.15 sys=0.03, real=7.19 secs]
      5645.076: [Full GC (Ergonomics) [PSYoungGen: 2110976K->2056255K(2378752K)] [ParOldGen: 5592119K->5592118K(5592576K)] 7703095K->7648373K(7971328K), [Metaspace: 106611K->106611K(1144832K)], 9.0688109 secs] [Times: user=35.37 sys=0.05, real=9.07 secs]
      5654.184: [Full GC (Ergonomics) [PSYoungGen: 2110976K->2057241K(2378752K)] [ParOldGen: 5592118K->5592118K(5592576K)] 7703094K->7649360K(7971328K), [Metaspace: 106612K->106612K(1144832K)], 8.9738715 secs] [Times: user=35.01 sys=0.04, real=8.98 secs]
      5663.196: [Full GC (Ergonomics) [PSYoungGen: 2110976K->2058197K(2378752K)] [ParOldGen: 5592118K->5592118K(5592576K)] 7703094K->7650315K(7971328K), [Metaspace: 106612K->106612K(1144832K)], 7.5961004 secs] [Times: user=29.77 sys=0.03, real=7.60 secs]
      5670.837: [Full GC (Ergonomics) [PSYoungGen: 2110976K->2059257K(2378752K)] [ParOldGen: 5592118K->5592118K(5592576K)] 7703094K->7651375K(7971328K), [Metaspace: 106612K->106612K(1144832K)], 8.9570115 secs] [Times: user=34.98 sys=0.04, real=8.96 secs]
      ....
      

      It happens for 8GB and 12GB heap being used.

      11-Sep-2017 09:52:18.614 WARNING [Grizzly-worker(3)] org.glassfish.grizzly.filterchain.DefaultFilterChain.execute GRIZZLY0013: Exception during FilterChain execution
       java.lang.OutOfMemoryError: GC overhead limit exceeded
              at java.util.Arrays.copyOf(Arrays.java:3332)
              at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
              at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
              at java.lang.StringBuilder.append(StringBuilder.java:136)
              at org.forgerock.opendj.ldap.Dn.toString(Dn.java:922)
              at com.sun.identity.idm.AMIdentity.<init>(AMIdentity.java:172)
              at com.sun.identity.idm.AMIdentity.<init>(AMIdentity.java:158)
              at org.forgerock.openam.uma.rest.UmaPolicyApplicationListener.getIdentity(UmaPolicyApplicationListener.java:214)
              at org.forgerock.openam.uma.rest.UmaPolicyApplicationListener.identityChanged(UmaPolicyApplicationListener.java:141)
              at com.sun.identity.idm.IdRepoListener.objectChanged(IdRepoListener.java:185)
              at org.forgerock.openam.idrepo.ldap.psearch.DJLDAPv3PersistentSearch$PSearchResultEntryHandler.handle(DJLDAPv3PersistentSearch.java:129)
              at com.iplanet.services.ldap.event.LDAPv3PersistentSearch$PersistentSearchResultHandler.handleEntry(LDAPv3PersistentSearch.java:371)
              at org.forgerock.opendj.ldap.LdapConnectionFactory$ConnectionImpl$3.handleEntry(LdapConnectionFactory.java:919)
              at org.forgerock.opendj.ldap.spi.SearchResultLdapPromiseImpl.handleEntry(SearchResultLdapPromiseImpl.java:58)
              at org.forgerock.opendj.grizzly.LdapClientFilter.searchResultEntry(LdapClientFilter.java:218)
              at org.forgerock.opendj.grizzly.LdapClientFilter.handleRead(LdapClientFilter.java:330)
              at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
              at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
              at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
              at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
              at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
              at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
              at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:526)
              at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
              at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
              at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
              at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137)
              at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:593)
              at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:573)
              at java.lang.Thread.run(Thread.java:745)
      

        Attachments

        1. am500.jmap.gz
          75 kB
        2. am500-thread-dumps.txt.gz
          90 kB
        3. gc.log.export-config
          14 kB
        4. gc.log.import-config
          101 kB

          Issue Links

            Activity

              People

              • Assignee:
                jonthomas Jonathan Thomas
                Reporter:
                bthalmayr Bernhard Thalmayr
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: