Uploaded image for project: 'OpenAM'
  1. OpenAM
  2. OPENAM-8004

refresh_token load: bottleneck at OpenAMOAuth2UrisFactory causing high perf regression (compared to AM12.0.2)

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 13.0.0
    • Fix Version/s: 13.0.0
    • Component/s: oauth2, performance
    • Labels:
    • Environment:
      AM13.0.0 DJ3.0.0

      Description

      After applying a patch for OPENAM-7579 on top of OpenAM13 with OIDC and an external DJ3.0.0 CTS
      Running mixed oauth2 load (with most sequences starting with a refresh_token call to generate a new access_token).
      We don't get any more contention at: com.sun.identity.sm.ServiceConfigImpl.getFromCache
      (fixed by the patch).

      However the throughput is going down very quickly: for instance at AM Oauth2 CRUD level we see nb READ going from 2k/sec to 200/sec in about 20 minutes.
      (See AM13.0.0_OAuth2_CRUD.svg, where the patch has been applied at about 2:30 pm)

      As a comparison, running the exact same test with AM12.0.2 we can steadily sustain 2.8k READ/sec during 10h.
      (See AM12.0.2_OAuth2_CRUD.svg)

      Doing some jstack while the load is running we see many threads stuck in:

      "http-bio-8081-exec-994" #1394 daemon prio=5 os_prio=0 tid=0x00007f9d44d45000 nid=0x1ce77 waiting for monitor entry [0x00007f9cd958d000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at org.forgerock.openam.oauth2.OpenAMOAuth2UrisFactory.get(OpenAMOAuth2UrisFactory.java:101)
      	- waiting to lock <0x00000005cfba3710> (a java.util.HashMap)
      	at org.forgerock.openam.oauth2.OpenAMOAuth2UrisFactory.get(OpenAMOAuth2UrisFactory.java:78)
      	at org.forgerock.openam.oauth2.OpenAMTokenStore.createOpenIDToken(OpenAMTokenStore.java:237)
      	at org.forgerock.openidconnect.OpenIDTokenIssuer.issueToken(OpenIDTokenIssuer.java:85)
      	at org.forgerock.openam.oauth2.OpenAMScopeValidator.additionalDataToReturnFromTokenEndpoint(OpenAMScopeValidator.java:455)
      	at org.forgerock.openam.oauth2.OpenAMOAuth2ProviderSettings.additionalDataToReturnFromTokenEndpoint(OpenAMOAuth2ProviderSettings.java:469)
      	at org.forgerock.oauth2.core.AuthorizationCodeGrantTypeHandler.handle(AuthorizationCodeGrantTypeHandler.java:156)
      	at org.forgerock.oauth2.core.GrantTypeHandler.handle(GrantTypeHandler.java:82)
      	at org.forgerock.oauth2.core.AccessTokenServiceImpl.requestAccessToken(AccessTokenServiceImpl.java:92)
      	at org.forgerock.oauth2.restlet.TokenEndpointResource.token(TokenEndpointResource.java:82)
      	at sun.reflect.GeneratedMethodAccessor104.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:483)
      	at org.restlet.resource.ServerResource.doHandle(ServerResource.java:520)
      	at org.restlet.resource.ServerResource.post(ServerResource.java:1377)
      	at org.restlet.resource.ServerResource.doHandle(ServerResource.java:620)
      	at org.restlet.resource.ServerResource.doNegotiatedHandle(ServerResource.java:678)
      	at org.restlet.resource.ServerResource.doConditionalHandle(ServerResource.java:356)
      	at org.restlet.resource.ServerResource.handle(ServerResource.java:1043)
      	at org.restlet.resource.Finder.handle(Finder.java:236)
      	at org.restlet.routing.Filter.doHandle(Filter.java:150)
      	at org.restlet.routing.Filter.handle(Filter.java:197)
      	at org.restlet.routing.Filter.doHandle(Filter.java:150)
      	at org.restlet.routing.Filter.handle(Filter.java:197)
      	at org.restlet.routing.Filter.doHandle(Filter.java:150)
      	at org.restlet.routing.Filter.handle(Filter.java:197)
      	at org.restlet.routing.Router.doHandle(Router.java:422)
      	at org.forgerock.openam.rest.service.RestletRealmRouter.doHandle(RestletRealmRouter.java:121)
      	at org.restlet.routing.Router.handle(Router.java:639)
      	at org.restlet.routing.Filter.doHandle(Filter.java:150)
      	at org.restlet.routing.Filter.handle(Filter.java:197)
      	at org.restlet.routing.Filter.doHandle(Filter.java:150)
      	at org.restlet.routing.Filter.handle(Filter.java:197)
      	at org.restlet.routing.Filter.doHandle(Filter.java:150)
      	at org.restlet.engine.application.StatusFilter.doHandle(StatusFilter.java:140)
      	at org.restlet.routing.Filter.handle(Filter.java:197)
      	at org.restlet.routing.Filter.doHandle(Filter.java:150)
      	at org.restlet.routing.Filter.handle(Filter.java:197)
      	at org.restlet.engine.CompositeHelper.handle(CompositeHelper.java:202)
      	at org.restlet.engine.application.ApplicationHelper.handle(ApplicationHelper.java:75)
      	at org.restlet.Application.handle(Application.java:385)
      	at org.restlet.routing.Filter.doHandle(Filter.java:150)
      	at org.restlet.routing.Filter.handle(Filter.java:197)
      	at org.restlet.routing.Router.doHandle(Router.java:422)
      	at org.restlet.routing.Router.handle(Router.java:639)
      	at org.restlet.routing.Filter.doHandle(Filter.java:150)
      	at org.restlet.routing.Filter.handle(Filter.java:197)
      	at org.restlet.routing.Router.doHandle(Router.java:422)
      	at org.restlet.routing.Router.handle(Router.java:639)
      	at org.restlet.routing.Filter.doHandle(Filter.java:150)
      	at org.restlet.routing.Filter.handle(Filter.java:197)
      	at org.restlet.engine.CompositeHelper.handle(CompositeHelper.java:202)
      	at org.restlet.Component.handle(Component.java:408)
      	at org.restlet.Server.handle(Server.java:507)
      	at org.restlet.engine.connector.ServerHelper.handle(ServerHelper.java:63)
      	at org.restlet.engine.adapter.HttpServerHelper.handle(HttpServerHelper.java:143)
      	at org.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:1117)
      	at org.forgerock.openam.rest.RestEndpointServlet.service(RestEndpointServlet.java:130)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
      	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
      	at org.forgerock.openam.validation.ResponseValidationFilter.doFilter(ResponseValidationFilter.java:44)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
      	at com.sun.identity.setup.AMSetupFilter.doFilter(AMSetupFilter.java:106)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
      	at org.forgerock.openam.audit.context.AuditContextFilter.doFilter(AuditContextFilter.java:51)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
      	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
      	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
      	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
      	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
      	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
      	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:957)
      	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
      	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423)
      	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079)
      	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620)
      	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
      	- locked <0x00000005b40fc430> (a org.apache.tomcat.util.net.SocketWrapper)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      	at java.lang.Thread.run(Thread.java:745)
      

      See attached jstack for complete output.

        Attachments

        1. AM12.0.2_OAuth2_CRUD.svg
          208 kB
        2. AM13.0.0_OAuth2_CRUD.svg
          87 kB
        3. jstack.Boo.latest
          6.05 MB
        4. jstack.new
          7.07 MB

          Issue Links

            Activity

              People

              • Assignee:
                peter.major Peter Major [X] (Inactive)
                Reporter:
                sberthol Sebastien Bertholet [X] (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: