Uploaded image for project: 'OpenAM'
  1. OpenAM
  2. OPENAM-12169

REST SMS deadlocks when processing notifications

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 14.0.0, 14.1.0, 14.1.1, 14.5.0, 14.5.1
    • Fix Version/s: 6.0.0, 5.5.2
    • Component/s: None
    • Labels:
      None
    • Needs backport:
      Yes
    • Support Ticket IDs:
    • Needs QA verification:
      No
    • Functional tests:
      No
    • Are the reproduction steps defined?:
      Yes and I used the same an in the description

      Description

      Bug description

      The following deadlock can be seen at times:

      "default task-42":
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x000000076ee6a8c8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
      	at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
      	at org.forgerock.openam.core.rest.sms.SmsRequestHandler.handleCreate(SmsRequestHandler.java:792)
      	at org.forgerock.json.resource.Router.handleCreate(Router.java:264)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleCreate(FilterChain.java:65)
      	at org.forgerock.authz.filter.crest.AuthorizationFilters$AuthorizationFilter.lambda$filterCreate$1(AuthorizationFilters.java:249)
      	at org.forgerock.authz.filter.crest.AuthorizationFilters$AuthorizationFilter$$Lambda$1267/278940421.apply(Unknown Source)
      	at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:252)
      	at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:216)
      	at org.forgerock.authz.filter.crest.AuthorizationFilters$AuthorizationFilter.filterCreate(AuthorizationFilters.java:247)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleCreate(FilterChain.java:63)
      	at org.forgerock.json.resource.FilterChain.handleCreate(FilterChain.java:228)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleCreate(FilterChain.java:65)
      	at org.forgerock.openam.rest.fluent.AuditFilter.filterCreate(AuditFilter.java:102)
      	at org.forgerock.openam.rest.fluent.AuditFilterWrapper.filterCreate(AuditFilterWrapper.java:58)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleCreate(FilterChain.java:63)
      	at org.forgerock.openam.rest.fluent.CrestLoggingFilter.filterCreate(CrestLoggingFilter.java:84)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleCreate(FilterChain.java:63)
      	at org.forgerock.openam.rest.ContextFilter.filterCreate(ContextFilter.java:55)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleCreate(FilterChain.java:63)
      	at org.forgerock.openam.rest.AuthenticationEnforcer.filterCreate(AuthenticationEnforcer.java:138)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleCreate(FilterChain.java:63)
      	at org.forgerock.json.resource.FilterChain.handleCreate(FilterChain.java:228)
      	at org.forgerock.json.resource.Router.handleCreate(Router.java:264)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleCreate(FilterChain.java:65)
      	at org.forgerock.openam.rest.ContextFilter.filterCreate(ContextFilter.java:55)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleCreate(FilterChain.java:63)
      	at org.forgerock.json.resource.FilterChain.handleCreate(FilterChain.java:228)
      	at org.forgerock.json.resource.InternalConnection.createAsync(InternalConnection.java:40)
      	at org.forgerock.json.resource.http.RequestRunner.visitCreateRequest(RequestRunner.java:162)
      	at org.forgerock.json.resource.http.RequestRunner.visitCreateRequest(RequestRunner.java:82)
      	at org.forgerock.json.resource.Requests$CreateRequestImpl.accept(Requests.java:250)
      	at org.forgerock.json.resource.http.RequestRunner.handleResult(RequestRunner.java:128)
      	at org.forgerock.json.resource.http.HttpAdapter$$Lambda$1237/2026833368.apply(Unknown Source)
      	at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:252)
      	at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:241)
      	at org.forgerock.json.resource.http.HttpAdapter.doRequest(HttpAdapter.java:712)
      	at org.forgerock.json.resource.http.HttpAdapter.doCreate(HttpAdapter.java:580)
      	at org.forgerock.json.resource.http.HttpAdapter.handle(HttpAdapter.java:272)
      	at org.forgerock.http.handler.Handlers$HandlerDescribableAsDescribableHandler.handle(Handlers.java:146)
      	at org.forgerock.http.filter.OptionsFilter.filter(OptionsFilter.java:69)
      	at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:53)
      	at org.forgerock.http.routing.Router.handle(Router.java:100)
      	at org.forgerock.openam.rest.RealmContextFilter.filter(RealmContextFilter.java:80)
      	at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:53)
      	at org.forgerock.http.routing.Router.handle(Router.java:100)
      	at org.forgerock.http.routing.Router.handle(Router.java:100)
      	at org.forgerock.openam.rest.RealmRoutingFactory$ChfRealmRouter.handle(RealmRoutingFactory.java:139)
      	at org.forgerock.http.routing.Router.handle(Router.java:100)
      	at org.forgerock.openam.rest.RealmRoutingFactory$ChfRealmRouter.handle(RealmRoutingFactory.java:139)
      	at org.forgerock.http.handler.Handlers$UndescribedAsDescribableHandler.handle(Handlers.java:179)
      	at org.forgerock.openam.rest.RealmRoutingFactory$HostnameFilter.filter(RealmRoutingFactory.java:116)
      	at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:53)
      	at org.forgerock.http.routing.Router.handle(Router.java:100)
      	at org.forgerock.http.routing.Router.handle(Router.java:100)
      	at org.forgerock.openam.rest.CsrfFilter.filter(CsrfFilter.java:93)
      	at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:53)
      	at org.forgerock.http.routing.ResourceApiVersionRoutingFilter.filter(ResourceApiVersionRoutingFilter.java:59)
      	at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:53)
      	at org.forgerock.caf.authentication.framework.AuthenticationFramework.grantAccess(AuthenticationFramework.java:188)
      	at org.forgerock.caf.authentication.framework.AuthenticationFramework.lambda$onValidateRequestSuccess$1(AuthenticationFramework.java:181)
      	at org.forgerock.caf.authentication.framework.AuthenticationFramework$$Lambda$1233/944429371.apply(Unknown Source)
      	at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:252)
      	at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:241)
      	at org.forgerock.caf.authentication.framework.AuthenticationFramework.validateRequest(AuthenticationFramework.java:144)
      	at org.forgerock.caf.authentication.framework.AuthenticationFramework.processMessage(AuthenticationFramework.java:134)
      	at org.forgerock.caf.authentication.framework.AuthenticationFilter.filter(AuthenticationFilter.java:84)
      	at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:53)
      	at org.forgerock.openam.http.GuiceHandler.handle(GuiceHandler.java:51)
      	at org.forgerock.openam.http.HttpRoute$6.handle(HttpRoute.java:206)
      	at org.forgerock.http.routing.Router.handle(Router.java:100)
      	at org.forgerock.openam.dpro.session.ProofOfPossessionTokenFilter.filter(ProofOfPossessionTokenFilter.java:88)
      	at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:53)
      	at org.forgerock.http.swagger.OpenApiRequestFilter.filter(OpenApiRequestFilter.java:62)
      	at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:53)
      	at org.forgerock.openam.http.ApiDescriptorFilter.filter(ApiDescriptorFilter.java:139)
      	at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:53)
      	at org.forgerock.openam.http.OpenAMHttpApplication$1.filter(OpenAMHttpApplication.java:74)
      	at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:53)
      	at org.forgerock.http.filter.TransactionIdInboundFilter.filter(TransactionIdInboundFilter.java:75)
      	at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:53)
      	at org.forgerock.http.servlet.HttpFrameworkServlet.service(HttpFrameworkServlet.java:258)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      	at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
      	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
      	at io.undertow.websockets.jsr.JsrWebSocketFilter.doFilter(JsrWebSocketFilter.java:130)
      	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
      	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
      	at org.forgerock.openam.rest.ProtocolVersionFilter.doFilter(ProtocolVersionFilter.java:65)
      	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
      	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
      	at org.forgerock.openam.validation.ResponseValidationFilter.doFilter(ResponseValidationFilter.java:36)
      	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
      	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
      	at org.forgerock.openam.headers.SetHeadersFilter.doFilter(SetHeadersFilter.java:80)
      	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
      	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
      	at com.sun.identity.setup.AMSetupFilter.doFilter(AMSetupFilter.java:111)
      	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
      	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
      	at org.forgerock.openam.audit.context.AuditContextFilter.doFilter(AuditContextFilter.java:46)
      	at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
      	at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
      	at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
      	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
      	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
      	at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
      	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
      	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
      	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
      	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
      	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
      	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
      	at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
      	at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
      	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      	at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
      	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      	at org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
      	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
      	at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
      	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
      	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
      	at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
      	at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
      	at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
      	at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction$$Lambda$865/1080683934.call(Unknown Source)
      	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
      	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$866/291554617.call(Unknown Source)
      	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
      	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$866/291554617.call(Unknown Source)
      	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
      	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$866/291554617.call(Unknown Source)
      	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
      	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$866/291554617.call(Unknown Source)
      	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
      	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
      	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
      	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:326)
      	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:812)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      "smIdmThreadPool":
      	at org.forgerock.openam.core.rest.sms.SmsRequestHandler.notifyDescriptorChange(SmsRequestHandler.java:957)
      	- waiting to lock <0x00000007706bcfd8> (a org.forgerock.openam.core.rest.sms.SmsRequestHandler)
      	at org.forgerock.openam.core.rest.sms.SmsRequestHandler.refreshServiceRoute(SmsRequestHandler.java:448)
      	at org.forgerock.openam.core.rest.sms.SmsRequestHandler.objectChanged(SmsRequestHandler.java:411)
      	at com.sun.identity.sm.SMSNotificationManager.sendNotifications(SMSNotificationManager.java:281)
      	at com.sun.identity.sm.SMSNotificationManager$LocalChangeNotifcationTask.run(SMSNotificationManager.java:357)
      	at org.forgerock.openam.audit.context.AuditRequestContextPropagatingRunnable.run(AuditRequestContextPropagatingRunnable.java:34)
      	at com.iplanet.am.util.ThreadPool$WorkerThread.run(ThreadPool.java:312)
      "smIdmThreadPool":
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x000000076ee6a8c8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
      	at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
      	at org.forgerock.openam.core.rest.sms.SmsRequestHandler.api(SmsRequestHandler.java:906)
      	at org.forgerock.openam.core.rest.sms.SmsRequestHandler.api(SmsRequestHandler.java:121)
      	at org.forgerock.services.routing.AbstractRouter.buildApi(AbstractRouter.java:353)
      	at org.forgerock.services.routing.AbstractRouter.notifyDescriptorChange(AbstractRouter.java:334)
      	at org.forgerock.openam.core.rest.sms.SmsRequestHandler.notifyDescriptorChange(SmsRequestHandler.java:963)
      	- locked <0x00000007706bcfd8> (a org.forgerock.openam.core.rest.sms.SmsRequestHandler)
      	at org.forgerock.openam.core.rest.sms.SmsRequestHandler.refreshServiceRoute(SmsRequestHandler.java:440)
      	at org.forgerock.openam.core.rest.sms.SmsRequestHandler.objectChanged(SmsRequestHandler.java:411)
      	at com.sun.identity.sm.SMSNotificationManager.sendNotifications(SMSNotificationManager.java:281)
      	at com.sun.identity.sm.SMSNotificationManager$LocalChangeNotifcationTask.run(SMSNotificationManager.java:357)
      	at org.forgerock.openam.audit.context.AuditRequestContextPropagatingRunnable.run(AuditRequestContextPropagatingRunnable.java:34)
      	at com.iplanet.am.util.ThreadPool$WorkerThread.run(ThreadPool.java:312)
      

      How to reproduce the issue

      Run the amster shell tests several times

      Expected behaviour

      There is no deadlock.

      Current behaviour

      There is.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                peter.major Peter Major [X] (Inactive)
                Reporter:
                peter.major Peter Major [X] (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: