Uploaded image for project: 'OpenIDM'
  1. OpenIDM
  2. OPENIDM-13802

With a large data set, delete managed users may fail with a NPE

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 7.0.0
    • Fix Version/s: None
    • Labels:
    • Environment:
      IDM latest master runs in a Google Kubenetes cluster
    • Target Version/s:

      Description

      When doing vertical scalability test noticed that with 500K managed users created in one IDM pod, delete managed user test may fail with a 500 error, and IDM log has this NPE:

      WARNING: Resource exception: 500 Internal Server Error: "Internal Server Error"
      org.forgerock.json.resource.InternalServerErrorException: Internal Server Error
              at org.forgerock.openidm.managed.ManagedObjectSet.deleteInstance(ManagedObjectSet.java:1235)
              at org.forgerock.json.resource.InterfaceCollectionInstance.handleDelete(InterfaceCollectionInstance.java:42)
              at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:74)
              at org.forgerock.json.resource.Resources$CollectionInstanceIdContextFilter.filterDelete(Resources.java:514)
              at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:72)
              at org.forgerock.json.resource.FilterChain.handleDelete(FilterChain.java:237)
              at org.forgerock.json.resource.Router.handleDelete(Router.java:279)
              at org.forgerock.openidm.managed.ManagedObjectService$ManagedObjectSetRequestHandler.handleDelete(ManagedObjectService.java:272)
              at org.forgerock.json.resource.FilterChain.handleDelete(FilterChain.java:237)
              at org.forgerock.json.resource.Router.handleDelete(Router.java:279)
              at org.forgerock.openidm.managed.ManagedObjectService.handleDelete(ManagedObjectService.java:455)
              at org.forgerock.json.resource.Router.handleDelete(Router.java:279)
              at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:74)
              at org.forgerock.openidm.selfservice.impl.HashKbaFilter.filterDelete(HashKbaFilter.java:110)
              at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:72)
              at org.forgerock.json.resource.ResourceApiVersionRoutingFilter.filterDelete(ResourceApiVersionRoutingFilter.java:55)
              at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:72)
              at org.forgerock.json.resource.FilterChain.handleDelete(FilterChain.java:237)
              at org.forgerock.json.resource.Router.handleDelete(Router.java:279)
              at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:74)
              at org.forgerock.openidm.authz.DelegatedAdminFilter.lambda$filterDelete$2(DelegatedAdminFilter.java:227)
              at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:263)
              at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:227)
              at org.forgerock.openidm.authz.DelegatedAdminFilter.filterRequest(DelegatedAdminFilter.java:282)
              at org.forgerock.openidm.authz.DelegatedAdminFilter.filterDelete(DelegatedAdminFilter.java:226)
              at org.forgerock.json.resource.Filters$ConditionalFilter.filterDelete(Filters.java:62)
              at org.forgerock.json.resource.Filters$ConditionalFilter.filterDelete(Filters.java:62)
              at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:72)
              at org.forgerock.json.resource.Filters$ConditionalFilter.filterDelete(Filters.java:64)
              at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:72)
              at org.forgerock.openidm.authz.DelegatedAdminFilter.lambda$filterDelete$2(DelegatedAdminFilter.java:227)
              at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:263)
              at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:227)
              at org.forgerock.openidm.authz.DelegatedAdminFilter.filterRequest(DelegatedAdminFilter.java:282)
              at org.forgerock.openidm.authz.DelegatedAdminFilter.filterDelete(DelegatedAdminFilter.java:226)
              at org.forgerock.json.resource.Filters$ConditionalFilter.filterDelete(Filters.java:62)
              at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:72)
              at org.forgerock.openidm.audit.filter.AuditFilter.lambda$filterDelete$2(AuditFilter.java:127)
      at org.forgerock.openidm.audit.filter.AuditFilter.logAuditAccessEntry(AuditFilter.java:171)
              at org.forgerock.openidm.audit.filter.AuditFilter.filterDelete(AuditFilter.java:127)
              at org.forgerock.openidm.router.filter.MutableFilterDecorator.filterDelete(MutableFilterDecorator.java:78)
              at org.forgerock.json.resource.Filters$ConditionalFilter.filterDelete(Filters.java:62)
              at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:72)
              at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$3.lambda$filterDelete$2(ServletConnectionFactory.java:385)
              at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$3.handleRequestWithLogging(ServletConnectionFactory.java:430)
              at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$3.filterDelete(ServletConnectionFactory.java:385)
              at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:72)
              at org.forgerock.openidm.router.filter.PassthroughFilter.filterDelete(PassthroughFilter.java:54)
              at org.forgerock.openidm.router.filter.MutableFilterDecorator.filterDelete(MutableFilterDecorator.java:78)
              at org.forgerock.openidm.router.filter.MutableFilterDecorator.filterDelete(MutableFilterDecorator.java:78)
              at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:72)
              at org.forgerock.openidm.router.filter.PassthroughFilter.filterDelete(PassthroughFilter.java:54)
              at org.forgerock.openidm.router.filter.MutableFilterDecorator.filterDelete(MutableFilterDecorator.java:78)
              at org.forgerock.json.resource.Filters$ConditionalFilter.filterDelete(Filters.java:62)
              at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:72)
              at org.forgerock.json.resource.FilterChain.handleDelete(FilterChain.java:237)
              at org.forgerock.json.resource.InternalConnection.deleteAsync(InternalConnection.java:47)
              at org.forgerock.json.resource.AbstractConnectionWrapper.deleteAsync(AbstractConnectionWrapper.java:119)
              at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$InternalConnectionWrapper.lambda$deleteAsync$7(ServletConnectionFactory.java:329)
              at org.forgerock.openidm.metrics.MetricsCollector.time(MetricsCollector.java:112)
              at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$InternalConnectionWrapper.time(ServletConnectionFactory.java:286)
              at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$InternalConnectionWrapper.deleteAsync(ServletConnectionFactory.java:329)
              at org.forgerock.json.resource.http.RequestRunner.visitDeleteRequest(RequestRunner.java:195)
              at org.forgerock.json.resource.http.RequestRunner.visitDeleteRequest(RequestRunner.java:84)
              at org.forgerock.json.resource.Requests$DeleteRequestImpl.accept(Requests.java:313)
              at org.forgerock.json.resource.http.RequestRunner.handleResult(RequestRunner.java:132)
              at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:263)
              at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:252)
              at org.forgerock.json.resource.http.HttpAdapter.doRequest(HttpAdapter.java:739)
              at org.forgerock.json.resource.http.HttpAdapter.doDelete(HttpAdapter.java:347)
              at org.forgerock.json.resource.http.HttpAdapter.handle(HttpAdapter.java:303)
              at org.forgerock.http.handler.Handlers$HandlerDescribableAsDescribableHandler.handle(Handlers.java:146)
              at org.forgerock.http.filter.OptionsFilter.filter(OptionsFilter.java:69)
              at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:53)
              at org.forgerock.http.routing.Router.handle(Router.java:100)
              at org.forgerock.openidm.auth.ProfileEnhancementCheckFilter.filter(ProfileEnhancementCheckFilter.java:145)
              at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:53)
              at org.forgerock.openidm.auth.LoginCountFilter.filter(LoginCountFilter.java:75)
      at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:53)
              at org.forgerock.caf.authentication.framework.AuthenticationFramework.grantAccess(AuthenticationFramework.java:188)
              at org.forgerock.caf.authentication.framework.AuthenticationFramework.lambda$onValidateRequestSuccess$1(AuthenticationFramework.java:181)
              at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:263)
              at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:252)
              at org.forgerock.caf.authentication.framework.AuthenticationFramework.validateRequest(AuthenticationFramework.java:144)
              at org.forgerock.caf.authentication.framework.AuthenticationFramework.processMessage(AuthenticationFramework.java:134)
              at org.forgerock.caf.authentication.framework.AuthenticationFilter.filter(AuthenticationFilter.java:85)
              at org.forgerock.openidm.auth.AuthFilterWrapper.filter(AuthFilterWrapper.java:85)
              at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:53)
              at org.forgerock.http.filter.TransactionIdInboundFilter.filter(TransactionIdInboundFilter.java:86)
              at org.forgerock.http.handler.Handlers$1.handle(Handlers.java:53)
              at org.forgerock.http.servlet.HttpFrameworkServlet.service(HttpFrameworkServlet.java:265)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
              at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
              at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:51)
              at jdk.internal.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
              at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
              at java.base/java.lang.reflect.Method.invoke(Unknown Source)
              at org.forgerock.openidm.servletregistration.impl.ServletRegistrationSingleton$FilterProxy.invoke(ServletRegistrationSingleton.java:306)
              at com.sun.proxy.$Proxy51.doFilter(Unknown Source)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
              at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:311)
              at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:265)
              at jdk.internal.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
              at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
              at java.base/java.lang.reflect.Method.invoke(Unknown Source)
              at org.forgerock.openidm.servletregistration.impl.ServletRegistrationSingleton$FilterProxy.invoke(ServletRegistrationSingleton.java:306)
              at com.sun.proxy.$Proxy51.doFilter(Unknown Source)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
              at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
              at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
              at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
              at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
              at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
              at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
              at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340)at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:293)
              at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
              at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
              at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
              at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
              at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
              at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
              at org.eclipse.jetty.server.Server.handle(Server.java:503)
              at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
              at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
              at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
              at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
              at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
              at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
              at java.base/java.lang.Thread.run(Unknown Source)
      Caused by: java.lang.NullPointerException
              at org.forgerock.openidm.meta.impl.MetaBehaviorImpl.lambda$null$9(MetaBehaviorImpl.java:206)
              at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:263)
              at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:227)
              at org.forgerock.openidm.meta.impl.MetaBehaviorImpl.lambda$null$12(MetaBehaviorImpl.java:205)
              at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:263)
              at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:227)
              at org.forgerock.openidm.managed.ManagedObjectSet.deleteInstance(ManagedObjectSet.java:1204)
              ... 140 more
      

      Note:
      It happened three times among four tests(with IDM CPU limits set as 12500, 10000 and 7500, not with limit 15000), attached are the conf file used in Pyrock test and IDM logs.
      To reproduce using pyrock:
      1. Clone test repo lodestar from ssh://git@stash.forgerock.org:7999/qa/lodestar.git
      2. Cluster is set as Gatling, DS and IDM are deployed to different nodes.
      3. Put the attached config file as lodestar/pyrock/tests/stress/idm/simple_managed_users_ds_shared_repo/conf.yaml
      4. Under lodestar/pyrock, run test

      ./run.py simple_managed_users_ds_shared_repo
      

        Attachments

        1. conf.yaml
          5 kB
        2. idm-log
          87 kB

          Activity

            People

            Assignee:
            brmiller Brendan Miller
            Reporter:
            Tinghua.Xu Tinghua Xu
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: