Uploaded image for project: 'OpenIDM'
  1. OpenIDM
  2. OPENIDM-7407

Backport OPENIDM-6923: IDM stops responding during create with managed role

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: OpenIDM 4.5.0
    • Fix Version/s: OpenIDM 4.5.1
    • Labels:
      None
    • Environment:
      CentOS Linux release 7.2, mysql 5.5

      Description

      Adding 100K managed objects with REST create put (example below) eventually stops responding at some point.. The tps here is 10 create/ops/sec, so not a high load. It's not possible to connect to openidm at this point.

      HTTP request:
      PUT http://morbier.internal.forgerock.com:8080/openidm/managed/user/xjgs482516
      headers=
      X-OpenIDM-Password: openidm-admin
      Connection: close
      X-OpenIDM-Username: openidm-admin
      Accept: application/json
      Content-Type: application/json
      If-None-Match: *
      User-Agent: Robot/Gatling
      Content-Length: 347
      Host: morbier.internal.forgerock.com:8080
      compositeByteData={
      "givenName" : "givenname",
      "sn" : "Keane",
      "mail" : "xjgs482516@example.com",
      "telephoneNumber" : "444-444-4444",
      "password" : "Passw0rd",
      "displayName" : "Nickie.Keane",
      "description" : "Managed User",
      "userName" : "xjgs482516",
      "roles" : [

      Unknown macro: {"_ref"}

      ]
      }
      =========================
      HTTP response:

      Looking at jstack there are 55 threads blocked in:

      "qtp1616933873-2308" #2308 prio=5 os_prio=0 tid=0x00007f77fc002800 nid=0x2a093 waiting for monitor entry [0x00007f76649c5000]
      java.lang.Thread.State: BLOCKED (on object monitor)
      at com.fasterxml.jackson.core.sym.CharsToNameCanonicalizer.makeChild(CharsToNameCanonicalizer.java:308)

      • waiting to lock <0x00000000c1561490> (a com.fasterxml.jackson.core.sym.CharsToNameCanonicalizer)
        at com.fasterxml.jackson.core.JsonFactory._createParser(JsonFactory.java:1300)
        at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:889)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2852)
        at org.forgerock.openidm.repo.jdbc.impl.GenericQueryResultMapper.mapQueryToObject(GenericTableHandler.java:771)
        at org.forgerock.openidm.repo.jdbc.impl.query.TableQueries.query(TableQueries.java:340)
        at org.forgerock.openidm.repo.jdbc.impl.GenericTableHandler.query(GenericTableHandler.java:653)
        at org.forgerock.openidm.repo.jdbc.impl.JDBCRepoService.query(JDBCRepoService.java:707)
        at org.forgerock.openidm.repo.jdbc.impl.JDBCRepoService.handleQuery(JDBCRepoService.java:605)
        at org.forgerock.json.resource.Router.handleQuery(Router.java:314)
        at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:103)
        at org.forgerock.json.resource.Filters$ConditionalFilter.filterQuery(Filters.java:92)
        at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:101)
        at org.forgerock.json.resource.Filters$ConditionalFilter.filterQuery(Filters.java:92)
        at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:101)
        at org.forgerock.json.resource.Filters$ConditionalFilter.filterQuery(Filters.java:92)
        at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:101)
        at org.forgerock.openidm.audit.filter.AuditFilter$5.apply(AuditFilter.java:160)
        at org.forgerock.openidm.audit.filter.AuditFilter$5.apply(AuditFilter.java:157)
        at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:255)
        at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:244)
        at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:223)
        etc..

      and 131 parked in TIMED_WAITING in getting a connection in the pool.

      "qtp1616933873-2310" #2310 prio=5 os_prio=0 tid=0x00007f76f4010800 nid=0x2a095 waiting on condition [0x00007f76647c2000]
      java.lang.Thread.State: TIMED_WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)

      • parking to wait for <0x00000000c17da168> (a com.zaxxer.hikari.util.QueuedSequenceSynchronizer$Synchronizer)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedLongSynchronizer.doAcquireSharedNanos(AbstractQueuedLongSynchronizer.java:815)
        at java.util.concurrent.locks.AbstractQueuedLongSynchronizer.tryAcquireSharedNanos(AbstractQueuedLongSynchronizer.java:1106)
        at com.zaxxer.hikari.util.QueuedSequenceSynchronizer.waitUntilSequenceExceeded(QueuedSequenceSynchronizer.java:92)
        at com.zaxxer.hikari.util.ConcurrentBag.borrow(ConcurrentBag.java:169)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:165)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:147)
        at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:83)
        at org.forgerock.openidm.repo.jdbc.impl.JDBCRepoService.getConnection(JDBCRepoService.java:830)
        at org.forgerock.openidm.repo.jdbc.impl.JDBCRepoService.query(JDBCRepoService.java:703)
        at org.forgerock.openidm.repo.jdbc.impl.JDBCRepoService.handleQuery(JDBCRepoService.java:605)
        etc

      Full stack is attached.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              cgdrake Chris Drake
              Reporter:
              cgdrake Chris Drake
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: