Uploaded image for project: 'OpenIDM'
  1. OpenIDM
  2. OPENIDM-13532

Under load IDM container may fail to respond to requests

    Details

    • Target Version/s:
    • Story Points:
      3

      Description

      Using the simple managed users test from Jenny at https://stash.forgerock.org/projects/QA/repos/lodestar/pull-requests/298/overview, we see that IDM container may fail to respond to http requests under certain load.
      One request is to read managed users, the Gatling simulation has a concurrency 40 and test duration 900 seconds, here is one instance of the following failure(from Gatling report):

      19:57:25.505 [WARN ] i.g.h.a.ResponseProcessor - Request 'Managed user Read' failed: status.find.in(200,304,201,202,203,204,205,206,207,208,209), but actually found 404
      19:57:25.520 [DEBUG] i.g.h.a.ResponseProcessor -
      >>>>>>>>>>>>>>>>>>>>>>>>>>
      Request:
      Managed user Read: KO status.find.in(200,304,201,202,203,204,205,206,207,208,209), but actually found 404
      =========================
      Session:
      Session(Managed user Read,14,Map(email -> H-M-11248@example.com, gatling.http.cache.dns -> io.gatling.http.resolver.ShuffleJdkNameResolver@67ebc807, 4db0359d-94a5-49e0-8b43-d92245c61463 -> 2261, username -> H-M-11248, id -> 626e36f2-11d0-4a48-8abf-7c7d56a05cfa, givenname -> Mbwozbno, timestamp.4db0359d-94a5-49e0-8b43-d92245c61463 -> 1562183630532, familyname -> Hrciso, gatling.http.cookies -> CookieJar(Map(CookieKey(route,tinghuaxu.iam.performance-team.forgeops.com,/openidm) -> StoredCookie(route=1562183631.056.10828.74002; path=/openidm; secure; HTTPOnly,true,false,1562183631136), CookieKey(session-jwt,tinghuaxu.iam.performance-team.forgeops.com,/) -> StoredCookie(session-jwt=eyJ0eXAiOiJKV1QiLCJraWQiOiJvcGVuaWRtLWp3dHNlc3Npb25obWFjLWtleSIsImN0eSI6IkpXVCIsImFsZyI6IkhTMjU2In0.ZXlKMGVYQWlPaUpLVjFRaUxDSnJhV1FpT2lKdmNHVnVhV1J0TFd4dlkyRnNhRzl6ZENJc0ltVnVZeUk2SWtFeE1qaERRa010U0ZNeU5UWWlMQ0poYkdjaU9pSlNVMEV4WHpVaWZRLk9mWlUxb2ROVTlqRWJRbkNrX0Uzd3hqcHR5MVAwT0JWcmNEVU5veDRndHNBSkdOYXdRa1F2cVpMM3YyODl3QmVJbXk4UDZFZ0FCc01sdHJ0SS1MUjc2Z2JIc3RCa09YTFd5dTJJRU5sdkNKMDkwY3piMXlCanF6VjZsVVZSckJGdWJtd0U0OHNONWtRdGhTTXI2SE16bG9xendoS1VTbk9rUVRLaTRIQnRnREFnSzQ3UTk1OGVTSkxrUWxjZTM5eTNXUjdZZGUzSzVjeVhVQkZZZDJnZzl6eUpKeFBoRkJPWlFnVGx0RTktaE1waGtFdUUyVE8zcFJ3ek9JUmRic19HZm9VZ2Y3a05yRzlJbDRMSGdPN1NoYXBRV04zUlhYZVZpWHhkU25BUFdfLXJndVVRRE5oaE12YWJfN1lmcDNVUlFHaEptSFN1ZHpSR2lFMW9kNjZBdy5mMGZ0Q2VhWDZ4MGt6ZzVXVzA4Uk5BLjBoek1YS2x3RV9Uem1LbGdnRlFmZkMwLVAtSlF0VWtiTWV6T2Y0Q1pRd05OZFd2VzhyMF9YWEpkQllLUnFlaVo1blhoRWtndHUyOUhVekJLa0tXRVpjc3pmelZabGVkSEdhcGVTeE5FNm94WTF0WnpFMC14bTlNMWE4QWd6QTFfbmdlVFBRb2VnYnhOMGRXVG5NUUxuYlJUTVpseElpcUVKd2VKVXZMQ09sbXgycGs2RUdXTzNtOXQ4UGtiZlBIcmNuR3g5QWdPdUJvSVlhT1NHRzZWa1RyVnF3MEk1RzBwamJFa2RXMzZzX3dLaVBxZDBzc3VYMll5TDJ4d21FczBvVXN3cTVIM2UtbVlVUFNBbFV0SGpkNEFIakNPYkxqSFVCRVJvYzdzYXJLUUVNanR1YWxISzF1ZXdWUkVsVkdhN091Y2stS3laelBxUUhGRmk3Zl8zcGNXeDBrZW05YmxQbkV2UFM0Rl80N0tyRHR5VVZCYUFJN2FnOW5UNl9xNkRYZWNINmE1TUhVOXNVVVJvRENhQ0g4YU9TMTlobS1HbUFkeWhzdTZ6ZlYwMEVIaWg1NzZYYWExWnNPN2s2WGd5ME0xeVRTUVpQZEYwUXAyX0JSV3E0VWhaUTJaaDltVzZSSmZsY1VMNUJMRmxabi1tN1ZJT0IzX1ZxOGI3ejhtMGFpakREMkVnaFBGQ2daRmxwNE84TFNTSlljalg1TlZJUmJoX0oxNG9WMEFpVmVnd2JrRVpUZk9tR1B4WVFnZ3FzZTdGOVhrd1dsNFFYUWt4ZGh2Y216dVY0bmo4V3RRd1BMTktHcXpkenZTQnhDeXotNWE0WVlsLXlfOTRCMThRM3FGNFdGbjNLamZ1eHhkX3lFTlpRMVB5aHhoZHdneHlsYjU3ekgzd3hUeEFQbzVpYmR1Z2lZU1JYX0pKLTJyQUV3X1FYYlNjWVFMSkp4RnFBSTd4cnlZMHdIVUY3TXZSZ09lTDU2Q1lvWXRVLXQ5TVkxT2RaQVZmejBVcVdqX29jd1VxT1ZtQlY4VE5TS3pfU25qbHJNd0Y3djZpbWhUTmdIbzc5THdBMzdWbHllNjExR2VmTGhHRjh3bWs4NzEudDA2ckZwXzlpSUR6b3Q5WmZGQ3BRQQ.9Ka3aB2Yl9n9a3zz_XWecigLtcryPAsUVdyVtT_3IAQ; path=/; HTTPOnly,true,false,1562183813977)))),1562183630532,234,KO,List(ExitAsapLoopBlock(4db0359d-94a5-49e0-8b43-d92245c61463,io.gatling.core.session.package$RichExpression$$$Lambda$379/1268974304@1e7135e7,io.gatling.core.action.Exit@7881702c)),io.gatling.core.protocol.ProtocolComponentsRegistry$$Lambda$440/1482429406@1825de23)=========================
      HTTP request:
      GET https://tinghuaxu.iam.performance-team.forgeops.com:443/openidm/managed/user/626e36f2-11d0-4a48-8abf-7c7d56a05cfa
      headers=
      Content-Type: application/json
      User-Agent: Robot/Gatling
      X-OpenIDM-Username: openidm-admin
      X-OpenIDM-Password: openidm-admin
      Cookie: route=1562183631.056.10828.74002; session-jwt=eyJ0eXAiOiJKV1QiLCJraWQiOiJvcGVuaWRtLWp3dHNlc3Npb25obWFjLWtleSIsImN0eSI6IkpXVCIsImFsZyI6IkhTMjU2In0.ZXlKMGVYQWlPaUpLVjFRaUxDSnJhV1FpT2lKdmNHVnVhV1J0TFd4dlkyRnNhRzl6ZENJc0ltVnVZeUk2SWtFeE1qaERRa010U0ZNeU5UWWlMQ0poYkdjaU9pSlNVMEV4WHpVaWZRLk9mWlUxb2ROVTlqRWJRbkNrX0Uzd3hqcHR5MVAwT0JWcmNEVU5veDRndHNBSkdOYXdRa1F2cVpMM3YyODl3QmVJbXk4UDZFZ0FCc01sdHJ0SS1MUjc2Z2JIc3RCa09YTFd5dTJJRU5sdkNKMDkwY3piMXlCanF6VjZsVVZSckJGdWJtd0U0OHNONWtRdGhTTXI2SE16bG9xendoS1VTbk9rUVRLaTRIQnRnREFnSzQ3UTk1OGVTSkxrUWxjZTM5eTNXUjdZZGUzSzVjeVhVQkZZZDJnZzl6eUpKeFBoRkJPWlFnVGx0RTktaE1waGtFdUUyVE8zcFJ3ek9JUmRic19HZm9VZ2Y3a05yRzlJbDRMSGdPN1NoYXBRV04zUlhYZVZpWHhkU25BUFdfLXJndVVRRE5oaE12YWJfN1lmcDNVUlFHaEptSFN1ZHpSR2lFMW9kNjZBdy5mMGZ0Q2VhWDZ4MGt6ZzVXVzA4Uk5BLjBoek1YS2x3RV9Uem1LbGdnRlFmZkMwLVAtSlF0VWtiTWV6T2Y0Q1pRd05OZFd2VzhyMF9YWEpkQllLUnFlaVo1blhoRWtndHUyOUhVekJLa0tXRVpjc3pmelZabGVkSEdhcGVTeE5FNm94WTF0WnpFMC14bTlNMWE4QWd6QTFfbmdlVFBRb2VnYnhOMGRXVG5NUUxuYlJUTVpseElpcUVKd2VKVXZMQ09sbXgycGs2RUdXTzNtOXQ4UGtiZlBIcmNuR3g5QWdPdUJvSVlhT1NHRzZWa1RyVnF3MEk1RzBwamJFa2RXMzZzX3dLaVBxZDBzc3VYMll5TDJ4d21FczBvVXN3cTVIM2UtbVlVUFNBbFV0SGpkNEFIakNPYkxqSFVCRVJvYzdzYXJLUUVNanR1YWxISzF1ZXdWUkVsVkdhN091Y2stS3laelBxUUhGRmk3Zl8zcGNXeDBrZW05YmxQbkV2UFM0Rl80N0tyRHR5VVZCYUFJN2FnOW5UNl9xNkRYZWNINmE1TUhVOXNVVVJvRENhQ0g4YU9TMTlobS1HbUFkeWhzdTZ6ZlYwMEVIaWg1NzZYYWExWnNPN2s2WGd5ME0xeVRTUVpQZEYwUXAyX0JSV3E0VWhaUTJaaDltVzZSSmZsY1VMNUJMRmxabi1tN1ZJT0IzX1ZxOGI3ejhtMGFpakREMkVnaFBGQ2daRmxwNE84TFNTSlljalg1TlZJUmJoX0oxNG9WMEFpVmVnd2JrRVpUZk9tR1B4WVFnZ3FzZTdGOVhrd1dsNFFYUWt4ZGh2Y216dVY0bmo4V3RRd1BMTktHcXpkenZTQnhDeXotNWE0WVlsLXlfOTRCMThRM3FGNFdGbjNLamZ1eHhkX3lFTlpRMVB5aHhoZHdneHlsYjU3ekgzd3hUeEFQbzVpYmR1Z2lZU1JYX0pKLTJyQUV3X1FYYlNjWVFMSkp4RnFBSTd4cnlZMHdIVUY3TXZSZ09lTDU2Q1lvWXRVLXQ5TVkxT2RaQVZmejBVcVdqX29jd1VxT1ZtQlY4VE5TS3pfU25qbHJNd0Y3djZpbWhUTmdIbzc5THdBMzdWbHllNjExR2VmTGhHRjh3bWs4NzEudDA2ckZwXzlpSUR6b3Q5WmZGQ3BRQQ.9Ka3aB2Yl9n9a3zz_XWecigLtcryPAsUVdyVtT_3IAQ
      Host: tinghuaxu.iam.performance-team.forgeops.com
      Accept: */*
      =========================
      HTTP response:
      status=
      404 Not Found
      headers=
      Server: nginx/1.15.10
      Date: Wed, 03 Jul 2019 19:57:25 GMT
      Content-Type: application/json;charset=utf-8
      Content-Length: 162
      Connection: keep-alive
      Cache-Control: no-cache
      Content-API-Version: protocol=2.1,resource=1.0
      X-Content-Type-Options: nosniff
      Strict-Transport-Security: max-age=15724800; includeSubDomains
      
      body=
      {"code":404,"reason":"Not Found","message":"No Results Returned: The search request succeeded but did not return any search result entries when one was expected"}
      <<<<<<<<<<<<<<<<<<<<<<<<<
      

      When manually run curl command on the managed user, the user is actually there:

      curl -k  -H "X-OpenIDM-Username: openidm-admin" -H "X-OpenIDM-Pasiam.performance-team.forgeops.com:443/openidm/managed/user/626e36f2-11d0-4a48-8abf-7c7d56a05cfa |python -mjson.tool
        % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                       Dload  Upload   Total   Spent    Left  Speed
      100   296  100   296    0     0    161      0  0:00:01  0:00:01 --:--:--   161
      {
          "_id": "626e36f2-11d0-4a48-8abf-7c7d56a05cfa",
          "_rev": "00000000ad033761",
          "accountStatus": "active",
          "description": "Managed User",
          "effectiveAssignments": [],
          "effectiveRoles": [],
          "givenName": "Mbwozbno",
          "mail": "H-M-11248@example.com",
          "sn": "Hrciso",
          "telephoneNumber": "444-444-4444",
          "userName": "H-M-11248"
      }
      

      Here is an instance with create managed user using POST:

      ================================================================================
      
      20:34:42.873 [WARN ] i.g.h.a.ResponseProcessor - Request 'Managed user create put' failed: status.find.in(200,304,201,202,203,204,205,206,207,208,209), but actually found 400
      20:34:42.886 [DEBUG] i.g.h.a.ResponseProcessor - 
      >>>>>>>>>>>>>>>>>>>>>>>>>>
      Request:
      Managed user create put: KO status.find.in(200,304,201,202,203,204,205,206,207,208,209), but actually found 400
      =========================
      Session:
      Session(Managed user create put,4,Map(city -> city1, a36f95dd-3930-4a11-96a3-d6c99bda2d37 -> 2270, email -> H-T-276933@example.com, gatling.http.cache.dns -> io.gatling.http.resolver.ShuffleJdkNameResolver@4f8d42e8, username -> H-T-276933, description -> This is the description for Hpjihu Tbnjazlt, country -> usa, fullname -> Hpjihu Tbnjazlt, roles -> [], givenname -> Tbnjazlt, manager -> [], postalcode -> 27195, timestamp.a36f95dd-3930-4a11-96a3-d6c99bda2d37 -> 1562185700383, familyname -> Hpjihu, gatling.http.cookies -> CookieJar(Map(CookieKey(route,tinghuaxu.iam.performance-team.forgeops.com,/openidm) -> StoredCookie(route=1562185700.668.10826.450737; path=/openidm; secure; HTTPOnly,true,false,1562185700819), CookieKey(session-jwt,tinghuaxu.iam.performance-team.forgeops.com,/) -> StoredCookie(session-jwt=eyJ0eXAiOiJKV1QiLCJraWQiOiJvcGVuaWRtLWp3dHNlc3Npb25obWFjLWtleSIsImN0eSI6IkpXVCIsImFsZyI6IkhTMjU2In0.ZXlKMGVYQWlPaUpLVjFRaUxDSnJhV1FpT2lKdmNHVnVhV1J0TFd4dlkyRnNhRzl6ZENJc0ltVnVZeUk2SWtFeE1qaERRa010U0ZNeU5UWWlMQ0poYkdjaU9pSlNVMEV4WHpVaWZRLllWWWZGTmNNWVR5RlMteXB6Q3ViVjFPa2Y3RjVIZ2RQay0zYWNNazF0TkVGV3RUaVpCUXVNMFVxYU9fc0pTNXhtNHZCZWRYbDcwR3FuS01TV1VlQWFrc0dRQ1Q0LUZtMmdkYUYxX29oQTlsMTF5OVJKYm82RnFRaG44VlpYRXR4YjdNYnJscTdMRTA4LUk2dTBiUDVHbUloMXdycWhTc1hfRTA2VDRCcGtIUENEdUVXUUNtcHE5QUttN2ZDMkk0VVEyUUhWcnNTMUFDeWlqNXg3Tkw1UmhvTEpBQkZrbjhWTWRlZzZqSDQweHM0VkpzSTRVOXV6bUFxVlhuQU9QN2w5X0U1dHlnc3JSNGQyaWxXMXJKVW9jMy1lclBLOVlRS2xqZEY1MWRKQnhUNUdPTVA3Zmk2eGl0Vm9LY3VQazFicUJuNXhQX1A4OFJPMExfN0hTNkJoZy4yeHRjYkdSWlY1elBqOVROTmVtNUNRLmRlVFByNU1MWUNGZTlFTU9fS0NzQkpFMkt2TGptbnRRek9hbldwREtPTkUteEVjR2FFMGdmTXF6Rjd2enFqWGdTNlRyZ2twcHZLVkxpbzQ5Z3p4Yi1YbTdyVGp1S3IzQTI4TWtHV3EzRkVmR2lIb2NRUTNUQVVndWZvbW1jSWFaMkF1Z0ZjR2ZqQnhZeGRGTkp4VHZoMlZFb1Q2cTN5bkc0VFRiNzBzeWMzVzNoRlZ5TlRxaEhrRGdqZnVtMEZ6b2JycklwUmg4UjhSbkRTdHBvaVloT1FYOGVaVHM1c0pqd3lPVHhlSHBMTDVERjhBc3JfQm53eTdKYlE1S1pQN0RZTmZvaVdlQ0p4RkhtdDQ3VXg1VXpCSmlTQy00N3pPRTZtNWQ4NngtblRjTW5NLWwzWmw5N19GVnFodG0zZ3dpZ2tLNXd0VTlNLXE0aFdTang3Z1VIQTg4ZFJ5TGhsSXhrdXBkRmMzdXNrZEFXQ2IwN1JrWXkyLU1HWTlJNzRfSHlJb0h1M2ZLZ0dmT3JHS2VpSjg4RW9oX2Y5QUxZenZsaXlwalhpREQ0TmU4LUJOYlAwV3dGZ0dpZm5SdVd1TUY1Ymt4S21lN1l4VWpxWHhJcTZTbU9aaGcyd1JUVVpXQURvdVV5bzFSbGlsQXgxZWNjbU1jNGFoQndRR2g0UzU5UGZIQVdhQk1ZRkRxd0Y3cDdkaThjX3lQVFBiLVN6bjRCZnE5QnhFQjZEMGpnMEFDcDlBa3pZaGViUFFpUmEySS1sZ2t3WW5kQTZDNXZJclJqYWtqeE5KODFYa1ZsMmRCcWpZVG5PMTR2ZFZmWGVYWGdnX1RRUzBENmw5ZURIVFlJS0FQanlPcTJ5T0VRRDA4eV90RjJBVW1HQVloQ1p2ZHVaVWItNVdBMnZSN1RVM3Y1b3RBODhIYjctT0NEdUE1elZYUmxEaU1DM2FvM3haRXR1RU9CeVRXeVdQYnNsUUdwdUNoWDNEcUxmNTJ5UkZmOTFzX0V1elpENy13a2pnNGRUN19zVVl1OWdaaFRVVVRvTzI2RzFBVkVPTDJQbDIxak1uZnJwV3JGWllPbkxnd2pGbldEOE15U0lKUFd0al8ucC10Y3p0SU1zX0taR0prSzBWV2VsUQ.kOFMcHAihN1TIOscUMvAvn1EI9OdwY4wr_alR4PRbKU; path=/; HTTPOnly,true,false,1562186066026))), password -> Pa_ssw0rd),1562185700383,341,KO,List(ExitAsapLoopBlock(a36f95dd-3930-4a11-96a3-d6c99bda2d37,io.gatling.core.session.package$RichExpression$$$Lambda$387/1108517064@62bf58c4,io.gatling.core.action.Exit@fcbb5e4)),io.gatling.core.protocol.ProtocolComponentsRegistry$$Lambda$444/1622863652@43174d34)
      =========================
      HTTP request:
      POST https://tinghuaxu.iam.performance-team.forgeops.com:443/openidm/managed/user/?_action=create
      headers=
      X-OpenIDM-Password: openidm-admin
      X-OpenIDM-Username: openidm-admin
      Accept: application/json
      Content-Type: application/json
      User-Agent: Robot/Gatling
      Cookie: route=1562185700.668.10826.450737; session-jwt=eyJ0eXAiOiJKV1QiLCJraWQiOiJvcGVuaWRtLWp3dHNlc3Npb25obWFjLWtleSIsImN0eSI6IkpXVCIsImFsZyI6IkhTMjU2In0.ZXlKMGVYQWlPaUpLVjFRaUxDSnJhV1FpT2lKdmNHVnVhV1J0TFd4dlkyRnNhRzl6ZENJc0ltVnVZeUk2SWtFeE1qaERRa010U0ZNeU5UWWlMQ0poYkdjaU9pSlNVMEV4WHpVaWZRLllWWWZGTmNNWVR5RlMteXB6Q3ViVjFPa2Y3RjVIZ2RQay0zYWNNazF0TkVGV3RUaVpCUXVNMFVxYU9fc0pTNXhtNHZCZWRYbDcwR3FuS01TV1VlQWFrc0dRQ1Q0LUZtMmdkYUYxX29oQTlsMTF5OVJKYm82RnFRaG44VlpYRXR4YjdNYnJscTdMRTA4LUk2dTBiUDVHbUloMXdycWhTc1hfRTA2VDRCcGtIUENEdUVXUUNtcHE5QUttN2ZDMkk0VVEyUUhWcnNTMUFDeWlqNXg3Tkw1UmhvTEpBQkZrbjhWTWRlZzZqSDQweHM0VkpzSTRVOXV6bUFxVlhuQU9QN2w5X0U1dHlnc3JSNGQyaWxXMXJKVW9jMy1lclBLOVlRS2xqZEY1MWRKQnhUNUdPTVA3Zmk2eGl0Vm9LY3VQazFicUJuNXhQX1A4OFJPMExfN0hTNkJoZy4yeHRjYkdSWlY1elBqOVROTmVtNUNRLmRlVFByNU1MWUNGZTlFTU9fS0NzQkpFMkt2TGptbnRRek9hbldwREtPTkUteEVjR2FFMGdmTXF6Rjd2enFqWGdTNlRyZ2twcHZLVkxpbzQ5Z3p4Yi1YbTdyVGp1S3IzQTI4TWtHV3EzRkVmR2lIb2NRUTNUQVVndWZvbW1jSWFaMkF1Z0ZjR2ZqQnhZeGRGTkp4VHZoMlZFb1Q2cTN5bkc0VFRiNzBzeWMzVzNoRlZ5TlRxaEhrRGdqZnVtMEZ6b2JycklwUmg4UjhSbkRTdHBvaVloT1FYOGVaVHM1c0pqd3lPVHhlSHBMTDVERjhBc3JfQm53eTdKYlE1S1pQN0RZTmZvaVdlQ0p4RkhtdDQ3VXg1VXpCSmlTQy00N3pPRTZtNWQ4NngtblRjTW5NLWwzWmw5N19GVnFodG0zZ3dpZ2tLNXd0VTlNLXE0aFdTang3Z1VIQTg4ZFJ5TGhsSXhrdXBkRmMzdXNrZEFXQ2IwN1JrWXkyLU1HWTlJNzRfSHlJb0h1M2ZLZ0dmT3JHS2VpSjg4RW9oX2Y5QUxZenZsaXlwalhpREQ0TmU4LUJOYlAwV3dGZ0dpZm5SdVd1TUY1Ymt4S21lN1l4VWpxWHhJcTZTbU9aaGcyd1JUVVpXQURvdVV5bzFSbGlsQXgxZWNjbU1jNGFoQndRR2g0UzU5UGZIQVdhQk1ZRkRxd0Y3cDdkaThjX3lQVFBiLVN6bjRCZnE5QnhFQjZEMGpnMEFDcDlBa3pZaGViUFFpUmEySS1sZ2t3WW5kQTZDNXZJclJqYWtqeE5KODFYa1ZsMmRCcWpZVG5PMTR2ZFZmWGVYWGdnX1RRUzBENmw5ZURIVFlJS0FQanlPcTJ5T0VRRDA4eV90RjJBVW1HQVloQ1p2ZHVaVWItNVdBMnZSN1RVM3Y1b3RBODhIYjctT0NEdUE1elZYUmxEaU1DM2FvM3haRXR1RU9CeVRXeVdQYnNsUUdwdUNoWDNEcUxmNTJ5UkZmOTFzX0V1elpENy13a2pnNGRUN19zVVl1OWdaaFRVVVRvTzI2RzFBVkVPTDJQbDIxak1uZnJwV3JGWllPbkxnd2pGbldEOE15U0lKUFd0al8ucC10Y3p0SU1zX0taR0prSzBWV2VsUQ.kOFMcHAihN1TIOscUMvAvn1EI9OdwY4wr_alR4PRbKU
      Content-Length: 281
      Host: tinghuaxu.iam.performance-team.forgeops.com
      compositeByteData={
      		    "userName" : "H-T-276933",
      		    "givenName" : "Tbnjazlt",
                  "sn" : "Hpjihu",
                  "mail" : "H-T-276933@example.com",
                  "telephoneNumber" : "444-444-4444",
                  "password" : "Pa_ssw0rd",
                  "description" : "Managed User"
              }
      =========================
      HTTP response:
      status=
      400 Bad Request
      headers= 
      Server: nginx/1.15.10
      Date: Wed, 03 Jul 2019 20:34:42 GMT
      Content-Type: application/json;charset=utf-8
      Content-Length: 120
      Connection: keep-alive
      Cache-Control: no-cache
      Content-API-Version: protocol=2.1,resource=1.0
      X-Content-Type-Options: nosniff
      Strict-Transport-Security: max-age=15724800; includeSubDomains
      
      body=
      {"code":400,"reason":"Bad Request","message":"The referenced object 'internal/role/openidm-authorized', does not exist"}
      

      I tried to create the managed users manually and it was ok:

      curl -k  -H "Content-Type: application/json" -H "X-OpenIDM-Username: openidm-admin" -H "X-OpenIDM-Password: openidm-admin" --request POST https://tinghuaxu.iam.performance-team.forgeops.com:443/openidm/managed/user/?_action=create --data '{"userName" : "H-T-276933", "givenName" : "Tbnjazlt", "sn" : "Hpjihu", "mail" : "H-T-276933@example.com", "telephoneNumber" : "444-444-4444", "password" : "Pa_ssw0rd", "description" : "Managed User" }'
      {"_id":"8cff9653-d9f5-4fcc-85a2-cb75c453013c","_rev":"0000000010dc366c","userName":"H-T-276933","accountStatus":"active","description":"Managed User","givenName":"Tbnjazlt","sn":"Hpjihu","telephoneNumber":"444-444-4444","mail":"H-T-276933@example.com"}
      

      To reproduce it using pyrock in lodestar:
      1. clone lodestar/pyrock:
      https://stash.forgerock.org/projects/QA/repos/lodestar/browse
      2. cd lodestar/pyrock and run the test as, the test used the default config, two IDM pods, and each of read/update/delete/create POST runs about 15 minutes.
      ./run.py simple_managed_users_ds_shared_repo
      3. Put the URL file:/Users/...../lodestar/results/pyrock/simple_managed_users_ds_shared_repo/global.html on browser
      and check report for read/update/create.

      Note:

      The symptom can be easier to see when I disabled notification service using notificationFactory endpoint, It can be seen with concurrency 35 or 30.
      IDM CPU usage is relative high, attached is one cpu monitoring output during one test, the cpu/request and limit are in the config file attached.

        Attachments

        1. conf.yaml
          4 kB
        2. idm-0.log
          146 kB
        3. idm0-update-error.log
          106 kB
        4. idm-1.log
          145 kB
        5. two-idm-pods-con-40-ds.png
          two-idm-pods-con-40-ds.png
          197 kB

          Issue Links

            Activity

              People

              • Assignee:
                Tinghua.Xu Tinghua Xu
                Reporter:
                Tinghua.Xu Tinghua Xu
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: