Uploaded image for project: 'OpenIDM'
  1. OpenIDM
  2. OPENIDM-15275

Significant lower throughput for read-with-no-link-expansion relationship test in GKE cloud



    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 7.0.0
    • Fix Version/s: None
    • Component/s: Performance
    • Labels:
    • Environment:
      7.0.0 platform runs on GKE cloud
    • Target Version/s:


      In GKE ds-shared-repo there is a large throughput difference between the tests read-with-no-link-expansion and query-with-no-link-expansion:

      The results in the attachment show:

      Test Concurrency tps mrt
      read-user-filter-with-no-link-expansion 35 196 178
      query-user-filter-with-no-link-expansion 35 2954 12

      These results should be similar. The data is the same. Only difference being that:

      read , url = 'managed/user/{id}'
      query, url = 'managed/user/?_queryFilter=_id+eq+"{id}"&_fields=*' 

      Grabbing the logs for each IDM request using transaction id from DJ access logs. Output in attachment.

      wc -l /tmp/*.txt
         45 /tmp/00ed3d12-6cf5-440b-b9e1-a37ecc06fd8b-692524.txt
          4 /tmp/e3c915f0-0c24-4eea-bf3a-f158ff7b024e-848781.txt

      This shows that for read there are 45 ldap operations and for query 4 ldap operations. Each ldap operation seems quick, but this *11 difference for read adds up both in response time on DS backend but also in the cost of processing on IDM side.

      This shows that there is significant fan-out of operations for the read with no-link-expansion.

      To reproduce it using PyRock:

      run tests:

      lodestar/pyrock/run.py managed_users_with_relationships

      and Check the PyRock report:

      As a side note investigation of the same test with IDM using DS as an External Repo for read yields much better results: read-user-with-no-link-expansion has throughput 1259 vs. query-user-fileter-with-no-link-expansion with 1453.

      Or from another aspect: No big throughput difference between read-user-no-link-expansion(136 from a different test) and read-user-with-all-link-expansion(135) in GKE. But we see big difference between the two tests: 1278 for read user with no link expansion and 85 for read user with all link expansion in ds as external repo

      Running the read test on same object without the relationships, roles, manager also gives over 2.5K reads/sec on tps.




            brmiller Brendan Miller
            Tinghua.Xu Tinghua Xu
            0 Vote for this issue
            1 Start watching this issue