Uploaded image for project: 'OpenIDM'
  1. OpenIDM
  2. OPENIDM-15275

Significant lower throughput for read-with-no-link-expansion relationship test in GKE cloud

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 7.0.0
    • Fix Version/s: None
    • Component/s: Performance
    • Labels:
    • Environment:
      7.0.0 platform runs on GKE cloud
    • Target Version/s:

      Description

      In GKE ds-shared-repo there is a large throughput difference between the tests read-with-no-link-expansion and query-with-no-link-expansion:

      The results in the attachment show:

      Test Concurrency tps mrt
      read-user-filter-with-no-link-expansion 35 196 178
      query-user-filter-with-no-link-expansion 35 2954 12

      These results should be similar. The data is the same. Only difference being that:

      read , url = 'managed/user/{id}'
      query, url = 'managed/user/?_queryFilter=_id+eq+"{id}"&_fields=*' 
      

      Grabbing the logs for each IDM request using transaction id from DJ access logs. Output in attachment.

      wc -l /tmp/*.txt
         45 /tmp/00ed3d12-6cf5-440b-b9e1-a37ecc06fd8b-692524.txt
          4 /tmp/e3c915f0-0c24-4eea-bf3a-f158ff7b024e-848781.txt
      

      This shows that for read there are 45 ldap operations and for query 4 ldap operations. Each ldap operation seems quick, but this *11 difference for read adds up both in response time on DS backend but also in the cost of processing on IDM side.

      This shows that there is significant fan-out of operations for the read with no-link-expansion.

      To reproduce it using PyRock:

      run tests:

      lodestar/pyrock/run.py managed_users_with_relationships

      and Check the PyRock report:

      Note:
      As a side note investigation of the same test with IDM using DS as an External Repo for read yields much better results: read-user-with-no-link-expansion has throughput 1259 vs. query-user-fileter-with-no-link-expansion with 1453.

      Or from another aspect: No big throughput difference between read-user-no-link-expansion(136 from a different test) and read-user-with-all-link-expansion(135) in GKE. But we see big difference between the two tests: 1278 for read user with no link expansion and 85 for read user with all link expansion in ds as external repo

      Running the read test on same object without the relationships, roles, manager also gives over 2.5K reads/sec on tps.

        Attachments

          Activity

            People

            Assignee:
            brmiller Brendan Miller
            Reporter:
            Tinghua.Xu Tinghua Xu
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: