Uploaded image for project: 'OpenIDM'
  1. OpenIDM
  2. OPENIDM-15322

Query on relationship endpoint takes much longer time to return with external DS as repo

    Details

    • Target Version/s:

      Description

      The test preloads 30K managed users first, with each user has 10 roles. Using external DS as repo, the following query would take about 28 seconds to return when external DS is used as repo. 

      curl  --header "X-OpenIDM-Username: openidm-admin"  --header "X-OpenIDM-Password: openidm-admin" --request GET "http://localhost:8080/openidm/managed/user/A-A-10518/roles?_queryFilter=true&_fields=*_ref" 

       

      Checked on DS log, there are some unindexed searches taking about 3 seconds, one is as below:

      {
        "eventName": "DJ-LDAP",
        "client": {
          "ip": "172.16.204.13",
          "port": 60014
        },
        "server": {
          "ip": "172.16.204.13",
          "port": 31389
        },
        "request": {
          "protocol": "LDAP",
          "operation": "SEARCH",
          "connId": 347,
          "msgId": 8910,
          "dn": "ou=user,ou=managed,dc=openidm,dc=forgerock,dc=com",
          "scope": "one",
          "filter": "(fr-idm-managed-user-roles:nameAndOptionalJsonEqualityMatchingRule:=uid=role27,ou=role,ou=managed,dc=openidm,dc=forgerock,dc=com)",
          "attrs": [
            "uid",
            "objectClass",
            "etag",
            "fr-idm-managed-user-json",
            "fr-idm-managed-user-roles"
          ]
        },
        "transactionId": "ff07a83a-b8d6-49fb-857f-a79948eda2ff-14602554",
        "response": {
          "status": "SUCCESSFUL",
          "statusCode": "0",
          "elapsedTime": 3218,
          "elapsedTimeUnits": "MILLISECONDS",
          "additionalItems": {
            "unindexed": null
          },
          "nentries": 4290
        },
        "timestamp": "2020-08-25T00:16:20.129Z",
        "_id": "ff07a83a-b8d6-49fb-857f-a79948eda2ff-14602581"
      } 

      Attached are the ds access log with the query run and the output with command on the log:

      jq 'select(.response.elapsedTime>=1000)'  ldap-access.audit.json

       

      To reproduce the symptom using Pyforge, run the following command:

      ./run-pybot.py -n -v -c perf -s relationship.QueryRelationShipEndpoint -t IDM_Query_Relationship_Endpoint_Fields_Ref OpenIDM 

      then the query above.

       

      Note: 

      The same query returned in less or about one second using MySQL as repo.

      Check the query output between MySQL and DS(not the same but not significantly different):

      -rw-rw-r-- 1 testuser testuser  8253228 Aug 25 02:16 ds.out
      -rw-rw-r-- 1 testuser testuser  7591678 Aug 25 01:52 mysql.out 

        Attachments

        1. ds-log.zip
          1.79 MB
        2. jq.out
          10 kB

          Issue Links

            Activity

              People

              • Assignee:
                jason Jason Lemay
                Reporter:
                Tinghua.Xu Tinghua Xu
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: