Uploaded image for project: 'OpenDJ'
  1. OpenDJ
  2. OPENDJ-8088

Rest2Ldap optimization: simplify attribute lists where possible

    XMLWordPrintable

Details

    Description

      The following search request has been observed being sent from IDM to DS via Rest2Ldap:

      response: {
         elapsedTimeUnits: "MILLISECONDS"
         status: "SUCCESSFUL"
         elapsedTime: 12608937  <----- 3.5h
         statusCode: "0"
         nentries: 652763       <----- 650k users
         additionalItems: {
         unindexed: null
         }
      }
      operation: "SEARCH"
      dn: "ou=user,o=alpha,o=root,ou=identities"
      scope: "one"
      connId: 453
      msgId: 2571
      protocol: "LDAP"
      filter: "(&(objectClass=person)(objectClass=organizationalPerson)(objectClass=inetOrgPerson)(objectClass=iplanet-am-user-service)(objectClass=devicePrintProfilesContainer)(objectClass=deviceProfilesContainer)(objectClass=kbaInfoContainer)(objectClass=fr-idm-managed-user-explicit)(objectClass=forgerock-am-dashboard-service)(objectClass=inetuser)(objectClass=iplanet-am-auth-configuration-service)(objectClass=iplanet-am-managed-person)(objectClass=iPlanetPreferences)(objectClass=oathDeviceProfilesContainer)(objectClass=pushDeviceProfilesContainer)(objectClass=sunAMAuthAccountLockout)(objectClass=sunFMSAML2NameIdentifier)(objectClass=webauthnDeviceProfilesContainer)(objectClass=fr-idm-hybrid-obj)(objectClass=fr-ext-attrs)(objectClass=top))"
      attrs: [
         0: "objectClass"
         1: "fr-idm-uuid"
         2: "etag"
         3: "co"
         4: "mail"
         5: "fr-attr-str1"
         6: "fr-idm-managed-user-memberoforgid"
         7: "fr-attr-str2"
         8: "fr-attr-idate5"
         9: "fr-attr-str3"
         10: "fr-attr-idate4"
         11: "postalCode"
         12: "fr-attr-str4"
         13: "fr-attr-idate3"
         14: "fr-attr-str5"
         15: "fr-attr-istr5"
         16: "fr-attr-istr4"
         17: "userPassword"
         18: "fr-attr-istr3"
         19: "fr-attr-istr2"
         20: "fr-attr-istr1"
         21: "fr-attr-imulti3"
         22: "fr-attr-int5"
         23: "fr-attr-imulti4"
         24: "fr-attr-int4"
         25: "fr-idm-consentedMapping"
         26: "fr-attr-imulti5"
         27: "fr-attr-int3"
         28: "fr-attr-int2"
         29: "fr-attr-imulti1"
         30: "fr-attr-imulti2"
         31: "fr-attr-int1"
         32: "givenName"
         33: "st"
         34: "street"
         35: "telephoneNumber"
         36: "l"
         37: "displayName"
         38: "fr-idm-effectiveAssignment"
         39: "description"
         40: "inetUserStatus"
         41: "fr-attr-multi1"
         42: "fr-attr-date3"
         43: "fr-attr-date2"
         44: "fr-attr-multi3"
         45: "fr-attr-date5"
         46: "fr-attr-multi2"
         47: "fr-attr-date4"
         48: "fr-idm-lastSync"
         49: "fr-attr-multi5"
         50: "iplanet-am-user-alias-list"
         51: "fr-attr-multi4"
         52: "fr-idm-kbaInfo"
         53: "fr-attr-iint4"
         54: "fr-attr-iint3"
         55: "isMemberOf"
         56: "fr-attr-iint2"
         57: "sn"
         58: "fr-attr-iint1"
         59: "fr-attr-date1"
         60: "fr-attr-iint5"
         61: "fr-idm-preferences"
         62: "cn"
         63: "uid"
         64: "fr-attr-idate2"
         65: "fr-attr-idate1"
         66: "fr-idm-effectiveRole"
      ]
      

      The attribute list contains 66 elements, most of which are user attributes. The method getBestMatch in org.forgerock.opendj.ldap.AttributeFilter is O with respect to the size of the attribute list which, in this case corresponds to the entire list of attributes in the entry, meaning that attribute filtering is O(n^2) with respect to the number of attributes in the entry when read by Rest2Ldap.

      Suggested fix:

      • reduce the size of the attribute list sent by Rest2Ldap. Once the list of user attributes increases beyond a certain threshold then just request all user attributes. The assumption is that the additional server side computation and additional bandwidth required for the search request itself outweighs any potential bandwidth gains in the search response
      • it is reasonable to assume that all user attributes will be mapped by Rest2Ldap, so we should ensure that when there are no _fields specified in the CREST API request that Rest2Ldap just requests all user attributes. Note that Rest2Ldap may also map some operational attributes, so some attribute mapping is still required
      • consider optimizing the AttributeFilter class to avoid the O(n^2) behavior as much as possible. One option could be to store the requested attributes in a Map and adding a fast-path:
                    private AttributeDescription getBestMatch(final AttributeDescription attributeDescription) {
                        AttributeDescription bestMatch = null;
        
                        if (isMatchedByIncludeAll(attributeDescription.getAttributeType())) {
                            bestMatch = attributeDescription;
                            // Fall-through because the the attribute description may need to be renamed.
                        }
        
                        // Fast-path for exact match.
                        final AttributeDescription exactMatch = requestedAttributes.get(attributeDescription);
                        if (exactMatch != null) {
                            return exactMatch;
                        }
        
                        // Slow-path for sub-type matching.
                        for (final AttributeDescription requestedAttribute : requestedAttributes) {
                            if (attributeDescription.isSubTypeOf(requestedAttribute)) {
                                bestMatch = requestedAttribute;
                                break;
                            }
                        }
        
                        if (bestMatch == null) {
                            return null;
                        }
        
                        // Try to align the name of the attribute as close as possible to the best match.
                        if (bestMatch.getAttributeType().equals(attributeDescription.getAttributeType())) {
                            return bestMatch.withOptions(attributeDescription.getOptions());
                        } else {
                            return attributeDescription.withoutAnyOptions()
                                                       .withOptions(bestMatch.getOptions())
                                                       .withOptions(attributeDescription.getOptions());
                        }
                    }
        

      WARNING: attribute filtering is quite a subtle piece of functionality. Care should be taken to ensure LDAP compliance.

      Attachments

        Issue Links

          Activity

            People

              ylecaillez Yannick Lecaillez
              matthew Matthew Swift
              Yannick Lecaillez Yannick Lecaillez
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: