[OPENDJ-6503] returnNullForMissingProperties=true supersedes _field specification Created: 30/Jul/19  Updated: 08/Nov/19  Resolved: 09/Aug/19

Status: Done
Project: OpenDJ
Component/s: rest
Affects Version/s: 7.0.0
Fix Version/s: Not applicable

Type: Bug Priority: Major
Reporter: Dirk Hogan Assignee: Jean-Noël Rouvignac
Resolution: Not a defect Votes: 0
Labels: None

Attachments: File repo.ds-external-explicit-managed-user.json    
Issue Links:
Depends
is required by OPENIDM-12683 Embedded DJ explicit user object has ... Closed
is required by OPENIDM-13375 REST2LDAP: Null source on query-all-ids Closed
Relates
is related to OPENIDM-13657 Implement Filter to map all queryId-b... Closed
Epic Link: Common Repo
Story Points: 3
Dev Assignee: Jean-Noël Rouvignac

 Description   

IDM defines the following rest2ldap query:

"query-all-ids": {
 "_queryFilter": "true",
 "_fields": "_id,_rev"
}

If http://localhost:8080/openidm/managed/user?_queryId=query-all-ids is dispatched against a DS instance configured with an explicit mapping (https://stash.forgerock.org/projects/OPENIDM/repos/openidm/browse/openidm-samples/repo-config/src/main/resources/opendj/conf/repo.ds-explicit-managed-user.json) and the default rest2ldap configuration of

"returnNullForMissingProperties": true

then the returned values will include all existing fields other than _id and _rev, all set to null, despite the fact that some of these fields do contain data.

If the above-referenced repo-configuration file is updated with

"returnNullForMissingProperties": false

then only the _id and _rev fields are returned, as expected. 

 Comments   
Comment by Jean-Noël Rouvignac [ 31/Jul/19 ]

I cannot recreate the problem. I tried like this:
https://stash.forgerock.org/projects/OPENDJ/repos/opendj/pull-requests/4565/ (see the last commit)

Comment by Dirk Hogan [ 31/Jul/19 ]

Note the problem does not occur against managed/user instances created with this rest2ldap definition:

"managed/user" : {
  "dnTemplate" : "ou=user,ou=managed,dc=openidm,dc=forgerock,dc=com",
  "objectClasses" : [ "uidObject", "fr-idm-managed-user"],
  "jsonAttribute" : "fr-idm-managed-user-json",
  "jsonQueryEqualityMatchingRule" : "caseIgnoreJsonQueryMatchManagedUser"
}

But it does occur against managed/user instances created with this rest2ldap definition:

"managed/user": {
  "dnTemplate": "ou=user,ou=managed,dc=openidm,dc=forgerock,dc=com",
  "objectClasses": [
    "person",
    "organizationalPerson",
    "inetOrgPerson",
    "fr-idm-managed-user-explicit"
  ],
  "properties": {
    "_id": {
      "type": "simple",
      "ldapAttribute": "uid",
      "isRequired": true,
      "writability": "createOnly"
    },
    "userName": {
      "type": "simple",
      "ldapAttribute": "cn"
    },
    "password": {
      "type": "json",
      "ldapAttribute": "fr-idm-password"
    },
    "accountStatus": {
      "type": "simple",
      "ldapAttribute": "fr-idm-accountStatus"
    },
    "effectiveRoles": {
      "type": "json",
      "ldapAttribute": "fr-idm-effectiveRole",
      "isMultiValued": true
    },
    "effectiveAssignments": {
      "type": "json",
      "ldapAttribute": "fr-idm-effectiveAssignment",
      "isMultiValued": true
    },
    "postalCode": {
      "type": "simple",
      "ldapAttribute": "postalCode"
    },
    "stateProvince": {
      "type": "simple",
      "ldapAttribute": "st"
    },
    "postalAddress": {
      "type": "simple",
      "ldapAttribute": "postalAddress"
    },
    "displayName": {
      "type": "simple",
      "ldapAttribute": "displayName"
    },
    "description": {
      "type": "simple",
      "ldapAttribute": "description"
    },
    "country": {
      "type": "simple",
      "ldapAttribute": "co"
    },
    "address2": {
      "type": "simple",
      "ldapAttribute": "postalAddress"
    },
    "city": {
      "type": "simple",
      "ldapAttribute": "l"
    },
    "givenName": {
      "type": "simple",
      "ldapAttribute": "givenName"
    },
    "sn": {
      "type": "simple",
      "ldapAttribute": "sn"
    },
    "telephoneNumber": {
      "type": "simple",
      "ldapAttribute": "telephoneNumber"
    },
    "mail": {
      "type": "simple",
      "ldapAttribute": "mail"
    },
    "siteImage": {
      "type": "simple",
      "ldapAttribute": "jpegPhoto"
    },
    "lastSync": {
      "type": "json",
      "ldapAttribute": "fr-idm-lastSync"
    },
    "consentedMappings": {
      "type": "json",
      "ldapAttribute": "fr-idm-consentedMapping",
      "isMultiValued": true
    },
    "kbaInfo": {
      "type": "json",
      "ldapAttribute": "fr-idm-kbaInfo",
      "isMultiValued": true
    },
    "preferences": {
      "type": "json",
      "ldapAttribute": "fr-idm-preferences"
    }
  }
}

Note also that these are amalgams of Resource/SubResource attributes, that are pulled-apart and composed to the appropriate rest2ldap Resource/SubResource definitions.

I will update this Jira with the definitions IDM ultimately provides to rest2ldap.

 

Comment by Dirk Hogan [ 31/Jul/19 ]

Here are the rest2ldap definitions dispatched by IDM. The first is for the 'generic' mapping, which does not display the problem. Note the Resource/SubResource definitions of interest are managed_user and managed/user, respectively:

{  
   "managed_user":{  
      "superType":"frapi:openidm:rest2ldap:base:1.0",
      "objectClasses":[  
         "uidObject",
         "fr-idm-managed-user"
      ],
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"uid",
            "isRequired":true,
            "writability":"createOnly"
         },
         "fullobject":{  
            "type":"json",
            "ldapAttribute":"fr-idm-managed-user-json",
            "jsonQueryEqualityMatchingRule":"caseIgnoreJsonQueryMatchManagedUser"
         }
      }
   },
   "managed_role":{  
      "superType":"frapi:openidm:rest2ldap:base:1.0",
      "objectClasses":[  
         "uidObject",
         "fr-idm-managed-role"
      ],
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"uid",
            "isRequired":true,
            "writability":"createOnly"
         },
         "fullobject":{  
            "type":"json",
            "ldapAttribute":"fr-idm-managed-role-json",
            "jsonQueryEqualityMatchingRule":"caseIgnoreJsonQueryMatchManagedRole"
         }
      }
   },
   "cluster":{  
      "superType":"frapi:openidm:rest2ldap:base:1.0",
      "objectClasses":[  
         "uidObject",
         "fr-idm-cluster-obj"
      ],
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"uid",
            "isRequired":true,
            "writability":"createOnly"
         },
         "fullobject":{  
            "type":"json",
            "ldapAttribute":"fr-idm-cluster-json",
            "jsonQueryEqualityMatchingRule":"caseIgnoreJsonQueryMatchClusterObject"
         }
      }
   },
   "relationships":{  
      "superType":"frapi:openidm:rest2ldap:base:1.0",
      "objectClasses":[  
         "uidObject",
         "fr-idm-relationship"
      ],
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"uid",
            "isRequired":true,
            "writability":"createOnly"
         },
         "fullobject":{  
            "type":"json",
            "ldapAttribute":"fr-idm-relationship-json",
            "jsonQueryEqualityMatchingRule":"caseIgnoreJsonQueryMatchRelationship"
         }
      }
   },
   "internal_user":{  
      "objectClasses":[  
         "uidObject",
         "fr-idm-internal-user"
      ],
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"uid",
            "isRequired":true,
            "writability":"createOnly"
         },
         "password":{  
            "type":"json",
            "ldapAttribute":"fr-idm-password"
         },
         "_rev":{  
            "type":"simple",
            "ldapAttribute":"etag",
            "writability":"readOnly"
         }
      }
   },
   "internal_role":{  
      "objectClasses":[  
         "fr-idm-internal-role"
      ],
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"cn",
            "isRequired":true,
            "writability":"createOnly"
         },
         "name":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-name"
         },
         "description":{  
            "type":"simple",
            "ldapAttribute":"description"
         },
         "temporalConstraints":{  
            "type":"json",
            "ldapAttribute":"fr-idm-temporal-constraints",
            "isMultiValued":true
         },
         "condition":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-condition"
         },
         "privileges":{  
            "type":"json",
            "ldapAttribute":"fr-idm-privilege",
            "isMultiValued":true
         },
         "_rev":{  
            "type":"simple",
            "ldapAttribute":"etag",
            "writability":"readOnly"
         }
      }
   },
   "link":{  
      "objectClasses":[  
         "uidObject",
         "fr-idm-link"
      ],
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"uid",
            "isRequired":true,
            "writability":"createOnly"
         },
         "linkType":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-link-type"
         },
         "linkQualifier":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-link-qualifier"
         },
         "firstId":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-link-firstId"
         },
         "secondId":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-link-secondId"
         },
         "_rev":{  
            "type":"simple",
            "ldapAttribute":"etag",
            "writability":"readOnly"
         }
      }
   },
   "clusteredrecontargetids":{  
      "objectClasses":[  
         "uidObject",
         "fr-idm-recon-clusteredTargetIds"
      ],
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"uid",
            "isRequired":true,
            "writability":"createOnly"
         },
         "reconId":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-recon-id"
         },
         "targetIds":{  
            "type":"json",
            "ldapAttribute":"fr-idm-recon-targetIds"
         },
         "_rev":{  
            "type":"simple",
            "ldapAttribute":"etag",
            "writability":"readOnly"
         }
      }
   },
   "locks":{  
      "objectClasses":[  
         "uidObject",
         "fr-idm-lock"
      ],
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"uid",
            "isRequired":true,
            "writability":"createOnly"
         },
         "nodeId":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-lock-nodeid"
         },
         "_rev":{  
            "type":"simple",
            "ldapAttribute":"etag",
            "writability":"readOnly"
         }
      }
   },
   "sync_queue":{  
      "objectClasses":[  
         "uidObject",
         "fr-idm-syncqueue"
      ],
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"uid",
            "isRequired":true,
            "writability":"createOnly"
         },
         "syncAction":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-syncqueue-syncaction"
         },
         "resourceCollection":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-syncqueue-resourcecollection"
         },
         "resourceId":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-syncqueue-resourceid"
         },
         "mapping":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-syncqueue-mapping"
         },
         "objectRev":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-syncqueue-objectRev"
         },
         "oldObject":{  
            "type":"json",
            "ldapAttribute":"fr-idm-syncqueue-oldobject"
         },
         "newObject":{  
            "type":"json",
            "ldapAttribute":"fr-idm-syncqueue-newobject"
         },
         "context":{  
            "type":"json",
            "ldapAttribute":"fr-idm-syncqueue-context"
         },
         "state":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-syncqueue-state"
         },
         "nodeId":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-syncqueue-nodeid"
         },
         "remainingRetries":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-syncqueue-remainingretries"
         },
         "createDate":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-syncqueue-createdate"
         },
         "_rev":{  
            "type":"simple",
            "ldapAttribute":"etag",
            "writability":"readOnly"
         }
      }
   },
   "recon_assoc":{  
      "objectClasses":[  
         "fr-idm-reconassoc"
      ],
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-reconassoc-reconid",
            "isRequired":true
         },
         "mapping":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-reconassoc-mapping"
         },
         "sourceResourceCollection":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-reconassoc-sourceresourcecollection"
         },
         "targetResourceCollection":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-reconassoc-targetresourcecollection"
         },
         "isAnalysis":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-reconassoc-isanalysis"
         },
         "finishTime":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-reconassoc-finishtime"
         },
         "_rev":{  
            "type":"simple",
            "ldapAttribute":"etag",
            "writability":"readOnly"
         }
      },
      "subResources":{  
         "entry":{  
            "type":"collection",
            "resource":"recon-assoc-entry",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            }
         }
      }
   },
   "recon-assoc-entry":{  
      "objectClasses":[  
         "uidObject",
         "fr-idm-reconassocentry"
      ],
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"uid",
            "isRequired":true
         },
         "reconId":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-reconassocentry-reconid"
         },
         "linkQualifier":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-reconassocentry-linkqualifier"
         },
         "status":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-reconassocentry-status"
         },
         "situation":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-reconassocentry-situation"
         },
         "action":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-reconassocentry-action"
         },
         "phase":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-reconassocentry-phase"
         },
         "sourceObjectId":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-reconassocentry-sourceObjectId"
         },
         "targetObjectId":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-reconassocentry-targetObjectId"
         },
         "exception":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-reconassocentry-exception"
         },
         "message":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-reconassocentry-message"
         },
         "messageDetail":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-reconassocentry-messagedetail"
         },
         "ambiguousTargetObjectIds":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-reconassocentry-ambiguoustargetobjectids"
         },
         "mapping":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-reconassoc-mapping"
         },
         "sourceResourceCollection":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-reconassoc-sourceresourcecollection"
         },
         "targetResourceCollection":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-reconassoc-targetresourcecollection"
         },
         "isAnalysis":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-reconassoc-isanalysis"
         },
         "_rev":{  
            "type":"simple",
            "ldapAttribute":"etag",
            "writability":"readOnly"
         }
      }
   },
   "repo":{  
      "subResources":{  
         "config":{  
            "type":"collection",
            "dnTemplate":"ou=config,dc=openidm,dc=forgerock,dc=com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"generic"
         },
         "ui":{  
            "type":"collection",
            "dnTemplate":"ou=ui,dc=openidm,dc=forgerock,dc=com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"generic"
         },
         "ui/__ou__":{  
            "type":"collection",
            "dnTemplate":"ou=ui,dc=openidm,dc=forgerock,dc=com",
            "resource":"frapi:openidm:rest2ldap:ou:1.0",
            "namingStrategy":{  
               "dnAttribute":"ou",
               "type":"clientDnNaming"
            }
         },
         "managed":{  
            "type":"collection",
            "dnTemplate":"ou=managed,dc=openidm,dc=forgerock,dc=com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"generic"
         },
         "managed/__ou__":{  
            "type":"collection",
            "dnTemplate":"ou=managed,dc=openidm,dc=forgerock,dc=com",
            "resource":"frapi:openidm:rest2ldap:ou:1.0",
            "namingStrategy":{  
               "dnAttribute":"ou",
               "type":"clientDnNaming"
            }
         },
         "managed/user":{  
            "type":"collection",
            "dnTemplate":"ou=user,ou=managed,dc=openidm,dc=forgerock,dc=com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"managed_user"
         },
         "managed/role":{  
            "type":"collection",
            "dnTemplate":"ou=role,ou=managed,dc=openidm,dc=forgerock,dc=com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"managed_role"
         },
         "scheduler":{  
            "type":"collection",
            "dnTemplate":"ou=scheduler,dc=openidm,dc=forgerock,dc=com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"generic"
         },
         "scheduler/__ou__":{  
            "type":"collection",
            "dnTemplate":"ou=scheduler,dc=openidm,dc=forgerock,dc=com",
            "resource":"frapi:openidm:rest2ldap:ou:1.0",
            "namingStrategy":{  
               "dnAttribute":"ou",
               "type":"clientDnNaming"
            }
         },
         "cluster":{  
            "type":"collection",
            "dnTemplate":"ou=cluster,dc=openidm,dc=forgerock,dc=com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"cluster"
         },
         "cluster/__ou__":{  
            "type":"collection",
            "dnTemplate":"ou=cluster,dc=openidm,dc=forgerock,dc=com",
            "resource":"frapi:openidm:rest2ldap:ou:1.0",
            "namingStrategy":{  
               "dnAttribute":"ou",
               "type":"clientDnNaming"
            }
         },
         "relationships":{  
            "type":"collection",
            "dnTemplate":"ou=relationships,dc=openidm,dc=forgerock,dc=com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"relationships"
         },
         "updates":{  
            "type":"collection",
            "dnTemplate":"ou=updates,dc=openidm,dc=forgerock,dc=com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"generic"
         },
         "reconprogressstate":{  
            "type":"collection",
            "dnTemplate":"ou=reconprogressstate,dc=openidm,dc=forgerock,dc=com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"generic"
         },
         "jsonstorage":{  
            "type":"collection",
            "dnTemplate":"ou=jsonstorage,dc=openidm,dc=forgerock,dc=com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"generic"
         },
         "internal/usermeta":{  
            "type":"collection",
            "dnTemplate":"ou=usermeta,ou=internal,dc=openidm,dc=forgerock,dc=com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"generic"
         },
         "internal/notification":{  
            "type":"collection",
            "dnTemplate":"ou=notification,ou=internal,dc=openidm,dc=forgerock,dc=com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"generic"
         },
         "file":{  
            "type":"collection",
            "dnTemplate":"ou=file,dc=openidm,dc=forgerock,dc=com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"generic"
         },
         "default":{  
            "type":"collection",
            "dnTemplate":"ou=generic,dc=openidm,dc=forgerock,dc=com",
            "resource":"frapi:openidm:rest2ldap:ou:1.0",
            "namingStrategy":{  
               "dnAttribute":"ou",
               "type":"clientDnNaming"
            }
         },
         "internal/user":{  
            "type":"collection",
            "dnTemplate":"ou=users,ou=internal,dc=openidm,dc=forgerock,dc=com",
            "resource":"internal_user",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            }
         },
         "internal/role":{  
            "type":"collection",
            "dnTemplate":"ou=roles,ou=internal,dc=openidm,dc=forgerock,dc=com",
            "resource":"internal_role",
            "namingStrategy":{  
               "dnAttribute":"cn",
               "type":"clientDnNaming"
            }
         },
         "link":{  
            "type":"collection",
            "dnTemplate":"ou=links,dc=openidm,dc=forgerock,dc=com",
            "resource":"link",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            }
         },
         "clusteredrecontargetids":{  
            "type":"collection",
            "dnTemplate":"ou=clusteredrecontargetids,dc=openidm,dc=forgerock,dc=com",
            "resource":"clusteredrecontargetids",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            }
         },
         "locks":{  
            "type":"collection",
            "dnTemplate":"ou=locks,dc=openidm,dc=forgerock,dc=com",
            "resource":"locks",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            }
         },
         "sync/queue":{  
            "type":"collection",
            "dnTemplate":"ou=queue,ou=sync,dc=openidm,dc=forgerock,dc=com",
            "resource":"sync_queue",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            }
         },
         "recon/assoc":{  
            "type":"collection",
            "dnTemplate":"ou=assoc,ou=recon,dc=openidm,dc=forgerock,dc=com",
            "resource":"recon_assoc",
            "namingStrategy":{  
               "dnAttribute":"fr-idm-reconassoc-reconid",
               "type":"clientDnNaming"
            }
         },
         "recon/assoc/entry":{  
            "type":"collection",
            "dnTemplate":null,
            "resource":"recon-assoc-entry",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            }
         }
      }
   },
   "frapi:openidm:rest2ldap:base:1.0":{  
      "isAbstract":true,
      "objectClasses":[  
         "top"
      ],
      "resourceTypeProperty":"_schema",
      "properties":{  
         "_schema":{  
            "type":"resourceType"
         },
         "_rev":{  
            "type":"simple",
            "ldapAttribute":"etag",
            "writability":"readOnly"
         },
         "_meta":{  
            "type":"object",
            "properties":{  
               "created":{  
                  "type":"simple",
                  "ldapAttribute":"createTimestamp",
                  "writability":"readOnly"
               },
               "lastModified":{  
                  "type":"simple",
                  "ldapAttribute":"modifyTimestamp",
                  "writability":"readOnly"
               }
            }
         }
      }
   },
   "frapi:openidm:rest2ldap:ou:1.0":{  
      "superType":"frapi:openidm:rest2ldap:base:1.0",
      "objectClasses":[  
         "organizationalUnit"
      ],
      "subResources":{  
         "__ou__":{  
            "type":"collection",
            "dnTemplate":"",
            "resource":"frapi:openidm:rest2ldap:base:1.0",
            "namingStrategy":{  
               "type":"clientDnNaming",
               "dnAttribute":"ou"
            }
         },
         "":{  
            "type":"collection",
            "dnTemplate":"",
            "resource":"generic",
            "namingStrategy":{  
               "type":"clientDnNaming",
               "dnAttribute":"uid"
            }
         }
      },
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"ou",
            "isRequired":true,
            "writability":"createOnly"
         },
         "displayName":{  
            "type":"simple",
            "ldapAttribute":"ou",
            "isRequired":true,
            "writability":"readOnly"
         },
         "description":{  
            "type":"simple"
         }
      }
   },
   "generic":{  
      "superType":"frapi:openidm:rest2ldap:base:1.0",
      "objectClasses":[  
         "uidObject",
         "fr-idm-generic-obj"
      ],
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"uid",
            "isRequired":true,
            "writability":"createOnly"
         },
         "fullobject":{  
            "type":"json",
            "ldapAttribute":"fr-idm-json"
         }
      }
   }
}

 

And here are the definitions for the 'explicit' mapping, which does show the problem:

{  
   "managed_role":{  
      "superType":"frapi:openidm:rest2ldap:base:1.0",
      "objectClasses":[  
         "uidObject",
         "fr-idm-managed-role"
      ],
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"uid",
            "isRequired":true,
            "writability":"create Only"
         },
         "fullobject":{  
            "type":"json",
            "ldapAttribute":"fr-idm-managed-role-json",
            "jsonQueryEqualityMatchingRule":"caseIgnoreJsonQueryMatchManagedRole"
         }
      }
   },
   "cluster":{  
      "superType":"frapi:openidm:rest2ldap:base:1.0",
      "objectClasses":[  
         "uidObject",
         "fr-idm-cluste r-obj"
      ],
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"uid",
            "isRequired":true,
            "writability":"createOnly"
         },
         "fullobject":{  
            "type":"json",
            "ldapAttribute":"fr-idm-cluster-json",
            "jsonQueryEqualityMatchingRule":"caseIgnoreJsonQueryMatchClusterObject"
         }
      }
   },
   "relationships":{  
      "superType":"frapi:openidm:rest2ldap:base:1.0",
      "objectClasses":[  
         "uidObject",
         "fr-idm-relationship"
      ],
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"uid",
            "isRequired":true,
            "writability":"createOnly"
         },
         "fullobject":{  
            "type ":"json",
            "ldapAttribute":"fr-idm-relationship-json",
            "jsonQueryEqualityMatchingRule":"caseIgnoreJsonQueryMatchRelationship"
         }
      }
   },
   "internal_user":{  
      "objectClasses":[  
         "uidObject",
         "fr-idm-internal-user"
      ],
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"uid",
            "isRequired":true,
            "writability":"createOnly"
         },
         "password":{  
            "type":"json",
            "ldapAttribute":"fr-idm-password"
         },
         "_rev":{  
            "type":"simple",
            "ldapAttribute":"etag",
            "writability":"readOnly"
         }
      }
   },
   "internal_role":{  
      "objectClasses":[  
         "fr-idm-internal-role"
      ],
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"cn",
            "isRequired":true,
            "writability":"createOnly"
         },
         "name":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-name"
         },
         "description":{  
            "type":"simple",
            "ldapAttribute":"description"
         },
         "temporalConstraints ":{  
            "type":"json",
            "ldapAttribute":"fr-idm-temporal-constraints",
            "isMultiValued":true
         },
         "condition":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-condition"
         },
         "privileges":{  
            "type":"json",
            "ldapAttribute":"fr-idm-privilege",
            "isMultiValued":true
         },
         "_rev":{  
            "t ype":"simple",
            "ldapAttribute":"etag",
            "writability":"readOnly"
         }
      }
   },
   "link":{  
      "objectClasses":[  
         "uidObject",
         "fr-idm-link"
      ],
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"uid",
            "isRequired":true,
            "writability":"createOnly"
         },
         "linkType":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-link-type"
         },
         "linkQualifier":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-link-qualifier"
         },
         "firstId":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-link-firstId"
         },
         "secondId":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-link-second Id"
         },
         "_rev":{  
            "type":"simple",
            "ldapAttribute":"etag",
            "writability":"readOnly"
         }
      }
   },
   "clusteredrecontargetids":{  
      "objectClasses":[  
         "uidObject",
         "fr-idm-recon-clusteredTargetIds"
      ],
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"uid",
            "isRequired":true,
            "writability":"createOnly"
         },
         "reconId":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-recon-id"
         },
         "targetIds":{  
            "type":"json",
            "ldapAttribute":"fr-idm-recon-targetIds"
         },
         "_rev":{  
            "type":"simple",
            "ldapAttribute":"etag",
            "writability":"readOnly"
         }
      }
   },
   "locks":{  
      "objectClasses":[  
         "uidObject",
         "fr-idm-lock"
      ],
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"uid",
            "isRequired":true,
            "writability":"createOnly"
         },
         "nodeId":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-lock-nodeid"
         },
         "_rev":{  
            "type":"simple",
            "ldapAttribute":"etag",
            "writability":"readOnly"
         }
      }
   },
   "sync_queue":{  
      "objectClasses":[  
         "uidObject",
         "fr-idm-syncqueue"
      ],
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"uid",
            "isRequired":true,
            "writability":"createOnly"
         },
         "syncAction":{  
            "type":" simple",
            "ldapAttribute":"fr-idm-syncqueue-syncaction"
         },
         "resourceCollection":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-syncqueue-resourcecollection"
         },
         "resourceId":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-syncqueue-resourceid"
         },
         "mapping":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-syncqueue-mapping"
         },
         "objectRev":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-syncqueue-objectRev"
         },
         "oldObject":{  
            "type":"json",
            "ldapAttribute":"fr-idm-syncqueue-oldobject"
         },
         "newObject":{  
            "type":"json",
            "ldapAttribute":"fr-idm-syncq ueue-newobject"
         },
         "context":{  
            "type":"json",
            "ldapAttribute":"fr-idm-syncqueue-context"
         },
         "state":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-syncqueue-state"
         },
         "nodeId":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-syncqueue-nodeid"
         },
         "remainingRetries":{  
            "type ":"simple",
            "ldapAttribute":"fr-idm-syncqueue-remainingretries"
         },
         "createDate":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-syncqueue-createdate"
         },
         "_rev":{  
            "type":"simple",
            "ldapAttribute":"etag",
            "writability":"readOnly"
         }
      }
   },
   "managed_user":{  
      "objectClasses":[  
         "person",
         "organizationalPerson",
         "inetOrgPerson",
         "fr-idm-managed-user-explicit"
      ],
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"uid",
            "isRequired":true,
            "writability":"createOnly"
         },
         "userName":{  
            "type":"simple",
            "ldapAttribute":"cn"
         },
         "password":{  
            "type":"json",
            "ldapAttribute":"fr-idm-password"
         },
         "accountStatus":{  
            "type":"simple",
            "ldapAttribute":"fr-idm-accountStatus"
         },
         "effectiveRoles":{  
            "type":"json",
            "ldapAttribute":"fr-idm-effectiveRole",
            "isMultiValued":true
         },
         "effectiveAssignments":{  
            "type":"json",
            "ldapAttribute":"fr-idm-effectiveAssignment",
            "isMultiValued":true
         },
         "postalCode":{  
            "type":"simple",
            "ldapAttribute":"postalCode"
         },
         "stateProvince":{  
            "type":"simple",
            "ldapAttribute":"st"
         },
         "postalAddress":{  
            "type":"simple",
            "ldapAttribute":"post alAddress"
         },
         "displayName":{  
            "type":"simple",
            "ldapAttribute":"displayName"
         },
         "description":{  
            "type":"simple",
            "ldapAttribute":"description"
         },
         "country":{  
            "type":"simple",
            "ldapAttribute":"co"
         },
         "address2":{  
            "type":"simple",
            "ldapAttribute":"postalAddress"
         },
         "city":{  
            "type":"simple",
            "ldapAttribute":"l"
         },
         "givenName":{  
            "type":"simple",
            "ldapAttribute":"givenName"
         },
         "sn":{  
            "type":"simple",
            "ldapAttribute":"sn"
         },
         "telephoneNumber":{  
            "type":"simple",
            "ldapAttribute":"telephoneNumber"
         },
         "mail":{  
            "type":"simp le",
            "ldapAttribute":"mail"
         },
         "siteImage":{  
            "type":"simple",
            "ldapAttribute":"jpegPhoto"
         },
         "lastSync":{  
            "type":"json",
            "ldapAttribute":"fr-idm-lastSync"
         },
         "consentedMappings":{  
            "type":"json",
            "ldapAttribute":"fr-idm-consentedMapping",
            "isMultiValued":true
         },
         "kbaInfo":{  
            "type":"json",
            "ldapAttribute":"fr-idm-kbaInfo",
            "isMultiValued":true
         },
         "preferences":{  
            "type":"json",
            "ldapAttribute":"fr-idm-preferences"
         },
         "_rev":{  
            "type":"simple",
            "ldapAttribute":"etag",
            "writability":"readOnly"
         }
      }
   },
   "repo":{  
      "subResource s":{  
         "config":{  
            "type":"collection",
            "dnTemplate":"ou=config,dc=openidm,dc=forgerock,dc=com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"generic"
         },
         "ui":{  
            "type":"collection",
            "dnTemplate":"ou=ui,dc=openidm,dc=forgerock,dc=c om",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"generic"
         },
         "ui/__ou__":{  
            "type":"collection",
            "dnTemplate":"ou=ui,dc=openidm,dc=forgerock,dc=com",
            "resource":"frapi:openidm:rest2ldap:ou:1.0",
            "namingStrategy":{  
               "dnAttribute":"ou",
               "type":"clientDnNaming"
            }
         },
         "managed":{  
            "type":"collection",
            "dnTemplate":"ou=managed,dc=openidm,dc=forgerock,dc=com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"generic"
         },
         "managed/__ou__":{  
            "type":"collection",
            "dnTe mplate":"ou=managed,dc=openidm,dc=forgerock,dc=com",
            "resource":"frapi:openidm:rest2ldap:ou:1.0",
            "namingStrategy":{  
               "dnAttribute":"ou",
               "type":"clientDnNaming"
            }
         },
         "managed/role":{  
            "type":"collection",
            "dnTemplate":"ou=role,ou=managed,dc=openidm,dc=forgerock,dc= com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"managed_role"
         },
         "scheduler":{  
            "type":"collection",
            "dnTemplate":"ou=scheduler,dc=openidm,dc=forgerock,dc=com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"generic"
         },
         "scheduler/__ou__":{  
            "type":"collection",
            "dnTemplate":"ou=scheduler,dc=openidm,dc=forgerock,dc=com",
            "resource":"frapi:openidm:rest2ldap:ou:1.0",
            "namingStrategy":{  
               "dnAttribute":"ou",
               "type":"clientDnNaming"
            }
         },
         "cluster":{  
            "type":"c ollection",
            "dnTemplate":"ou=cluster,dc=openidm,dc=forgerock,dc=com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"cluster"
         },
         "cluster/__ou__":{  
            "type":"collection",
            "dnTemplate":"ou=cluster,dc=openidm,dc=forgerock,dc=com",
            "reso urce":"frapi:openidm:rest2ldap:ou:1.0",
            "namingStrategy":{  
               "dnAttribute":"ou",
               "type":"clientDnNaming"
            }
         },
         "relationships":{  
            "type":"collection",
            "dnTemplate":"ou=relationships,dc=openidm,dc=forgerock,dc=com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"cli entDnNaming"
            },
            "resource":"relationships"
         },
         "updates":{  
            "type":"collection",
            "dnTemplate":"ou=updates,dc=openidm,dc=forgerock,dc=com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"generic"
         },
         "reconprogressstate":{  
            "type":"col lection",
            "dnTemplate":"ou=reconprogressstate,dc=openidm,dc=forgerock,dc=com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"generic"
         },
         "jsonstorage":{  
            "type":"collection",
            "dnTemplate":"ou=jsonstorage,dc=openidm,dc=forgerock,dc=c om",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"generic"
         },
         "internal/usermeta":{  
            "type":"collection",
            "dnTemplate":"ou=usermeta,ou=internal,dc=openidm,dc=forgerock,dc=com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clie ntDnNaming"
            },
            "resource":"generic"
         },
         "internal/notification":{  
            "type":"collection",
            "dnTemplate":"ou=notification,ou=internal,dc=openidm,dc=forgerock,dc=com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"generic"
         },
         "file":{  
            "type":"collection",
            "dnTemplate":"ou=file,dc=openidm,dc=forgerock,dc=com",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            },
            "resource":"generic"
         },
         "default":{  
            "type":"collection",
            "dnTemplate":"ou=generic,dc=openidm,dc=forgerock,dc=com",
            "resour ce":"frapi:openidm:rest2ldap:ou:1.0",
            "namingStrategy":{  
               "dnAttribute":"ou",
               "type":"clientDnNaming"
            }
         },
         "internal/user":{  
            "type":"collection",
            "dnTemplate":"ou=users,ou=internal,dc=openidm,dc=forgerock,dc=com",
            "resource":"internal_user",
            "namingStrategy":{  
               "dn Attribute":"uid",
               "type":"clientDnNaming"
            }
         },
         "internal/role":{  
            "type":"collection",
            "dnTemplate":"ou=roles,ou=internal,dc=openidm,dc=forgerock,dc=com",
            "resource":"internal_role",
            "namingStrategy":{  
               "dnAttribute":"cn",
               "type":"clientDnNaming"
            }
         },
         "link":{  
            "ty pe":"collection",
            "dnTemplate":"ou=links,dc=openidm,dc=forgerock,dc=com",
            "resource":"link",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            }
         },
         "clusteredrecontargetids":{  
            "type":"collection",
            "dnTemplate":"ou=clusteredrecontargetids,dc=openidm, dc=forgerock,dc=com",
            "resource":"clusteredrecontargetids",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            }
         },
         "locks":{  
            "type":"collection",
            "dnTemplate":"ou=locks,dc=openidm,dc=forgerock,dc=com",
            "resource":"locks",
            "namingStrategy":{  
               "dnAttri bute":"uid",
               "type":"clientDnNaming"
            }
         },
         "sync/queue":{  
            "type":"collection",
            "dnTemplate":"ou=queue,ou=sync,dc=openidm,dc=forgerock,dc=com",
            "resource":"sync_queue",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            }
         },
         "managed/user":{  
            "type":" collection",
            "dnTemplate":"ou=user,ou=managed,dc=openidm,dc=forgerock,dc=com",
            "resource":"managed_user",
            "namingStrategy":{  
               "dnAttribute":"uid",
               "type":"clientDnNaming"
            }
         }
      }
   },
   "frapi:openidm:rest2ldap:base:1.0":{  
      "isAbstract":true,
      "objectClasses":[  
         "top"
      ],
      "r esourceTypeProperty":"_schema",
      "properties":{  
         "_schema":{  
            "type":"resourceType"
         },
         "_rev":{  
            "type":"simple",
            "ldapAttribute":"etag",
            "writability":"readOnly"
         },
         "_meta":{  
            "type":"object",
            "properties":{  
               "created":{  
                  "type":"simple",
                  "ldapAttribute":"createTi mestamp",
                  "writability":"readOnly"
               },
               "lastModified":{  
                  "type":"simple",
                  "ldapAttribute":"modifyTimestamp",
                  "writability":"readOnly"
               }
            }
         }
      }
   },
   "frapi:openidm:rest2ldap:ou:1.0":{  
      "superType":"frapi:openidm:rest2ldap:base:1.0",
      "objectClasses":[  
         "organizationalUnit"
      ],
      "subResources":{  
         "__ou__":{  
            "type":"collection",
            "dnTemplate":"",
            "resource":"frapi:openidm:rest2ldap:base:1.0",
            "namingStrategy":{  
               "type":"clientDnNaming",
               "dnAttribute":"ou"
            }
         },
         "":{  
            "type":"collection",
            "dnTemplate":"",
            "resource":"generic",
            "namingStr ategy":{  
               "type":"clientDnNaming",
               "dnAttribute":"uid"
            }
         }
      },
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"ou",
            "isRequired":true,
            "writability":"createOnly"
         },
         "displayName":{  
            "type":"simple",
            "ldapAttribute":"ou",
            "isRequired":true,
            "writability":"readOnly"
         },
         "description":{  
            "type":"simple"
         }
      }
   },
   "generic":{  
      "superType":"frapi:openidm:rest2ldap:base:1.0",
      "objectClasses":[  
         "uidObject",
         "fr-idm-generic-obj"
      ],
      "properties":{  
         "_id":{  
            "type":"simple",
            "ldapAttribute":"uid",
            "isRequired":true,
            "writability":"createOnly"
         },
         "fullobject":{  
            "type":"json",
            "ldapAttribute":"fr-idm-json"
         }
      }
   }
}

 

Comment by Dirk Hogan [ 31/Jul/19 ]

Some additional debugging information, comparing the ResourceResponse values between the generic and explicit mappings. In both cases, the queries sent to DS are exactly the same, as expected:

{ "method": "query", "totalPagedResultsPolicy": "NONE", "resourcePath": "managed/user", "queryFilter": "true", "pageSize": "0", "queryExpression": "null", "additionalParameters": {  }, "sortKeys": [ ], "pagedResultsOffset": "0", "fields": [ /_id, /_rev ], "pagedResultsCookie": "null", "queryId": "null" }

Note however, the ResourceResponse for the generic case includes the fullObject json field, even though it was not in the fields specification:

{ "id": "dhogan1", "rev": "000000007ea476f6", "content": { "_schema": "managed_user", "_rev": "000000007ea476f6", "_meta": { "created": null, "lastModified": null }, "_id": "dhogan1", "fullobject": null } }

The field

"fullobject" : null

does not appear in the the output because, in the generic case, properties nested under in fullobject are 'hoisted' into the root, and fullobject discarded.

I add this information only as a warning: it may not be that the generic vs. explicit mapping is somehow at the root of the problem, as it appears to occur in both cases. Though it is not clear that including 

"fullobject" : null

 is the same as e.g.

"userName" : null

as fullobject is the container for all state other than id and rev, while an explicitly mapped field like userName is not. 

Comment by Dirk Hogan [ 02/Aug/19 ]

Reproduction steps using IDM:

  1. download, or build IDM 7 snapshot .zip file. If building, .zip file will be in openidm-zip/target
  2. unzip file and cd into openidm directory
  3. remove conf/repo.ds.json
  4. download https://stash.forgerock.org/projects/OPENIDM/repos/openidm/browse/openidm-samples/repo-config/src/main/resources/opendj/conf/repo.ds-explicit-managed-user.json and copy it into the conf directory, renaming it as repo.ds.json (if building, file can be found in openidm-samples/repo-config/src/main/resources/opendj/repo.ds-explicit-managed-user.json)
  5. start idm with command 
    ./startup.sh jpda
  1. create a user with the following command: 
    curl -X PUT \
  http://localhost:8080/openidm/managed/user/bjensen \
  -H 'Content-Type: application/json' \
  -H 'X-OpenIDM-Password: openidm-admin' \
  -H 'X-OpenIDM-Username: openidm-admin' \
  -d '{"preferences":{"updates":false,"marketing":false},"mail":"babs.jensen@forgerock.com","sn":"Jensen","givenName":"Babs","userName":"bjensen"}'
  1. Query the users: 
    curl -X GET \
  'http://localhost:8080/openidm/managed/user?_queryId=query-all-ids' \
  -H 'Content-Type: application/json' \
  -H 'X-OpenIDM-password: openidm-admin' \
  -H 'X-OpenIDM-username: openidm-admin'
  1. observe that many fields are returned as null. 

The query-all-ids query is defined in conf/repo.ds.json: 

"query-all-ids": {
  "_queryFilter": "true",
  "_fields": "_id,_rev"
}

Note that repo.ds.json also defines

"returnNullForMissingProperties": true

Presumably, the semantics for this setting are not 'return null for all properties not specified in the _fields parameter'. Please clarify if this assumption is wrong.

If you built locally, you can set a breakpoint in ExplicitDJTypeHandler#handleQuery to catch the query right before it enters rest2ldap.

 

Comment by Jean-Noël Rouvignac [ 06/Aug/19 ]

I think I have found the issue.
However, I do not know if the issue lives in IDM or DS land.

Root cause

IDM is doing a query substitution that is not expected by Rest2LDAP.

Originally, I could not reproduce it in my unit tests because the code surrounding rest2ldap is a bit different compared to what IDM does.
The code where behaviour differs is in org.forgerock.json.resource.InternalConnection.queryAsync(Context, QueryRequest, QueryResourceHandler).
Here, there is a call to Resources.filterResource() which should filter the fields of the resource. See code of InternalConnection.java.

In IDM's case, the query requests that comes in only contains a query id (no fields at all).
This query ID is resolved by IDM code in ExplicitDJTypeHandler.processQueryId() (code here), and that resolved query (with fields) is then passed to DS in ExplicitDJTypeHandler.handleQuery() (code here).
The JSON resources returned by this call contain a bunch of null values.
When these JSON resources finally reach Resources.filterResource() in InternalConnection.queryAsync(),
that query request has no fields, and thus no filtering happens.
IDM is given the resources with the null fields. Whether they should have been filtered is an open question (see below).

In DS unit tests, the query request has the correct fields passed to Resources.filterResource() (in InternalConnection.queryAsync()).
Thus the filtering happens and the resources are filtered.

I have managed to add a unit test that reproduces the case of IDM.

What is the expected contract?

From Requests class:

    /**
     * Returns the list of fields which should be included with each JSON resource returned by this request. The
     * returned list may be modified if permitted by this query request. An empty list indicates that all fields should
     * be included.
     * <p>
     * <b>NOTE:</b> field filtering alters the structure of a JSON resource and MUST only be performed once while
     * processing a request. It is therefore the responsibility of front-end implementations (e.g. HTTP listeners,
     * Servlets, etc) to perform field filtering. Request handler and resource provider implementations SHOULD NOT
     * filter fields, but MAY choose to optimise their processing in order to return a resource containing only the
     * fields targeted by the field filters.
     *
     * @return The list of fields which should be included with each JSON resource returned by this request (never
     * {@code null}).
     */
    List<JsonPointer> getFields();

Request handler and resource provider implementations SHOULD NOT filter fields,

So does it mean DS request handler should not filter fields either?

Comment by Dirk Hogan [ 06/Aug/19 ]

In an attempt to falsify the hypothesis that a field-less queryId QueryRequest does not allow the field-based Resource-filtering resident in the InternalConnection to appropriately filter the fields in the rest2ldap response, I deployed an IDM instance configured with the explicit mapping to consume external DS. In such a deployment, rest2ldap will only see the queryFilter-and-fields based QueryRequest, allowing the CHF/CREST layers exposing rest2ldap functionality to HTTP to appropriately filter the response, and return only the fields specified in the queryFilter. This was unfortunately not the case - I got back a response with a bunch of null fields.

Reproduction steps:

  1. unzip DS-6.5.2.zip 
  2. ./setup  directory-server  --rootUserDN "cn=Directory Manager"  --rootUserPassword password  --hostname localhost  --ldapPort 31389  --adminConnectorPort 34444  --profile idm-repo  --set idm-repo/domain:forgerock.com  --acceptLicense 
  1. Copy the repo.ds-external-explicit-managed-user.json, attached to this JIRA, and renamed as repo.ds.json, to the conf directory of your IDM installation
  2. Create and query the users as in the reproduction steps described in my previous comment.
  3. Note if you are deploying a 7.0-based DS, the bindDn field in the repo.ds.json file has to be changed to uid=admin and make this same change to the --rootUserDN value in the DS setup command above.
Comment by Jean-Noël Rouvignac [ 08/Aug/19 ]

I have setup everything as you described and I debugged it.

I am still seeing the exact same problem because of the exact same reason.

Using DS as an internal or external data store does not change the fact that rest2ldap is executed inside IDM, and thus the same problem described in my previous comment still applies.

Comment by Dirk Hogan [ 08/Aug/19 ]

Ha - that also makes sense. Let me look and think about it a bit more.

Jean-Noël Rouvignac what do you think about:

https://stash.forgerock.org/projects/COMMONS/repos/forgerock-commons/commits/250365bd096749facda2e4cf57ad16e538591bf0#rest/json-resource-servlet/src/main/java/org/forgerock/json/resource/servlet/RequestRunner.java 

In other words, the rest2ldap endpoint on a stand-alone DS instance is taking an http request, and as such, could be functioning as the 'front-end implementation' responsible for field filtering. I am assuming that this servlet/http-listener only sees the queryFilter-with-fields, and as a front-end, should/could perform field filtering. Are you saying that the http listener actually sees the field-less queryId filter?

We are currently working on a queryId->queryFilter translation filter for both DS and jdbc, and are wondering if it should be a servlet or CHF or CREST filter. If you are saying that external DS, in its capacity as an http listener, will not be filtering, then the IDM translation filter almost has to be a servlet or CHF filter because filtering at the CREST layer will be too late, as the CREST layer was initially traversed by the field-less queryId QueryRequest. 

Comment by Jean-Noël Rouvignac [ 08/Aug/19 ]

If you are saying that external DS, in its capacity as an http listener, will not be filtering, ...

 
Actually, the HTTP listener is never hit DS side.
Everything happens via LDAP
Said otherwise, rest2ldap (in the IDM JVM process) handles all the CREST calls, and translates them only LDAP calls.

Comment by Jean-Noël Rouvignac [ 09/Aug/19 ]

In case the problem must be fixed DS side (I don't think that is the case), here is a patch that fixes the problem for queries.
Other type of CREST operations may also need to be fixed:

diff --git a/opendj-rest2ldap/src/main/java/org/forgerock/opendj/rest2ldap/SubResourceImpl.java b/opendj-rest2ldap/src/main/java/org/forgerock/opendj/rest2ldap/SubResourceImpl.java
index bca085f3a5..2adf32ff7d 100644
--- a/opendj-rest2ldap/src/main/java/org/forgerock/opendj/rest2ldap/SubResourceImpl.java
+++ b/opendj-rest2ldap/src/main/java/org/forgerock/opendj/rest2ldap/SubResourceImpl.java
@@ -11,6 +11,7 @@ import static java.util.Collections.emptyList;
 import static org.forgerock.i18n.LocalizableMessage.raw;
 import static org.forgerock.json.resource.ResourceException.FORBIDDEN;
 import static org.forgerock.json.resource.ResourceException.newResourceException;
+import static org.forgerock.json.resource.Resources.filterResource;
 import static org.forgerock.json.resource.Responses.newActionResponse;
 import static org.forgerock.json.resource.Responses.newQueryResponse;
 import static org.forgerock.json.resource.Responses.newResourceResponse;
@@ -878,7 +879,7 @@ final class SubResourceImpl {
                     final RoutingContext routingContext = newRoutingContext(context, entry.getName(), subType);
                     final PropertyMapper propertyMapper = subType.getPropertyMapper();
                     return readPropertyMapperSingle(propertyMapper, routingContext, subType, entry)
-                            .map(json -> newResourceResponse(id, revision, json))
+                            .map(json -> newResourceResponse(id, revision, filterResource(json, request.getFields())))
                             .toObservable();
                 } else if (ldapResponse instanceof Result) {
                     updateCookie(request, (Result) ldapResponse);
diff --git a/opendj-rest2ldap/src/test/java/org/forgerock/opendj/rest2ldap/BasicRequestsTest.java b/opendj-rest2ldap/src/test/java/org/forgerock/opendj/rest2ldap/BasicRequestsTest.java
index 692ba84dd0..c21fa6b2a6 100644
--- a/opendj-rest2ldap/src/test/java/org/forgerock/opendj/rest2ldap/BasicRequestsTest.java
+++ b/opendj-rest2ldap/src/test/java/org/forgerock/opendj/rest2ldap/BasicRequestsTest.java
@@ -11,7 +11,6 @@ import static java.util.Arrays.asList;
 import static java.util.Collections.emptyList;
 import static org.assertj.core.api.Assertions.assertThat;
 import static org.forgerock.json.JsonPointer.ptr;
-import static org.forgerock.json.JsonValue.array;
 import static org.forgerock.json.JsonValue.field;
 import static org.forgerock.json.JsonValue.json;
 import static org.forgerock.json.JsonValue.object;
@@ -629,15 +628,7 @@ public final class BasicRequestsTest extends ForgeRockTestCase {
                                  resources);
         assertThat(result.getTotalPagedResults()).isEqualTo(NO_COUNT);
         assertThat(resources).hasSize(5);
-        // the missing properties are replaced by null values
-        checkResourcesAreEqual(resources.get(0),
-                               filteredJsonValue().put("description", null)
-                                                  .put("fullobject", null)
-                                                  .put("multiNumber", null)
-                                                  .put("name", object(field("displayName", null),
-                                                                      field("surname", null)))
-                                                  .put("schemas", array("urn:scim:schemas:core:1.0"))
-                                                  .put("singleNumber", null));
+        checkResourcesAreEqual(resources.get(0), filteredJsonValue());
     }
 
     private Promise<QueryResponse, ResourceException> delegateBySubstitutingQuery(RequestHandler requestHandler,
Comment by Dirk Hogan [ 09/Aug/19 ]

Yes - I also believe that it is an issue on the IDM side. I've filed OPENIDM-13657 to take care of this, and related, issues. I think it would be fine to close this Jira - it does not appear to be a defect in rest2ldap.

Comment by Jean-Noël Rouvignac [ 09/Aug/19 ]

Closing as per the latest comment from Dirk.

From our understanding of the design, the fields must be filtered by the callers of the request handler.

Side note on efficiency: Rest2LDAP is busy generating null JsonValue that will be filtered later. A small optimization would be to not generate them in the first place:

 * ... Request handler and resource provider
 * implementations SHOULD NOT filter fields, but MAY choose to optimise their
 * processing in order to return a resource containing only the fields targeted
 * by the field filters.
Comment by Matthew Swift [ 07/Nov/19 ]

Moved to closed state because the fixVersion has already been released.

Generated at Wed Mar 03 02:54:28 UTC 2021 using Jira 7.13.12#713012-sha1:6e07c38070d5191bbf7353952ed38f111754533a.