[OPENIDM-13953] embedded DJ explicit mapping returns null instead of [] Created: 17/Oct/19  Updated: 04/Jun/20  Resolved: 29/May/20

Status: Closed
Project: OpenIDM
Component/s: Module - Repository DS
Affects Version/s: 7.0.0
Fix Version/s: 7.0.0

Type: Bug Priority: Major
Reporter: Ladislav Folta Assignee: Dirk Hogan
Resolution: Fixed Votes: 0
Labels: LEWIS, regression
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OpenIDM: 7.0.0-SNAPSHOT 55394c2


Issue Links:
Duplicate
is duplicated by OPENIDM-12125 effectiveAssignments and effectiveRol... Closed
Target Version/s:
Verified Version/s:
Story Points: 3
Sprint: 2020.06 - IDM, 2020.07 - IDM
Epic Link: DS as a shared repo

 Description   

When creating user with on embedded dj explicit mapping, the effectiveAssignments is returned as null instead of an empty array like all other repositories do.

 	curl --header "X-OpenIDM-Username: openidm-admin" --header "X-OpenIDM-Password: openidm-admin" --header "If-None-Match: *" --header "Content-Type: application/json" --data '{"id": "ricksutter", "userName": "rsutter", "givenName": "rick", "sn": "sutter", "mail": "rick@example.com", "telephoneNumber": "6669876987", "password": "Th3Password", "description": "Just another user", "roles": [{"_ref": "managed/role/role1"}, {"_ref": "managed/role/role2"}]}' --request PUT "http://localhost:8080/openidm/managed/user/ricksutter"

Response Content:
{
	"effectiveAssignments": null,  <=== THIS
...
	"_id": "ricksutter",
	"_rev": "00000000f67000a4",
	"userName": "rsutter",
	"accountStatus": "active",
	"effectiveRoles": [{
		"_ref": "managed/role/role1"
	}, {
		"_ref": "managed/role/role2"
	}],
	"postalCode": null,
	"description": "Just another user",
	"givenName": "rick",
	"sn": "sutter",
	"telephoneNumber": "6669876987",
	"mail": "rick@example.com",
	"id": "ricksutter",
	"history": null
}

vs generic embedded dj:

curl --header "X-OpenIDM-Username: openidm-admin" --header "X-OpenIDM-Password: openidm-admin" --header "If-None-Match: *" --header "Content-Type: application/json" --data '{"id": "ricksutter", "userName": "rsutter", "givenName": "rick", "sn": "sutter", "mail": "rick@example.com", "telephoneNumber": "6669876987", "password": "Th3Password", "description": "Just another user", "roles": [{"_ref": "managed/role/role1"}, {"_ref": "managed/role/role2"}]}' --request PUT "http://localhost:8080/openidm/managed/user/ricksutter"

Response Content:
{
	"effectiveAssignments": []  <=== THIS
	"_id": "ricksutter",
	"_rev": "00000000db180904",
	"id": "ricksutter",
	"userName": "rsutter",
	"givenName": "rick",
	"sn": "sutter",
	"mail": "rick@example.com",
	"telephoneNumber": "6669876987",
	"description": "Just another user",
	"accountStatus": "active",
	"effectiveRoles": [{
		"_ref": "managed/role/role1"
	}, {
		"_ref": "managed/role/role2"
	}],
}

or mysql explicit:

 	curl --header "X-OpenIDM-Username: openidm-admin" --header "X-OpenIDM-Password: openidm-admin" --header "If-None-Match: *" --header "Content-Type: application/json" --data '{"id": "ricksutter", "userName": "rsutter", "givenName": "rick", "sn": "sutter", "mail": "rick@example.com", "telephoneNumber": "6669876987", "password": "Th3Password", "description": "Just another user", "roles": [{"_ref": "managed/role/role1"}, {"_ref": "managed/role/role2"}]}' --request PUT "http://localhost:8080/openidm/managed/user/ricksutter"

Response Content:
{
	"effectiveAssignments": [], <=== THIS
	"_id": "ricksutter",
	"_rev": "0",
	"userName": "rsutter",
	"accountStatus": "active",
	"givenName": "rick",
	"description": "Just another user",
	"sn": "sutter",
	"telephoneNumber": "6669876987",
	"mail": "rick@example.com",
	"kbaInfo": null,
	"preferences": null,
	"consentedMappings": null,
	"effectiveRoles": [{
		"_ref": "managed/role/role1"
	}, {
		"_ref": "managed/role/role2"
	}],
	"active": null,
	"id": "ricksutter",
	"history": null
...
}


 Comments   
Comment by Brendan Miller [ 23/Oct/19 ]

Ladislav Folta As you marked this a regression, can you provide when/where it broke?

Comment by Ladislav Folta [ 30/Oct/19 ]

I went all the way back to M1 tag in January and it's still there. So it's been there for really long time.

Comment by Dirk Hogan [ 07/May/20 ]

The linked commentary very useful for understanding:

https://bugster.forgerock.org/jira/browse/OPENIDM-12125?focusedCommentId=141184&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-141184

Comment by Dirk Hogan [ 08/May/20 ]

The issue is complicated. If https://bugster.forgerock.org/jira/browse/OPENDJ-5722 were implemented, DS would always return [] for an unspecified multi-valued field, but these are not the semantics for JDBC, which can distinguish between an empty list and null for multi-valued fields. Thus even if virtual property calculation were always mandated to return [] for an empty, multi-valued field (which is not currently the case for effectiveRoles/Assignments, or their java-based replacement), the DS/jdbc discrepancy would persist for non-virtual multi-valued fields: [] can be explicitly stored in jdbc, but not in DS, thereby allowing jdbc to distinguish null and an empty list for multi-valued fields. If DS always returned [] for null multi-valued attributes, effectiveRoles and effectiveAssignments might converge, but non-virtual multi-valued fields would not. If they are not explicitly specified in jdbc, null is returned, but post-OPENDJ-5722, DS will always return []. If they are explicitly specified as [], jdbc will return [], but pre-OPENDJ-5722 will return null. 

We discussed implementing these 'convergence' semantics in ContentFilteringSupport, but this would still leave the discrepancy at the repo layer, and prior to the managed prepareResponse processing, where ContentFilteringSupport is engaged. This discrepancy means that e.g. scripts would have to handle both cases. It would also leave the repo API divergent between DS and jdbc.

We discussed implementing this at a Filter in front of the repo layer, much like the 'edge augmentation' Filter for DS. Such a Filter would have to know about the field types defined in the managed layer, a bleeding across semantic layers which also should be avoided.    

Comment by Dirk Hogan [ 11/May/20 ]

The decision was made to add a filter layer to managed/internal, via the AugmentingIDMConnectionFactoryProxy, to process requests as they return from the repo, and replace any null-valued fields of type array, as specified by the managed/internal schema, with '[]'.

Comment by Ladislav Folta [ 04/Jun/20 ]

Verified OK on 7.0.0-SNAPSHOT dfd37e7

Generated at Wed Nov 25 07:34:51 UTC 2020 using Jira 7.13.12#713012-sha1:6e07c38070d5191bbf7353952ed38f111754533a.