[OPENIDM-11601] OpenIDM Needs to Rework how it Queries the repo for Assignments and Roles Created: 28/Aug/18  Updated: 27/Jul/20  Resolved: 18/May/20

Status: Closed
Project: OpenIDM
Component/s: None
Affects Version/s: OpenIDM 5.0.0, OpenIDM 6.0.0
Fix Version/s: 7.0.0

Type: Improvement Priority: Major
Reporter: Jeremy Barras [X] (Inactive) Assignee: Dirk Hogan
Resolution: Fixed Votes: 0
Labels: CLARK
Σ Remaining Estimate: Not Specified Remaining Estimate: Not Specified
Σ Time Spent: Not Specified Time Spent: Not Specified
Σ Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relates
relates to OPENIDM-14528 Relationship signal propagation not w... Closed
relates to OPENIDM-11785 Syncing change notifications from "no... Closed
relates to OPENIDM-14093 Handle role temporal constraints with... Closed
relates to OPENIDM-14518 Write functional tests for additional... Closed
is related to OPENIDM-14314 Performance degradation when using qu... Closed
is related to OPENIDM-14189 adjust test cases to reflect that eff... Closed
Sub-Tasks:
Key
Summary
Type
Status
Assignee
OPENIDM-14222 Traverse the queryConfig elements of ... Sub-task Closed Dirk Hogan  
OPENIDM-14223 Parse the VirtualPropertyQueryState i... Sub-task Closed Dirk Hogan  
Target Version/s:
Verified Version/s:
Story Points: 5
Sprint: 2019.16 - IDM, 2019.17 - IDM, 2020.01 - IDM, 2020.02 - IDM, 2020.03 - IDM, 2020.04 - IDM, 2020.05 - IDM, 2020.06 - IDM, 2020.07 - IDM
Support Ticket IDs:

 Description   

Description:
We are seeing poor performance in our load testing of IDM 6.0 with Postgresql as the repo. We are testing with about 70k users.

Troubleshooting:
In looking at the performance tuning information in Postgresql and using p6spy (https://github.com/p6spy/p6spy) we noticed a couple of things that seem to be contributing to the poor performance.

1. IDM is querying for conditional roles, individual role, and individual assignments hundreds of thousands of times each during the course of a recon (on average 8-10 queries per user). We reviewed the queries with our DBA but he wasn't able to offer any help in speeding this up.

2. When a user is being recon'd from Banner to managed/user, the onRecon.groovy script does not query for all assignments so the implicit syncs that are triggered to the downstream systems like AD have to make an additional query for their assignments.

Request:
Reduce the number of unnecessary queries IDM makes to the repo through caching or another mechanism so IDM can scale better.

Blockage:
Can't proceed in our efforts to upgrade to OpenIDM 6.0 with the current performance levels. Our workaround looks like it may work.

Workaround:
Role and Assignment data is static for us and changes maybe once every six months or so.
1. We have a script now that will cache this static data into the properties section of script.json.
2. onRecon.groovy doesn't do anything any longer.
3. conditionalRoles.js, effectiveRoles.js, and effectiveAssignments.js now use the global variables from script.json instead of querying the repo.



 Comments   
Comment by Rachel Louden [ 07/Sep/18 ]

We have added a call to our update cache script for the role and assignment onPostUpdate, onPostCreate, and onPostDelete events.

 

 

Comment by Dirk Hogan [ 27/Jul/20 ]

Verified with the relationship_derived_virtual_props functional test suite, and the provisioning_roles test suite which contains enhanced checks to verify the correctness of effectiveRoles and effectiveAssignments, obtained both from /managed and /repo.

Generated at Thu Apr 22 20:07:45 UTC 2021 using Jira 8.16.0#816000-sha1:a455b91378454416b49bbc88d03e653cb9815ed5.