[OPENIDM-14714] Response time increased significantly with certain read operations of managed users with relationship using DS as repo Created: 12/May/20  Updated: 20/Jun/20  Resolved: 11/Jun/20

Status: Closed
Project: OpenIDM
Component/s: Module - Relationships
Affects Version/s: 7.0.0
Fix Version/s: 7.0.0

Type: Bug Priority: Critical
Reporter: Tinghua Xu Assignee: Dirk Hogan
Resolution: Cannot Reproduce Votes: 0
Labels: LEWIS, Performance, regression
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

IDM 7.0.0 master(c286e03) runs on a CentOS 7 machine, Java11, external DJ(7.0.0-M2020-6.1) as repo. Vs. IDM 6.5.0.3 with DJ 6.5.2


Attachments: File config.cfg-6.5.0     File config.cfg-7.0.0     File idm-metrics-read-no-expansion-6.5.0-dj     File idm-metrics-read-no-expansion-7.0.0-dj     File idm-metrics-read-role-expansion-6.5.0-dj     File idm-metrics-read-role-expansion-7.0.0-dj     File metrics-comparison-read-no-expansion-6.5.x-7.0.0-dj     File metrics-comparison-read-role-expansion-6.5.x-7.0.0-dj    
Issue Links:
Relates
relates to OPENIDM-14715 Response time increased significantly... Closed
Target Version/s:
Verified Version/s:
Story Points: 3
Sprint: 2020.07 - IDM, 2020.08 - IDM

 Description   

The test preload 30K users with managed roles and managers, then either read the user w/o link expansion as managed/user/A-A-123 or w/ link expansion on roles like managed/user/A-A-123?_fields=,roles/. Between 6.5.0.3 and 7.0.0, we see mean response time(throughput drops) jumped significantly:

Tests 6.5.0 Throughput 6.5.0 Mean response time 7.0.0
Throughput
7.0.0 Mean response time Mean response time(ms) diff in percentage
Read w/o expansion 126 237 103 290 22
Read w/ expansion on roles 121 247 95 313 27

 

Attached are the idm metrics collected during the tests. From both tests, noticed that, 7.0.0 needs more connections to DS and the mean response time on the connection is higher than 6.5(F1 is for 6.5, F2 is for 7.0) 

id : repo.ds.get-connection
F1-mean: 4.599533146591592
F1-count: 1787293
F2-mean: 7.708673457367644
F2-count: 3831051
F2-F1 total: 21311607.761350643

To reproduce the symptom:
Use the config attached and run the following tests between 6.5 and 7.0:

./run-pybot.py -v -c perf -s relationship.ReadManUsersWithRolesManager -t IDM_Read_User_With_No_Link_Expansion OpenIDM

or 

./run-pybot.py -v -c perf -s relationship.ReadManUsersWithRolesManager -t IDM_Read_User_With_Roles_Link_Expansion OpenIDM


 Comments   
Comment by Dirk Hogan [ 11/Jun/20 ]

Cannot reproduce. The fact that https://bugster.forgerock.org/jira/browse/OPENIDM-14715 could not be reproduced makes it very unlikely that this is a bug - the problem would have to be in IDM's DS repo layer. I have been unable to run both IDM 7.0 and IDM 6.5.3 bits against the same DS version. Until that is possible, there is no way to determine whether the putative latency increase results from IDM or DS changes. If the degradation can only be 'pinpointed' to 1.5 years of changes, then at least this 'pinpointing' has to be limited to a single product. 

Please reopen iff a >10% degradation can be consistently reproduced between IDM 6.5.3 and 7.0 bits running against the same DS version, with precise reproduction steps documenting how to make this work in pyforge. 

Comment by Tinghua Xu [ 20/Jun/20 ]

Due to the significant improvement merged after this Jira was filed and the difficulty to bisect trace old commit. I'm ok we don't do anything on this. In current master, we have mean response time/throughput pairs:

For read w/o link expansion: 30/994

For read w/ link expansion: 139/108

both are much better than 6.5.0

 

Generated at Sat Nov 28 10:39:56 UTC 2020 using Jira 7.13.12#713012-sha1:6e07c38070d5191bbf7353952ed38f111754533a.