[OPENAM-6545] ServerInfoResource should attempt to cache ServiceConfigs per realm rather than creating one on each request Created: 06/Aug/15  Updated: 20/Nov/16  Resolved: 13/Aug/15

Status: Resolved
Project: OpenAM
Component/s: performance, rest
Affects Version/s: 12.0.0, 12.0.1, 13.0.0
Fix Version/s: 12.0.2, 12.0.3, 13.0.0

Type: Bug Priority: Major
Reporter: Mark de Reeper Assignee: Mark de Reeper
Resolution: Fixed Votes: 0
Labels: EDISON, release-notes
Remaining Estimate: 0h
Time Spent: 8h
Original Estimate: 0h

Target Version/s:
Sprint: Sustaining Sprint 10
Support Ticket IDs:
Verified Version/s:

 Description   

During the investigation of a performance issue it was observed that ServerInfoResource.getServiceConfig() was constructing a ServiceConfigManager on each invocation which under heavy loads can lead to blocked threads in ServiceConfigImpl

"http-bio-8080-exec-124" daemon prio=5 tid=0x00007fd4648a0800 nid=0x2c603 waiting for monitor entry [0x000000012cca5000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at java.util.Collections$SynchronizedMap.get(Collections.java:2037)
	- waiting to lock <0x00000007c2942d30> (a java.util.Collections$SynchronizedMap)
	at com.sun.identity.sm.ServiceConfigImpl.getFromCache(ServiceConfigImpl.java:600)
	at com.sun.identity.sm.ServiceConfigImpl.getInstance(ServiceConfigImpl.java:486)
	at com.sun.identity.sm.ServiceConfigImpl.getInstance(ServiceConfigImpl.java:464)
	at com.sun.identity.sm.ServiceConfigManagerImpl.getOrganizationConfig(ServiceConfigManagerImpl.java:254)
	at com.sun.identity.sm.ServiceConfigManager.getOrganizationConfig(ServiceConfigManager.java:277)
	at org.forgerock.openam.forgerockrest.server.ServerInfoResource.getServiceConfig(ServerInfoResource.java:235)

Caching the ServerConfig per realm is one approach to solving the problem.



 Comments   
Comment by Mark de Reeper [ 13/Aug/15 ]

Fixed in r15114 and r15115.

Comment by Nemanja Lukic [ 29/Sep/15 ]

Verified in: OpenAM 12.0.2 Build 15797 (2015-September-21 17:41)

See OPENAM-6501 for description about how it was tested.

Comment by Richard Hruza [ 05/May/16 ]

I executed a jmeter HTTP requests:
GET /openam/json/serverinfo/*
GET /openam/json/subrealm/serverinfo/*

These requests were executed in 4 threads for 5000 loops = 40 000 requests. 40000 requests were done in 27.3s .I did not observe error in the debug log. See also OPENAM-6501 for more details.

OpenAM 12.0.3-RC2 Build 4dbe218a05 (2016-April-25 17:57)
Creating summariser <summary>
Created the tree successfully using /opt/riso/stability-testing/Get_server_info_with_subrealm.jmx
Starting the test @ Thu May 05 10:12:31 BST 2016 (1462439551895)
Waiting for possible shutdown message on port 4445
summary +      1 in   0.1s =   17.9/s Avg:    56 Min:    56 Max:    56 Err:     0 (0.00%) Active: 3 Started: 3 Finished: 0
summary +  39999 in  27.3s = 1465.3/s Avg:     2 Min:     0 Max:   121 Err:     0 (0.00%) Active: 0 Started: 0 Finished: 0
summary =  40000 in  27.3s = 1465.3/s Avg:     2 Min:     0 Max:   121 Err:     0 (0.00%)
Tidying up ...    @ Thu May 05 10:12:59 BST 2016 (1462439579857)
... end of run
Generated at Tue Oct 27 02:53:49 UTC 2020 using Jira 7.13.12#713012-sha1:6e07c38070d5191bbf7353952ed38f111754533a.