Major 1. authenticate with amadmin
curl --request POST --header "X-OpenAM-Username: amadmin" --header "X-OpenAM-Password: cangetin" --header "Content-Type: application/json" --data "{}" "http://openam.example.com:18080/openam/json/authenticate"
2. try to delete non-existent policy for (policy datastore's max connection +1) times
$curl -X DELETE -k --header "iPlanetDirectoryPro: AQSfcxBgucoN2I-jgWlAxeJN9LYzZ-GdAl0vBc.*AAJTSQACMDEAAlNLABQtNzY4ODk5NTM0MDY4NTk4NzM3NAACUzEAAA..*" --header "Content-Type: application/json" "http://openam.example.com:18080/openam/json/policies/mynonexistentpolicy"
2-repro. you will see response not coming back on 11th time
You will see threads waiting on the connection to be available with PromiseImpl. But since SMSLdapObject doesn't close connection and release it to the CachedConnectionPool, it just sits and wait forever.
"http-bio-1300-exec-8" daemon prio=10 tid=0x00007fb53006d800 nid=0xc284 in Object.wait() [0x00007fb4f972d000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000fb31bcc0> (a org.forgerock.util.promise.PromiseImpl) at java.lang.Object.wait(Object.java:503) at org.forgerock.util.promise.PromiseImpl.await(PromiseImpl.java:572) - locked <0x00000000fb31bcc0> (a org.forgerock.util.promise.PromiseImpl) at org.forgerock.util.promise.PromiseImpl.getOrThrow(PromiseImpl.java:143) at org.forgerock.opendj.ldap.CachedConnectionPool.getConnection(CachedConnectionPool.java:789) at com.sun.identity.sm.ldap.SMDataLayer.getConnection(SMDataLayer.java:107) at com.sun.identity.sm.ldap.SMSLdapObject.getConnection(SMSLdapObject.java:574) at com.sun.identity.sm.ldap.SMSLdapObject.searchSubOrganizationNames(SMSLdapObject.java:910) at com.sun.identity.sm.ldap.SMSLdapObject.searchSubOrgNames(SMSLdapObject.java:887)
In the normal case where ldap operations succeed, CachedConnectionPool$ConnectionResultHandler.handleResult() will publishConnection so the connection will be available for the next incoming request. However, CachedConnectionPool$ConnectionFailureHandler.handleException() just puts the connection in pending state and wait for the subclass of this pool to handle exception. The solution would be to put close() in finally/exception clause or handle exception properly with failure handler class.
A quick test on the below code confirmed fix to the problem
private ConnectionEntryReader searchObjectsEx(SSOToken token,
String startDN, String filter, int numOfEntries, int timeLimit,
boolean sortResults, boolean ascendingOrder, Connection conn
) throws SSOException, SMSException {
:
try {
ConnectionEntryReader iterResults = conn.search(request);
iterResults.hasNext();
results = iterResults;
break;
} catch (LdapException e) {
:
} finally {
+ IOUtils.closeIfNotNull(conn);
}
}
return results;
}