Uploaded image for project: 'OpenAM'
  1. OpenAM
  2. OPENAM-13740

File descriptor / Connection leak when LDAP connection handshake fails/times out

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 14.1.0, 14.1.1, 5.5.1, 6.0.0, 6.0.0.1, 6.0.0.2, 6.0.0.3, 6.0.0.4, 6.0.0.5
    • Fix Version/s: 6.5.0, 5.5.2
    • Component/s: CTS
    • Labels:
    • Target Version/s:
    • Support Ticket IDs:

      Description

      Bug description

      CTS connection leaks are observed during CTS failover

      How to reproduce the issue

      Setup 2 external CTS store

      "org.forgerock.services.cts.store.location" : "external",
       "org.forgerock.services.cts.store.root.suffix" : "ou=famrecords,ou=openam-session,ou=tokens,dc=cts,dc=example,dc=com",
       "org.forgerock.services.cts.store.max.connections" : "10"
       },
       "amconfig.org.forgerock.services.cts.store.external.section" : {
       "org.forgerock.services.cts.store.ssl.enabled" : false,
       "org.forgerock.services.cts.store.directory.name" : "openam.internal.example.com:41389,openam.internal.example.com:51389",
       "org.forgerock.services.cts.store.loginid" : "cn=Directory Manager",
       "org.forgerock.services.cts.store.password" : null,
       "org.forgerock.services.cts.store.heartbeat" : "10",
       "org.forgerock.services.cts.store.affinity.enabled" : false

       

      Fire a first curl authentication command against the OpenAM

      curl -s --request POST --header 'Accept-API-Version: resource=2.0, protocol=1.0' --header 'X-OpenAM-Username: demo' --header 'X-OpenAM-Password: changeit' --header 'Content-Type: application/json' --data '{}' http://openam.internal.example.com:8080/openam/json/authenticate

      run this command to observe the connection statistics

      lsof -p < openam process> | grep  < cts port or hostname >

      In my case, both CTS 's ports are 41389 and 51389

      CTS1 - 41389

      CTS2 - 51389

      lsof -p 31330 | grep 1389 
      
      java 31330 iplanet 379u IPv6 560711 0t0 TCP openam.internal.example.com:60688->openam.internal.example.com:41389 (ESTABLISHED)
      java 31330 iplanet 394u IPv6 560733 0t0 TCP openam.internal.example.com:60694->openam.internal.example.com:41389 (ESTABLISHED)
      java 31330 iplanet 398u IPv6 561542 0t0 TCP openam.internal.example.com:60692->openam.internal.example.com:41389 (ESTABLISHED)
      java 31330 iplanet 422u IPv6 560755 0t0 TCP openam.internal.example.com:60702->openam.internal.example.com:41389 (ESTABLISHED)
      java 31330 iplanet 424u IPv6 561555 0t0 TCP openam.internal.example.com:60700->openam.internal.example.com:41389 (ESTABLISHED)

       Stop the CTS store at port 41389 

      I run this command to temporary disabled the port

      kill -stop 30605 

      The following message was observed

      WARNING: Connection factory 'CachedConnectionPool(size=0[in:0 + out:0 + pending:0], maxSize=10, blocked=0, ldapClient=org.forgerock.opendj.ldap.LdapClientImpl@70d3e47)' is no longer operational: Connect Error: The connection attempt to server openam.internal.example.com/172.28.1.112:41389 has failed because the connection timeout period of 10000 ms was exceeded
      org.forgerock.opendj.ldap.LoadBalancer:10/10/2018 02:13:06:104 PM SGT: Thread[OpenDJ LDAP SDK Default Scheduler,5,main]: TransactionId[e6c56d67-ea11-439c-a800-3bcad7690831-91]
      Starting monitoring thread

       

      Fire a second curl authentication command against the OpenAM to trigger the failover

      curl -s --request POST --header 'Accept-API-Version: resource=2.0, protocol=1.0' --header 'X-OpenAM-Username: demo' --header 'X-OpenAM-Password: changeit' --header 'Content-Type: application/json' --data '{}' http://openam.internal.example.com:8080/openam/json/authenticate

      Grep for the connection statistics again

      lsof -p 31330 | grep 41389 | wc -l
       114  <=========

      Recover the CTS1 at port 41389 again

      kill -cont 30605 

      Re-run the lsof command again

       

      lsof -p 31330 | grep 41389 | wc -l
      
      145   <===========

       

      There is a connection leak. You should see a lot of "ESTABLISHED" connections.

      Expected behaviour
      No connection leak 
      
      Current behaviour
      There is a connection leak
      

       

      Workaround

      ==========

      Restart OpenAM
      (Upgrade to AM6.5) (does not affect AM 6.5.0 as it used SDK DJ 6.5)

       

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jonthomas Jonathan Thomas
                Reporter:
                sam.phua Sam Phua
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: