Uploaded image for project: 'OpenAM'
  1. OpenAM
  2. OPENAM-17591

Session quota destroy next expiring action can fail when two new sessions attempt to read and update the same expiring session

    XMLWordPrintable

Details

    • Rank:
      1|i0426w:
    • AM Sustaining Sprint 84, AM Sustaining Sprint 85
    • 2
    • Yes
    • No
    • Yes and I used the same an in the description, Yes but I used my own steps. (If so, please add them in a new comment)
    • 0
    • Future
    • None

    Description

      Description:

      Session quota destroy next expiring session can fail if there are two logins at the same time and they compete to destroy the same next expiring session.

      The end user sees MAX_SESSION_REACHED authentication failure and the audit logs indicate the same.

      Reproduction steps:

      A customer has only seen this occasionally. Its not been reproducible internally but may be possible to simulate using the following:

      1. Deploy AM, external config and user store (so that CTS is external to AM).
      2. Login as amadmin, navigate to Configure -> Global Services -> Session, Session Quotas and enable quotas. Leave other settings as defaults, so action is Destroy Next Expiring token, read timeout is 6000. On Dynamic attributes page, max set max number of sessions allowed to 2.
      3. Login as user demo (to create first session for user) using e.g.: curl -X POST -k -H 'Content-type: application/octet-stream' -H 'X-OpenAM-username: demo' -H 'X-OpenAM-Password: changeit' --header 'Accept-API-Version: protocol=1.0,resource=2.0' 'https://openam.amtest2.com:8443/access/json/authenticate  Note the ssoToken returned.
      1. Repeat the login.
      2. Using the debugger, set a breakpoint in DestroyNextExpiringAction.action, somewhere in the call stack below line 83 (6.5.2.2) for sessionToRemove = sessionCache.getSession(sessionD); (InMemoryCtsSessionCacheStep.getSessionByID so that the session has been read but the update has not yet been attempted).
      3. Perform a new login.
      4. When the above breakpoint is hit, run an ldap command direct to CTS to delete the token about to be read by the first thread.
      5. Then let the debugger thread continue.

      Expected behaviour

      The logic should attempt to remove another expiring session and then return a successful new one for the current authentication.

      Current behaviour:

      The authentication fails with MAX_SESSION_REACHED.

      Code analysis:

      The error and stack trace seen when this fails is:

       

      12:15:16:802 PM GMT: Thread[idleTimeout1193,5,main]: TransactionId[ea715237-f676-4ce5-b135-3e98ae7cb4b6-9102822]
      ERROR: Unable to persist session to data store, check documentation for CTS configuration to ensure reads and writes may occur, and all appropriate virtual attributes are enabled.
      org.forgerock.openam.cts.exceptions.CoreTokenException:
      CTS:
      CTS: Operation failed:
      Result Code: No Such Entry
      Diagnostic Message: Entry coreTokenId=CiE1xZsbTzTED1qw5Sgkm02bxQ8=,ou=famrecords,ou=openam-session,ou=tokens,dc=hsbc,dc=com cannot be modified because no such entry exists in the server
      Matched DN: ou=famrecords,ou=openam-session,ou=tokens,dc=hsbc,dc=com
              at org.forgerock.openam.cts.impl.DataLayerExceptionMapper.map(DataLayerExceptionMapper.java:29)
              at org.forgerock.openam.cts.impl.queue.AsyncResultHandler.getResults(AsyncResultHandler.java:91)
              at org.forgerock.openam.cts.monitoring.impl.queue.TokenMonitoringResultHandler.getResults(TokenMonitoringResultHandler.java:42)
              at org.forgerock.openam.cts.monitoring.impl.queue.TokenMonitoringResultHandler.getResults(TokenMonitoringResultHandler.java:19)
              at org.forgerock.openam.cts.CTSPersistentStoreImpl.upsert(CTSPersistentStoreImpl.java:158)
      . . .
      com.iplanet.dpro.session.service.SessionProvider.getSession(SessionProvider.java:43)
              at org.forgerock.openam.session.service.DestroyNextExpiringAction.action(DestroyNextExpiringAction.java:83)
              at com.iplanet.dpro.session.service.SessionConstraint.checkQuotaAndPerformAction(SessionConstraint.java:146)
              at org.forgerock.openam.session.cts.CtsSessionBuilder.build(CtsSessionBuilder.java:94)
              at com.iplanet.dpro.session.monitoring.MonitoredBuilder.build(MonitoredBuilder.java:93)
              at com.sun.identity.authentication.service.LoginState.produceSessionFromState(LoginState.java:1127)
      . . .
      Caused by: org.forgerock.openam.sm.datalayer.api.LdapOperationFailedException:
      CTS: Operation failed:
      Result Code: No Such Entry
      Diagnostic Message: Entry coreTokenId=CiE1xZsbTzTED1qw5Sgkm02bxQ8=,ou=famrecords,ou=openam-session,ou=tokens,dc=hsbc,dc=com cannot be modified because no such entry exists in the server
      Matched DN: ou=famrecords,ou=openam-session,ou=tokens,dc=hsbc,dc=com
              at org.forgerock.openam.cts.impl.LdapAdapter.update(LdapAdapter.java:190)
              at org.forgerock.openam.sm.datalayer.impl.tasks.UpsertTask.performTask(UpsertTask.java:62)
      

      i.e. the logic that performs the getSession looks to also be performing an update (update of expiry perhaps)? This fails because another thread has already deleted the token.

       

       

       

      Attachments

        Activity

          People

            lawrence.yarham Lawrence Yarham
            lawrence.yarham Lawrence Yarham
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: