[OPENAM-10332] Quota constraints exceeded - Interim Fix Created: 05/Jan/17  Updated: 02/Sep/20  Resolved: 22/Sep/17

Status: Resolved
Project: OpenAM
Component/s: CTS
Affects Version/s: 12.0.2
Fix Version/s: 13.5.1, 14.5.0

Type: Bug Priority: Major
Reporter: Andy Hall Assignee: Ken Stubbings
Resolution: Fixed Votes: 0
Labels: EDISON, Must-Fix
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by OPENAM-10603 Login page error "Maximum sessions li... Resolved
Relates
relates to OPENAM-5864 Quota constraints exceeded in multi-i... Closed
is related to OPENAM-16031 Intermittent error message when concu... Resolved
Target Version/s:
Sprint: AM Sustaining Sprint 39
Story Points: 3
Needs backport:
No
Support Ticket IDs:
Verified Version/s:
Needs QA verification:
No
Functional tests:
No
Are the reproduction steps defined?:
Yes and I used the same an in the description

 Description   

See OPENAM-5864 for Full Description, but in some circumstances, Quotas are being exceeded.
Current thinking is that this is a CTS race condition.
The long term solution will be addressed under 5864 so this issue represents work needed for an interim solution.

One idea would be a service which would run after sessions are created and persisted, mopping up any discrepancies in session quotas allowance and number of running sessions. (Belt and braces)



 Comments   
Comment by Andy Hall [ 05/Jan/17 ]

OPENAM-5864 covers the longer term approach.
This is the interim fix.

Comment by Robert Wapshott [ 02/Feb/17 ]

Copied from this comment

Confirmed, we are able to reproduce an issue which looks highly related to the described issue:

  • Site
  • Server part of site
  • Session Failover enabled
  • Session Quota enabled
  • Active Sessions limited to 2

The Apache Bench command then can demonstrate this.

No concurrency:

SERVER=rwapshott.forgerock.com:8080/openam && ab -m POST -T "Content-Type: application/json" -H "X-OpenAM-Username: demo" -H "X-OpenAM-Password: changeit" -n 100 -c 1 http://$SERVER/json/authenticate

Will leave the server with only two sessions for the demo user.

However moving to multiple concurrency:

SERVER=rwapshott.forgerock.com:8080/openam && ab -m POST -T "Content-Type: application/json" -H "X-OpenAM-Username: demo" -H "X-OpenAM-Password: changeit" -n 100 -c 2 http://$SERVER/json/authenticate

We can see that there are now more than two sessions for the demo user in the sessions tab.

The current working hypothesis is as follows: there is a concurrency bug in the DestroyNextExpiringAction class which is not aware of the number of sessions to destroy and so will not consistently destroy them.

Comment by Ken Stubbings [ 07/Feb/17 ]

Note - the problem is reproducible on a single server, so the concurrency issue is not specifically caused by multi server concurrency issues but general concurrency issues.

Comment by Ken Stubbings [ 07/Feb/17 ]

The problem seems to be that when there are multiple concurrent requests to create a session for a use the delete oldest session action becomes unreliable (possibly because it is being called multiple times for the same session) and it becomes difficult to know how many sessions are currently in the process of being created for the purpose of calculating the session count.
Attempts to resolve these problems without massive restructuring of the code (bearing in mind that these areas are scheduled for significant refactor after the release of 14.0.0 and have already changed a lot) have not been successful so currently efforts are focused on creating a post authentication plugin to act as a session count checker.

Comment by Ken Stubbings [ 14/Mar/17 ]

Fix is in sustaining 12.0.x, awaiting port to 13.5.x

Comment by Ken Stubbings [ 14/Mar/17 ]

Note that the nature of the fix for this was to allow the session quota exceeded action objects to handle a situation where more than one excess session can be handled. It is not possible to support multi server concurrence and guarantee that there will not be sessions in excess of the quota from time to time. By allowing session quota exceeded actions to tidy up more than once excess sessions we can allow the system to recover when there are too many rather than staying in the exceeded state.
Other alternative approaches had the affect of increasing the collisions on deletion and thereby increasing the likelihood of getting into the state where there were excess sessions.

Comment by Jonathan Thomas [ 16/Mar/17 ]

Ken Stubbings It's an the target branches - reopen if needed.

Comment by Sachiko Wallace [ 24/May/17 ]

There is a possibility that sessions that was retrieved by the following code in SessionConstraint. checkQuotaAndPerformAction() could've been cleared by some other instances of OpenAM :

    // Step 2: get the information (session id and expiration
    // time) of all sessions for the given user from all
    // AM servers and/or session repository
    Map sessions = null;
    try {
        sessions  =
        SessionCount.getAllSessionsByUUID(is.getUUID());

And if this sessions list is handed over to action handler class such as DestroyNextExpiringAction.java, it might throw SessionException and wrongly marks rejectNewSession=true.

Also, shouldn't removed++; occur after destroySession went through?

Comment by Sachiko Wallace [ 15/Jun/17 ]

additional fix only merged into 13.5.x since 12.0.5 is not likely to be released.

Comment by Philip Anderson [ 06/Jul/17 ]

Verified on 13.5.1-RC5

Comment by Andy Hall [ 07/Sep/17 ]

Reopening to forward port Intermim Fix to 14.5.0

Generated at Thu Sep 24 14:21:29 UTC 2020 using Jira 7.13.12#713012-sha1:6e07c38070d5191bbf7353952ed38f111754533a.