[OPENAM-5864] Quota constraints exceeded in multi-instance with LB and CTS enabled Created: 21/Apr/15  Updated: 06/Oct/17  Resolved: 06/Oct/17

Status: Closed
Project: OpenAM
Component/s: CTS, session
Affects Version/s: 11.0.2, 12.0.2, 13.0.0
Fix Version/s: 14.5.0

Type: Bug Priority: Major
Reporter: Nathalie Hoet Assignee: Unassigned
Resolution: Fixed Votes: 3
Labels: AME
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates
is related to OPENAM-10332 Quota constraints exceeded - Interim Fix Resolved
Sprint: AM Sustaining Sprint 15
Support Ticket IDs:

 Description   

Environment:

2 OpenAM 11.0.2 instances behind HA_Proxy
Session Failover enabled (CTS within embedded config in my test)
Site defined
ha_proxy load balancing in round robin between am1 and am2
Quota constraint set to 2 per user
DESTROY_NEXT_EXPIRING set up

Test:

A simple curl that throws the same authentication request for the same user; loop 5000 times:

curl --request POST --header "X-OpenAM-Username: testuser" --header "X-OpenAM-Password: password" --header "Content-Type: application/json" --data "{}" http://lb.example.com:8080/openam/json/authenticate

In the Session log file, you can see that the number of sessions is being checked in order to decide whether the quota has been reached. At the beginning of the logs I can see it stays at 2 existing sessions - as it should -, then it moves on to 3 and does not go back down.

amCoreTokenService:04/16/2015 05:21:07:041 PM BST: Thread[http-bio-18080-exec-31,5,main]
CTS: Querying Sessions by User Id. Found 3 Sessions.
UUID: id=testuser,ou=user,dc=openam,dc=forgerock,dc=org


 Comments   
Comment by Simon Harding [ 28/Apr/16 ]

Also observed on AM 12.0.2 and AM 13.0.0 using both single AM instance (all embedded DJ) and multiple instance (all embedded and with an external CTS).

Concurrent attempts to authenticate are not subject to quota constraints. For example, quota constrainsts turned on with DESTROY_LAST works correctly with no concurrency:

ab -m POST -T "Content-Type: application/json" -H "X-OpenAM-Username: demo" -H "X-OpenAM-Password: changeit" -n 100 -c 1 http://am.example.com:8080/am1300/json/authenticate

But allows extra sessions to be created with concurrency:

ab -m POST -T "Content-Type: application/json" -H "X-OpenAM-Username: demo" -H "X-OpenAM-Password: changeit" -n 100 -c 2 http://am.example.com:8080/am1300/json/authenticate

Also observed using concurrently executed bash scripts consisting of curl commands in a while loop.

Comment by Robert Wapshott [ 30/Jan/17 ]

Confirmed, we are able to reproduce an issue which looks highly related to the described issue:

  • Site
  • Server part of site
  • Session Failover enabled
  • Session Quota enabled
  • Active Sessions limited to 2

The Apache Bench command then can demonstrate this.

No concurrency:

SERVER=rwapshott.forgerock.com:8080/openam && ab -m POST -T "Content-Type: application/json" -H "X-OpenAM-Username: demo" -H "X-OpenAM-Password: changeit" -n 100 -c 1 http://$SERVER/json/authenticate

Will leave the server with only two sessions for the demo user.

However moving to multiple concurrency:

SERVER=rwapshott.forgerock.com:8080/openam && ab -m POST -T "Content-Type: application/json" -H "X-OpenAM-Username: demo" -H "X-OpenAM-Password: changeit" -n 100 -c 2 http://$SERVER/json/authenticate

We can see that there are now more than two sessions for the demo user in the sessions tab.

The current working hypothesis is as follows: there is a concurrency bug in the DestroyNextExpiringAction class which is not aware of the number of sessions to destroy and so will not consistently destroy them.

Comment by Andy Hall [ 06/Oct/17 ]

This was fixed in the linked issue.

Generated at Sun Sep 27 23:04:56 UTC 2020 using Jira 7.13.12#713012-sha1:6e07c38070d5191bbf7353952ed38f111754533a.