Uploaded image for project: 'OpenAM Agents'
  1. OpenAM Agents
  2. AMAGENTS-4050

Saving agent config in quick succession whilst under load can lead to continual 403s

    XMLWordPrintable

    Details

    • 2
    • 2021.03 - Agents, 2021.04 - Agents, 2021.05 - Agents, 2021.06 - Agents, 2021.07 - Agents, 2021.09 - Agents, 2021.10 - Agents

      Description

      Description
      In a reverse proxy environment, whilst under load of 800 connections on a not enforced url, go to am and saving multiple times can lead to an outage until the load stops. Unix only.

      Reproduction
      Environment and performance test is the same as AMAGENTS-3986. Tested on 5.8.0-RC1 and patch which resolves AMAGENTS-3986.
      Once load is running go to am console and edit the agent definition. Click save in this pattern.
      while [ 1 -eq 1]; do wc -l debug.log
      Save. wait 10s
      Save. Save. wait 10s
      Save. Save. Save (in 2 seconds)

      Expected result
      No significant errors should occur, or if they did, then they should be of short duration

      Actual result
      After the 1st save and waiting 10s, there is no ill effect (no change in debug.log)
      After the 3rd save and waiting 10s, there is not ill effect (no change in debug.log)
      After the 6th save, debug.log keeps growing and growing with messages like this:

      2021-02-05 14:42:31 GMT ERROR [55c86aa8-fb83-9371-7017-f559411ee7cc]: agent authentication and profile failure
      2021-02-05 14:42:31 GMT ERROR [55c86aa8-fb83-9371-7017-f559411ee7cc]: config_fetch(): failed to load configuration for agent: / webapp, error
      2021-02-05 14:42:31 GMT ERROR [55c86aa8-fb83-9371-7017-f559411ee7cc]: amagent_auth_handler(): failed to get agent configuration instance, error: error
      2021-02-05 14:42:31 GMT ERROR [1c14e7c7-2627-d829-a2dc-8b84dd0782b9]: agent authentication and profile failure
      2021-02-05 14:42:31 GMT ERROR [1c14e7c7-2627-d829-a2dc-8b84dd0782b9]: config_fetch(): failed to load configuration for agent: / webapp, error
      2021-02-05 14:42:31 GMT ERROR [1c14e7c7-2627-d829-a2dc-8b84dd0782b9]: amagent_auth_handler(): failed to get agent configuration instance, error: error

      ith load but no config click
      15:09
      1 debug.log
      1 debug.log
      1 debug.log
      1 debug.log
      1 debug.log
      1 debug.log
      1 debug.log
      15:09
      after clicking 3 times
      1 debug.log
      145 debug.log
      178 debug.log
      310 debug.log
      439 debug.log
      580 debug.log
      733 debug.log
      913 debug.log
      991 debug.log
      1042 debug.log
      1210 debug.log
      1342 debug.log
      1498 debug.log
      1729 debug.log
      1756 debug.log
      2047 debug.log
      2107 debug.log
      2197 debug.log
      2392 debug.log
      2557 debug.log

      Errors also increase in the load test until load is stopped (I waited 10 minutes, but the error growth was still continual)

      Workaround
      Don't save config like this

        Attachments

          Issue Links

            Activity

              People

              nick.james Nicholas James
              alex.levin@forgerock.com Alex Levin
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: