[OPENDJ-5979] Server does not validate sum of memory used by JE backend caches after upgrade Created: 01/Feb/19  Updated: 08/Nov/19  Resolved: 25/Feb/19

Status: Done
Project: OpenDJ
Component/s: config, core apis
Affects Version/s: 6.5.1, 6.5.0, 7.0.0
Fix Version/s: 7.0.0

Type: Bug Priority: Critical
Reporter: Ondrej Fuchsik Assignee: Chris Ridd
Resolution: Fixed Votes: 0
Labels: Verified

Issue Links:
Backport
is backported by OPENDJ-5999 Backport OPENDJ-5979: Server does not... Done
Epic Link: Bugs 7.0
Story Points: 1
QA Assignee: Ondrej Fuchsik

 Description   

Found with latest 7.0.0-SNAPSHOT rev. 1db9934fae1. 

This issue is same as OPENDJ-5584 but it only happens when upgrading from older version. I tried with 5.0.0 release.

To reproduce the issue run following commands:

export OPENDJ_VERSION='["4.0.0", "7.0.0-SNAPSHOT"]'
python3 configure.py -s
python3 run-pybot.py -v -s Backends_Group.GlobalConfigurationSettings dj

The same test works fine when OPENDJ_VERSION is 6.5.1-SNAPSHOT or 7.0.0-SNAPSHOT. From first look it looks like a missing upgrade task.


Steps to reproduce:

  • Configure 5.0.0
  • Upgrade to 7.0.0
  • Start the instance
  • Set je-backend-shared-cache-enabled to False
  • Stop the instance
  • Start the instance

Last point should return rc==1 and start-ds output should looks like an example below, but the instance starts with rc==0 and the output doesn't contain message about an error.

[01/Feb/2019:02:57:29 +0000] category=CORE severity=NOTICE msgID=134 msg=ForgeRock Directory Services 7.0.0-SNAPSHOT (build 20190201004143, revision number 1db9934fae1b36709d9b2fe2ca61e94e5d7b3f27) starting up
[01/Feb/2019:02:57:29 +0000] category=JVM severity=NOTICE msgID=21 msg=Installation Directory: /root/workspace/OpenDJ-7.0.x/tests_daily/Configs/results/20190201-023500/backends_group/GlobalConfigurationSettings/ProfileDJ/opendj 
[01/Feb/2019:02:57:29 +0000] category=JVM severity=NOTICE msgID=23 msg=Instance Directory: /root/workspace/OpenDJ-7.0.x/tests_daily/Configs/results/20190201-023500/backends_group/GlobalConfigurationSettings/ProfileDJ/opendj 
[01/Feb/2019:02:57:29 +0000] category=JVM severity=NOTICE msgID=17 msg=JVM Information: 1.8.0_181-b13 by Oracle Corporation, 64-bit architecture, 28631367680 bytes heap size 
[01/Feb/2019:02:57:29 +0000] category=JVM severity=NOTICE msgID=18 msg=JVM Host: localhost, running Linux 3.10.0-862.3.2.el7.x86_64 amd64, 337829969920 bytes physical memory size, number of processors available 48 
[01/Feb/2019:02:57:29 +0000] category=JVM severity=NOTICE msgID=19 msg=JVM Arguments: "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-Dorg.opends.server.scriptName=start-ds" 
[01/Feb/2019:02:57:30 +0000] category=BACKEND severity=NOTICE msgID=513 msg=The database backend amCts containing 5 entries has started 
[01/Feb/2019:02:57:30 +0000] category=BACKEND severity=NOTICE msgID=513 msg=The database backend idmRepo containing 38 entries has started 
[01/Feb/2019:02:57:30 +0000] category=CONFIG severity=ERROR msgID=116 msg=An error occurred while trying to initialize a backend loaded from class org.opends.server.backends.jeb.JEBackend with the information in configuration entry ds-cfg-backend-id=dsEvaluation,cn=Backends,cn=config: Configuration attribute ds-cfg-db-cache-percent has a value of 50% but the JVM has only 0% available. This backend will be disabled 
[01/Feb/2019:02:57:30 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.core.DirectoryServer (alert type org.opends.server.DirectoryServerShutdown, alert ID org.opends.messages.core-141): The Directory Server has started the shutdown process. The shutdown was initiated by an instance of class org.opends.server.core.DirectoryServer and the reason provided for the shutdown was An error occurred while attempting to bootstrap the Directory Server: An error occurred while trying to initialize a backend loaded from class org.opends.server.backends.jeb.JEBackend with the information in configuration entry ds-cfg-backend-id=dsEvaluation,cn=Backends,cn=config: Configuration attribute ds-cfg-db-cache-percent has a value of 50% but the JVM has only 0% available. This backend will be disabled\ [01/Feb/2019:02:57:30 +0000] category=BACKEND severity=NOTICE msgID=370 msg=The backend idmRepo is now taken offline 
[01/Feb/2019:02:57:30 +0000] category=BACKEND severity=NOTICE msgID=370 msg=The backend amCts is now taken offline 
[01/Feb/2019:02:57:30 +0000] category=CORE severity=NOTICE msgID=203 msg=The Directory Server is now stopped

Without upgrade and direct configuration of 7.0.0 at the beginning the return rc of last step is equal to 1 and the output is same as above example.



 Comments   
Comment by Ondrej Fuchsik [ 06/Feb/19 ]

After all I think this is not an issue. The test use profiles to setup the instance. In fact multiple profiles to have multiple backends. The issue is we upgrade from 4.0.0 (5.0.0) and the framework use a fallback to regular setup (no profiles in old version) and configure only one backend with db-cache-percent set to 50. It is obvious it can't behave same, so it's test to update.

I tried the upgrade scenario with profiles with 6.5.0 => 7.0.0 and 6.5.0 => 6.5.1 and works like it should. The only thing to consider is if je-shared-cache-enable property should be set to True or to False after upgrade. Currently the default value is True.

Comment by Matthew Swift [ 06/Feb/19 ]

I agree, this does sound like a surprising change in behavior. I think we should have an upgrade task that disables the shared cache. Do you agree Yannick Lecaillez?

Comment by Yannick Lecaillez [ 06/Feb/19 ]

I agree, we should not change behavior during upgrade.
I hope users are reading release notes so that they know this shared-cache feature now exists.

Comment by Ondrej Fuchsik [ 08/Feb/19 ]

Verified with 7.0.0-SNAPSHOT rev. d934122aeb7 and I added new upgrade tests for this.

Comment by Ondrej Fuchsik [ 25/Feb/19 ]

There is an issue when doing upgrade from 6.5.0 (je-shared-cache-enabled default value is True) to 6.5.1. After upgrade the value of je-shared-cache-enabled is False.

When I set the value of this property to False or True before upgrade the value hasn't changed.

From my point of view, the upgrade task should has got target version 6.5.0 instead of 6.5.1.

Comment by Ondrej Fuchsik [ 07/Mar/19 ]

Verified last problem with 7.0.0-SNAPSHOT c092b9442f5.

Generated at Wed Oct 21 10:09:51 UTC 2020 using Jira 7.13.12#713012-sha1:6e07c38070d5191bbf7353952ed38f111754533a.