Found with upgrade from 4.0.0 to 7.0.0-SNAPSHOT (77267f54c4a)
Sometimes, during upgrade while going through the rebuild indexes in post upgrade tasks we are getting an error:
/root/workspace/OpenDJ-7.0.x/tests_ds_upgrade_paths_je/results/20190812-130110/upgrade_group/UpgradeTasks/DJ_TASKS5/opendj/upgrade -n --acceptLicense --force 16:09:18.259 WARN ERROR: -- rc -- returned 1, expected to be in [0] -- stdout -- >>>> OpenDJ Upgrade Utility * OpenDJ will be upgraded from version 5.5.2.6e236778e2c9addd9bd4053299ec4da35a295443 to 7.0.0.77267f54c4aa41f96bde5c518c88263550b7b983 * See '/root/workspace/OpenDJ-7.0.x/tests_ds_upgrade_paths_je/results/20190812-130110/upgrade_group/UpgradeTasks/DJ_TASKS5/opendj/logs/upgrade.log' for a detailed log of this operation >>>> Preparing to upgrade OpenDJ 6.5.0 changed the indexing algorithm for replication metadata. Its index must be rebuilt which may take a long time; without a working index every server start will take longer than normal. Do you want to rebuild the index automatically at the end of the upgrade? (yes/no) yes OpenDJ 7.0.0 changed the indexing algorithm for TelephoneNumber equality and substring matching rules. All TelephoneNumber syntax based attribute indexes must be rebuilt which may take a long time. Do you want to rebuild the indexes automatically at the end of the upgrade? (yes/no) yes The upgrade is ready to proceed. Do you wish to continue? (yes/no) yes >>>> Performing upgrade Removing configuration entries for the monitor providers. 0% Removing configuration entries for the monitor providers............ 100% Removing file '/root/workspace/OpenDJ-7.0.x/tests_ds_upgrade_paths_je/results/20190812-130110/upgrade_group/UpgradeTasks/DJ_TASKS5/opendj/bin/control-panel' . 0% '/root/workspace/OpenDJ-7.0.x/tests_ds_upgrade_paths_je/results/20190812-130110/upgrade_group/UpgradeTasks/DJ_TASKS5/opendj/bin/control-panel' . 100% Removing file '/root/workspace/OpenDJ-7.0.x/tests_ds_upgrade_paths_je/results/20190812-130110/upgrade_group/UpgradeTasks/DJ_TASKS5/opendj/bin/ControlPanel.app' . 0% '/root/workspace/OpenDJ-7.0.x/tests_ds_upgrade_paths_je/results/20190812-130110/upgrade_group/UpgradeTasks/DJ_TASKS5/opendj/bin/ControlPanel.app' . 100% Removing file '/root/workspace/OpenDJ-7.0.x/tests_ds_upgrade_paths_je/results/20190812-130110/upgrade_group/UpgradeTasks/DJ_TASKS5/opendj/bat/control-panel.bat' . 0% '/root/workspace/OpenDJ-7.0.x/tests_ds_upgrade_paths_je/results/20190812-130110/upgrade_group/UpgradeTasks/DJ_TASKS5/opendj/bat/control-panel.bat' . 100% Removing file '/root/workspace/OpenDJ-7.0.x/tests_ds_upgrade_paths_je/results/20190812-130110/upgrade_group/UpgradeTasks/DJ_TASKS5/opendj/lib/forgerock-persistit-core.jar' . 0% '/root/workspace/OpenDJ-7.0.x/tests_ds_upgrade_paths_je/results/20190812-130110/upgrade_group/UpgradeTasks/DJ_TASKS5/opendj/lib/forgerock-persistit-core.jar' . 100% Replacing low durability settings in JE backends. 0% Replacing low durability settings in JE backends.................... 100% Replacing high durability settings in JE backends. 0% Replacing high durability settings in JE backends................... 100% Replacing medium durability settings in JE backends. 0% Replacing medium durability settings in JE backends................. 100% Replacing medium durability settings in JE backends. 0% Replacing medium durability settings in JE backends................. 100% Renaming 'ds-cfg-json-schema' object class to 'ds-cfg-json-query-equality-matching-rule'. 0% 'ds-cfg-json-query-equality-matching-rule'.......................... 100% Migrating root DN 'cn=myself'. 0% Migrating root DN 'cn=myself'....................................... 100% Removing root DN users from configuration. 0% Removing root DN users from configuration........................... 100% Segregating mutable and immutable files. 0% Segregating mutable and immutable files............................. 100% Update admin-backend.ldif file location. 0% Update admin-backend.ldif file location............................. 100% Update tasks backend file location. 0% Update tasks backend file location.................................. 100% Replacing all pin related configuration attributes by a single pin configuration attribute. 0% configuration attribute............................................. 100% Removing send and receive window size configuration in replication. 0% Removing send and receive window size configuration in replication.. 100% Replacing compute change number setting in replication server. 0% Replacing compute change number setting in replication server....... 100% Replacing compute change number setting in replication server. 0% Replacing compute change number setting in replication server....... 100% Replacing "reject unauthenticated requests" policy in global configuration. 0% configuration....................................................... 100% Replacing "reject unauthenticated requests" policy in global configuration. 0% configuration....................................................... 100% Removing configuration for assured replication. 0% Removing configuration for assured replication...................... 100% Removing configuration for assured replication. 0% Removing configuration for assured replication...................... 100% Removing generation-id data from configuration. 0% Removing generation-id data from configuration...................... 100% Removing synchronization state data from configuration. 0% Removing synchronization state data from configuration.............. 100% Renaming the proxy backend configuration property 'service discovery mechanism' to 'shard'. 0% discovery mechanism' to 'shard'..................................... 100% Adding objectClass to JSON, CSV, and External access logger configurations. 0% configurations...................................................... 100% Use the old JE backend caches instead of the new shared cache. 0% Use the old JE backend caches instead of the new shared cache....... 100% Removing configuration for replication monitoring. 0% Removing configuration for replication monitoring................... 100% Setting new global server ID. 0% Setting new global server ID........................................ 100% Removing global server ID from replication domains and replication server. 0% server............................... 50% server.............................................................. 100% Set the proxy backend configuration property 'hash-function' to MD5. 0% Set the proxy backend configuration property 'hash-function' to MD5. 100% Update replication SSL configuration. 0% Update replication SSL configuration................................ 100% Adding Mail Servers. 0% Adding Mail Servers................................................. 100% Renaming ds-cfg-connection-pool-min-size attributes to ds-cfg-bind-connection-pool-min-size. 0% ds-cfg-bind-connection-pool-min-size................................ 100% Renaming ds-cfg-connection-pool-max-size attributes to ds-cfg-bind-connection-pool-max-size. 0% ds-cfg-bind-connection-pool-max-size................................ 100% Renaming ds-cfg-connection-pool-idle-timeout attributes to ds-cfg-bind-connection-pool-idle-timeout. 0% ds-cfg-bind-connection-pool-idle-timeout............................ 100% Set the database cache mode 'ds-cfg-db-cache-mode' to 'cache-ln'. 0% Set the database cache mode 'ds-cfg-db-cache-mode' to 'cache-ln'.... 100% Add 'inheritFromDNParent' attribute type to the 'inheritedCollectiveAttributeSubentry' object class. 0% 'inheritedCollectiveAttributeSubentry' object class................. 100% Unable to retrieve the hostname from the admin backend using the truststore as source of keys; 'advertised-listen-address' attribute in global configuration will use the local hostname as value. Cause: Unable to look up server key id '0F6BFF984B6F4F1001B3A480BE76F2E8' in the admin backend Adding 'listen-address' and 'advertised-listen-address' attributes to the global configuration. 0% to the global configuration......................................... 100% Removing 'listen-address' attributes that are redundant with default value provided by the global configuration. 0% default value provided by the global configuration.................. 100% Replacing schema file '02-config.ldif'. 0% Replacing schema file '02-config.ldif'...... 20% Replacing schema file '02-config.ldif'.............................. 100% Archiving concatenated schema. 0% Archiving concatenated schema....................................... 100% >>>> OpenDJ was successfully upgraded to version 7.0.0.77267f54c4aa41f96bde5c518c88263550b7b983 >>>> Performing post upgrade tasks Rebuilding index(es) '[ds-sync-hist.changeSequenceNumberOrderingMatch, .telephoneNumberMatch, .telephoneNumberSubstringsMatch]' for base dn(s) '[dc=com]'. 0% dn(s) '[dc=com]'............. 25% dn(s) '[dc=com]'.................................................... FAIL Rebuilding index(es) '[ds-sync-hist.changeSequenceNumberOrderingMatch, .telephoneNumberMatch, .telephoneNumberSubstringsMatch]' for base dn(s) '[dc=com]'.................................................... FAIL [!] An error occurred during post upgrade task. Process aborted. Please check log for further details >>>> Post upgrade tasks complete * See '/root/workspace/OpenDJ-7.0.x/tests_ds_upgrade_paths_je/results/20190812-130110/upgrade_group/UpgradeTasks/DJ_TASKS5/opendj/logs/upgrade.log' for a detailed log of this operation
And in the log files:
--- Check for error patterns in "/root/workspace/OpenDJ-7.0.x/tests_ds_upgrade_paths_je/results/20190812-130110/upgrade_group/UpgradeTasks/DJ_TASKS5/opendj/logs/upgrade.log" file --- 16:11:15.923 WARN (ERROR) [12/08/2019:14:09:16 +0000] category=TOOLS seq=140 severity=SEVERE msg=An error occurred while attempting to perform index rebuild: Other: ConcurrentModificationException(HashMap.java:1442)
To reproduce:
./run-pybot.py -s upgrade_group.TemplateCorruption -t Template_00core_Schema_File_With_Missing_Mandatory_Line opendj
- is related to
-
OPENDJ-6505 ConcurrentModificationException in Topology
-
- Done
-
-
OPENDJ-6710 Use a single method for generating log error messages and stack traces if needed
-
- Dev in Progress
-