Uploaded image for project: 'OpenDJ'
  1. OpenDJ
  2. OPENDJ-6528

Randomly getting ConcurrentModificationException during upgrade

    Details

    • Type: Bug
    • Status: Dev backlog
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 7.0.0
    • Fix Version/s: None
    • Component/s: tools, upgrade
    • Labels:
      None

      Description

      Found with upgrade from 4.0.0 to 7.0.0-SNAPSHOT (77267f54c4a)

      Sometimes, during upgrade while going through the rebuild indexes in post upgrade tasks we are getting an error:

       	/root/workspace/OpenDJ-7.0.x/tests_ds_upgrade_paths_je/results/20190812-130110/upgrade_group/UpgradeTasks/DJ_TASKS5/opendj/upgrade -n --acceptLicense --force 	
      16:09:18.259 	WARN 	ERROR:
      -- rc --
      returned 1, expected to be in [0]
      -- stdout --
      
      >>>> OpenDJ Upgrade Utility
      
       * OpenDJ will be upgraded from version
       5.5.2.6e236778e2c9addd9bd4053299ec4da35a295443 to
       7.0.0.77267f54c4aa41f96bde5c518c88263550b7b983
       * See
       '/root/workspace/OpenDJ-7.0.x/tests_ds_upgrade_paths_je/results/20190812-130110/upgrade_group/UpgradeTasks/DJ_TASKS5/opendj/logs/upgrade.log'
       for a detailed log of this operation
      
      >>>> Preparing to upgrade
      
        OpenDJ 6.5.0 changed the indexing algorithm for replication metadata. Its
        index must be rebuilt which may take a long time; without a working index
        every server start will take longer than normal. Do you want to rebuild the
        index automatically at the end of the upgrade? (yes/no) yes
      
        OpenDJ 7.0.0 changed the indexing algorithm for TelephoneNumber equality and
        substring matching rules. All TelephoneNumber syntax based attribute indexes
        must be rebuilt which may take a long time. Do you want to rebuild the
        indexes automatically at the end of the upgrade? (yes/no) yes
      
        The upgrade is ready to proceed. Do you wish to continue? (yes/no) yes
      
      
      >>>> Performing upgrade
      
        Removing configuration entries for the monitor providers.   0%       Removing configuration entries for the monitor providers............   100%     
        Removing file
      '/root/workspace/OpenDJ-7.0.x/tests_ds_upgrade_paths_je/results/20190812-130110/upgrade_group/UpgradeTasks/DJ_TASKS5/opendj/bin/control-panel'
      .   0%     '/root/workspace/OpenDJ-7.0.x/tests_ds_upgrade_paths_je/results/20190812-130110/upgrade_group/UpgradeTasks/DJ_TASKS5/opendj/bin/control-panel'
      .   100%     
        Removing file
      '/root/workspace/OpenDJ-7.0.x/tests_ds_upgrade_paths_je/results/20190812-130110/upgrade_group/UpgradeTasks/DJ_TASKS5/opendj/bin/ControlPanel.app'
      .   0%     '/root/workspace/OpenDJ-7.0.x/tests_ds_upgrade_paths_je/results/20190812-130110/upgrade_group/UpgradeTasks/DJ_TASKS5/opendj/bin/ControlPanel.app'
      .   100%     
        Removing file
      '/root/workspace/OpenDJ-7.0.x/tests_ds_upgrade_paths_je/results/20190812-130110/upgrade_group/UpgradeTasks/DJ_TASKS5/opendj/bat/control-panel.bat'
      .   0%     '/root/workspace/OpenDJ-7.0.x/tests_ds_upgrade_paths_je/results/20190812-130110/upgrade_group/UpgradeTasks/DJ_TASKS5/opendj/bat/control-panel.bat'
      .   100%     
        Removing file
      '/root/workspace/OpenDJ-7.0.x/tests_ds_upgrade_paths_je/results/20190812-130110/upgrade_group/UpgradeTasks/DJ_TASKS5/opendj/lib/forgerock-persistit-core.jar'
      .   0%     '/root/workspace/OpenDJ-7.0.x/tests_ds_upgrade_paths_je/results/20190812-130110/upgrade_group/UpgradeTasks/DJ_TASKS5/opendj/lib/forgerock-persistit-core.jar'
      .   100%     
        Replacing low durability settings in JE backends.   0%       Replacing low durability settings in JE backends....................   100%     
        Replacing high durability settings in JE backends.   0%       Replacing high durability settings in JE backends...................   100%     
        Replacing medium durability settings in JE backends.   0%       Replacing medium durability settings in JE backends.................   100%     
        Replacing medium durability settings in JE backends.   0%       Replacing medium durability settings in JE backends.................   100%     
        Renaming 'ds-cfg-json-schema' object class to
        'ds-cfg-json-query-equality-matching-rule'.   0%       'ds-cfg-json-query-equality-matching-rule'..........................   100%     
        Migrating root DN 'cn=myself'.   0%       Migrating root DN 'cn=myself'.......................................   100%     
        Removing root DN users from configuration.   0%       Removing root DN users from configuration...........................   100%     
        Segregating mutable and immutable files.   0%       Segregating mutable and immutable files.............................   100%     
        Update admin-backend.ldif file location.   0%       Update admin-backend.ldif file location.............................   100%     
        Update tasks backend file location.   0%       Update tasks backend file location..................................   100%     
        Replacing all pin related configuration attributes by a single pin
        configuration attribute.   0%       configuration attribute.............................................   100%     
        Removing send and receive window size configuration in replication.   0%       Removing send and receive window size configuration in replication..   100%     
        Replacing compute change number setting in replication server.   0%       Replacing compute change number setting in replication server.......   100%     
        Replacing compute change number setting in replication server.   0%       Replacing compute change number setting in replication server.......   100%     
        Replacing "reject unauthenticated requests" policy in global
        configuration.   0%       configuration.......................................................   100%     
        Replacing "reject unauthenticated requests" policy in global
        configuration.   0%       configuration.......................................................   100%     
        Removing configuration for assured replication.   0%       Removing configuration for assured replication......................   100%     
        Removing configuration for assured replication.   0%       Removing configuration for assured replication......................   100%     
        Removing generation-id data from configuration.   0%       Removing generation-id data from configuration......................   100%     
        Removing synchronization state data from configuration.   0%       Removing synchronization state data from configuration..............   100%     
        Renaming the proxy backend configuration property 'service
        discovery mechanism' to 'shard'.   0%       discovery mechanism' to 'shard'.....................................   100%     
        Adding objectClass to JSON, CSV, and External access logger
        configurations.   0%       configurations......................................................   100%     
        Use the old JE backend caches instead of the new shared cache.   0%       Use the old JE backend caches instead of the new shared cache.......   100%     
        Removing configuration for replication monitoring.   0%       Removing configuration for replication monitoring...................   100%     
        Setting new global server ID.   0%       Setting new global server ID........................................   100%     
        Removing global server ID from replication domains and replication
        server.   0%       server...............................   50%       server..............................................................   100%     
        Set the proxy backend configuration property 'hash-function' to MD5.   0%       Set the proxy backend configuration property 'hash-function' to MD5.   100%     
        Update replication SSL configuration.   0%       Update replication SSL configuration................................   100%     
        Adding Mail Servers.   0%       Adding Mail Servers.................................................   100%     
        Renaming ds-cfg-connection-pool-min-size attributes to
        ds-cfg-bind-connection-pool-min-size.   0%       ds-cfg-bind-connection-pool-min-size................................   100%     
        Renaming ds-cfg-connection-pool-max-size attributes to
        ds-cfg-bind-connection-pool-max-size.   0%       ds-cfg-bind-connection-pool-max-size................................   100%     
        Renaming ds-cfg-connection-pool-idle-timeout attributes to
        ds-cfg-bind-connection-pool-idle-timeout.   0%       ds-cfg-bind-connection-pool-idle-timeout............................   100%     
        Set the database cache mode 'ds-cfg-db-cache-mode' to 'cache-ln'.   0%       Set the database cache mode 'ds-cfg-db-cache-mode' to 'cache-ln'....   100%     
        Add 'inheritFromDNParent' attribute type to the
        'inheritedCollectiveAttributeSubentry' object class.   0%       'inheritedCollectiveAttributeSubentry' object class.................   100%     
      Unable to retrieve the hostname from the admin backend using the truststore as
      source of keys; 'advertised-listen-address' attribute in global configuration
      will use the local hostname as value. Cause: Unable to look up server key id
      '0F6BFF984B6F4F1001B3A480BE76F2E8' in the admin backend
        Adding 'listen-address' and 'advertised-listen-address' attributes
        to the global configuration.   0%       to the global configuration.........................................   100%     
        Removing 'listen-address' attributes that are redundant with
        default value provided by the global configuration.   0%       default value provided by the global configuration..................   100%     
        Replacing schema file '02-config.ldif'.   0%       Replacing schema file '02-config.ldif'......   20%       Replacing schema file '02-config.ldif'..............................   100%     
        Archiving concatenated schema.   0%       Archiving concatenated schema.......................................   100%     
      
      >>>> OpenDJ was successfully upgraded to version
      7.0.0.77267f54c4aa41f96bde5c518c88263550b7b983
      
      
      >>>> Performing post upgrade tasks
      
        Rebuilding index(es)
        '[ds-sync-hist.changeSequenceNumberOrderingMatch,
        .telephoneNumberMatch, .telephoneNumberSubstringsMatch]' for base
        dn(s) '[dc=com]'.   0%       dn(s) '[dc=com]'.............   25%       dn(s) '[dc=com]'....................................................   FAIL
        Rebuilding index(es)
        '[ds-sync-hist.changeSequenceNumberOrderingMatch,
        .telephoneNumberMatch, .telephoneNumberSubstringsMatch]' for base
        dn(s) '[dc=com]'....................................................   FAIL
        [!] An error occurred during post upgrade task. Process aborted. Please
        check log for further details
      
      >>>> Post upgrade tasks complete
      
       * See
       '/root/workspace/OpenDJ-7.0.x/tests_ds_upgrade_paths_je/results/20190812-130110/upgrade_group/UpgradeTasks/DJ_TASKS5/opendj/logs/upgrade.log'
       for a detailed log of this operation
      

      And in the log files:

       	--- Check for error patterns in "/root/workspace/OpenDJ-7.0.x/tests_ds_upgrade_paths_je/results/20190812-130110/upgrade_group/UpgradeTasks/DJ_TASKS5/opendj/logs/upgrade.log" file --- 	
      16:11:15.923 	WARN 	(ERROR) [12/08/2019:14:09:16 +0000] category=TOOLS seq=140 severity=SEVERE msg=An error occurred while attempting to perform index rebuild:  Other: ConcurrentModificationException(HashMap.java:1442)
      

      To reproduce:

       ./run-pybot.py -s upgrade_group.TemplateCorruption -t Template_00core_Schema_File_With_Missing_Mandatory_Line opendj
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                cforel carole forel
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: