Uploaded image for project: 'OpenIDM'
  1. OpenIDM
  2. OPENIDM-13342

Config changes made via admin UI on one cluster node may not be propagated to the other nodes

    XMLWordPrintable

    Details

      Description

      To reproduce this:

      1. Set up a two node cluster, node1 and node2.
      Do not change the out of box setting of fileinstall.enabled, repo.enabled, fileinstall.enableConfigSave. This implies they are all set to true.

      2. Shut down node2.

      3. Make some configuration change to node1 via admin UI. For example, add a new attribute to a connector, or to a managed object.

      4. Observe that the change is made to the repo configobjects, and it's also propagated to the corresponding conf .json file on node1.

      5. Start node2. The change may or may not be seen on node2 admin UI, and subsequently the conf .json file. If the change is not propagated, the entry in configobjects is overwritten by node2.

      Normally node2 gets the change. To induce the other outcome, we can stress the system resources. For example, I could reproduce the issue by running both IDM nodes and one DS instance on the same vm with 1 cpu and 4GB memory.

      If the change is propagated on node 2, the config listener is registered before ClusterManager processes pending cluster event.

      Jun 04, 2019 8:25:10 AM org.forgerock.openidm.cluster.ClusterManager register
      FINE: Registering listener config
      ...
      Jun 04, 2019 8:25:10 AM org.forgerock.openidm.cluster.ClusterManager processPendingEvents
      FINE: Querying cluster events
      Jun 04, 2019 8:25:10 AM org.forgerock.openidm.repo.jdbc.impl.JDBCRepoService query
      FINEST: Full id: cluster/events Extracted type: cluster/events
      Jun 04, 2019 8:25:10 AM org.forgerock.openidm.repo.jdbc.impl.JDBCRepoService getTableHandler
      FINE: Use table handler configured for cluster for type cluster/events 
      Jun 04, 2019 8:25:10 AM org.forgerock.openidm.repo.jdbc.impl.query.TableQueries query
      FINE: Querying {_pageSize=2147483647, instanceId=node2, _queryFilter=null, _pagedResultsCookie={}, _queryExpression=null, _pagedResultsOffset=0, _resource=cluster/events, _queryId=query-cluster-events, _sortKeys=[]}
      Jun 04, 2019 8:25:10 AM org.forgerock.openidm.repo.jdbc.impl.query.TableQueries resolveQuery
      FINE: Prepared statement: HikariProxyPreparedStatement@1814586188 wrapping com.mysql.jdbc.JDBC42PreparedStatement@332afcef: SELECT fullobject FROM openidm5500.clusterobjects obj INNER JOIN openidm5500.clusterobjectproperties prop1 ON obj.id = prop1.clusterobjects_id INNER JOIN openidm5500.clusterobjectproperties prop2 ON obj.id = prop2.clusterobjects_id WHERE (prop1.propkey = '/type' AND prop1.propvalue = 'event') AND (prop2.propkey = '/instanceId' AND prop2.propvalue = 'node2')
      Jun 04, 2019 8:25:10 AM org.forgerock.openidm.repo.jdbc.impl.GenericResultSetMapper mapToObject
      FINEST: Query result for queryId: query-cluster-events type: cluster/events converted obj: {type=event, instanceId=node2, event={type=CUSTOM, instanceId=node1, listenerId=config, details={action=UPDATE, resourcePath=provisioner.openicf/dj, resourceId=null, obj={name=dj, connectorRef={bundleName=org.forgerock.openicf.connectors.ldap-connector, bundleVersion=[1.4.0.0,1.5.0.0),...
      

      If the change is not propagated, the config listener is not yet registered when ClusterManager gets the pending event.

      Jun 04, 2019 9:07:36 AM org.forgerock.openidm.cluster.ClusterManager processPendingEvents
      FINE: Querying cluster events
      Jun 04, 2019 9:07:36 AM org.forgerock.openidm.repo.jdbc.impl.JDBCRepoService query
      FINEST: Full id: cluster/events Extracted type: cluster/events
      Jun 04, 2019 9:07:36 AM org.forgerock.openidm.repo.jdbc.impl.JDBCRepoService getTableHandler
      FINE: Use table handler configured for cluster for type cluster/events 
      Jun 04, 2019 9:07:36 AM org.forgerock.openidm.repo.jdbc.impl.query.TableQueries query
      FINE: Querying {_pageSize=2147483647, instanceId=node2, _queryFilter=null, _pagedResultsCookie={}, _queryExpression=null, _pagedResultsOffset=0, _resource=cluster/events, _queryId=query-cluster-events, _sortKeys=[]}
      Jun 04, 2019 9:07:36 AM org.forgerock.openidm.repo.jdbc.impl.query.TableQueries resolveQuery
      FINE: Prepared statement: HikariProxyPreparedStatement@1458052373 wrapping com.mysql.jdbc.JDBC42PreparedStatement@9e37ff2: SELECT fullobject FROM openidm5500.clusterobjects obj INNER JOIN openidm5500.clusterobjectproperties prop1 ON obj.id = prop1.clusterobjects_id INNER JOIN openidm5500.clusterobjectproperties prop2 ON obj.id = prop2.clusterobjects_id WHERE (prop1.propkey = '/type' AND prop1.propvalue = 'event') AND (prop2.propkey = '/instanceId' AND prop2.propvalue = 'node2')
      Jun 04, 2019 9:07:36 AM org.forgerock.openidm.repo.jdbc.impl.GenericResultSetMapper mapToObject
      FINEST: Query result for queryId: query-cluster-events type: cluster/events converted obj: {type=event, instanceId=node2, event={type=CUSTOM, instanceId=node1, listenerId=config, details={action=UPDATE, resourcePath=provisioner.openicf/dj, 
      ...
      Jun 04, 2019 9:07:41 AM org.forgerock.openidm.cluster.ClusterManager register
      FINE: Registering listener config
      
      [5.5.0.0-mysql-node2]$ ./startup.sh
      Executing ./startup.sh...
      Using OPENIDM_HOME:   /opt/fg/5.5.0.0-mysql-node2
      Using PROJECT_HOME:   /opt/fg/5.5.0.0-mysql-node2
      Using OPENIDM_OPTS:   -Xmx1024m -Xms1024m
      Using LOGGING_CONFIG: -Djava.util.logging.config.file=/opt/fg/5.5.0.0-mysql-node2/conf/logging.properties
      -> Jun 04, 2019 9:07:36 AM org.forgerock.openidm.cluster.ClusterManager processPendingEvents
      WARNING: No listener config available to receive event { "type": "CUSTOM", "instanceId": "node1", "listenerId": "config", "details": { "action": "UPDATE", "resourcePath": "provisioner.openicf/dj", "resourceId": null, "obj": { "name": "dj",...
      

        Attachments

        1. race-condition.log
          2.65 MB
        2. success.log
          3.83 MB

          Issue Links

            Activity

              People

              Assignee:
              naren.koganti Naren Koganti
              Reporter:
              yinyan.cao Yinyan Cao
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: