Uploaded image for project: 'OpenIDM'
  1. OpenIDM
  2. OPENIDM-5697

Cluster state failure yields permanent persistent schedule failure in cluster when a cluster node is shutdown

    Details

    • Sprint:
      OpenIDM Sprint 68
    • Story Points:
      3
    • Support Ticket IDs:

      Description

      When testing IDM 4 GA cluster with persistent schedules, processing such as liveSync is critically impacted (persistent schedule(s) effectively stop) when either node in, say, a two node cluster is shutdown. The schedule then resumes, either

      1. when the node is brought back up, or,
      2. if the remaining node is also shutdown, when then one is subsequently brought back up.

      Either way, if this is correct we don’t have 100% functional availability in a cluster for such processing.

      When the node that is not running the schedule is shut down the other node running the schedule immediately logs the following and stops processing the schedule

      Apr 19, 2016 6:13:35 AM org.forgerock.openidm.quartz.impl.RepoJobStore triggerFired
      FINE: Trigger scheduler-service-group.trigger-activeSynchroniser_systemPartnerAccount has fired
      Apr 19, 2016 6:13:35 AM org.forgerock.openidm.quartz.impl.RepoJobStore getTriggerFromRepo
      FINEST: Getting trigger trigger-activeSynchroniser_systemPartnerAccount in group scheduler-service-group from repo
      Apr 19, 2016 6:13:35 AM org.forgerock.openidm.servlet.internal.ServletConnectionFactory$5 filterRead
      FINEST: Request: { "resourcePath": "repo/scheduler/triggers/scheduler-service-group_$x$x$_trigger-activeSynchroniser_systemPartnerAccount", "method": "read", "fields": [  ] }
      Apr 19, 2016 6:13:35 AM org.forgerock.openidm.repo.jdbc.impl.GenericTableHandler read
      FINEST: Populating prepared statement SELECT obj.rev, obj.fullobject FROM openidm.objecttypes objtype, openidm.schedulerobjects obj WHERE obj.objecttypes_id = objtype.id AND objtype.objecttype = ? AND obj.objectid  = ? for scheduler/triggers/scheduler-service-group_$x$x$_trigger-activeSynchroniser_systemPartnerAccount
      Apr 19, 2016 6:13:35 AM org.forgerock.openidm.repo.jdbc.impl.GenericTableHandler read
      FINE: Executing: SELECT obj.rev, obj.fullobject FROM openidm.objecttypes objtype, openidm.schedulerobjects obj WHERE obj.objecttypes_id = objtype.id AND objtype.objecttype = ? AND obj.objectid  = ?
      Apr 19, 2016 6:13:35 AM org.forgerock.openidm.repo.jdbc.impl.JDBCRepoService read
      FINE: ResourceException in read of scheduler/triggers/scheduler-service-group_$x$x$_trigger-activeSynchroniser_systemPartnerAccount
      org.forgerock.json.resource.NotFoundException: Object scheduler/triggers/scheduler-service-group_$x$x$_trigger-activeSynchroniser_systemPartnerAccount not found in scheduler/triggers
      at org.forgerock.json.resource.ResourceException.newResourceException(ResourceException.java:227)
      at org.forgerock.json.resource.ResourceException.getException(ResourceException.java:309)
      at org.forgerock.openidm.repo.jdbc.impl.GenericTableHandler.read(GenericTableHandler.java:238)
      at org.forgerock.openidm.repo.jdbc.impl.JDBCRepoService.read(JDBCRepoService.java:268)
      at org.forgerock.openidm.repo.jdbc.impl.JDBCRepoService.handleRead(JDBCRepoService.java:239)
      at org.forgerock.json.resource.Router.handleRead(Router.java:324)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:109)
      at org.forgerock.json.resource.Filters$ConditionalFilter.filterRead(Filters.java:102)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:107)
      at org.forgerock.json.resource.Filters$ConditionalFilter.filterRead(Filters.java:102)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:107)
      at org.forgerock.json.resource.Filters$ConditionalFilter.filterRead(Filters.java:102)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:107)
      at org.forgerock.openidm.audit.filter.AuditFilter.filterRead(AuditFilter.java:166)
      at org.forgerock.json.resource.Filters$ConditionalFilter.filterRead(Filters.java:100)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:107)
      at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$5.filterRead(ServletConnectionFactory.java:527)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:107)
      at org.forgerock.openidm.maintenance.impl.PassthroughFilter.filterRead(PassthroughFilter.java:72)
      at org.forgerock.openidm.maintenance.impl.MaintenanceService.filterRead(MaintenanceService.java:273)
      at org.forgerock.json.resource.Filters$ConditionalFilter.filterRead(Filters.java:100)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:107)
      at org.forgerock.json.resource.FilterChain.handleRead(FilterChain.java:237)
      at org.forgerock.json.resource.InternalConnection.readAsync(InternalConnection.java:89)
      at org.forgerock.json.resource.AbstractAsynchronousConnection.read(AbstractAsynchronousConnection.java:79)
      at org.forgerock.json.resource.AbstractConnectionWrapper.read(AbstractConnectionWrapper.java:198)
      at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$1$1.read(ServletConnectionFactory.java:199)
      at org.forgerock.json.resource.AbstractConnectionWrapper.read(AbstractConnectionWrapper.java:198)
      at org.forgerock.openidm.quartz.impl.RepoJobStore.readFromRepo(RepoJobStore.java:1578)
      at org.forgerock.openidm.quartz.impl.RepoJobStore.getTriggerFromRepo(RepoJobStore.java:2073)
      at org.forgerock.openidm.quartz.impl.RepoJobStore.getTriggerWrapper(RepoJobStore.java:2116)
      at org.forgerock.openidm.quartz.impl.RepoJobStore.triggerFired(RepoJobStore.java:1409)
      at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:319)
      
      Apr 19, 2016 6:13:35 AM org.forgerock.openidm.servlet.internal.ServletConnectionFactory$4 handleException
      FINEST: Resource exception: 404 Not Found: "Object scheduler/triggers/scheduler-service-group_$x$x$_trigger-activeSynchroniser_systemPartnerAccount not found in scheduler/triggers"
      org.forgerock.json.resource.NotFoundException: Object scheduler/triggers/scheduler-service-group_$x$x$_trigger-activeSynchroniser_systemPartnerAccount not found in scheduler/triggers
      at org.forgerock.json.resource.ResourceException.newResourceException(ResourceException.java:227)
      at org.forgerock.json.resource.ResourceException.getException(ResourceException.java:309)
      at org.forgerock.openidm.repo.jdbc.impl.GenericTableHandler.read(GenericTableHandler.java:238)
      at org.forgerock.openidm.repo.jdbc.impl.JDBCRepoService.read(JDBCRepoService.java:268)
      at org.forgerock.openidm.repo.jdbc.impl.JDBCRepoService.handleRead(JDBCRepoService.java:239)
      at org.forgerock.json.resource.Router.handleRead(Router.java:324)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:109)
      at org.forgerock.json.resource.Filters$ConditionalFilter.filterRead(Filters.java:102)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:107)
      at org.forgerock.json.resource.Filters$ConditionalFilter.filterRead(Filters.java:102)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:107)
      at org.forgerock.json.resource.Filters$ConditionalFilter.filterRead(Filters.java:102)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:107)
      at org.forgerock.openidm.audit.filter.AuditFilter.filterRead(AuditFilter.java:166)
      at org.forgerock.json.resource.Filters$ConditionalFilter.filterRead(Filters.java:100)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:107)
      at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$5.filterRead(ServletConnectionFactory.java:527)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:107)
      at org.forgerock.openidm.maintenance.impl.PassthroughFilter.filterRead(PassthroughFilter.java:72)
      at org.forgerock.openidm.maintenance.impl.MaintenanceService.filterRead(MaintenanceService.java:273)
      at org.forgerock.json.resource.Filters$ConditionalFilter.filterRead(Filters.java:100)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:107)
      at org.forgerock.json.resource.FilterChain.handleRead(FilterChain.java:237)
      at org.forgerock.json.resource.InternalConnection.readAsync(InternalConnection.java:89)
      at org.forgerock.json.resource.AbstractAsynchronousConnection.read(AbstractAsynchronousConnection.java:79)
      at org.forgerock.json.resource.AbstractConnectionWrapper.read(AbstractConnectionWrapper.java:198)
      at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$1$1.read(ServletConnectionFactory.java:199)
      at org.forgerock.json.resource.AbstractConnectionWrapper.read(AbstractConnectionWrapper.java:198)
      at org.forgerock.openidm.quartz.impl.RepoJobStore.readFromRepo(RepoJobStore.java:1578)
      at org.forgerock.openidm.quartz.impl.RepoJobStore.getTriggerFromRepo(RepoJobStore.java:2073)
      at org.forgerock.openidm.quartz.impl.RepoJobStore.getTriggerWrapper(RepoJobStore.java:2116)
      at org.forgerock.openidm.quartz.impl.RepoJobStore.triggerFired(RepoJobStore.java:1409)
      at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:319)
      
      Apr 19, 2016 6:13:35 AM org.forgerock.openidm.quartz.impl.RepoJobStore triggerFired
      WARNING: Error setting trigger fired, trigger does not exist
      Apr 19, 2016 6:13:35 AM org.forgerock.openidm.quartz.impl.RepoJobStore getTriggerFromRepo
      FINEST: Getting trigger trigger-activeSynchroniser_systemPartnerAccount in group scheduler-service-group from repo
      Apr 19, 2016 6:13:35 AM org.forgerock.openidm.servlet.internal.ServletConnectionFactory$5 filterRead
      FINEST: Request: { "resourcePath": "repo/scheduler/triggers/scheduler-service-group_$x$x$_trigger-activeSynchroniser_systemPartnerAccount", "method": "read", "fields": [  ] }
      Apr 19, 2016 6:13:35 AM org.forgerock.openidm.repo.jdbc.impl.GenericTableHandler read
      FINEST: Populating prepared statement SELECT obj.rev, obj.fullobject FROM openidm.objecttypes objtype, openidm.schedulerobjects obj WHERE obj.objecttypes_id = objtype.id AND objtype.objecttype = ? AND obj.objectid  = ? for scheduler/triggers/scheduler-service-group_$x$x$_trigger-activeSynchroniser_systemPartnerAccount
      Apr 19, 2016 6:13:35 AM org.forgerock.openidm.repo.jdbc.impl.GenericTableHandler read
      FINE: Executing: SELECT obj.rev, obj.fullobject FROM openidm.objecttypes objtype, openidm.schedulerobjects obj WHERE obj.objecttypes_id = objtype.id AND objtype.objecttype = ? AND obj.objectid  = ?
      Apr 19, 2016 6:13:35 AM org.forgerock.openidm.repo.jdbc.impl.JDBCRepoService read
      FINE: ResourceException in read of scheduler/triggers/scheduler-service-group_$x$x$_trigger-activeSynchroniser_systemPartnerAccount
      org.forgerock.json.resource.NotFoundException: Object scheduler/triggers/scheduler-service-group_$x$x$_trigger-activeSynchroniser_systemPartnerAccount not found in scheduler/triggers
      at org.forgerock.json.resource.ResourceException.newResourceException(ResourceException.java:227)
      at org.forgerock.json.resource.ResourceException.getException(ResourceException.java:309)
      at org.forgerock.openidm.repo.jdbc.impl.GenericTableHandler.read(GenericTableHandler.java:238)
      at org.forgerock.openidm.repo.jdbc.impl.JDBCRepoService.read(JDBCRepoService.java:268)
      at org.forgerock.openidm.repo.jdbc.impl.JDBCRepoService.handleRead(JDBCRepoService.java:239)
      at org.forgerock.json.resource.Router.handleRead(Router.java:324)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:109)
      at org.forgerock.json.resource.Filters$ConditionalFilter.filterRead(Filters.java:102)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:107)
      at org.forgerock.json.resource.Filters$ConditionalFilter.filterRead(Filters.java:102)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:107)
      at org.forgerock.json.resource.Filters$ConditionalFilter.filterRead(Filters.java:102)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:107)
      at org.forgerock.openidm.audit.filter.AuditFilter.filterRead(AuditFilter.java:166)
      at org.forgerock.json.resource.Filters$ConditionalFilter.filterRead(Filters.java:100)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:107)
      at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$5.filterRead(ServletConnectionFactory.java:527)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:107)
      at org.forgerock.openidm.maintenance.impl.PassthroughFilter.filterRead(PassthroughFilter.java:72)
      at org.forgerock.openidm.maintenance.impl.MaintenanceService.filterRead(MaintenanceService.java:273)
      at org.forgerock.json.resource.Filters$ConditionalFilter.filterRead(Filters.java:100)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:107)
      at org.forgerock.json.resource.FilterChain.handleRead(FilterChain.java:237)
      at org.forgerock.json.resource.InternalConnection.readAsync(InternalConnection.java:89)
      at org.forgerock.json.resource.AbstractAsynchronousConnection.read(AbstractAsynchronousConnection.java:79)
      at org.forgerock.json.resource.AbstractConnectionWrapper.read(AbstractConnectionWrapper.java:198)
      at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$1$1.read(ServletConnectionFactory.java:199)
      at org.forgerock.json.resource.AbstractConnectionWrapper.read(AbstractConnectionWrapper.java:198)
      at org.forgerock.openidm.quartz.impl.RepoJobStore.readFromRepo(RepoJobStore.java:1578)
      at org.forgerock.openidm.quartz.impl.RepoJobStore.getTriggerFromRepo(RepoJobStore.java:2073)
      at org.forgerock.openidm.quartz.impl.RepoJobStore.getTriggerWrapper(RepoJobStore.java:2116)
      at org.forgerock.openidm.quartz.impl.RepoJobStore.releaseAcquiredTrigger(RepoJobStore.java:677)
      at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:340)
      
      Apr 19, 2016 6:13:35 AM org.forgerock.openidm.servlet.internal.ServletConnectionFactory$4 handleException
      FINEST: Resource exception: 404 Not Found: "Object scheduler/triggers/scheduler-service-group_$x$x$_trigger-activeSynchroniser_systemPartnerAccount not found in scheduler/triggers"
      org.forgerock.json.resource.NotFoundException: Object scheduler/triggers/scheduler-service-group_$x$x$_trigger-activeSynchroniser_systemPartnerAccount not found in scheduler/triggers
      at org.forgerock.json.resource.ResourceException.newResourceException(ResourceException.java:227)
      at org.forgerock.json.resource.ResourceException.getException(ResourceException.java:309)
      at org.forgerock.openidm.repo.jdbc.impl.GenericTableHandler.read(GenericTableHandler.java:238)
      at org.forgerock.openidm.repo.jdbc.impl.JDBCRepoService.read(JDBCRepoService.java:268)
      at org.forgerock.openidm.repo.jdbc.impl.JDBCRepoService.handleRead(JDBCRepoService.java:239)
      at org.forgerock.json.resource.Router.handleRead(Router.java:324)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:109)
      at org.forgerock.json.resource.Filters$ConditionalFilter.filterRead(Filters.java:102)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:107)
      at org.forgerock.json.resource.Filters$ConditionalFilter.filterRead(Filters.java:102)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:107)
      at org.forgerock.json.resource.Filters$ConditionalFilter.filterRead(Filters.java:102)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:107)
      at org.forgerock.openidm.audit.filter.AuditFilter.filterRead(AuditFilter.java:166)
      at org.forgerock.json.resource.Filters$ConditionalFilter.filterRead(Filters.java:100)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:107)
      at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$5.filterRead(ServletConnectionFactory.java:527)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:107)
      at org.forgerock.openidm.maintenance.impl.PassthroughFilter.filterRead(PassthroughFilter.java:72)
      at org.forgerock.openidm.maintenance.impl.MaintenanceService.filterRead(MaintenanceService.java:273)
      at org.forgerock.json.resource.Filters$ConditionalFilter.filterRead(Filters.java:100)
      at org.forgerock.json.resource.FilterChain$Cursor.handleRead(FilterChain.java:107)
      at org.forgerock.json.resource.FilterChain.handleRead(FilterChain.java:237)
      at org.forgerock.json.resource.InternalConnection.readAsync(InternalConnection.java:89)
      at org.forgerock.json.resource.AbstractAsynchronousConnection.read(AbstractAsynchronousConnection.java:79)
      at org.forgerock.json.resource.AbstractConnectionWrapper.read(AbstractConnectionWrapper.java:198)
      at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$1$1.read(ServletConnectionFactory.java:199)
      at org.forgerock.json.resource.AbstractConnectionWrapper.read(AbstractConnectionWrapper.java:198)
      at org.forgerock.openidm.quartz.impl.RepoJobStore.readFromRepo(RepoJobStore.java:1578)
      at org.forgerock.openidm.quartz.impl.RepoJobStore.getTriggerFromRepo(RepoJobStore.java:2073)
      at org.forgerock.openidm.quartz.impl.RepoJobStore.getTriggerWrapper(RepoJobStore.java:2116)
      at org.forgerock.openidm.quartz.impl.RepoJobStore.releaseAcquiredTrigger(RepoJobStore.java:677)
      at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:340)
      
      Apr 19, 2016 6:13:35 AM org.forgerock.openidm.quartz.impl.RepoJobStore releaseAcquiredTrigger
      FINE: Cannot release acquired trigger trigger-activeSynchroniser_systemPartnerAccount in group scheduler-service-group, trigger does not exist
      

      When the node running the schedule is shut down the other running node does not take it over. If I then bring that node down and up again it then processes the schedule.  

      Furthermore, possibly related to timing/recovery, the cluster can get into a state where the schedule alternates between nodes on each invocation, rather than staying executing on one node.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jason Jason Lemay
                Reporter:
                johan Johan Gardner
                QA Assignee:
                Ladislav Folta
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: