Uploaded image for project: 'OpenDJ'
  1. OpenDJ
  2. OPENDJ-6832

Deadlock between delete and sorted unindexed search

    Details

    • Type: Bug
    • Status: Done
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 7.0.0
    • Fix Version/s: 7.0.0
    • Component/s: backends
    • Labels:
      None

      Description

      Chris Drake has reported the following deadlock while performing a delete and a search.

      "EventPollExecutor-0" #276 prio=5 os_prio=31 tid=0x00007fc44f73d800 nid=0xa307 in Object.wait() [0x000070000f192000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	at com.sleepycat.je.txn.LockManager.waitForLock(LockManager.java:525)
      	at com.sleepycat.je.txn.LockManager.lock(LockManager.java:353)
      	- locked <0x000000075855a388> (a com.sleepycat.je.txn.ThreadLocker)
      	at com.sleepycat.je.txn.BasicLocker.lockInternal(BasicLocker.java:125)
      	at com.sleepycat.je.txn.Locker.lock(Locker.java:496)
      	at com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:3674)
      	at com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:3399)
      	at com.sleepycat.je.dbi.CursorImpl.lockLNAndCheckDefunct(CursorImpl.java:2177)
      	at com.sleepycat.je.dbi.CursorImpl.searchExact(CursorImpl.java:1989)
      	at com.sleepycat.je.Cursor.searchExact(Cursor.java:4210)
      	at com.sleepycat.je.Cursor.searchNoDups(Cursor.java:4071)
      	at com.sleepycat.je.Cursor.search(Cursor.java:3842)
      	- locked <0x000000075868d348> (a com.sleepycat.je.Cursor)
      	at com.sleepycat.je.Cursor.getInternal(Cursor.java:1290)
      	at com.sleepycat.je.Database.get(Database.java:1403)
      	at com.sleepycat.je.Database.get(Database.java:1462)
      	at org.opends.server.backends.jeb.JEStorage$WriteableTransactionImpl.read(JEStorage.java:544)
      	at org.opends.server.backends.pluggable.ID2Entry.get(ID2Entry.java:1223)
      	at org.opends.server.backends.pluggable.EntryContainer.getEntryOrNull0(EntryContainer.java:1951)
      	at org.opends.server.backends.pluggable.EntryContainer.getEntryOrNull(EntryContainer.java:1356)
      	at org.opends.server.backends.pluggable.EntryContainer.doUnindexedSortedSearch(EntryContainer.java:1315)
      	at org.opends.server.backends.pluggable.EntryContainer.searchNotIndexed(EntryContainer.java:1049)
      	at org.opends.server.backends.pluggable.EntryContainer.access$2000(EntryContainer.java:207)
      	at org.opends.server.backends.pluggable.EntryContainer$1.run(EntryContainer.java:845)
      	at org.opends.server.backends.pluggable.EntryContainer$1.run(EntryContainer.java:687)
      	at org.opends.server.backends.jeb.JEStorage.read(JEStorage.java:1061)
      	at org.opends.server.backends.pluggable.TracedStorage.read(TracedStorage.java:412)
      	at org.opends.server.backends.pluggable.EntryContainer.search(EntryContainer.java:687)
      	at org.opends.server.backends.pluggable.BackendImpl.search(BackendImpl.java:481)
      	at org.opends.server.core.SearchOperation.processSearch(SearchOperation.java:869)
      	at org.opends.server.core.SearchOperation.processLocalSearch(SearchOperation.java:824)
      	at org.opends.server.core.SearchOperation.run(SearchOperation.java:763)
      	at org.opends.server.protocols.internal.AbstractInternalClientConnection.handleRequest(AbstractInternalClientConnection.java:375)
      	at org.opends.server.protocols.internal.AbstractInternalClientConnection.lambda$handle$0(AbstractInternalClientConnection.java:358)
      	at org.opends.server.protocols.internal.AbstractInternalClientConnection$$Lambda$320/1911920986.subscribe(Unknown Source)
      	at io.reactivex.internal.operators.flowable.FlowableCreate.subscribeActual(FlowableCreate.java:72)
      	at io.reactivex.Flowable.subscribe(Flowable.java:13234)
      	at io.reactivex.Flowable.subscribe(Flowable.java:13180)
      	at io.reactivex.internal.operators.flowable.FlowableLift.subscribeActual(FlowableLift.java:49)
      	at io.reactivex.Flowable.subscribe(Flowable.java:13234)
      	at io.reactivex.internal.operators.flowable.FlowableDoOnEach.subscribeActual(FlowableDoOnEach.java:50)
      	at io.reactivex.Flowable.subscribe(Flowable.java:13234)
      	at io.reactivex.internal.operators.flowable.FlowableDoFinally.subscribeActual(FlowableDoFinally.java:47)
      	at io.reactivex.Flowable.subscribe(Flowable.java:13234)
      	at io.reactivex.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:38)
      	at io.reactivex.Flowable.subscribe(Flowable.java:13234)
      	at io.reactivex.internal.operators.flowable.FlowableDoOnEach.subscribeActual(FlowableDoOnEach.java:50)
      	at io.reactivex.Flowable.subscribe(Flowable.java:13234)
      	at io.reactivex.internal.operators.flowable.FlowableOnErrorNext.subscribeActual(FlowableOnErrorNext.java:39)
      	at io.reactivex.Flowable.subscribe(Flowable.java:13234)
      	at io.reactivex.internal.operators.flowable.FlowablePublish.connect(FlowablePublish.java:130)
      	at io.reactivex.flowables.ConnectableFlowable.connect(ConnectableFlowable.java:64)
      	at org.opends.server.protocols.internal.AbstractInternalClientConnection$InternalConnection.search(AbstractInternalClientConnection.java:156)
      	at org.forgerock.opendj.ldap.AbstractSynchronousConnection.searchAsync(AbstractSynchronousConnection.java:133)
      	at org.forgerock.opendj.rest2ldap.RequestControlConnectionDecorator.searchAsync(RequestControlConnectionDecorator.java:112)
      	at org.forgerock.opendj.rest2ldap.CachedReadConnectionDecorator.searchAsync(CachedReadConnectionDecorator.java:173)
      	at org.forgerock.opendj.ldap.AbstractConnection.searchAsync(AbstractConnection.java:364)
      	at org.forgerock.opendj.rest2ldap.SubResourceImpl$1ObservableSearchResultHandler.subscribeActual(SubResourceImpl.java:744)
      	at io.reactivex.Observable.subscribe(Observable.java:11194)
      	at io.reactivex.internal.operators.observable.ObservableConcatMapEager.subscribeActual(ObservableConcatMapEager.java:54)
      	at io.reactivex.Observable.subscribe(Observable.java:11194)
      	at io.reactivex.internal.operators.observable.ObservableDematerialize.subscribeActual(ObservableDematerialize.java:29)
      	at io.reactivex.Observable.subscribe(Observable.java:11194)
      	at io.reactivex.internal.operators.observable.ObservableMap.subscribeActual(ObservableMap.java:33)
      	at io.reactivex.Observable.subscribe(Observable.java:11194)
      	at io.reactivex.internal.operators.observable.ObservablePublish.connect(ObservablePublish.java:116)
      	at io.reactivex.observables.ConnectableObservable.connect(ConnectableObservable.java:63)
      	at org.forgerock.opendj.rest2ldap.SubResourceImpl$3.apply(SubResourceImpl.java:832)
      	at org.forgerock.opendj.rest2ldap.SubResourceImpl$3.apply(SubResourceImpl.java:781)
      	at org.forgerock.util.promise.PromiseImpl$1.executeCallback(PromiseImpl.java:434)
      	at org.forgerock.util.promise.PromiseImpl$1.handleStateChange(PromiseImpl.java:422)
      	at org.forgerock.util.promise.PromiseImpl.handleCompletion(PromiseImpl.java:538)
      	at org.forgerock.util.promise.PromiseImpl.addOrFireListener(PromiseImpl.java:526)
      	at org.forgerock.util.promise.PromiseImpl.thenAsync(PromiseImpl.java:416)
      	at org.forgerock.util.promise.PromiseImpl.thenAsync(PromiseImpl.java:407)
      	at org.forgerock.util.promise.PromiseImpl.thenAsync(PromiseImpl.java:388)
      	at org.forgerock.opendj.rest2ldap.SubResourceImpl.query(SubResourceImpl.java:565)
      	at org.forgerock.opendj.rest2ldap.SubResourceCollection$CollectionHandler.handleQuery(SubResourceCollection.java:361)
      	at org.forgerock.json.resource.Router.handleQuery(Router.java:317)
      	at org.forgerock.opendj.rest2ldap.SubResource$SubResourceHandler.lambda$handleQuery$4(SubResource.java:161)
      	at org.forgerock.opendj.rest2ldap.SubResource$SubResourceHandler$$Lambda$694/1156721533.apply(Unknown Source)
      	at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:263)
      	at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:227)
      	at org.forgerock.opendj.rest2ldap.SubResource$SubResourceHandler.handleQuery(SubResource.java:161)
      	at org.forgerock.json.resource.Router.handleQuery(Router.java:317)
      	at org.forgerock.opendj.rest2ldap.DescribableRequestHandler.handleQuery(DescribableRequestHandler.java:74)
      	at org.forgerock.openidm.repo.opendj.impl.RetryRequestHandlerProxy.lambda$handleQuery$6(RetryRequestHandlerProxy.java:123)
      	at org.forgerock.openidm.repo.opendj.impl.RetryRequestHandlerProxy$$Lambda$692/2110134357.get(Unknown Source)
      	at org.forgerock.openidm.repo.opendj.impl.RetryRequestHandlerProxy.afterRetrying(RetryRequestHandlerProxy.java:67)
      	at org.forgerock.openidm.repo.opendj.impl.RetryRequestHandlerProxy.handleQuery(RetryRequestHandlerProxy.java:123)
      	at org.forgerock.openidm.repo.opendj.impl.ExplicitDJTypeHandler.handleQuery(ExplicitDJTypeHandler.java:268)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:94)
      	at org.forgerock.openidm.repo.opendj.impl.PopulateMissingEdgesFilter.filterQuery(PopulateMissingEdgesFilter.java:124)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:92)
      	at org.forgerock.openidm.repo.opendj.impl.TransformRelationshipQuery.filterQuery(TransformRelationshipQuery.java:88)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:92)
      	at org.forgerock.openidm.repo.opendj.impl.OpenDJConnectionContextFilter.lambda$filterQuery$5(OpenDJConnectionContextFilter.java:118)
      	at org.forgerock.openidm.repo.opendj.impl.OpenDJConnectionContextFilter$$Lambda$687/1402530079.apply(Unknown Source)
      	at org.forgerock.openidm.repo.opendj.impl.OpenDJConnectionContextFilter.lambda$withConnectionContext$0(OpenDJConnectionContextFilter.java:80)
      	at org.forgerock.openidm.repo.opendj.impl.OpenDJConnectionContextFilter$$Lambda$662/1756984047.get(Unknown Source)
      	at org.forgerock.openidm.metrics.MetricsCollector.time(MetricsCollector.java:112)
      	at org.forgerock.openidm.repo.opendj.impl.OpenDJConnectionContextFilter.withConnectionContext(OpenDJConnectionContextFilter.java:75)
      	at org.forgerock.openidm.repo.opendj.impl.OpenDJConnectionContextFilter.filterQuery(OpenDJConnectionContextFilter.java:117)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:92)
      	at org.forgerock.json.resource.FilterChain.handleQuery(FilterChain.java:249)
      	at org.forgerock.openidm.repo.opendj.impl.OpenDJRepoService.lambda$handleQuery$9(OpenDJRepoService.java:221)
      	at org.forgerock.openidm.repo.opendj.impl.OpenDJRepoService$$Lambda$686/1117512499.get(Unknown Source)
      	at org.forgerock.openidm.metrics.MetricsCollector.time(MetricsCollector.java:112)
      	at org.forgerock.openidm.repo.opendj.impl.OpenDJRepoService.handleQuery(OpenDJRepoService.java:216)
      	at org.forgerock.json.resource.Router.handleQuery(Router.java:317)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:94)
      	at org.forgerock.json.resource.ResourceApiVersionRoutingFilter.filterQuery(ResourceApiVersionRoutingFilter.java:71)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:92)
      	at org.forgerock.json.resource.FilterChain.handleQuery(FilterChain.java:249)
      	at org.forgerock.json.resource.Router.handleQuery(Router.java:317)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:94)
      	at org.forgerock.json.resource.Filters$ConditionalFilter.filterQuery(Filters.java:84)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:92)
      	at org.forgerock.json.resource.Filters$ConditionalFilter.filterQuery(Filters.java:84)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:92)
      	at org.forgerock.json.resource.Filters$ConditionalFilter.filterQuery(Filters.java:84)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:92)
      	at org.forgerock.openidm.audit.filter.AuditFilter.lambda$filterQuery$4(AuditFilter.java:139)
      	at org.forgerock.openidm.audit.filter.AuditFilter$$Lambda$1007/466161677.get(Unknown Source)
      	at org.forgerock.openidm.audit.filter.AuditFilter.logAuditAccessEntry(AuditFilter.java:171)
      	at org.forgerock.openidm.audit.filter.AuditFilter.filterQuery(AuditFilter.java:139)
      	at org.forgerock.openidm.router.filter.MutableFilterDecorator.filterQuery(MutableFilterDecorator.java:90)
      	at org.forgerock.json.resource.Filters$ConditionalFilter.filterQuery(Filters.java:82)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:92)
      	at org.forgerock.openidm.servlet.internal.ResourceFilters$3.lambda$filterQuery$4(ResourceFilters.java:194)
      	at org.forgerock.openidm.servlet.internal.ResourceFilters$3$$Lambda$1006/940294106.get(Unknown Source)
      	at org.forgerock.openidm.servlet.internal.ResourceFilters$3.handleRequestWithLogging(ResourceFilters.java:222)
      	at org.forgerock.openidm.servlet.internal.ResourceFilters$3.filterQuery(ResourceFilters.java:194)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:92)
      	at org.forgerock.openidm.filter.QueryIdToQueryFilterTransformFilter.filterQuery(QueryIdToQueryFilterTransformFilter.java:191)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:92)
      	at org.forgerock.openidm.router.filter.PassthroughFilter.filterQuery(PassthroughFilter.java:66)
      	at org.forgerock.openidm.router.filter.MutableFilterDecorator.filterQuery(MutableFilterDecorator.java:90)
      	at org.forgerock.openidm.router.filter.MutableFilterDecorator.filterQuery(MutableFilterDecorator.java:90)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:92)
      	at org.forgerock.json.resource.Filters$ConditionalFilter.filterQuery(Filters.java:84)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:92)
      	at org.forgerock.json.resource.Filters$ConditionalFilter.filterQuery(Filters.java:84)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:92)
      	at org.forgerock.json.resource.FilterChain.handleQuery(FilterChain.java:249)
      	at org.forgerock.json.resource.InternalConnection.queryAsync(InternalConnection.java:74)
      	at org.forgerock.json.resource.AbstractConnectionWrapper.queryAsync(AbstractConnectionWrapper.java:185)
      	at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$InternalConnectionWrapper.lambda$queryAsync$13(ServletConnectionFactory.java:335)
      	at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$InternalConnectionWrapper$$Lambda$1111/758353887.get(Unknown Source)
      	at org.forgerock.openidm.metrics.MetricsCollector.time(MetricsCollector.java:112)
      	at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$InternalConnectionWrapper.time(ServletConnectionFactory.java:257)
      	at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$InternalConnectionWrapper.queryAsync(ServletConnectionFactory.java:334)
      	at org.forgerock.json.resource.AbstractConnectionWrapper.queryAsync(AbstractConnectionWrapper.java:185)
      	at org.forgerock.openidm.sync.impl.queue.SynchronizationQueueService.lambda$null$8(SynchronizationQueueService.java:151)
      	at org.forgerock.openidm.sync.impl.queue.SynchronizationQueueService$$Lambda$1408/83399464.apply(Unknown Source)
      	at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:263)
      	at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:227)
      	at org.forgerock.openidm.sync.impl.queue.SynchronizationQueueService.lambda$poll$9(SynchronizationQueueService.java:151)
      	at org.forgerock.openidm.sync.impl.queue.SynchronizationQueueService$$Lambda$1407/1949977669.get(Unknown Source)
      	at org.forgerock.openidm.metrics.MetricsCollector.time(MetricsCollector.java:112)
      	at org.forgerock.openidm.sync.impl.queue.SynchronizationQueueService.poll(SynchronizationQueueService.java:148)
      	at org.forgerock.openidm.sync.impl.queue.SynchronizationQueueConsumer.lambda$new$0(SynchronizationQueueConsumer.java:139)
      	at org.forgerock.openidm.sync.impl.queue.SynchronizationQueueConsumer$$Lambda$1374/1259688551.run(Unknown Source)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      
      "Processing-68069acb-93dd-4e55-9584-49536bfaa669" #275 prio=5 os_prio=31 tid=0x00007fc46b008800 nid=0xa407 in Object.wait() [0x000070000f08d000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	at com.sleepycat.je.txn.LockManager.waitForLock(LockManager.java:525)
      	at com.sleepycat.je.txn.LockManager.lock(LockManager.java:353)
      	- locked <0x00000007586b8e28> (a com.sleepycat.je.txn.Txn)
      	at com.sleepycat.je.txn.Txn.lockInternal(Txn.java:545)
      	at com.sleepycat.je.txn.ReadCommittedLocker.lockInternal(ReadCommittedLocker.java:89)
      	at com.sleepycat.je.txn.Locker.lock(Locker.java:496)
      	at com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:3674)
      	at com.sleepycat.je.dbi.CursorImpl.deleteCurrentRecord(CursorImpl.java:917)
      	at com.sleepycat.je.Cursor.deleteNoNotify(Cursor.java:2370)
      	- locked <0x00000007586c5ae8> (a com.sleepycat.je.Transaction)
      	at com.sleepycat.je.Cursor.deleteNotify(Cursor.java:2311)
      	at com.sleepycat.je.Cursor.deleteInternal(Cursor.java:2220)
      	at com.sleepycat.je.Database.deleteInternal(Database.java:1229)
      	at com.sleepycat.je.Database.delete(Database.java:1185)
      	at com.sleepycat.je.Database.delete(Database.java:1291)
      	at org.opends.server.backends.jeb.JEStorage$WriteableTransactionImpl.delete(JEStorage.java:512)
      	at org.opends.server.backends.pluggable.VLVIndex.updateIndex(VLVIndex.java:404)
      	at org.opends.server.backends.pluggable.IndexBuffer$DefaultIndexBuffer.flush(IndexBuffer.java:177)
      	at org.opends.server.backends.pluggable.IndexBuffer.flush(IndexBuffer.java:301)
      	at org.opends.server.backends.pluggable.EntryContainer$2.run(EntryContainer.java:1849)
      	at org.opends.server.backends.jeb.JEStorage.write(JEStorage.java:1085)
      	at org.opends.server.backends.pluggable.TracedStorage.write(TracedStorage.java:441)
      	at org.opends.server.backends.pluggable.EntryContainer.deleteEntry(EntryContainer.java:1758)
      	at org.opends.server.backends.pluggable.EntryContainer.deleteEntry(EntryContainer.java:1751)
      	at org.opends.server.backends.pluggable.BackendImpl.deleteEntry(BackendImpl.java:426)
      	at org.opends.server.core.DeleteOperation.processDelete(DeleteOperation.java:334)
      	at org.opends.server.core.DeleteOperation.processLocalDelete(DeleteOperation.java:231)
      	at org.opends.server.core.DeleteOperation.run(DeleteOperation.java:201)
      	at org.opends.server.protocols.internal.AbstractInternalClientConnection.handleRequest(AbstractInternalClientConnection.java:375)
      	at org.opends.server.protocols.internal.AbstractInternalClientConnection.lambda$handle$0(AbstractInternalClientConnection.java:358)
      	at org.opends.server.protocols.internal.AbstractInternalClientConnection$$Lambda$320/1911920986.subscribe(Unknown Source)
      	at io.reactivex.internal.operators.flowable.FlowableCreate.subscribeActual(FlowableCreate.java:72)
      	at io.reactivex.Flowable.subscribe(Flowable.java:13234)
      	at io.reactivex.Flowable.subscribe(Flowable.java:13180)
      	at io.reactivex.internal.operators.flowable.FlowableLift.subscribeActual(FlowableLift.java:49)
      	at io.reactivex.Flowable.subscribe(Flowable.java:13234)
      	at io.reactivex.internal.operators.flowable.FlowableDoOnEach.subscribeActual(FlowableDoOnEach.java:50)
      	at io.reactivex.Flowable.subscribe(Flowable.java:13234)
      	at io.reactivex.internal.operators.flowable.FlowableDoFinally.subscribeActual(FlowableDoFinally.java:47)
      	at io.reactivex.Flowable.subscribe(Flowable.java:13234)
      	at io.reactivex.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:38)
      	at io.reactivex.Flowable.subscribe(Flowable.java:13234)
      	at io.reactivex.internal.operators.flowable.FlowableDoOnEach.subscribeActual(FlowableDoOnEach.java:50)
      	at io.reactivex.Flowable.subscribe(Flowable.java:13234)
      	at io.reactivex.internal.operators.flowable.FlowableOnErrorNext.subscribeActual(FlowableOnErrorNext.java:39)
      	at io.reactivex.Flowable.subscribe(Flowable.java:13234)
      	at io.reactivex.internal.operators.flowable.FlowableFilter.subscribeActual(FlowableFilter.java:37)
      	at io.reactivex.Flowable.subscribe(Flowable.java:13234)
      	at io.reactivex.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:38)
      	at io.reactivex.Flowable.subscribe(Flowable.java:13234)
      	at io.reactivex.Flowable.blockingLast(Flowable.java:5576)
      	at org.opends.server.protocols.internal.AbstractInternalClientConnection$InternalConnection.performRequest(AbstractInternalClientConnection.java:203)
      	at org.opends.server.protocols.internal.AbstractInternalClientConnection$InternalConnection.delete(AbstractInternalClientConnection.java:122)
      	at org.forgerock.opendj.ldap.AbstractSynchronousConnection.deleteAsync(AbstractSynchronousConnection.java:93)
      	at org.forgerock.opendj.rest2ldap.RequestControlConnectionDecorator.deleteAsync(RequestControlConnectionDecorator.java:86)
      	at org.forgerock.opendj.rest2ldap.CachedReadConnectionDecorator.deleteAsync(CachedReadConnectionDecorator.java:156)
      	at org.forgerock.opendj.ldap.AbstractConnection$2.visitRequest(AbstractConnection.java:205)
      	at org.forgerock.opendj.ldap.AbstractConnection$2.visitRequest(AbstractConnection.java:197)
      	at org.forgerock.opendj.ldap.messages.DeleteRequestImpl.accept(DeleteRequestImpl.java:38)
      	at org.forgerock.opendj.ldap.AbstractConnection.applyChangeAsync(AbstractConnection.java:218)
      	at org.forgerock.opendj.ldap.AbstractConnection.applyChangeAsync(AbstractConnection.java:190)
      	at org.forgerock.opendj.rest2ldap.SubResourceImpl.delete(SubResourceImpl.java:436)
      	at org.forgerock.opendj.rest2ldap.SubResourceImpl.lambda$delete$8(SubResourceImpl.java:419)
      	at org.forgerock.opendj.rest2ldap.SubResourceImpl$$Lambda$1481/716149672.apply(Unknown Source)
      	at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:263)
      	at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:227)
      	at org.forgerock.opendj.rest2ldap.SubResourceImpl.delete(SubResourceImpl.java:419)
      	at org.forgerock.opendj.rest2ldap.SubResourceCollection$InstanceHandler.handleDelete(SubResourceCollection.java:399)
      	at org.forgerock.json.resource.Router.handleDelete(Router.java:279)
      	at org.forgerock.opendj.rest2ldap.SubResource$SubResourceHandler.lambda$handleDelete$2(SubResource.java:147)
      	at org.forgerock.opendj.rest2ldap.SubResource$SubResourceHandler$$Lambda$1480/1569304095.apply(Unknown Source)
      	at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:263)
      	at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:227)
      	at org.forgerock.opendj.rest2ldap.SubResource$SubResourceHandler.handleDelete(SubResource.java:147)
      	at org.forgerock.json.resource.Router.handleDelete(Router.java:279)
      	at org.forgerock.opendj.rest2ldap.DescribableRequestHandler.handleDelete(DescribableRequestHandler.java:63)
      	at org.forgerock.openidm.repo.opendj.impl.RetryRequestHandlerProxy.lambda$handleDelete$3(RetryRequestHandlerProxy.java:105)
      	at org.forgerock.openidm.repo.opendj.impl.RetryRequestHandlerProxy$$Lambda$1479/1127899243.get(Unknown Source)
      	at org.forgerock.openidm.repo.opendj.impl.RetryRequestHandlerProxy.afterRetrying(RetryRequestHandlerProxy.java:67)
      	at org.forgerock.openidm.repo.opendj.impl.RetryRequestHandlerProxy.handleDelete(RetryRequestHandlerProxy.java:105)
      	at org.forgerock.openidm.repo.opendj.impl.ExplicitDJTypeHandler.handleDelete(ExplicitDJTypeHandler.java:144)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:74)
      	at org.forgerock.openidm.repo.opendj.impl.PopulateMissingEdgesFilter.filterDelete(PopulateMissingEdgesFilter.java:102)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:72)
      	at org.forgerock.openidm.repo.opendj.impl.OpenDJConnectionContextFilter.lambda$filterDelete$3(OpenDJConnectionContextFilter.java:105)
      	at org.forgerock.openidm.repo.opendj.impl.OpenDJConnectionContextFilter$$Lambda$1478/1017489697.apply(Unknown Source)
      	at org.forgerock.openidm.repo.opendj.impl.OpenDJConnectionContextFilter.lambda$withConnectionContext$0(OpenDJConnectionContextFilter.java:80)
      	at org.forgerock.openidm.repo.opendj.impl.OpenDJConnectionContextFilter$$Lambda$662/1756984047.get(Unknown Source)
      	at org.forgerock.openidm.metrics.MetricsCollector.time(MetricsCollector.java:112)
      	at org.forgerock.openidm.repo.opendj.impl.OpenDJConnectionContextFilter.withConnectionContext(OpenDJConnectionContextFilter.java:75)
      	at org.forgerock.openidm.repo.opendj.impl.OpenDJConnectionContextFilter.filterDelete(OpenDJConnectionContextFilter.java:105)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:72)
      	at org.forgerock.json.resource.FilterChain.handleDelete(FilterChain.java:237)
      	at org.forgerock.openidm.repo.opendj.impl.OpenDJRepoService.lambda$handleDelete$8(OpenDJRepoService.java:203)
      	at org.forgerock.openidm.repo.opendj.impl.OpenDJRepoService$$Lambda$1477/606760801.get(Unknown Source)
      	at org.forgerock.openidm.metrics.MetricsCollector.time(MetricsCollector.java:112)
      	at org.forgerock.openidm.repo.opendj.impl.OpenDJRepoService.handleDelete(OpenDJRepoService.java:199)
      	at org.forgerock.json.resource.Router.handleDelete(Router.java:279)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:74)
      	at org.forgerock.json.resource.ResourceApiVersionRoutingFilter.filterDelete(ResourceApiVersionRoutingFilter.java:55)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:72)
      	at org.forgerock.json.resource.FilterChain.handleDelete(FilterChain.java:237)
      	at org.forgerock.json.resource.Router.handleDelete(Router.java:279)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:74)
      	at org.forgerock.json.resource.Filters$ConditionalFilter.filterDelete(Filters.java:64)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:72)
      	at org.forgerock.json.resource.Filters$ConditionalFilter.filterDelete(Filters.java:64)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:72)
      	at org.forgerock.json.resource.Filters$ConditionalFilter.filterDelete(Filters.java:64)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:72)
      	at org.forgerock.openidm.audit.filter.AuditFilter.lambda$filterDelete$2(AuditFilter.java:127)
      	at org.forgerock.openidm.audit.filter.AuditFilter$$Lambda$1476/737586040.get(Unknown Source)
      	at org.forgerock.openidm.audit.filter.AuditFilter.logAuditAccessEntry(AuditFilter.java:171)
      	at org.forgerock.openidm.audit.filter.AuditFilter.filterDelete(AuditFilter.java:127)
      	at org.forgerock.openidm.router.filter.MutableFilterDecorator.filterDelete(MutableFilterDecorator.java:78)
      	at org.forgerock.json.resource.Filters$ConditionalFilter.filterDelete(Filters.java:62)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:72)
      	at org.forgerock.openidm.servlet.internal.ResourceFilters$3.lambda$filterDelete$2(ResourceFilters.java:182)
      	at org.forgerock.openidm.servlet.internal.ResourceFilters$3$$Lambda$1475/1117236976.get(Unknown Source)
      	at org.forgerock.openidm.servlet.internal.ResourceFilters$3.handleRequestWithLogging(ResourceFilters.java:222)
      	at org.forgerock.openidm.servlet.internal.ResourceFilters$3.filterDelete(ResourceFilters.java:182)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:72)
      	at org.forgerock.openidm.filter.QueryIdToQueryFilterTransformFilter.filterDelete(QueryIdToQueryFilterTransformFilter.java:166)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:72)
      	at org.forgerock.openidm.router.filter.PassthroughFilter.filterDelete(PassthroughFilter.java:54)
      	at org.forgerock.openidm.router.filter.MutableFilterDecorator.filterDelete(MutableFilterDecorator.java:78)
      	at org.forgerock.openidm.router.filter.MutableFilterDecorator.filterDelete(MutableFilterDecorator.java:78)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:72)
      	at org.forgerock.json.resource.Filters$ConditionalFilter.filterDelete(Filters.java:64)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:72)
      	at org.forgerock.json.resource.Filters$ConditionalFilter.filterDelete(Filters.java:64)
      	at org.forgerock.json.resource.FilterChain$Cursor.handleDelete(FilterChain.java:72)
      	at org.forgerock.json.resource.FilterChain.handleDelete(FilterChain.java:237)
      	at org.forgerock.json.resource.InternalConnection.deleteAsync(InternalConnection.java:47)
      	at org.forgerock.json.resource.AbstractConnectionWrapper.deleteAsync(AbstractConnectionWrapper.java:119)
      	at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$InternalConnectionWrapper.lambda$deleteAsync$7(ServletConnectionFactory.java:300)
      	at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$InternalConnectionWrapper$$Lambda$1474/1178326416.get(Unknown Source)
      	at org.forgerock.openidm.metrics.MetricsCollector.time(MetricsCollector.java:112)
      	at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$InternalConnectionWrapper.time(ServletConnectionFactory.java:257)
      	at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$InternalConnectionWrapper.deleteAsync(ServletConnectionFactory.java:300)
      	at org.forgerock.json.resource.AbstractConnectionWrapper.deleteAsync(AbstractConnectionWrapper.java:119)
      	at org.forgerock.openidm.sync.impl.queue.SynchronizationQueueService.lambda$null$2(SynchronizationQueueService.java:119)
      	at org.forgerock.openidm.sync.impl.queue.SynchronizationQueueService$$Lambda$1473/916150540.apply(Unknown Source)
      	at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:263)
      	at org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:227)
      	at org.forgerock.openidm.sync.impl.queue.SynchronizationQueueService.lambda$discard$3(SynchronizationQueueService.java:119)
      	at org.forgerock.openidm.sync.impl.queue.SynchronizationQueueService$$Lambda$1472/1818467896.get(Unknown Source)
      	at org.forgerock.openidm.metrics.MetricsCollector.time(MetricsCollector.java:112)
      	at org.forgerock.openidm.sync.impl.queue.SynchronizationQueueService.discard(SynchronizationQueueService.java:117)
      	at org.forgerock.openidm.sync.impl.queue.SynchronizationEventExecutor.lambda$submit$1(SynchronizationEventExecutor.java:122)
      	at org.forgerock.openidm.sync.impl.queue.SynchronizationEventExecutor$$Lambda$1414/49388225.call(Unknown Source)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at org.forgerock.openidm.sync.impl.queue.TaskRunningQueue.run(TaskRunningQueue.java:51)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      

      The delete thread has updated the id2entry index and is then blocked trying to update a VLV index. The search thread is cursoring through the VLV index and reading from id2entry. What I don't understand is that the search VLV index cursor is using read committed isolation, so it should not hold onto record locks for the duration of the txn: it should have released the lock before reading id2entry (we should check this).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                matthew Matthew Swift
                Reporter:
                matthew Matthew Swift
                Dev Assignee:
                Matthew Swift
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: