Uploaded image for project: 'OpenDJ'
  1. OpenDJ
  2. OPENDJ-1034

Improve the performance of the replication changelog DB

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 3.0.0, 2.8.0
    • Component/s: replication
    • Labels:
      None
    • Sprint:
      Sprint 14, Sprint 15, Sprint 16

      Description

      Purging records from the replication changelog is unnecessarily expensive because we are using JE, which forces us to delete records one at a time. In addition the current implementation can only remove at most 5000 changes per second which is not sufficient to keep up with constant heavy load. Furthermore, the JE based implementation will be required to maintain BTree inner-nodes, run cleaner threads, store keys as well as values, etc, all of which consume resources.

      Since a change log is a very special case DB which does not require much of the complexity provided by JE (random access, transaction support, etc), we should be able to roll our own. We have the following requirements:

      1. update: write a change to the head of the log
      2. purge: remove a block of changes from the tail of the log
      3. random seek: performed when finding the initial position within the log during recovery or changelog browsing
      4. sequential read: performed once positioned during recovery or browsing, and always in the direction of the head of the log

      We should optimize for (1), (2), and (4): item (3) is performed less frequently and we can tolerate latencies of 100s of milli-seconds if needed.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                JnRouvignac Jean-Noël Rouvignac
                Reporter:
                matthew Matthew Swift
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: