Uploaded image for project: 'OpenDJ'
  1. OpenDJ
  2. OPENDJ-1702

Replace MessageHandler lateQueue and msgQueue by cursors

    Details

    • Type: Task
    • Status: Done
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 5.5.0
    • Component/s: replication
    • Labels:
      None

      Description

      The use of msgQueue and lateQueue originally sounded like a useful optimization: When the RS is fast enough following what the DS does, it stays in memory, and when it becomes late it reads from the lateQueue, after populating it from cursors.

      However, after OPENDJ-1611, OPENDJ-1672 and the file based changelog, it became apparent that not only this design is brittle and error prone, but also probably not as efficient as it seems.

      If the flow of messages coming from the DS is small, then the RS will always be following. However, in such case, the I/O savings are not worth the added complexity because the RS is not under big load from this DS. In the case where the flow of messages coming from the DS is very big, the RS cannot follow and is resorting to the lateQueue which incurs cursoring anyway.

      However, with the file based changelog it is less than ideal to repeatedly have to open and close several cursors for reading 100 entries then only each time. The disk seeks are a big toll to pay to maintain a complex design, even more when the system is under heavy load.

      I think it might be better to replace the msgQueue and lateQueue by cursors into the changelogDB that would always remain open. This way, we would rely on the OS file system caching and virtual pages to speed up linear reads (changelogDB is append only).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                fabiop Fabio Pistolesi
                Reporter:
                JnRouvignac Jean-Noël Rouvignac
                Dev Assignee:
                Fabio Pistolesi
              • Votes:
                2 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: