Uploaded image for project: 'OpenDJ'
  1. OpenDJ
  2. OPENDJ-3133

dsreplication status reports M.C. (Missing Changes) when none exist.

    Details

    • Support Ticket IDs:
    • Sprint:
      OpenDJ Sprint 87

      Description

      dsreplication status reports Missing Changes when all databases and the changelogDb all have the same data; i.e. no missing changes/data difference.

      The following example build, models how an instance can be created from a system "snapshot" (AWS) of a current Master.

      1. Create 2 masters in a Multi-Master replication topology.
        • Each uses an instance.loc to split the cli's and libs from the data
        • instance.loc - /opt/instances/reproduction/masterX/opendjdata
        • Installation Path: /opt/instances/reproduction/masterX/opendj
        • Instance Path: /opt/instances/reproduction/masterX/opendjdata
      2. Stop Master 1
        • Snapshot the whole instance as would happen with AWS' snapshot feature.
        • cp -rp <master> <new master> will suffice.
      3. Start Master 1 and add an entry
        • Check replication status. 0 Missing Changes
      4. Using the snapshotted data above, create a new Master using an instance.loc
        • instance.loc - /opt/instances/reproduction/masterX/opendjdata
        • Remove opendj and all under opendjdata "except" db/
        • Setup with no backend db or suffixes.
        • Create a new backend with the topology suffix. bin/status now shows all entries from Master 1 now available.
        • Enable replication from Master 1 to Master3.
        • Check replication status. 1 Missing Changes

      With Master 1 & 2 built out, replication is reporting 0 M.C. (from Master1)

      Checking status for opendj.forgerock.com:4444	2016-06-15-152324
      
      opendj; bin/$ date; ./dsreplication status --adminUID admin --adminPasswordFile pass --hostname opendj.forgerock.com --port 4444 --trustAll
      Wed Jun 15 15:23:24 MDT 2016
      Suffix DN         : Server                    : Entries : Replication enabled : DS ID : RS ID : RS Port (1) : M.C. (2) : A.O.M.C. (3) : Security (4)
      ------------------:---------------------------:---------:---------------------:-------:-------:-------------:----------:--------------:-------------
      dc=example,dc=com : opendj.forgerock.com:4444 : 2003    : true                : 10000 : 20000 : 8989        : 0        :              : false
      dc=example,dc=com : opendj.forgerock.com:5444 : 2003    : true                : 31817 : 9865  : 9989        : 0        :              : true
      

      The following starts with enabling replication from Step 4 above.

      The Command is -> ./dsreplication enable --adminUID admin --adminPassword password --baseDN dc=example,dc=com --host1 opendj.forgerock.com --port1 4444 --bindDN1 "cn=Directory Manager" --bindPassword1 password --replicationPort1 8989 --host2 opendj.forgerock.com --port2 6444 --bindDN2 "cn=Directory Manager" --bindPassword2 password --replicationPort2 10989 --trustAll --no-prompt
      
      Running command: Wed Jun 15 15:25:53 MDT 2016
      
      
      Establishing connections ..... Done.
      Checking registration information ..... Done.
      Updating remote references on server opendj.forgerock.com:4444 ..... Done.
      Configuring Replication port on server opendj.forgerock.com:6444 ..... Done.
      Updating replication configuration for baseDN dc=example,dc=com on server
      opendj.forgerock.com:4444 .....Done.
      Updating replication configuration for baseDN dc=example,dc=com on server
      opendj.forgerock.com:5444 .....Done.
      Updating remote references on server opendj.forgerock.com:5444 ..... Done.
      Updating replication configuration for baseDN dc=example,dc=com on server
      opendj.forgerock.com:6444 .....Done.
      Updating registration configuration on server opendj.forgerock.com:4444 ..... Done.
      Updating registration configuration on server opendj.forgerock.com:5444 ..... Done.
      Updating registration configuration on server opendj.forgerock.com:6444 ..... Done.
      Updating replication configuration for baseDN cn=schema on server
      opendj.forgerock.com:4444 .....Done.
      Updating replication configuration for baseDN cn=schema on server
      opendj.forgerock.com:5444 .....Done.
      Updating replication configuration for baseDN cn=schema on server
      opendj.forgerock.com:6444 .....Done.
      Initializing registration information on server opendj.forgerock.com:6444 with
      the contents of server opendj.forgerock.com:4444 .....Done.
      Initializing schema on server opendj.forgerock.com:6444 with the contents of
      server opendj.forgerock.com:4444 .....Done.
      
      Replication has been successfully enabled.  Note that for replication to work
      you must initialize the contents of the base DNs that are being replicated
      (use dsreplication initialize to do so).
      

      Check replication (from Master1) after enabling replication from Master 1 to Master 3.

      Checking status for opendj.forgerock.com:4444	2016-06-15-152627
      
      opendj; bin/$ date; ./dsreplication status --adminUID admin --adminPasswordFile pass --hostname opendj.forgerock.com --port 4444 --trustAll
      Wed Jun 15 15:26:27 MDT 2016
      Suffix DN         : Server                    : Entries : Replication enabled : DS ID : RS ID : RS Port (1) : M.C. (2) : A.O.M.C. (3) : Security (4)
      ------------------:---------------------------:---------:---------------------:-------:-------:-------------:----------:--------------:-------------
      dc=example,dc=com : opendj.forgerock.com:4444 : 2003    : true                : 10000 : 20000 : 8989        : 0        :              : false
      dc=example,dc=com : opendj.forgerock.com:5444 : 2003    : true                : 31817 : 9865  : 9989        : 0        :              : true
      dc=example,dc=com : opendj.forgerock.com:6444 : 2003    : true                : 16699 : 15470 : 10989       : 1        :              : false
      

      All changelog searches show 1 Change.

      dn: cn=changelog
      objectClass: top
      objectClass: container
      cn: changelog
      subschemaSubentry: cn=schema
      hasSubordinates: true
      entryDN: cn=changelog
      
      dn: changeNumber=1,cn=changelog
      objectClass: top
      objectClass: changeLogEntry
      changeNumber: 1
      changeTime: 20160615212319Z
      changeType: add
      targetDN: uid=luser,ou=People,dc=example,dc=com
      changes:: b2JqZWN0Q2xhc3M6IG9yZ2FuaXphdGlvbmFsUGVyc29uCm9iamVjdENsYXNzOiB0b3AKb2JqZWN0Q2xhc3M6IHBlcnNvbgpvYmplY3RDbGFzczogaW5ldE9yZ1BlcnNvbgp1aWQ6IGx1c2VyCmdpdmVuTmFtZTogTG91CnNuOiBVc2VyCmNuOiBMb3UgVXNlcgp1c2VyUGFzc3dvcmQ6IHtTU0hBfUNrcno4MUJDYmUwRjhNc1lWNnoyanpwck4zT084SzRUVWZ1TFpnPT0KZW50cnlVVUlEOiBkNzE3OGNkNS1iOWY3LTQyZjQtYjIxNC1hMzE5YjdhOWI5MDcKY3JlYXRlVGltZXN0YW1wOiAyMDE2MDYxNTIxMjMxOVoKcHdkQ2hhbmdlZFRpbWU6IDIwMTYwNjE1MjEyMzE5LjUxMVoKY3JlYXRvcnNOYW1lOiBjbj1EaXJlY3RvcnkgTWFuYWdlcixjbj1Sb290IEROcyxjbj1jb25maWcK
      subschemaSubentry: cn=schema
      numSubordinates: 0
      hasSubordinates: false
      entryDN: changeNumber=1,cn=changelog
      replicationCSN: 0000015555f26f57271000000001
      replicaIdentifier: 10000
      changeInitiatorsName: cn=Directory Manager,cn=Root DNs,cn=config
      targetEntryUUID: d7178cd5-b9f7-42f4-b214-a319b7a9b907
      changeLogCookie: dc=example,dc=com:0000015555f26f57271000000001;
      
      # Total number of matching entries: 2
      

      Likewise, each exported backend ldif are exactly the same..again, no missing changes (data).

      opendj; reproduction/$ diff ./master2/opendj/bin/m1-example.ldif ./master2/opendj/bin/m1-example.ldif
      opendj; reproduction/$ diff ./master2/opendj/bin/m1-example.ldif ./master1/opendj/bin/m2-example.ldif
      opendj; reproduction/$ diff ./master2/opendj/bin/m1-example.ldif ./master3/opendj/bin/m3-example.ldif
      opendj; reproduction/$ 
      opendj; reproduction/$ diff ./master2/opendj/bin/m1-example.ldif ./master1/opendj/bin/m2-example.ldif
      opendj; reproduction/$ diff ./master2/opendj/bin/m1-example.ldif ./master3/opendj/bin/m3-example.ldif
      opendj; reproduction/$ diff ./master1/opendj/bin/m2-example.ldif ./master3/opendj/bin/m3-example.ldif
      

      Data attached:

      I captured the following files within each of the following steps:

      num.<file> = Index Num from below.

      1.changelogDb.2016-06-15-152228
      1.cn=repl.monitor.2016-06-15-152229
      1.head-dump.2016-06-15-152230
      1.replStatus.2016-06-15-152230
      2.changelogDb.2016-06-15-152322
      2.cn=repl.monitor.2016-06-15-152322
      2.head-dump.2016-06-15-152323
      2.replStatus.2016-06-15-152324
      3.changelogDb.2016-06-15-152626
      3.cn=repl.monitor.2016-06-15-152626
      3.head-dump.2016-06-15-152627
      3.replStatus.2016-06-15-152627
      

      Index 1: After M1 & M2 are MMR enabled.
      Index 2: After restart of M1 (post snapshot) and adding 1 entry to M1 (uid=luser)
      Index 3: After adding M3 to replication

      -rw-r--r--  1 opendj  opendj     268159 Jun 15 15:26 debug-data-2016-06-15-152636.zip
      -rw-r--r--  1 opendj  opendj     268886 Jun 15 15:27 debug-data-2016-06-15-152718.zip
      

      Each of the attached contains a copy of the opendjdata/changelogDb before and after stopping all servers.

      debug-data-2016-06-15-152636.zip: taken while the servers were running with a Missing Change.
      debug-data-2016-06-15-152718.zip: taken after all servers were stopped.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                JnRouvignac Jean-Noël Rouvignac
                Reporter:
                lee.trujillo Lee Trujillo
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: