Uploaded image for project: 'OpenDJ'
  1. OpenDJ
  2. OPENDJ-2969

changelogDb could not be read on OpenDJ instance startup

    Details

    • Support Ticket IDs:
    • Sprint:
      OpenDJ Sprint 81

      Description

      OpenDJ reports the following error during and after startup of the instance.

      msg=The replication server failed to start because the database /opt/app/opendj/changelogDb could not be read...
      

      There are no space issues at the time. Open file limits were not ruled out.

      Full log:

      [21/Apr/2016:23:00:48 +0200] category=UTIL severity=NOTICE msgID=org.opends.messages.runtime.21 msg=Installation Directory:  /opt/app/opendj
      [21/Apr/2016:23:00:48 +0200] category=UTIL severity=NOTICE msgID=org.opends.messages.runtime.23 msg=Instance Directory:      /opt/app/opendj
      [21/Apr/2016:23:00:48 +0200] category=UTIL severity=NOTICE msgID=org.opends.messages.runtime.17 msg=JVM Information: 1.8.0_77-b03 by Oracle Corporation, 6
      4-bit architecture, 2058354688 bytes heap size
      [21/Apr/2016:23:00:48 +0200] category=UTIL severity=NOTICE msgID=org.opends.messages.runtime.18 msg=JVM Host: opendj1.example.com, running Linux 2.6.32-57
      3.18.1.el6.x86_64 amd64, 4014825472 bytes physical memory size, number of processors available 2
      [21/Apr/2016:23:00:48 +0200] category=UTIL severity=NOTICE msgID=org.opends.messages.runtime.19 msg=JVM Arguments: "-Xms2G", "-Xmx2G", "-Dorg.opends.serve
      r.scriptName=start-ds"
      [21/Apr/2016:23:00:48 +0200] category=PLUGGABLE severity=NOTICE msgID=org.opends.messages.backend.513 msg=The database backend userRoot containing 55697 e
      ntries has started
      [21/Apr/2016:23:00:50 +0200] category=EXTENSIONS severity=NOTICE msgID=org.opends.messages.extension.221 msg=DIGEST-MD5 SASL mechanism using a server full
      y qualified domain name of: opendj1.example.com
      [21/Apr/2016:23:00:50 +0200] category=SYNC severity=ERROR msgID=org.opends.messages.replication.274 msg=The following log '/opt/app/opendj/changelogDb/2.d
      om/30966.server' must be released but it is not referenced."
      [21/Apr/2016:23:00:50 +0200] category=SYNC severity=ERROR msgID=org.opends.messages.replication.11 msg=The replication server failed to start because the 
      database /opt/app/opendj/changelogDb could not be read : Could not get or create replica DB for baseDN 'dc=example,dc=com', serverId '30966', generationId
       '102871'
      [21/Apr/2016:23:00:51 +0200] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.204 msg=Replication server RS(17907) started listening fo
      r new connections on address 0.0.0.0 port 8989
      [21/Apr/2016:23:00:51 +0200] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.62 msg=Directory server DS(28115) has connected to replic
      ation server RS(17907) for domain "cn=admin data" at opendj1.example.com/10.10.10.87:8989 with generation ID 162314
      [21/Apr/2016:23:00:52 +0200] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.62 msg=Directory server DS(30966) has connected to replic
      ation server RS(17907) for domain "dc=example,dc=com" at opendj1.example.com/10.10.10.87:8989 with generation ID 102871
      [21/Apr/2016:23:00:52 +0200] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.62 msg=Directory server DS(19144) has connected to replic
      ation server RS(17907) for domain "cn=schema" at opendj1.example.com/10.10.10.87:8989 with generation ID 8408
      [21/Apr/2016:23:00:52 +0200] category=PROTOCOL severity=NOTICE msgID=org.opends.messages.protocol.276 msg=Started listening for new connections on Adminis
      tration Connector 0.0.0.0 port 4444
      [21/Apr/2016:23:00:52 +0200] category=PROTOCOL severity=NOTICE msgID=org.opends.messages.protocol.276 msg=Started listening for new connections on LDAP Co
      nnection Handler 0.0.0.0 port 51389
      [21/Apr/2016:23:00:52 +0200] category=CORE severity=NOTICE msgID=org.opends.messages.core.135 msg=The Directory Server has started successfully
      [21/Apr/2016:23:00:52 +0200] category=CORE severity=NOTICE msgID=org.opends.messages.core.139 msg=The Directory Server has sent an alert notification gene
      rated by class org.opends.server.core.DirectoryServer (alert type org.opends.server.DirectoryServerStarted, alert ID org.opends.messages.core-135): The Di
      rectory Server has started successfully
      [21/Apr/2016:23:01:54 +0200] category=SYNC severity=ERROR msgID=org.opends.messages.replication.274 msg=The following log '/opt/app/opendj/changelogDb/2.d
      om/30966.server' must be released but it is not referenced."
      [21/Apr/2016:23:01:54 +0200] category=SYNC severity=ERROR msgID=org.opends.messages.replication.26 msg=Error trying to use the underlying database. The Replication Server is going to shut down: ChangelogException: Could not get or create replica DB for baseDN 'dc=example,dc=com', serverId '30966', generationId '102871' (ReplicationEnvironment.java:387 FileReplicaDB.java:151 FileReplicaDB.java:129 FileChangelogDB.java:278 FileChangelogDB.java:199 FileChangelogDB.java:812 ReplicationServerDomain.java:513 ReplicationServerDomain.java:337 ServerHandler.java:1140 ServerReader.java:113)
      [21/Apr/2016:23:01:54 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.63 msg=Replication server RS(17907) at opendj1.example.com/10.10.10.87:8989 has closed the connection to this directory server DS(30966). This directory server will now try to connect to another replication server in order to receive changes for the domain "dc=example,dc=com"
      [21/Apr/2016:23:01:54 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.63 msg=Replication server RS(17907) at opendj1.example.com/10.10.10.87:8989 has closed the connection to this directory server DS(19144). This directory server will now try to connect to another replication server in order to receive changes for the domain "cn=schema"
      [21/Apr/2016:23:01:54 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.63 msg=Replication server RS(17907) at opendj1.example.com/10.10.10.87:8989 has closed the connection to this directory server DS(28115). This directory server will now try to connect to another replication server in order to receive changes for the domain "cn=admin data"
      [21/Apr/2016:23:01:54 +0200] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.62 msg=Directory server DS(19144) has connected to replication server RS(8454) for domain "cn=schema" at opendj2.example.com/10.10.10.86:8989 with generation ID 8408
      [21/Apr/2016:23:01:54 +0200] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.62 msg=Directory server DS(30966) has connected to replication server RS(8454) for domain "dc=example,dc=com" at opendj2.example.com/10.10.10.86:8989 with generation ID 102871
      [21/Apr/2016:23:01:54 +0200] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.62 msg=Directory server DS(28115) has connected to replication server RS(8454) for domain "cn=admin data" at opendj2.example.com/10.10.10.86:8989 with generation ID 162314
      [21/Apr/2016:23:02:17 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.63 msg=Replication server RS(8454) at opendj2.example.com/10.10.10.86:8989 has closed the connection to this directory server DS(30966). This directory server will now try to connect to another replication server in order to receive changes for the domain "dc=example,dc=com"
      [21/Apr/2016:23:02:17 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.63 msg=Replication server RS(8454) at opendj2.example.com/10.10.10.86:8989 has closed the connection to this directory server DS(19144). This directory server will now try to connect to another replication server in order to receive changes for the domain "cn=schema"
      [21/Apr/2016:23:02:17 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.208 msg=Directory server DS(30966) was unable to connect to any replication servers for domain "dc=example,dc=com"
      [21/Apr/2016:23:02:17 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.208 msg=Directory server DS(19144) was unable to connect to any replication servers for domain "cn=schema"
      [21/Apr/2016:23:02:17 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.63 msg=Replication server RS(8454) at opendj2.example.com/10.10.10.86:8989 has closed the connection to this directory server DS(28115). This directory server will now try to connect to another replication server in order to receive changes for the domain "cn=admin data"
      [21/Apr/2016:23:02:17 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.208 msg=Directory server DS(28115) was unable to connect to any replication servers for domain "cn=admin data"
      

      Disabling replication, cleaning out the full changelogDb path and enabling replication is enough to clear the issue.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                nicolas.capponi@forgerock.com Nicolas Capponi
                Reporter:
                lee.trujillo Lee Trujillo
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: