[OPENDJ-2993] Backport OPENDJ-2969: changelogDb could not be read on OpenDJ instance startup Created: 04/May/16  Updated: 08/Nov/19  Resolved: 18/May/16

Status: Done
Project: OpenDJ
Component/s: backends, core server, replication
Affects Version/s: 3.0.0
Fix Version/s: Not applicable

Type: Bug Priority: Critical
Reporter: Chris Ridd Assignee: Chris Ridd
Resolution: Duplicate Votes: 0
Labels: None
Environment:

So far, Linux only.


Issue Links:
Backport
is a backport of OPENDJ-2969 changelogDb could not be read on Open... Done
Duplicate
duplicates OPENDJ-3012 Backport OPENDJ-2969: changelogDb cou... Done
is duplicated by OPENDJ-3012 Backport OPENDJ-2969: changelogDb cou... Done
Dev Assignee: Chris Ridd
Support Ticket IDs:

 Description   

OpenDJ reports the following error during and after startup of the instance.

msg=The replication server failed to start because the database /opt/app/opendj/changelogDb could not be read...

There are no space issues at the time. Open file limits were not ruled out.

Full log:

[21/Apr/2016:23:00:48 +0200] category=UTIL severity=NOTICE msgID=org.opends.messages.runtime.21 msg=Installation Directory:  /opt/app/opendj
[21/Apr/2016:23:00:48 +0200] category=UTIL severity=NOTICE msgID=org.opends.messages.runtime.23 msg=Instance Directory:      /opt/app/opendj
[21/Apr/2016:23:00:48 +0200] category=UTIL severity=NOTICE msgID=org.opends.messages.runtime.17 msg=JVM Information: 1.8.0_77-b03 by Oracle Corporation, 6
4-bit architecture, 2058354688 bytes heap size
[21/Apr/2016:23:00:48 +0200] category=UTIL severity=NOTICE msgID=org.opends.messages.runtime.18 msg=JVM Host: opendj1.example.com, running Linux 2.6.32-57
3.18.1.el6.x86_64 amd64, 4014825472 bytes physical memory size, number of processors available 2
[21/Apr/2016:23:00:48 +0200] category=UTIL severity=NOTICE msgID=org.opends.messages.runtime.19 msg=JVM Arguments: "-Xms2G", "-Xmx2G", "-Dorg.opends.serve
r.scriptName=start-ds"
[21/Apr/2016:23:00:48 +0200] category=PLUGGABLE severity=NOTICE msgID=org.opends.messages.backend.513 msg=The database backend userRoot containing 55697 e
ntries has started
[21/Apr/2016:23:00:50 +0200] category=EXTENSIONS severity=NOTICE msgID=org.opends.messages.extension.221 msg=DIGEST-MD5 SASL mechanism using a server full
y qualified domain name of: opendj1.example.com
[21/Apr/2016:23:00:50 +0200] category=SYNC severity=ERROR msgID=org.opends.messages.replication.274 msg=The following log '/opt/app/opendj/changelogDb/2.d
om/30966.server' must be released but it is not referenced."
[21/Apr/2016:23:00:50 +0200] category=SYNC severity=ERROR msgID=org.opends.messages.replication.11 msg=The replication server failed to start because the 
database /opt/app/opendj/changelogDb could not be read : Could not get or create replica DB for baseDN 'dc=example,dc=com', serverId '30966', generationId
 '102871'
[21/Apr/2016:23:00:51 +0200] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.204 msg=Replication server RS(17907) started listening fo
r new connections on address 0.0.0.0 port 8989
[21/Apr/2016:23:00:51 +0200] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.62 msg=Directory server DS(28115) has connected to replic
ation server RS(17907) for domain "cn=admin data" at opendj1.example.com/10.10.10.87:8989 with generation ID 162314
[21/Apr/2016:23:00:52 +0200] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.62 msg=Directory server DS(30966) has connected to replic
ation server RS(17907) for domain "dc=example,dc=com" at opendj1.example.com/10.10.10.87:8989 with generation ID 102871
[21/Apr/2016:23:00:52 +0200] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.62 msg=Directory server DS(19144) has connected to replic
ation server RS(17907) for domain "cn=schema" at opendj1.example.com/10.10.10.87:8989 with generation ID 8408
[21/Apr/2016:23:00:52 +0200] category=PROTOCOL severity=NOTICE msgID=org.opends.messages.protocol.276 msg=Started listening for new connections on Adminis
tration Connector 0.0.0.0 port 4444
[21/Apr/2016:23:00:52 +0200] category=PROTOCOL severity=NOTICE msgID=org.opends.messages.protocol.276 msg=Started listening for new connections on LDAP Co
nnection Handler 0.0.0.0 port 51389
[21/Apr/2016:23:00:52 +0200] category=CORE severity=NOTICE msgID=org.opends.messages.core.135 msg=The Directory Server has started successfully
[21/Apr/2016:23:00:52 +0200] category=CORE severity=NOTICE msgID=org.opends.messages.core.139 msg=The Directory Server has sent an alert notification gene
rated by class org.opends.server.core.DirectoryServer (alert type org.opends.server.DirectoryServerStarted, alert ID org.opends.messages.core-135): The Di
rectory Server has started successfully
[21/Apr/2016:23:01:54 +0200] category=SYNC severity=ERROR msgID=org.opends.messages.replication.274 msg=The following log '/opt/app/opendj/changelogDb/2.d
om/30966.server' must be released but it is not referenced."
[21/Apr/2016:23:01:54 +0200] category=SYNC severity=ERROR msgID=org.opends.messages.replication.26 msg=Error trying to use the underlying database. The Replication Server is going to shut down: ChangelogException: Could not get or create replica DB for baseDN 'dc=example,dc=com', serverId '30966', generationId '102871' (ReplicationEnvironment.java:387 FileReplicaDB.java:151 FileReplicaDB.java:129 FileChangelogDB.java:278 FileChangelogDB.java:199 FileChangelogDB.java:812 ReplicationServerDomain.java:513 ReplicationServerDomain.java:337 ServerHandler.java:1140 ServerReader.java:113)
[21/Apr/2016:23:01:54 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.63 msg=Replication server RS(17907) at opendj1.example.com/10.10.10.87:8989 has closed the connection to this directory server DS(30966). This directory server will now try to connect to another replication server in order to receive changes for the domain "dc=example,dc=com"
[21/Apr/2016:23:01:54 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.63 msg=Replication server RS(17907) at opendj1.example.com/10.10.10.87:8989 has closed the connection to this directory server DS(19144). This directory server will now try to connect to another replication server in order to receive changes for the domain "cn=schema"
[21/Apr/2016:23:01:54 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.63 msg=Replication server RS(17907) at opendj1.example.com/10.10.10.87:8989 has closed the connection to this directory server DS(28115). This directory server will now try to connect to another replication server in order to receive changes for the domain "cn=admin data"
[21/Apr/2016:23:01:54 +0200] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.62 msg=Directory server DS(19144) has connected to replication server RS(8454) for domain "cn=schema" at opendj2.example.com/10.10.10.86:8989 with generation ID 8408
[21/Apr/2016:23:01:54 +0200] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.62 msg=Directory server DS(30966) has connected to replication server RS(8454) for domain "dc=example,dc=com" at opendj2.example.com/10.10.10.86:8989 with generation ID 102871
[21/Apr/2016:23:01:54 +0200] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.62 msg=Directory server DS(28115) has connected to replication server RS(8454) for domain "cn=admin data" at opendj2.example.com/10.10.10.86:8989 with generation ID 162314
[21/Apr/2016:23:02:17 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.63 msg=Replication server RS(8454) at opendj2.example.com/10.10.10.86:8989 has closed the connection to this directory server DS(30966). This directory server will now try to connect to another replication server in order to receive changes for the domain "dc=example,dc=com"
[21/Apr/2016:23:02:17 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.63 msg=Replication server RS(8454) at opendj2.example.com/10.10.10.86:8989 has closed the connection to this directory server DS(19144). This directory server will now try to connect to another replication server in order to receive changes for the domain "cn=schema"
[21/Apr/2016:23:02:17 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.208 msg=Directory server DS(30966) was unable to connect to any replication servers for domain "dc=example,dc=com"
[21/Apr/2016:23:02:17 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.208 msg=Directory server DS(19144) was unable to connect to any replication servers for domain "cn=schema"
[21/Apr/2016:23:02:17 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.63 msg=Replication server RS(8454) at opendj2.example.com/10.10.10.86:8989 has closed the connection to this directory server DS(28115). This directory server will now try to connect to another replication server in order to receive changes for the domain "cn=admin data"
[21/Apr/2016:23:02:17 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.208 msg=Directory server DS(28115) was unable to connect to any replication servers for domain "cn=admin data"

Disabling replication, cleaning out the full changelogDb path and enabling replication is enough to clear the issue.



 Comments   
Comment by Chris Ridd [ 04/May/16 ]

Backporting the initial improvement to logging would be helpful before we get a final verified fix. This was in commit 33a4c18.

Comment by Matthew Swift [ 07/Nov/19 ]

Moved to closed state because the fixVersion has already been released.

Generated at Sat Jan 16 23:29:31 UTC 2021 using Jira 7.13.12#713012-sha1:6e07c38070d5191bbf7353952ed38f111754533a.