[OPENDJ-3012] Backport OPENDJ-2969: changelogDb could not be read on OpenDJ instance startup Created: 11/May/16  Updated: 08/Nov/19  Resolved: 21/Nov/16

Status: Done
Project: OpenDJ
Component/s: backends, core server, replication
Affects Version/s: 3.0.0
Fix Version/s: 3.0.1

Type: Bug Priority: Critical
Reporter: Chris Ridd Assignee: Chris Ridd
Resolution: Fixed Votes: 0
Labels: Verified, release-notes
Environment:

So far, Linux only.


Issue Links:
Backport
is a backport of OPENDJ-2969 changelogDb could not be read on Open... Done
Duplicate
duplicates OPENDJ-2993 Backport OPENDJ-2969: changelogDb cou... Done
is duplicated by OPENDJ-2993 Backport OPENDJ-2969: changelogDb cou... Done
QA Assignee: Christophe Sovant
Support Ticket IDs:

 Description   

OpenDJ reports the following error during and after startup of the instance.

msg=The replication server failed to start because the database /opt/app/opendj/changelogDb could not be read...

There are no space issues at the time. Open file limits were not ruled out.

Full log:

[21/Apr/2016:23:00:48 +0200] category=UTIL severity=NOTICE msgID=org.opends.messages.runtime.21 msg=Installation Directory:  /opt/app/opendj
[21/Apr/2016:23:00:48 +0200] category=UTIL severity=NOTICE msgID=org.opends.messages.runtime.23 msg=Instance Directory:      /opt/app/opendj
[21/Apr/2016:23:00:48 +0200] category=UTIL severity=NOTICE msgID=org.opends.messages.runtime.17 msg=JVM Information: 1.8.0_77-b03 by Oracle Corporation, 6
4-bit architecture, 2058354688 bytes heap size
[21/Apr/2016:23:00:48 +0200] category=UTIL severity=NOTICE msgID=org.opends.messages.runtime.18 msg=JVM Host: opendj1.example.com, running Linux 2.6.32-57
3.18.1.el6.x86_64 amd64, 4014825472 bytes physical memory size, number of processors available 2
[21/Apr/2016:23:00:48 +0200] category=UTIL severity=NOTICE msgID=org.opends.messages.runtime.19 msg=JVM Arguments: "-Xms2G", "-Xmx2G", "-Dorg.opends.serve
r.scriptName=start-ds"
[21/Apr/2016:23:00:48 +0200] category=PLUGGABLE severity=NOTICE msgID=org.opends.messages.backend.513 msg=The database backend userRoot containing 55697 e
ntries has started
[21/Apr/2016:23:00:50 +0200] category=EXTENSIONS severity=NOTICE msgID=org.opends.messages.extension.221 msg=DIGEST-MD5 SASL mechanism using a server full
y qualified domain name of: opendj1.example.com
[21/Apr/2016:23:00:50 +0200] category=SYNC severity=ERROR msgID=org.opends.messages.replication.274 msg=The following log '/opt/app/opendj/changelogDb/2.d
om/30966.server' must be released but it is not referenced."
[21/Apr/2016:23:00:50 +0200] category=SYNC severity=ERROR msgID=org.opends.messages.replication.11 msg=The replication server failed to start because the 
database /opt/app/opendj/changelogDb could not be read : Could not get or create replica DB for baseDN 'dc=example,dc=com', serverId '30966', generationId
 '102871'
[21/Apr/2016:23:00:51 +0200] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.204 msg=Replication server RS(17907) started listening fo
r new connections on address 0.0.0.0 port 8989
[21/Apr/2016:23:00:51 +0200] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.62 msg=Directory server DS(28115) has connected to replic
ation server RS(17907) for domain "cn=admin data" at opendj1.example.com/10.10.10.87:8989 with generation ID 162314
[21/Apr/2016:23:00:52 +0200] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.62 msg=Directory server DS(30966) has connected to replic
ation server RS(17907) for domain "dc=example,dc=com" at opendj1.example.com/10.10.10.87:8989 with generation ID 102871
[21/Apr/2016:23:00:52 +0200] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.62 msg=Directory server DS(19144) has connected to replic
ation server RS(17907) for domain "cn=schema" at opendj1.example.com/10.10.10.87:8989 with generation ID 8408
[21/Apr/2016:23:00:52 +0200] category=PROTOCOL severity=NOTICE msgID=org.opends.messages.protocol.276 msg=Started listening for new connections on Adminis
tration Connector 0.0.0.0 port 4444
[21/Apr/2016:23:00:52 +0200] category=PROTOCOL severity=NOTICE msgID=org.opends.messages.protocol.276 msg=Started listening for new connections on LDAP Co
nnection Handler 0.0.0.0 port 51389
[21/Apr/2016:23:00:52 +0200] category=CORE severity=NOTICE msgID=org.opends.messages.core.135 msg=The Directory Server has started successfully
[21/Apr/2016:23:00:52 +0200] category=CORE severity=NOTICE msgID=org.opends.messages.core.139 msg=The Directory Server has sent an alert notification gene
rated by class org.opends.server.core.DirectoryServer (alert type org.opends.server.DirectoryServerStarted, alert ID org.opends.messages.core-135): The Di
rectory Server has started successfully
[21/Apr/2016:23:01:54 +0200] category=SYNC severity=ERROR msgID=org.opends.messages.replication.274 msg=The following log '/opt/app/opendj/changelogDb/2.d
om/30966.server' must be released but it is not referenced."
[21/Apr/2016:23:01:54 +0200] category=SYNC severity=ERROR msgID=org.opends.messages.replication.26 msg=Error trying to use the underlying database. The Replication Server is going to shut down: ChangelogException: Could not get or create replica DB for baseDN 'dc=example,dc=com', serverId '30966', generationId '102871' (ReplicationEnvironment.java:387 FileReplicaDB.java:151 FileReplicaDB.java:129 FileChangelogDB.java:278 FileChangelogDB.java:199 FileChangelogDB.java:812 ReplicationServerDomain.java:513 ReplicationServerDomain.java:337 ServerHandler.java:1140 ServerReader.java:113)
[21/Apr/2016:23:01:54 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.63 msg=Replication server RS(17907) at opendj1.example.com/10.10.10.87:8989 has closed the connection to this directory server DS(30966). This directory server will now try to connect to another replication server in order to receive changes for the domain "dc=example,dc=com"
[21/Apr/2016:23:01:54 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.63 msg=Replication server RS(17907) at opendj1.example.com/10.10.10.87:8989 has closed the connection to this directory server DS(19144). This directory server will now try to connect to another replication server in order to receive changes for the domain "cn=schema"
[21/Apr/2016:23:01:54 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.63 msg=Replication server RS(17907) at opendj1.example.com/10.10.10.87:8989 has closed the connection to this directory server DS(28115). This directory server will now try to connect to another replication server in order to receive changes for the domain "cn=admin data"
[21/Apr/2016:23:01:54 +0200] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.62 msg=Directory server DS(19144) has connected to replication server RS(8454) for domain "cn=schema" at opendj2.example.com/10.10.10.86:8989 with generation ID 8408
[21/Apr/2016:23:01:54 +0200] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.62 msg=Directory server DS(30966) has connected to replication server RS(8454) for domain "dc=example,dc=com" at opendj2.example.com/10.10.10.86:8989 with generation ID 102871
[21/Apr/2016:23:01:54 +0200] category=SYNC severity=NOTICE msgID=org.opends.messages.replication.62 msg=Directory server DS(28115) has connected to replication server RS(8454) for domain "cn=admin data" at opendj2.example.com/10.10.10.86:8989 with generation ID 162314
[21/Apr/2016:23:02:17 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.63 msg=Replication server RS(8454) at opendj2.example.com/10.10.10.86:8989 has closed the connection to this directory server DS(30966). This directory server will now try to connect to another replication server in order to receive changes for the domain "dc=example,dc=com"
[21/Apr/2016:23:02:17 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.63 msg=Replication server RS(8454) at opendj2.example.com/10.10.10.86:8989 has closed the connection to this directory server DS(19144). This directory server will now try to connect to another replication server in order to receive changes for the domain "cn=schema"
[21/Apr/2016:23:02:17 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.208 msg=Directory server DS(30966) was unable to connect to any replication servers for domain "dc=example,dc=com"
[21/Apr/2016:23:02:17 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.208 msg=Directory server DS(19144) was unable to connect to any replication servers for domain "cn=schema"
[21/Apr/2016:23:02:17 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.63 msg=Replication server RS(8454) at opendj2.example.com/10.10.10.86:8989 has closed the connection to this directory server DS(28115). This directory server will now try to connect to another replication server in order to receive changes for the domain "cn=admin data"
[21/Apr/2016:23:02:17 +0200] category=SYNC severity=WARNING msgID=org.opends.messages.replication.208 msg=Directory server DS(28115) was unable to connect to any replication servers for domain "cn=admin data"

Disabling replication, cleaning out the full changelogDb path and enabling replication is enough to clear the issue.



 Comments   
Comment by Chris Ridd [ 13/May/16 ]

The idea was that OPENDJ-2993 was just to improve the logging and OPENDJ-3012 was for the fix, but it makes sense to use a single backport issue instead.

Comment by Christophe Sovant [ 23/May/16 ]

Verified with automated test using OpenDJ 3.0.1 rev 23a465e0fdf

Comment by Christophe Sovant [ 23/May/16 ]

Reopened to add Verified label

Comment by Quentin CASTEL [X] (Inactive) [ 20/Nov/16 ]

modification of the status, in order to migrate the 'Zendesk ID' field to 'Support Ticket ID' field.

Generated at Thu Jan 28 15:10:50 UTC 2021 using Jira 7.13.12#713012-sha1:6e07c38070d5191bbf7353952ed38f111754533a.