[OPENAM-8202] If the "Login Id" in the External Store Configuration(CTS) is set to incorrect value,CoreSystem debug log is full of duplicate error Created: 26/Jan/16  Updated: 26/Apr/17  Resolved: 16/Nov/16

Status: Closed
Project: OpenAM
Component/s: CTS
Affects Version/s: 13.0.0, 13.5.0
Fix Version/s: 13.5.1, 14.0.0

Type: Bug Priority: Major
Reporter: Richard Hruza Assignee: Quentin CASTEL [X] (Inactive)
Resolution: Fixed Votes: 2
Labels: EDISON, Must-Fix
Remaining Estimate: 4h
Time Spent: 4h
Original Estimate: 8h
Environment:

OpenAM 13.0.0 Build 5d4589530d (2016-January-14 21:15)
CTS: OpenDJ 2.6.3


Issue Links:
Regression
caused OPENAM-10043 Failing unit test ThreadMonitorTest Resolved
Target Version/s:
Sprint: AM Sustaining Sprint 29, AM Sustaining Sprint 30, AM Sustaining Sprint 31
Story Points: 3
Support Ticket IDs:
QA Assignee: Nemanja Lukic
Verified Version/s:

 Description   

CoreSystem debug log is growing extreme fast with duplicate error until disk will be full, when the "Login Id" in the External Store Configuration(CTS) is set to incorrect value.

Steps to reproduce:

1.) Install OpenAM with OpenDJ Directory Service for CTS
https://qa-backstage.forgerock.com/#!/docs/openam/13/install-guide#cts-opendj-preparation
2.) At the step "6.2.5. OpenAM CTS Configuration" set to incorrect value for "Login Id" (in my case correct value was "cn=Directory Manager" and I set it to "cn=Directory ManagerXXX" )
3.) Restart OpenAM
4.) Observe /openam/openam/debug/CoreSystem log

Observed result:

The log file is full of the same error and size of file is growing very fast. In my case log file had 11GB(my limit of free space on HDD) after few minutes.

Expected Result

Error appears only once or the process will finished after few times of restarting Thread WatchDog

CoreSystem debug log:

amThreadManager:01/26/2016 02:14:47:633 PM GMT: Thread[amThreadManager-1,5,ServerService ThreadGroup]: TransactionId[300c2122-7500-4b31-abdc-39a19681ff30-2]
ERROR: ThreadMonitor: Thread WatchDog detected error, restarting
java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Cannot start task executor
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.forgerock.openam.shared.concurrency.ThreadMonitor$WatchDog.run(ThreadMonitor.java:231)
	at org.forgerock.openam.audit.context.AuditRequestContextPropagatingRunnable.run(AuditRequestContextPropagatingRunnable.java:42)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Cannot start task executor
	at org.forgerock.openam.sm.datalayer.impl.SeriesTaskExecutorThread.run(SeriesTaskExecutorThread.java:85)
	... 6 more
Caused by: org.forgerock.openam.sm.datalayer.api.LdapOperationFailedException: 
CTS: Operation failed:
Result Code: Connect Error
Diagnostic Message: No operational connection factories available
Matched DN: 
	at org.forgerock.openam.sm.datalayer.providers.LdapConnectionFactoryProvider$LdapConnectionFactory.create(LdapConnectionFactoryProvider.java:158)
	at org.forgerock.openam.sm.datalayer.providers.LdapConnectionFactoryProvider$LdapConnectionFactory.create(LdapConnectionFactoryProvider.java:126)
	at org.forgerock.openam.cts.monitoring.impl.connections.MonitoredCTSConnectionFactory.create(MonitoredCTSConnectionFactory.java:71)
	at org.forgerock.openam.sm.datalayer.impl.SimpleTaskExecutor.start(SimpleTaskExecutor.java:59)
	at org.forgerock.openam.sm.datalayer.impl.SeriesTaskExecutorThread.run(SeriesTaskExecutorThread.java:83)
	... 6 more
Caused by: org.forgerock.opendj.ldap.AuthenticationException: Invalid Credentials
	at org.forgerock.opendj.ldap.LdapException.newLdapException(LdapException.java:150)
	at org.forgerock.opendj.ldap.spi.ResultLdapPromiseImpl.setResultOrError(ResultLdapPromiseImpl.java:142)
	at org.forgerock.opendj.grizzly.LDAPClientFilter$ClientResponseHandler.bindResult(LDAPClientFilter.java:192)
	at org.forgerock.opendj.io.LDAPReader.readBindResult(LDAPReader.java:217)
	at org.forgerock.opendj.io.LDAPReader.readProtocolOp(LDAPReader.java:550)
	at org.forgerock.opendj.io.LDAPReader.readMessage(LDAPReader.java:132)
	at org.forgerock.opendj.grizzly.LDAPBaseFilter.handleRead(LDAPBaseFilter.java:82)
	at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:283)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:132)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:111)
	at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:536)
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571)
	... 1 more

amThreadManager:01/26/2016 02:14:47:633 PM GMT: Thread[amThreadManager-8,5,ServerService ThreadGroup]: TransactionId[300c2122-7500-4b31-abdc-39a19681ff30-2]
ERROR: ThreadMonitor: Thread WatchDog detected error, restarting
java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Cannot start task executor
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.forgerock.openam.shared.concurrency.ThreadMonitor$WatchDog.run(ThreadMonitor.java:231)
	at org.forgerock.openam.audit.context.AuditRequestContextPropagatingRunnable.run(AuditRequestContextPropagatingRunnable.java:42)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Cannot start task executor
	at org.forgerock.openam.sm.datalayer.impl.SeriesTaskExecutorThread.run(SeriesTaskExecutorThread.java:85)
	... 6 more
Caused by: org.forgerock.openam.sm.datalayer.api.LdapOperationFailedException: 
CTS: Operation failed:
Result Code: Connect Error
Diagnostic Message: No operational connection factories available
Matched DN: 
	at org.forgerock.openam.sm.datalayer.providers.LdapConnectionFactoryProvider$LdapConnectionFactory.create(LdapConnectionFactoryProvider.java:158)
	at org.forgerock.openam.sm.datalayer.providers.LdapConnectionFactoryProvider$LdapConnectionFactory.create(LdapConnectionFactoryProvider.java:126)
	at org.forgerock.openam.cts.monitoring.impl.connections.MonitoredCTSConnectionFactory.create(MonitoredCTSConnectionFactory.java:71)
	at org.forgerock.openam.sm.datalayer.impl.SimpleTaskExecutor.start(SimpleTaskExecutor.java:59)
	at org.forgerock.openam.sm.datalayer.impl.SeriesTaskExecutorThread.run(SeriesTaskExecutorThread.java:83)
	... 6 more
Caused by: org.forgerock.opendj.ldap.AuthenticationException: Invalid Credentials
	at org.forgerock.opendj.ldap.LdapException.newLdapException(LdapException.java:150)
	at org.forgerock.opendj.ldap.spi.ResultLdapPromiseImpl.setResultOrError(ResultLdapPromiseImpl.java:142)
	at org.forgerock.opendj.grizzly.LDAPClientFilter$ClientResponseHandler.bindResult(LDAPClientFilter.java:192)
	at org.forgerock.opendj.io.LDAPReader.readBindResult(LDAPReader.java:217)
	at org.forgerock.opendj.io.LDAPReader.readProtocolOp(LDAPReader.java:550)
	at org.forgerock.opendj.io.LDAPReader.readMessage(LDAPReader.java:132)
	at org.forgerock.opendj.grizzly.LDAPBaseFilter.handleRead(LDAPBaseFilter.java:82)
	at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:283)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:132)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:111)
	at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:536)
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571)
	... 1 more

amThreadManager:01/26/2016 02:14:47:633 PM GMT: Thread[amThreadManager-12,5,ServerService ThreadGroup]: TransactionId[300c2122-7500-4b31-abdc-39a19681ff30-2]
ERROR: ThreadMonitor: Thread WatchDog detected error, restarting
java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Cannot start task executor
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.forgerock.openam.shared.concurrency.ThreadMonitor$WatchDog.run(ThreadMonitor.java:231)
	at org.forgerock.openam.audit.context.AuditRequestContextPropagatingRunnable.run(AuditRequestContextPropagatingRunnable.java:42)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Cannot start task executor
	at org.forgerock.openam.sm.datalayer.impl.SeriesTaskExecutorThread.run(SeriesTaskExecutorThread.java:85)
	... 6 more
Caused by: org.forgerock.openam.sm.datalayer.api.LdapOperationFailedException: 
CTS: Operation failed:
Result Code: Connect Error
Diagnostic Message: No operational connection factories available
Matched DN: 
	at org.forgerock.openam.sm.datalayer.providers.LdapConnectionFactoryProvider$LdapConnectionFactory.create(LdapConnectionFactoryProvider.java:158)
	at org.forgerock.openam.sm.datalayer.providers.LdapConnectionFactoryProvider$LdapConnectionFactory.create(LdapConnectionFactoryProvider.java:126)
	at org.forgerock.openam.cts.monitoring.impl.connections.MonitoredCTSConnectionFactory.create(MonitoredCTSConnectionFactory.java:71)
	at org.forgerock.openam.sm.datalayer.impl.SimpleTaskExecutor.start(SimpleTaskExecutor.java:59)
	at org.forgerock.openam.sm.datalayer.impl.SeriesTaskExecutorThread.run(SeriesTaskExecutorThread.java:83)
	... 6 more
Caused by: org.forgerock.opendj.ldap.AuthenticationException: Invalid Credentials
	at org.forgerock.opendj.ldap.LdapException.newLdapException(LdapException.java:150)
	at org.forgerock.opendj.ldap.spi.ResultLdapPromiseImpl.setResultOrError(ResultLdapPromiseImpl.java:142)
	at org.forgerock.opendj.grizzly.LDAPClientFilter$ClientResponseHandler.bindResult(LDAPClientFilter.java:192)
	at org.forgerock.opendj.io.LDAPReader.readBindResult(LDAPReader.java:217)
	at org.forgerock.opendj.io.LDAPReader.readProtocolOp(LDAPReader.java:550)
	at org.forgerock.opendj.io.LDAPReader.readMessage(LDAPReader.java:132)
	at org.forgerock.opendj.grizzly.LDAPBaseFilter.handleRead(LDAPBaseFilter.java:82)
	at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:283)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:132)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:111)
	at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:536)
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571)
	... 1 more


 Comments   
Comment by Paul psangree [X] (Inactive) [ 12/Mar/16 ]

I encountered this bug under identical conditions to those originally described. It caused a 30GB disk to become full in under a minute once tomcat was restarted, at which time OpenAM began to act erratically and ssh logins were no longer successful on the machine due to full disk. All this due to a typo in the ldap login id.

Comment by Richard Hruza [ 01/Jul/16 ]

I running into this case with 13.5 RC11. After Functional Tests (seems that FT messed my configuration) I got this exception and it consumes my HDD in few minutes (cca 20GB). It is annoying issue and should be fixed.

amThreadManager:07/01/2016 12:20:29:520 PM CEST: Thread[amThreadManager-5,5,main]: TransactionId[d89ea65f-7a7b-423f-ba5b-b5cfc204b731-0]
ERROR: ThreadMonitor: Thread WatchDog detected error, restarting
java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Cannot start task executor
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.forgerock.openam.shared.concurrency.ThreadMonitor$WatchDog.run(ThreadMonitor.java:231)
	at org.forgerock.openam.audit.context.AuditRequestContextPropagatingRunnable.run(AuditRequestContextPropagatingRunnable.java:42)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Cannot start task executor
	at org.forgerock.openam.sm.datalayer.impl.SeriesTaskExecutorThread.run(SeriesTaskExecutorThread.java:85)
	... 6 more
Caused by: org.forgerock.openam.sm.datalayer.api.LdapOperationFailedException: 
CTS: Operation failed:
Result Code: Connect Error
Diagnostic Message: No operational connection factories available
Matched DN: 
	at org.forgerock.openam.sm.datalayer.providers.LdapConnectionFactoryProvider$LdapConnectionFactory.create(LdapConnectionFactoryProvider.java:158)
	at org.forgerock.openam.sm.datalayer.providers.LdapConnectionFactoryProvider$LdapConnectionFactory.create(LdapConnectionFactoryProvider.java:126)
	at org.forgerock.openam.cts.monitoring.impl.connections.MonitoredCTSConnectionFactory.create(MonitoredCTSConnectionFactory.java:71)
	at org.forgerock.openam.sm.datalayer.impl.SimpleTaskExecutor.start(SimpleTaskExecutor.java:59)
	at org.forgerock.openam.sm.datalayer.impl.SeriesTaskExecutorThread.run(SeriesTaskExecutorThread.java:83)
	... 6 more
Caused by: org.forgerock.opendj.ldap.ConnectionException: Connect Error
	at org.forgerock.opendj.ldap.LdapException.newLdapException(LdapException.java:153)
	at org.forgerock.opendj.ldap.LdapException.newLdapException(LdapException.java:114)
	at org.forgerock.opendj.grizzly.GrizzlyLDAPConnectionFactory$CompletionHandlerAdapter.adaptConnectionException(GrizzlyLDAPConnectionFactory.java:123)
	at org.forgerock.opendj.grizzly.GrizzlyLDAPConnectionFactory$CompletionHandlerAdapter.failed(GrizzlyLDAPConnectionFactory.java:97)
	at org.glassfish.grizzly.nio.transport.TCPNIOConnectorHandler.connectAsync(TCPNIOConnectorHandler.java:193)
	at org.glassfish.grizzly.nio.transport.TCPNIOConnectorHandler.connect(TCPNIOConnectorHandler.java:86)
	at org.glassfish.grizzly.AbstractSocketConnectorHandler.connect(AbstractSocketConnectorHandler.java:85)
	at org.glassfish.grizzly.AbstractSocketConnectorHandler.connect(AbstractSocketConnectorHandler.java:56)
	at org.forgerock.opendj.grizzly.GrizzlyLDAPConnectionFactory.getConnectionAsync(GrizzlyLDAPConnectionFactory.java:208)
	at org.forgerock.opendj.ldap.LDAPConnectionFactory.getConnectionAsync(LDAPConnectionFactory.java:466)
	at org.forgerock.opendj.ldap.CachedConnectionPool.getConnectionAsync(CachedConnectionPool.java:806)
	at org.forgerock.opendj.ldap.LoadBalancer$MonitoredConnectionFactory.checkIfAvailable(LoadBalancer.java:170)
	at org.forgerock.opendj.ldap.LoadBalancer$MonitoredConnectionFactory.access$1500(LoadBalancer.java:94)
	at org.forgerock.opendj.ldap.LoadBalancer$MonitorRunnable.run(LoadBalancer.java:235)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	... 3 more
Caused by: java.nio.channels.UnresolvedAddressException
	at sun.nio.ch.Net.checkAddress(Net.java:101)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
	at org.glassfish.grizzly.nio.transport.TCPNIOConnectorHandler.connectAsync(TCPNIOConnectorHandler.java:137)
	... 16 more
Comment by Bernhard Thalmayr [ 20/Jul/16 ]

Whenever CTS is not available when OpenAM starts up disk space will be eaten up by CoreSystem log.

This could grow so fast that the JVM gets into out-of memory.

If embedded config store is used one has to be quite fast to use 'ldapmodify' to correct this situation. Furthermore if the free disk space also got below the threshold configured in OpenDJ, OpenDJ will not even update the entry in config store.

Comment by Nikolaos Giannopoulos [ 03/Aug/16 ]

This issue was quite challenging when it was hit. Unfortunately CTS account credentials were incorrect and as CTS was enabled as soon as OpenAM would start up CPU would hit 90%+ and 50GB of disk logs would get consumed quite rapidly. As the work was being done on a Vmware VM this made the VM unresponsive and the literally impossible to enable OpenAM debug logging to trouble shoot. No matter how many threads are setup in a pool this excessive logging should never happen and I cringe that such an issue could occur on a Production system. Really surprised this was not fixed in 13.5.

--Nikolaos

Comment by C-Weng C [ 14/Nov/16 ]

Question: - Would it be possible to disable logging the ExecutionException exception (dedepulication) if this is repeatedly happening (for some time) and only print it again after some time. (Yes the delay to retry would help too).

Comment by Joel Pearson [ 14/Nov/16 ]

I got severely bitten by this one over the weekend. I ended up logging 3 Terabytes over the weekend because opendj was shut down for some reason. Filled up the corporate NAS, whoops...

Comment by Filip Kubáň [X] (Inactive) [ 28/Nov/16 ]

Verified fix on: OpenAM 14.0.0-M7 Build eff0e96cfd (2016-November-25 16:23)

Comment by Nemanja Lukic [ 25/Apr/17 ]

Verified in RC2 for 13.5.1

Comment by Nemanja Lukic [ 25/Apr/17 ]

A side note, CoreSystem log keeps growing with the duplicate messages but on a much slower rate: approximately 960 kb per minute.

Generated at Thu Sep 24 14:54:30 UTC 2020 using Jira 7.13.12#713012-sha1:6e07c38070d5191bbf7353952ed38f111754533a.