jmap 的一次使用经历

一个负责的项目（tomcat容器内），重构部分代码后，发现cpu使用居高不下，而且http接口无法访问，

但是提供的dubbo接口能正常访问，感觉很奇怪。

首先想到先dump出heap分析下

jmap -dump:file=auto.dump 11598

提示：-bash: jmap: command not found

找到jdk的环境变量目录：

echo $JAVA_HOME

/usr/java/default/

./jmap -dump:file=/data/logs/apps/8342-autotender/auto.dump 11598

最后用visualVM分析了heap文件：

"ConsumeMessageThread_3" daemon prio=5 tid=82 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#212
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#22
at java.lang.Thread.run(Thread.java:745)

"localhost-startStop-1-EventThread" daemon prio=5 tid=23 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#95
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#206
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
Local Variable: java.util.concurrent.locks.ReentrantLock#293
Local Variable: java.util.concurrent.atomic.AtomicInteger#142
Local Variable: java.util.concurrent.LinkedBlockingQueue#11
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)

"Finalizer" daemon prio=8 tid=3 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
Local Variable: java.lang.ref.ReferenceQueue#210
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
Local Variable: java.lang.System$2#1

"Reference Handler" daemon prio=10 tid=2 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)

"DubboServerHandler-172.20.100.194:18342-thread-1" daemon prio=5 tid=65 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
Local Variable: java.util.concurrent.SynchronousQueue$TransferStack#12
Local Variable: java.util.concurrent.SynchronousQueue$TransferStack$SNode#94
at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925)
Local Variable: java.util.concurrent.SynchronousQueue#12
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
Local Variable: java.util.concurrent.ThreadPoolExecutor#25
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#35
at java.lang.Thread.run(Thread.java:745)

"PullMessageServiceScheduledThread" daemon prio=5 tid=55 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#221
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#126
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
Local Variable: java.util.concurrent.locks.ReentrantLock#369
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#6
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor#6
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#30
at java.lang.Thread.run(Thread.java:745)

"Signal Dispatcher" daemon prio=9 tid=4 RUNNABLE

"GC Daemon" daemon prio=2 tid=12 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at sun.misc.GC$Daemon.run(GC.java:117)

"DubboServerHandler-172.20.100.194:18342-thread-2" daemon prio=5 tid=66 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
Local Variable: java.util.concurrent.SynchronousQueue$TransferStack$SNode#93
at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#36
at java.lang.Thread.run(Thread.java:745)

"ConsumeMessageScheduledThread_1" daemon prio=5 tid=48 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#107
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#203
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
Local Variable: java.util.concurrent.locks.ReentrantLock#346
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#36
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#5
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor#5
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#25
at java.lang.Thread.run(Thread.java:745)

"New I/O client worker #1-3" daemon prio=5 tid=34 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#2
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
Local Variable: sun.nio.ch.Util$2#2
Local Variable: java.util.Collections$UnmodifiableSet#18
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:38)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:165)
Local Variable: sun.nio.ch.EPollSelectorImpl#3
Local Variable: org.jboss.netty.channel.socket.nio.NioWorker#3
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
Local Variable: java.lang.String#76960
Local Variable: org.jboss.netty.util.ThreadRenamingRunnable#3
Local Variable: java.lang.String#76959
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
Local Variable: org.jboss.netty.util.internal.DeadLockProofWorker$1#3
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#10
at java.lang.Thread.run(Thread.java:745)

"DubboClientReconnectTimer-thread-1" daemon prio=5 tid=25 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#89
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#187
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
Local Variable: java.util.concurrent.locks.ReentrantLock#272
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#8
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#2
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor#2
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#18
at java.lang.Thread.run(Thread.java:745)

"Druid-ConnectionPool-Destroy-742457638" daemon prio=5 tid=17 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at com.alibaba.druid.pool.DruidDataSource$DestroyConnectionThread.run(DruidDataSource.java:1898)

"DubboClientReconnectTimer-thread-2" daemon prio=5 tid=30 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#186
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#17
at java.lang.Thread.run(Thread.java:745)

"New I/O client worker #1-4" daemon prio=5 tid=36 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#5
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
Local Variable: sun.nio.ch.Util$2#5
Local Variable: java.util.Collections$UnmodifiableSet#36
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:38)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:165)
Local Variable: org.jboss.netty.channel.socket.nio.NioWorker#2
Local Variable: sun.nio.ch.EPollSelectorImpl#1
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
Local Variable: java.lang.String#76957
Local Variable: java.lang.String#76958
Local Variable: org.jboss.netty.util.ThreadRenamingRunnable#2
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
Local Variable: org.jboss.netty.util.internal.DeadLockProofWorker$1#2
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#8
at java.lang.Thread.run(Thread.java:745)

"MySQL Statement Cancellation Timer" daemon prio=5 tid=75 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
Local Variable: java.lang.Object#45
at java.util.TimerThread.run(Timer.java:505)

"MQClientFactoryScheduledThread" daemon prio=5 tid=47 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#128
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#185
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#50
Local Variable: java.util.concurrent.locks.ReentrantLock#371
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#4
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor#4
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#31
at java.lang.Thread.run(Thread.java:745)

"New I/O client worker #1-1" daemon prio=5 tid=27 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#4
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
Local Variable: java.util.Collections$UnmodifiableSet#35
Local Variable: sun.nio.ch.Util$2#4
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:38)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:165)
Local Variable: sun.nio.ch.EPollSelectorImpl#5
Local Variable: org.jboss.netty.channel.socket.nio.NioWorker#5
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
Local Variable: java.lang.String#76964
Local Variable: java.lang.String#76963
Local Variable: org.jboss.netty.util.ThreadRenamingRunnable#5
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
Local Variable: java.util.concurrent.ThreadPoolExecutor#17
Local Variable: org.jboss.netty.util.internal.DeadLockProofWorker$1#5
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#14
at java.lang.Thread.run(Thread.java:745)

"localhost-startStop-1" daemon prio=5 tid=14 RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
Local Variable: java.io.FileDescriptor#15
at java.net.SocketInputStream.read(SocketInputStream.java:122)
Local Variable: byte[]#10178
Local Variable: java.net.SocketInputStream#1
at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:114)
at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:161)
at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:189)
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2549)
Local Variable: com.mysql.jdbc.util.ReadAheadInputStream#1
Local Variable: byte[]#10176
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3002)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2991)
Local Variable: com.mysql.jdbc.Buffer#388
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3532)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2002)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2163)
Local Variable: com.mysql.jdbc.MysqlIO#1
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2624)
Local Variable: java.lang.String#57265
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2127)
Local Variable: com.mysql.jdbc.StatementImpl$CancelTask#44
at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1362)
Local Variable: com.mysql.jdbc.JDBC4Connection#1
Local Variable: com.mysql.jdbc.JDBC4PreparedStatement#15
Local Variable: com.mysql.jdbc.Buffer#396
at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:2931)
at com.alibaba.druid.filter.FilterEventAdapter.preparedStatement_execute(FilterEventAdapter.java:440)
Local Variable: com.alibaba.druid.filter.stat.StatFilter#1
at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:2929)
Local Variable: com.alibaba.druid.filter.FilterChainImpl#348
at com.alibaba.druid.proxy.jdbc.PreparedStatementProxyImpl.execute(PreparedStatementProxyImpl.java:118)
Local Variable: com.alibaba.druid.proxy.jdbc.PreparedStatementProxyImpl#8
at com.alibaba.druid.pool.DruidPooledPreparedStatement.execute(DruidPooledPreparedStatement.java:493)
at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:57)
Local Variable: org.apache.ibatis.executor.statement.PreparedStatementHandler#44
at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:70)
at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:57)
Local Variable: org.apache.ibatis.session.Configuration#1
Local Variable: com.alibaba.druid.pool.DruidPooledPreparedStatement#44
Local Variable: org.apache.ibatis.executor.statement.RoutingStatementHandler#44
at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:267)
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:141)
Local Variable: org.apache.ibatis.executor.SimpleExecutor#44
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:98)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:73)
Local Variable: org.apache.ibatis.mapping.BoundSql#88
Local Variable: org.apache.ibatis.cache.CacheKey#44
at sun.reflect.GeneratedMethodAccessor128.invoke(<unknown string>)
Local Variable: sun.reflect.GeneratedMethodAccessor128#1
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
Local Variable: org.apache.ibatis.executor.CachingExecutor#44
Local Variable: sun.reflect.DelegatingMethodAccessorImpl#399
at org.apache.ibatis.plugin.Invocation.proceed(Invocation.java:46)
at com.weidai.cat.plugin.database.CatMybatisIntercepter.intercept(CatMybatisIntercepter.java:50)
Local Variable: java.lang.String#54144
Local Variable: org.apache.ibatis.plugin.Invocation#44
Local Variable: com.dianping.cat.message.internal.DefaultTransaction#88
Local Variable: java.lang.String#22492
Local Variable: com.weidai.cat.plugin.database.CatMybatisIntercepter#1
at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:57)
Local Variable: java.util.HashSet#519
Local Variable: java.lang.reflect.Method#17298
Local Variable: org.apache.ibatis.plugin.Plugin#44
Local Variable: java.lang.Object[]#7008
at com.sun.proxy.$Proxy40.query(<unknown string>)
Local Variable: com.sun.proxy.$Proxy40#44
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:101)
Local Variable: org.apache.ibatis.mapping.MappedStatement#5
Local Variable: org.apache.ibatis.session.RowBounds#1
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:95)
at sun.reflect.GeneratedMethodAccessor135.invoke(<unknown string>)
Local Variable: sun.reflect.GeneratedMethodAccessor135#1
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
Local Variable: sun.reflect.DelegatingMethodAccessorImpl#272
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:355)
Local Variable: java.lang.Object[]#7005
Local Variable: java.lang.reflect.Method#15046
Local Variable: org.apache.ibatis.session.defaults.DefaultSqlSession#44
Local Variable: org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor#13
at com.sun.proxy.$Proxy13.selectList(<unknown string>)
Local Variable: com.sun.proxy.$Proxy13#13
at org.mybatis.spring.SqlSessionTemplate.selectList(SqlSessionTemplate.java:195)
Local Variable: org.mybatis.spring.SqlSessionTemplate#13
Local Variable: java.lang.String#22292
at org.apache.ibatis.binding.MapperMethod.executeForMany(MapperMethod.java:120)
Local Variable: org.apache.ibatis.binding.MapperMethod$MapperParamMap#44
at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:90)
at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:40)
Local Variable: class com.weidai.autotender.dao.mapper.set.LazyAutoTenderSetDoMapper
Local Variable: org.apache.ibatis.binding.MapperProxy#6
Local Variable: org.apache.ibatis.binding.MapperMethod#44
Local Variable: java.lang.Object[]#6954
Local Variable: java.lang.reflect.Method#9787
at com.sun.proxy.$Proxy23.queryLazySets(<unknown string>)
Local Variable: com.sun.proxy.$Proxy23#1
at com.weidai.autotender.biz.impl.tender.LazyAutoTender.autoTender(LazyAutoTender.java:92)
Local Variable: java.util.ArrayList$Itr#1628
Local Variable: com.weidai.goodscore.facade.result.goods.GoodsListDTO#1
Local Variable: java.util.ArrayList#11483
Local Variable: java.lang.Long#847
Local Variable: com.weidai.autotender.biz.impl.tender.LazyAutoTender#1
Local Variable: java.lang.String#69961
Local Variable: java.lang.Boolean#45
at com.weidai.autotender.init.AutoTenderInitialize.onApplicationEvent(AutoTenderInitialize.java:48)
at com.weidai.autotender.init.AutoTenderInitialize.onApplicationEvent(AutoTenderInitialize.java:22)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:96)
Local Variable: java.util.LinkedList$ListItr#1
Local Variable: com.weidai.autotender.init.AutoTenderInitialize#1
Local Variable: org.springframework.context.event.SimpleApplicationEventMulticaster#1
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:334)
Local Variable: org.springframework.context.event.ContextRefreshedEvent#1
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:948)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)
Local Variable: java.lang.Object#4032
Local Variable: org.springframework.beans.factory.support.DefaultListableBeanFactory#1
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:410)
Local Variable: java.lang.String#33559
Local Variable: org.springframework.web.context.support.StandardServletEnvironment#1
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
Local Variable: org.apache.catalina.core.ApplicationContextFacade#1
Local Variable: org.springframework.web.context.support.XmlWebApplicationContext#1
Local Variable: org.apache.commons.logging.impl.Log4JLogger#25
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5068)
Local Variable: org.springframework.web.context.ContextLoaderListener#1
Local Variable: java.lang.Object[]#9070
Local Variable: org.apache.catalina.deploy.ApplicationListener[]#1
Local Variable: javax.servlet.ServletContextEvent#1
Local Variable: java.lang.Object[]#11868
Local Variable: java.util.ArrayList#7570
Local Variable: java.util.ArrayList#7571
Local Variable: javax.servlet.ServletContextEvent#2
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5584)
Local Variable: java.lang.String#37985
Local Variable: java.util.LinkedHashMap$EntryIterator#1
Local Variable: java.net.URLClassLoader#1
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
Local Variable: java.util.HashMap#2276
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1092)
Local Variable: java.io.File#105
Local Variable: org.apache.catalina.startup.HostConfig$DeployedApplication#1
Local Variable: class org.apache.catalina.startup.ContextConfig
Local Variable: org.apache.catalina.util.ContextName#1
Local Variable: java.io.File#103
Local Variable: org.apache.catalina.startup.ContextConfig#1
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1984)
Local Variable: org.apache.catalina.startup.HostConfig$DeployWar#1
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
Local Variable: java.util.concurrent.Executors$RunnableAdapter#2
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#1
at java.lang.Thread.run(Thread.java:745)

"ConsumeMessageThread_2" daemon prio=5 tid=81 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#215
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#23
at java.lang.Thread.run(Thread.java:745)

"NettyClientSelector_1" daemon prio=5 tid=44 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#7
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
Local Variable: java.util.Collections$UnmodifiableSet#175
Local Variable: com.alibaba.rocketmq.shade.io.netty.channel.nio.SelectedSelectionKeySet#1
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at com.alibaba.rocketmq.shade.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:639)
Local Variable: sun.nio.ch.EPollSelectorImpl#7
at com.alibaba.rocketmq.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:325)
Local Variable: com.alibaba.rocketmq.shade.io.netty.channel.nio.NioEventLoop#1
at com.alibaba.rocketmq.shade.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
Local Variable: com.alibaba.rocketmq.shade.io.netty.util.concurrent.SingleThreadEventExecutor$2#1
at java.lang.Thread.run(Thread.java:745)

"pool-2-thread-1" prio=5 tid=58 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#200
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#133
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
Local Variable: java.util.concurrent.locks.ReentrantLock#379
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#1
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#8
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor#8
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#33
at java.lang.Thread.run(Thread.java:745)

"New I/O client worker #1-5" daemon prio=5 tid=38 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#1
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
Local Variable: sun.nio.ch.Util$2#1
Local Variable: java.util.Collections$UnmodifiableSet#16
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:38)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:165)
Local Variable: org.jboss.netty.channel.socket.nio.NioWorker#1
Local Variable: sun.nio.ch.EPollSelectorImpl#2
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
Local Variable: java.lang.String#76956
Local Variable: java.lang.String#76955
Local Variable: org.jboss.netty.util.ThreadRenamingRunnable#1
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
Local Variable: org.jboss.netty.util.internal.DeadLockProofWorker$1#1
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#9
at java.lang.Thread.run(Thread.java:745)

"New I/O server boss #1 ([id: 0x0e6694f5, /0:0:0:0:0:0:0:0:18342])" daemon prio=5 tid=61 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#8
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
Local Variable: java.util.Collections$UnmodifiableSet#176
Local Variable: sun.nio.ch.Util$2#7
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
Local Variable: sun.nio.ch.EPollSelectorImpl#8
at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:240)
Local Variable: org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss#1
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
Local Variable: java.lang.String#97898
Local Variable: org.jboss.netty.util.ThreadRenamingRunnable#8
Local Variable: java.lang.String#97899
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
Local Variable: org.jboss.netty.util.internal.DeadLockProofWorker$1#8
Local Variable: java.util.concurrent.ThreadPoolExecutor#26
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#37
at java.lang.Thread.run(Thread.java:745)

"threadDeathWatcher-2-1" daemon prio=1 tid=76 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
Local Variable: io.netty.util.ThreadDeathWatcher$Watcher#1
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
Local Variable: io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator#1
at java.lang.Thread.run(Thread.java:745)

"NettyClientPublicExecutor_3" daemon prio=5 tid=78 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#181
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#29
at java.lang.Thread.run(Thread.java:745)

"JDWP Command Reader" daemon prio=10 tid=88 RUNNABLE

"cat-TcpSocketSender" daemon prio=5 tid=71 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#173
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#158
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
Local Variable: java.util.concurrent.TimeUnit$3#1
Local Variable: java.util.concurrent.locks.ReentrantLock#416
Local Variable: java.util.concurrent.LinkedBlockingQueue#28
Local Variable: java.util.concurrent.atomic.AtomicInteger#240
at com.dianping.cat.message.io.DefaultMessageQueue.poll(DefaultMessageQueue.java:49)
Local Variable: com.dianping.cat.message.io.DefaultMessageQueue#1
at com.dianping.cat.message.io.TcpSocketSender.run(TcpSocketSender.java:176)
Local Variable: com.dianping.cat.message.io.TcpSocketSender#1
Local Variable: io.netty.channel.DefaultChannelPromise#89
at java.lang.Thread.run(Thread.java:745)
at org.unidal.helper.Threads$RunnableThread.run(Threads.java:294)

"RebalanceService" daemon prio=5 tid=46 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at com.alibaba.rocketmq.common.ServiceThread.waitForRunning(ServiceThread.java:128)
at com.alibaba.rocketmq.client.impl.consumer.RebalanceService.run(RebalanceService.java:46)
at java.lang.Thread.run(Thread.java:745)

"localhost-startStop-1-SendThread(zk.wd.com:2181)" daemon prio=5 tid=22 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#6
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
Local Variable: java.util.Collections$UnmodifiableSet#170
Local Variable: sun.nio.ch.Util$2#6
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
Local Variable: sun.nio.ch.EPollSelectorImpl#6
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)

"ClientHouseKeepingService" daemon prio=5 tid=43 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
Local Variable: com.alibaba.rocketmq.remoting.netty.NettyRemotingClient$5#1
Local Variable: java.lang.Object#6383
at java.util.TimerThread.run(Timer.java:505)

"MultiThreadedHttpConnectionManager cleanup" daemon prio=5 tid=60 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
Local Variable: java.lang.ref.ReferenceQueue#275
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1122)

"JDWP Transport Listener: dt_socket" daemon prio=10 tid=5 RUNNABLE

"NettyClientPublicExecutor_2" daemon prio=5 tid=77 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#180
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#28
at java.lang.Thread.run(Thread.java:745)

"NettyClientPublicExecutor_4" daemon prio=5 tid=79 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#179
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#27
at java.lang.Thread.run(Thread.java:745)

"Catalina-startStop-1" daemon prio=5 tid=13 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:425)
Local Variable: java.util.concurrent.FutureTask$WaitNode#1
at java.util.concurrent.FutureTask.get(FutureTask.java:187)
at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:832)
Local Variable: java.util.concurrent.ThreadPoolExecutor#2
Local Variable: java.util.ArrayList$Itr#1625
Local Variable: java.util.ArrayList#3286
Local Variable: java.util.concurrent.FutureTask#1
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:495)
Local Variable: java.io.File#92
Local Variable: java.io.File#93
Local Variable: java.lang.String[]#356
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1713)
Local Variable: javax.management.ObjectName#38
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:337)
Local Variable: org.apache.catalina.startup.HostConfig#1
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
Local Variable: org.apache.catalina.util.LifecycleSupport#11
Local Variable: org.apache.catalina.LifecycleListener[]#11
Local Variable: org.apache.catalina.LifecycleEvent#1
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:394)
Local Variable: java.lang.String#37651
at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:339)
Local Variable: org.apache.catalina.LifecycleState#10
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1137)
Local Variable: java.util.ArrayList$Itr#1626
Local Variable: java.util.ArrayList#3287
Local Variable: org.apache.catalina.Container[]#1
at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:819)
Local Variable: java.lang.String#39940
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1572)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1562)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
Local Variable: org.apache.catalina.core.ContainerBase$StartChild#1
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
Local Variable: java.util.concurrent.ThreadPoolExecutor#3
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#2
at java.lang.Thread.run(Thread.java:745)

"commons-pool-EvictionTimer" daemon prio=5 tid=19 TIMED_WAITING
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
Local Variable: org.apache.commons.pool2.impl.BaseGenericObjectPool$Evictor#1
Local Variable: java.lang.Object#4654
at java.util.TimerThread.run(Timer.java:505)

"DubboResponseTimeoutScanTimer" daemon prio=5 tid=84 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture$RemotingInvocationTimeoutScan.run(DefaultFuture.java:300)
Local Variable: java.util.concurrent.ConcurrentHashMap$ValueIterator#7
Local Variable: com.alibaba.dubbo.remoting.exchange.support.DefaultFuture$RemotingInvocationTimeoutScan#1
at java.lang.Thread.run(Thread.java:745)

"cat-TcpSocketSender-ChannelManager" daemon prio=5 tid=72 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at com.dianping.cat.message.io.ChannelManager.run(ChannelManager.java:356)
Local Variable: com.dianping.cat.message.io.ChannelManager#1
Local Variable: java.util.ArrayList#11043
at java.lang.Thread.run(Thread.java:745)
at org.unidal.helper.Threads$RunnableThread.run(Threads.java:294)

"NettyClientPublicExecutor_1" daemon prio=5 tid=54 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#109
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#182
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
Local Variable: java.util.concurrent.LinkedBlockingQueue#14
Local Variable: java.util.concurrent.atomic.AtomicInteger#174
Local Variable: java.util.concurrent.locks.ReentrantLock#348
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
Local Variable: java.util.concurrent.ThreadPoolExecutor#23
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#26
at java.lang.Thread.run(Thread.java:745)

"JDWP Event Helper Thread" daemon prio=10 tid=6 RUNNABLE

"cat-merge-atomic-task" daemon prio=5 tid=73 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at com.dianping.cat.message.io.TcpSocketSender$MergeAtomicTask.run(TcpSocketSender.java:294)
Local Variable: com.dianping.cat.message.io.TcpSocketSender$MergeAtomicTask#1
at java.lang.Thread.run(Thread.java:745)
at org.unidal.helper.Threads$RunnableThread.run(Threads.java:294)

"dubbo-remoting-client-heartbeat-thread-2" daemon prio=5 tid=33 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#359
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#21
at java.lang.Thread.run(Thread.java:745)

"cat-StatusUpdateTask" daemon prio=5 tid=74 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at com.dianping.cat.status.StatusUpdateTask.run(StatusUpdateTask.java:177)
Local Variable: com.dianping.cat.message.internal.DefaultTransaction#90
Local Variable: com.dianping.cat.message.internal.DefaultMessageProducer#1
Local Variable: com.dianping.cat.status.StatusUpdateTask#1
at java.lang.Thread.run(Thread.java:745)
at org.unidal.helper.Threads$RunnableThread.run(Threads.java:294)

"DubboSaveRegistryCache-thread-1" daemon prio=5 tid=24 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#227
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#78
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
Local Variable: java.util.concurrent.LinkedBlockingQueue#10
Local Variable: java.util.concurrent.locks.ReentrantLock#253
Local Variable: java.util.concurrent.atomic.AtomicInteger#73
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
Local Variable: java.util.concurrent.ThreadPoolExecutor#13
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#4
at java.lang.Thread.run(Thread.java:745)

"Thread-9" daemon prio=5 tid=68 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#11
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
Local Variable: java.util.Collections$UnmodifiableSet#179
Local Variable: io.netty.channel.nio.SelectedSelectionKeySet#1
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
Local Variable: sun.nio.ch.EPollSelectorImpl#11
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
Local Variable: io.netty.channel.nio.NioEventLoop#1
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
Local Variable: io.netty.util.concurrent.SingleThreadEventExecutor$2#1
at java.lang.Thread.run(Thread.java:745)

"DubboRegistryFailedRetryTimer-thread-1" daemon prio=5 tid=20 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#93
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#188
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
Local Variable: java.util.concurrent.locks.ReentrantLock#290
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#16
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#3
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor#3
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#19
at java.lang.Thread.run(Thread.java:745)

"pool-3-thread-1" prio=5 tid=56 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#201
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#138
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#54
Local Variable: java.util.concurrent.locks.ReentrantLock#385
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#9
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor#9
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#34
at java.lang.Thread.run(Thread.java:745)

"dubbo-remoting-client-heartbeat-thread-1" daemon prio=5 tid=28 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#360
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#102
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#2
Local Variable: java.util.concurrent.locks.ReentrantLock#331
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#1
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor#1
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#20
at java.lang.Thread.run(Thread.java:745)

"NettyClientWorkerThread_2" daemon prio=5 tid=50 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#122
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#202
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
Local Variable: java.util.concurrent.locks.ReentrantLock#361
Local Variable: java.util.concurrent.atomic.AtomicInteger#186
at com.alibaba.rocketmq.shade.io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:231)
Local Variable: java.util.concurrent.LinkedBlockingQueue#20
Local Variable: com.alibaba.rocketmq.shade.io.netty.util.concurrent.ScheduledFutureTask#22
at com.alibaba.rocketmq.shade.io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
Local Variable: com.alibaba.rocketmq.shade.io.netty.util.concurrent.DefaultEventExecutor#1
at com.alibaba.rocketmq.shade.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
Local Variable: com.alibaba.rocketmq.shade.io.netty.util.concurrent.SingleThreadEventExecutor$2#2
at java.lang.Thread.run(Thread.java:745)

"Attach Listener" daemon prio=9 tid=94 RUNNABLE

"New I/O server worker #1-1" daemon prio=5 tid=63 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#10
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
Local Variable: java.util.Collections$UnmodifiableSet#178
Local Variable: sun.nio.ch.Util$2#9
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:38)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:165)
Local Variable: sun.nio.ch.EPollSelectorImpl#9
Local Variable: org.jboss.netty.channel.socket.nio.NioWorker#10
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
Local Variable: org.jboss.netty.util.ThreadRenamingRunnable#7
Local Variable: java.lang.String#97896
Local Variable: java.lang.String#97897
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
Local Variable: org.jboss.netty.util.internal.DeadLockProofWorker$1#7
Local Variable: java.util.concurrent.ThreadPoolExecutor#27
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#38
at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-1" prio=5 tid=57 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#131
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#222
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
Local Variable: java.util.concurrent.LinkedBlockingQueue#22
Local Variable: java.util.concurrent.atomic.AtomicInteger#206
Local Variable: java.util.concurrent.locks.ReentrantLock#381
at com.weidai.diamond.utils.io.watch.WatchService.take(WatchService.java:142)
Local Variable: com.weidai.diamond.utils.io.watch.WatchService#1
at com.weidai.diamond.client.processor.LocalConfigInfoProcessor$1.run(LocalConfigInfoProcessor.java:157)
Local Variable: com.weidai.diamond.client.processor.LocalConfigInfoProcessor$1#1
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
Local Variable: java.util.concurrent.Executors$RunnableAdapter#60
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#56
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor#7
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#32
at java.lang.Thread.run(Thread.java:745)

"PullMessageService" daemon prio=5 tid=45 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#124
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#184
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
Local Variable: java.util.concurrent.LinkedBlockingQueue#21
Local Variable: java.util.concurrent.locks.ReentrantLock#367
Local Variable: java.util.concurrent.atomic.AtomicInteger#192
at com.alibaba.rocketmq.client.impl.consumer.PullMessageService.run(PullMessageService.java:91)
Local Variable: com.alibaba.rocketmq.client.impl.consumer.PullMessageService#1
at java.lang.Thread.run(Thread.java:745)

"ZkClient-EventThread-21-zk.wd.com:2181" daemon prio=5 tid=21 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#209
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#100
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
Local Variable: java.util.concurrent.atomic.AtomicInteger#143
Local Variable: java.util.concurrent.locks.ReentrantLock#327
Local Variable: java.util.concurrent.LinkedBlockingQueue#12
at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:67)

"New I/O server worker #1-2" daemon prio=5 tid=64 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#9
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
Local Variable: java.util.Collections$UnmodifiableSet#177
Local Variable: sun.nio.ch.Util$2#8
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:38)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:165)
Local Variable: sun.nio.ch.EPollSelectorImpl#10
Local Variable: org.jboss.netty.channel.socket.nio.NioWorker#9
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
Local Variable: org.jboss.netty.util.ThreadRenamingRunnable#6
Local Variable: java.lang.String#97894
Local Variable: java.lang.String#97895
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
Local Variable: org.jboss.netty.util.internal.DeadLockProofWorker$1#6
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#39
at java.lang.Thread.run(Thread.java:745)

"Druid-ConnectionPool-Create-742457638" daemon prio=5 tid=16 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#72
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#229
at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:1824)

"NettyClientWorkerThread_1" daemon prio=5 tid=49 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#183
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#120
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
Local Variable: java.util.concurrent.TimeUnit$1#1
Local Variable: java.util.concurrent.atomic.AtomicInteger#183
Local Variable: java.util.concurrent.locks.ReentrantLock#359
at com.alibaba.rocketmq.shade.io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:231)
Local Variable: com.alibaba.rocketmq.shade.io.netty.util.concurrent.ScheduledFutureTask#23
Local Variable: java.util.concurrent.LinkedBlockingQueue#19
at com.alibaba.rocketmq.shade.io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
Local Variable: com.alibaba.rocketmq.shade.io.netty.util.concurrent.DefaultEventExecutor#2
at com.alibaba.rocketmq.shade.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
Local Variable: com.alibaba.rocketmq.shade.io.netty.util.concurrent.SingleThreadEventExecutor$2#5
at java.lang.Thread.run(Thread.java:745)

"ConsumeMessageThread_1" daemon prio=5 tid=80 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#104
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#214
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
Local Variable: java.util.concurrent.locks.ReentrantLock#343
Local Variable: java.util.concurrent.atomic.AtomicInteger#170
Local Variable: java.util.concurrent.LinkedBlockingQueue#13
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
Local Variable: java.util.concurrent.ThreadPoolExecutor#22
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#24
at java.lang.Thread.run(Thread.java:745)

"New I/O client worker #1-2" daemon prio=5 tid=31 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#3
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
Local Variable: java.util.Collections$UnmodifiableSet#21
Local Variable: sun.nio.ch.Util$2#3
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:38)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:165)
Local Variable: sun.nio.ch.EPollSelectorImpl#4
Local Variable: org.jboss.netty.channel.socket.nio.NioWorker#4
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
Local Variable: java.lang.String#76961
Local Variable: java.lang.String#76962
Local Variable: org.jboss.netty.util.ThreadRenamingRunnable#4
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
Local Variable: org.jboss.netty.util.internal.DeadLockProofWorker$1#4
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#12
at java.lang.Thread.run(Thread.java:745)

"main" prio=5 tid=1 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:425)
Local Variable: java.util.concurrent.FutureTask$WaitNode#2
at java.util.concurrent.FutureTask.get(FutureTask.java:187)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1120)
Local Variable: java.util.concurrent.FutureTask#2
Local Variable: org.apache.catalina.realm.LockOutRealm#1
Local Variable: org.apache.catalina.Container[]#2
Local Variable: java.util.ArrayList$Itr#1627
Local Variable: java.util.ArrayList#3288
at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:300)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
at org.apache.catalina.core.StandardService.startInternal(StandardService.java:444)
Local Variable: org.apache.catalina.core.StandardEngine#1
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
Local Variable: org.apache.catalina.core.StandardService#1
at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:756)
Local Variable: java.lang.Object#2578
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
at org.apache.catalina.startup.Catalina.start(Catalina.java:693)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
Local Variable: sun.reflect.NativeMethodAccessorImpl#35
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
Local Variable: org.apache.catalina.startup.Catalina#1
Local Variable: sun.reflect.DelegatingMethodAccessorImpl#38
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:294)
Local Variable: org.apache.catalina.startup.Bootstrap#1
Local Variable: java.lang.reflect.Method#2029
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:428)
Local Variable: java.lang.String#33501
Local Variable: java.lang.String[]#357

"DubboClientHandler-172.20.100.115:18119-thread-2" daemon prio=5 tid=90 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
Local Variable: java.util.concurrent.SynchronousQueue$TransferStack$SNode#77
Local Variable: java.util.concurrent.SynchronousQueue$TransferStack#10
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
Local Variable: java.util.concurrent.SynchronousQueue#10
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
Local Variable: java.util.concurrent.ThreadPoolExecutor#20
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#41
at java.lang.Thread.run(Thread.java:745)

"dubbo-remoting-server-heartbeat-thread-1" daemon prio=5 tid=62 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#363
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#150
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#55
Local Variable: java.util.concurrent.locks.ReentrantLock#398
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#11
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor#11
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#40
at java.lang.Thread.run(Thread.java:745)

这些日志中，我看不出到底哪出了问题（水平有限）。没办法，想到一个比较笨的方法，把日志级别放到debug级别

通过系统的日志，看看到底在干嘛。修改后，满屏的都是一个sql的执行，居然是个死循环，一直在执行一个sql。

检查代码，发现问题，修改后，问题解决。