背景
现在的需求是上游hive库中若干张表产生的增量数据要同步到redis中，这边代码使用线程池去执行推送任务。可是在线上发现频繁的的在创建线程
[INFO ] 2021-12-27 03:02:13.982 [DEV-INDICATOR-PUSH-5111971] [com.dmall.rcs.utils.ThreadPoolUtils] [] - DEV-INDICATOR-PUSH-5111978 has been created
[INFO ] 2021-12-27 03:02:13.989 [DEV-INDICATOR-PUSH-5111948] [com.dmall.rcs.utils.ThreadPoolUtils] [] - DEV-INDICATOR-PUSH-5111979 has been created
[INFO ] 2021-12-27 03:02:13.990 [DEV-INDICATOR-PUSH-5111961] [com.dmall.rcs.utils.ThreadPoolUtils] [] - DEV-INDICATOR-PUSH-5111980 has been created
[INFO ] 2021-12-27 03:02:13.993 [DEV-INDICATOR-PUSH-5111964] [com.dmall.rcs.utils.ThreadPoolUtils] [] - DEV-INDICATOR-PUSH-5111981 has been created
[INFO ] 2021-12-27 03:02:14.000 [DEV-INDICATOR-PUSH-5111920] [com.dmall.rcs.utils.ThreadPoolUtils] [] - DEV-INDICATOR-PUSH-5111982 has been created
[INFO ] 2021-12-27 03:02:14.001 [DEV-INDICATOR-PUSH-5111969] [com.dmall.rcs.utils.ThreadPoolUtils] [] - DEV-INDICATOR-PUSH-5111983 has been created
[INFO ] 2021-12-27 03:02:14.002 [DEV-INDICATOR-PUSH-5111922] [com.dmall.rcs.utils.ThreadPoolUtils] [] - DEV-INDICATOR-PUSH-5111984 has been created
[INFO ] 2021-12-27 03:02:14.003 [DEV-INDICATOR-PUSH-5111980] [com.dmall.rcs.utils.ThreadPoolUtils] [] - DEV-INDICATOR-PUSH-5111985 has been created
[INFO ] 2021-12-27 03:02:14.005 [DEV-INDICATOR-PUSH-5111985] [com.dmall.rcs.utils.ThreadPoolUtils] [] - DEV-INDICATOR-PUSH-5111986 has been created
[INFO ] 2021-12-27 03:02:14.006 [DEV-INDICATOR-PUSH-5111974] [com.dmall.rcs.utils.ThreadPoolUtils] [] - DEV-INDICATOR-PUSH-5111987 has been created
[INFO ] 2021-12-27 03:02:14.008 [DEV-INDICATOR-PUSH-5111977] [com.dmall.rcs.utils.ThreadPoolUtils] [] - DEV-INDICATOR-PUSH-5111988 has been created
[INFO ] 2021-12-27 03:02:14.010 [DEV-INDICATOR-PUSH-5111973] [com.dmall.rcs.utils.ThreadPoolUtils] [] - DEV-INDICATOR-PUSH-5111989 has been created
这边线程池参数设置代码：
ExecutorService pushThreadPool = ThreadPoolUtils.createBlockPool(12, 20, 200, 200, "DEV-INDICATOR-PUSH");
coreSize: 12, maxSize:20, maxIdleTime:200, queueSize:200 虽然按照hive读取的数据一条条推送但是根据配置按道理是不该这么频繁创建线程的。
问题原因
2.1 Jedis中的hmset的方法注意事项
/**
* Set the respective fields to the respective values. HMSET replaces old values with new values.
* <p>
* If key does not exist, a new key holding a hash is created.
* <p>
* <b>Time complexity:</b> O(N) (with N being the number of fields)
* @param key
* @param hash
* @return Return OK or Exception if hash is empty
*/
public String hmset(final String key, final Map<String, String> hash) {
checkIsInMultiOrPipeline();
client.hmset(key, hash);
return client.getStatusCodeReply();
}
public void hmset(final String key, final Map<String, String> hash) {
final Map<byte[], byte[]> bhash = new HashMap<byte[], byte[]>(hash.size());
for (final Entry<String, String> entry : hash.entrySet()) {
bhash.put(SafeEncoder.encode(entry.getKey()), SafeEncoder.encode(entry.getValue()));
}
hmset(SafeEncoder.encode(key), bhash);
}
public static byte[] encode(final String str) {
try {
if (str == null) {
throw new JedisDataException("value sent to redis cannot be null");
}
return str.getBytes(Protocol.CHARSET);
} catch (UnsupportedEncodingException e) {
throw new JedisException(e);
}
}
-
根据注释来说明
@return Return OK or Exception if hash is empty如果map没有内容将会报错，另外如果map里的key对应的value为null也会抛出JedisDataException此异常是一个运行时异常 -
线程池中的worker是这么执行代码的
try {
beforeExecute(wt, task);
Throwable thrown = null;
try {
task.run();
} catch (RuntimeException x) {
thrown = x; throw x;
} catch (Error x) {
thrown = x; throw x;
} catch (Throwable x) {
thrown = x; throw new Error(x);
} finally {
afterExecute(task, thrown);
}
} finally {
task = null;
w.completedTasks++;
w.unlock();
}
因为从需求上来说推数的数据很有可能是null那么转换成Map的话，很有可能它的value值也是null。根据上述代码SafeEncoder就会抛出RuntimeException，如果在线程池里没有进行异常处理的话那么就会导致线程池中的线程成为消亡状态。然后线程池根据线程池策略频繁的创建线程，毕竟原先的线程都因为异常挂掉了么。我们来看下面的例子:
ExecutorService executorService = new ThreadPoolExecutor(5, 15, 200, TimeUnit.SECONDS, new LinkedBlockingQueue<>(), new ThreadFactory() {
@Override
public Thread newThread(Runnable runnable) {
System.out.println("创建线程");
return new Thread(runnable);
}
});
for (int i = 0; i < Short.MAX_VALUE; i++) {
executorService.execute(new Runnable() {
@Override
public void run() {
try {
TimeUnit.SECONDS.sleep(5);
} catch (InterruptedException e) {
}
System.out.println(1 / 0);
}
});
}
}
上述代码的输出:
创建线程
创建线程
创建线程
创建线程
创建线程
创建线程
创建线程
创建线程
Exception in thread "Thread-3" 创建线程
创建线程
Exception in thread "Thread-2" Exception in thread "Thread-1" Exception in thread "Thread-4" Exception in thread "Thread-0" java.lang.ArithmeticException: / by zero
at com.company.Main$2.run(Main.java:26)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.lang.ArithmeticException: / by zero
at com.company.Main$2.run(Main.java:26)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.lang.ArithmeticException: / by zero
at com.company.Main$2.run(Main.java:26)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.lang.ArithmeticException: / by zero
at com.company.Main$2.run(Main.java:26)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.lang.ArithmeticException: / by zero
at com.company.Main$2.run(Main.java:26)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
创建线程
Exception in thread "Thread-8" java.lang.ArithmeticException: / by zero
at com.company.Main$2.run(Main.java:26)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
创建线程
创建线程
Exception in thread "Thread-5" 创建线程
创建线程
创建线程
创建线程
创建线程
Exception in thread "Thread-14" 创建线程
java.lang.ArithmeticException: / by zero
创建线程
at com.company.Main$2.run(Main.java:26)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Exception in thread "Thread-12" Exception in thread "Thread-13" java.lang.ArithmeticException: / by zero
at com.company.Main$2.run(Main.java:26)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Exception in thread "Thread-11" java.lang.ArithmeticException: / by zero
at com.company.Main$2.run(Main.java:26)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.lang.ArithmeticException: / by zero
at com.company.Main$2.run(Main.java:26)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
创建线程
Exception in thread "Thread-15" java.lang.ArithmeticException: / by zero
at com.company.Main$2.run(Main.java:26)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
创建线程
创建线程
Exception in thread "Thread-17" 创建线程
结论
- 非特殊原因使用线程池一定要在Runable代码里进行异常处理，减少不必要的麻烦。
- 任务出错的日志一定要合理的输出，毕竟线上跑，不输出到日志，错误就被无情的吃掉了。不利于排查