报错信息可以看出:由于没有正常的关闭连接,导致连接异常
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream. at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:198) at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40) at redis.clients.jedis.Protocol.process(Protocol.java:132) at redis.clients.jedis.Protocol.read(Protocol.java:196) at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:288) at redis.clients.jedis.Connection.getBinaryBulkReply(Connection.java:207) at redis.clients.jedis.Connection.getBulkReply(Connection.java:196) at redis.clients.jedis.Jedis.get(Jedis.java:98) at com.dinpay.bdp.rcp.streaming.WindowHzUtil.setData(WindowHzUtil.java:343) at com.dinpay.bdp.rcp.streaming.WindowHzUtil$12$1.call(WindowHzUtil.java:329) at com.dinpay.bdp.rcp.streaming.WindowHzUtil$12$1.call(WindowHzUtil.java:325) at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreach$1.apply(JavaRDDLike.scala:332) at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreach$1.apply(JavaRDDLike.scala:332) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28) at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$32.apply(RDD.scala:912) at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$32.apply(RDD.scala:912) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) ... 3 more
解决方法:采用jedis连接池实现连接,并关闭连接
import redis.clients.jedis.JedisPool; import redis.clients.jedis.JedisPoolConfig; private static JedisPool pool = null; public static JedisPool getJedisPool(){ JedisPoolConfig config = new JedisPoolConfig(); config.setMaxTotal(500); config.setMaxIdle(5); config.setMaxWaitMillis(1000 * 100); config.setTestOnBorrow(true); if(pool == null){ pool = new JedisPool(config, Constant.CODISPROXY, Constant.CODISPORT); } return pool; } 调用方式: private static Jedis jedis = null; try { jedis = CodisUtil.getJedisPool().getResource(); jedis.set()....jedis operations logger.info(tp._1() + ">>>>>>>>>>>>>" + tp._2()); logger.info("保存到Codis完成!"); } finally { //回收jedis连接 if(jedis !=null){ jedis.close(); } }
更多codis的介绍:
https://github.com/xetorthio/jedis/wiki/Getting-started#using-jedis-in-a-multithreaded-environment