最近公司有个项目，需要flink实时地对elasticsearch进行频繁的插入。但是在写入elasticsearch的时候出现了OOM内存溢出的异常，以及连接异常中断的错误。
报错如下：1.Caused by: java.lang.IllegalStateException: I/O reactor has been shut down 连接异常关闭。
2.java.lang.OutOfMemoryError: Direct buffer memory OOM内存溢出。
首先解决第一个异常，连接中断。网上很多人说是因为es的client调用了close方法，client请求在还没有完毕时就已经被关闭掉，导致后面的连接不可用，从而报出来这个异常。
但是我的代码一开始用的原生elasticsearch7.12来执行插入请求，没用调用close方法，所以异常可能是别的原因造成的。后面改为了flink封装的方法，需要手动关闭。
当然了，在解决这个问题之前，一定要保证代码本身执行没有问题，否则可能是其他的异常导致连接的关闭。
为了解决这个异常我们做了如下努力：
用flink封装的ElasticsearchSink代替es原生的client来执行插入的请求。（可能原生的也可以，但是我们在测试过程中发现，flink封装的效果更好，更不容易出错）
然后设置参数：
1.设置超时时间： requestBuilder.setConnectTimeout(60000); requestBuilder.setSocketTimeout(60000);这里两个超时时间都设置的一分钟。
2.设置最大连接数和刷新周期： esSinkBuilder.setBulkFlushMaxActions(1); esSinkBuilder.setBulkFlushMaxSizeMb(1); esSinkBuilder.setBulkFlushInterval(1);//刷新周期设置的1毫秒。
3.设置线程数量：
IOReactorConfig.custom().setIoThreadCount(5).build());
esSinkBuilder.setFailureHandler(new RetryRequestFailureHandler());//处理失败的Elasticsearch请求
这里sink每执行一次就要建立一次请求，所以要进行关闭。if（build！=null）build.close();
elasticsearch7.12版本使用了登录验证
完整代码如下：
`//operator为flink数据流
SingleOutputStreamOperator<JSONObject> operator;
//elasticsearch 地址
List<HttpHost> esAddresses = ESSinkUtil.getEsAddresses("locolhost1:9200,locolhost2:9200,locolhost3:9200,locolhost4:9200,locolhost5:9200");
//getEsAddresses实体类
public static List<HttpHost> getEsAddresses(String hosts) throws MalformedURLException {
String[] hostList = hosts.split(",");
List<HttpHost> addresses = new ArrayList<>();
for (String host : hostList) {
if (host.startsWith("http")) {
URL url = new URL(host);
addresses.add(new HttpHost(url.getHost(), url.getPort()));
} else {
String[] parts = host.split(":", 2);
if (parts.length > 1) {
addresses.add(new HttpHost(parts[0], Integer.parseInt(parts[1])));
} else {
throw new MalformedURLException("invalid elasticsearch hosts format");
}
}
}
return addresses;
}
//elasticsearch插入请求
ESSinkUtil.addSink(esAddresses, 1, 8, operator, new ElasticsearchSinkFunction<JSONObject>() {
@Override
public void process(JSONObject metric, RuntimeContext runtimeContext, RequestIndexer requestIndexer) {
requestIndexer.add(Requests.indexRequest()
.index(INDEX)
.id(metric.get("id"))
.source(JSON.toJSONString(metric), XContentType.JSON));
}
});
//flink封装的elasticsearch连接sink
public static <T> void addSink(List<HttpHost> hosts, int bulkFlushMaxActions, int parallelism,
SingleOutputStreamOperator<T> data, ElasticsearchSinkFunction<T> func) {
try {
ElasticsearchSink.Builder<T> esSinkBuilder = new ElasticsearchSink.Builder<>(hosts, func);
esSinkBuilder.setBulkFlushMaxActions(bulkFlushMaxActions);//每次最大插入数量
esSinkBuilder.setBulkFlushMaxSizeMb(1);//最大插入内存
esSinkBuilder.setBulkFlushInterval(1);//插入刷新周期
esSinkBuilder.setFailureHandler(new RetryRequestFailureHandler());//处理失败的Elasticsearch请求
//设置自定义http客户端配置
esSinkBuilder.setRestClientFactory(new RestClientFactory() {
@Override
public void configureRestClientBuilder(RestClientBuilder restClientBuilder) {
final CredentialsProvider credentialsProvider =new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY,new UsernamePasswordCredentials(USER, PASSWORD));
restClientBuilder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
//httpClientBuilder.disableAuthCaching();
httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
return httpClientBuilder.setDefaultIOReactorConfig(
IOReactorConfig.custom()
.setIoThreadCount(5)//设置线程数量为5
.build());
}
}).setRequestConfigCallback(new RestClientBuilder.RequestConfigCallback(){
@Override
public RequestConfig.Builder customizeRequestConfig(RequestConfig.Builder requestBuilder) {
requestBuilder.setConnectTimeout(5000);//连接超时时间
requestBuilder.setSocketTimeout(60000);
requestBuilder.setConnectionRequestTimeout(10000);
return requestBuilder;
}
});
}
});
//todo:xpack security
ElasticsearchSink<T> build = esSinkBuilder.build();
data.addSink(build).setParallelism(parallelism);
if (build!=null) build.close();//build用完后一定要关闭
} catch (Exception e) {
e.printStackTrace();
}
}
//处理失败的Elasticsearch请求
public class RetryRequestFailureHandler implements ActionRequestFailureHandler {
public RetryRequestFailureHandler() {
}
@Override
public void onFailure(ActionRequest actionRequest, Throwable throwable, int i, RequestIndexer requestIndexer) throws Throwable {
if (ExceptionUtils.findThrowable(throwable, EsRejectedExecutionException.class).isPresent()) {
requestIndexer.add(new ActionRequest[]{actionRequest});
} else {
if (ExceptionUtils.findThrowable(throwable, SocketTimeoutException.class).isPresent()) {
return;
} else {
Optional<IOException> exp = ExceptionUtils.findThrowable(throwable, IOException.class);
if (exp.isPresent()) {
IOException ioExp = exp.get();
if (ioExp != null && ioExp.getMessage() != null && ioExp.getMessage().contains("max retry timeout")) {
return;
}
}
}
throw throwable;
}
}
}
//下面原生elasticsearch建立client连接
public static RestHighLevelClient client(){
final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY,
new UsernamePasswordCredentials(USER, PASSWORD)); //es账号密码（默认用户名为elastic）
//创建带用户名密码的ES客户端对象
try {
if (null == client){
client = new RestHighLevelClient(RestClient.builder(new HttpHost(PRODUCE_HOST,PORT,SCHEMA)
,new HttpHost(PRODUCE_HOST2,PORT,SCHEMA),new HttpHost(PRODUCE_HOST3,PORT,SCHEMA),new HttpHost(PRODUCE_HOST4,PORT,SCHEMA)
,new HttpHost(PRODUCE_HOST5,PORT,SCHEMA))
//异步HTTPclient连接数配置
.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
@Override
public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
//httpClientBuilder.disableAuthCaching();
httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
return httpClientBuilder.setDefaultIOReactorConfig(
IOReactorConfig.custom()
.setIoThreadCount(5)
.build());
}
})
.setRequestConfigCallback(new RestClientBuilder.RequestConfigCallback(){
@Override
public RequestConfig.Builder customizeRequestConfig(RequestConfig.Builder requestBuilder) {
requestBuilder.setConnectTimeout(5000);
requestBuilder.setSocketTimeout(60000);
requestBuilder.setConnectionRequestTimeout(10000);
return requestBuilder;
}
})
);
}
} catch (Exception e) {
e.printStackTrace();
}
return client;
}`
然后解决OOM内存溢出的问题，我们出发点是代码里调整并行度，保证有足够的slot可用，暂定为8。然后环境配置里调大内存。一般来说内存溢出就是存在内存泄漏
还有可能是代码本身异常太多，导致程序异常。通过修改代码，找到可能出现异常的地方，进行修改。
接着就是给flink设置重启策略
上述操作弄好之后，flink的报错就消失了，之前任务一直跑不上去，放到ui上面马上就报红失败。
码字不易，如果问题解决了别忘了留言点赞噢