1.现象:
当dubbo provider 抛出异常时,dubbo consumer 在输出异常栈信息时,展示的都是provider 侧的线程栈,这是因为,异常的stackTrace实在在new Throwable()的时候生成的。
/**
* Constructs a new throwable with {@code null} as its detail message.
* The cause is not initialized, and may subsequently be initialized by a
* call to {@link #initCause}.
*
* <p>The {@link #fillInStackTrace()} method is called to initialize
* the stack trace data in the newly created throwable.
*/
public Throwable() {
fillInStackTrace();
}
/**
* Fills in the execution stack trace. This method records within this
* {@code Throwable} object information about the current state of
* the stack frames for the current thread.
*
* <p>If the stack trace of this {@code Throwable} {@linkplain
* Throwable#Throwable(String, Throwable, boolean, boolean) is not
* writable}, calling this method has no effect.
*
* @return a reference to this {@code Throwable} instance.
* @see java.lang.Throwable#printStackTrace()
*/
public synchronized Throwable fillInStackTrace() {
if (stackTrace != null ||
backtrace != null /* Out of protocol state */ ) {
fillInStackTrace(0);
stackTrace = UNASSIGNED_STACK;
}
return this;
}
//执行native方法
private native Throwable fillInStackTrace(int dummy);
这里的关键在fillInStackTrace native 方法,会将当前线程栈的信息填充进stackTrace中;但是分布式服务的调用链中各个服务,都是不同进程,更是不同线程,所以这里的stackTrace只会有发生异常的provider的线程栈信息。当consumer接收到异常时,哪怕log出来,也只有provider侧的相关信息,丢失了consumer侧的线程栈信息;而这一现象,在复杂的dubbo调用链中,是无法满足开发人员对异常分析的需求的;
2.解决办法:
为了解决这个问题,首先想到的是ExceptionFilter.class,dubbo自带的ExceptionFilter.class是只对provider生效的,对异常是否需要包装成RuntimeException进行判断;那我们可以相应的实现一个consumer侧的ConsumerExceptionFilter去实现,当有provider返回异常时,对异常栈进行追加当前consumer侧的线程栈;这样就变相的实现了跨线程的异常栈了;
这里是实现代码,为了避免异常栈过大,在代码实现时,追加的异常栈只取了当前服务的执行api行为的位置;(Filter要起作用,是要在org.apache.dubbo.rpc.Filter中添加自定义filter的配置)
@Activate(group = "consumer")
public class ConsumerExceptionFilter extends ListenableFilter {
public ConsumerExceptionFilter() {
super.listener = new ConsumerExceptionFilter.ExceptionListener();
}
@Override
public Result invoke(Invoker<?> invoker, Invocation invocation) throws RpcException {
return invoker.invoke(invocation);
}
static class ExceptionListener implements Listener {
private Logger logger = LoggerFactory.getLogger(ConsumerExceptionFilter.ExceptionListener.class);
@Override
public void onResponse(Result appResponse, Invoker<?> invoker, Invocation invocation) {
if (appResponse.hasException() && GenericService.class != invoker.getInterface()) {
try {
Throwable exception = appResponse.getException();
// directly throw if it's checked exception
if (!(exception instanceof RuntimeException) && (exception instanceof Exception)) {
return;
}
//这段代码的主要目的是为了将consumer方的部分stackTrace追加到provider抛出来的异常的stackTrace
//方便在复杂调用环境中,追踪异常位置
StackTraceElement[] stackTrace = exception.getStackTrace();
StackTraceElement[] newStackTrace = Arrays.copyOf(stackTrace, stackTrace.length + 1);
StackTraceElement[] consumerStackTrace = new RuntimeException().getStackTrace();
boolean meetProxyElement = false;
for (StackTraceElement consumerStackTraceElement : consumerStackTrace) {
if (meetProxyElement){
//这里为了节省资源,只追加一行stackTrace(执行api代码位置的stackTrace)
newStackTrace[newStackTrace.length-1] = consumerStackTraceElement;
break;
}
//dubbo的调用使用动态代理,所以stackTraceElement的className会是com.sun.proxy.$Proxy,
//它的下一个stackTraceElement就是真正的调用方位置
if(
Objects.equals(consumerStackTraceElement.getMethodName(), invocation.getMethodName())
&& consumerStackTraceElement.getClassName().startsWith("com.sun.proxy")){
meetProxyElement = true;
}
}
exception.setStackTrace(newStackTrace);
return;
} catch (Throwable e) {
logger.warn("Fail to ConsumerExceptionFilter when execute " + RpcContext.getContext().getRemoteHost() + ". service: " + invoker.getInterface().getName() + ", method: " + invocation.getMethodName() + ", exception: " + e.getClass().getName() + ": " + e.getMessage(), e);
return;
}
}
}
@Override
public void onError(Throwable e, Invoker<?> invoker, Invocation invocation) {
logger.error("Got unchecked and undeclared exception which from " + RpcContext.getContext().getRemoteHost() + ". service: " + invoker.getInterface().getName() + ", method: " + invocation.getMethodName() + ", exception: " + e.getClass().getName() + ": " + e.getMessage(), e);
}
}
}