1.简介
1.1 概述
In a distributed environment, inevitably some of the many service dependencies will fail. Hystrix is a library that helps you control the interactions between these distributed services by adding latency tolerance and fault tolerance logic. Hystrix does this by isolating points of access between the services, stopping cascading failures across them, and providing fallback options, all of which improve your system’s overall resiliency.
在分布式环境中，不可避免地会有许多服务依赖项中的某些失败。 Hystrix是一个库，可通过添加等待时间容限和容错逻辑来帮助您控制这些分布式服务之间的交互。 Hystrix通过隔离服务之间的访问点，停止服务之间的级联故障并提供后备选项来实现此目的，所有这些都可以提高系统的整体弹性。
1.2 特点
- Preventing any single dependency from using up all container (such as Tomcat) user threads: 防止任何单个依赖项耗尽所有容器（例如Tomcat）用户线程
- Shedding load and failing fast instead of queueing: 减少负载并快速失败，而不是排队
- Providing fallbacks wherever feasible to protect users from failure: 在可行的情况下提供备用，以保护用户免受故障的影响
- Using isolation techniques (such as bulkhead, swimlane, and circuit breaker patterns) to limit the impact of any one dependency: 使用隔离技术（如bulkhead，swimlane和 circuit breaker patterns）来限制任何一种依赖关系的影响
- Optimizing for time-to-discovery through near real-time metrics, monitoring, and alerting: 提供实时监控、报警等手段
- Optimizing for time-to-recovery by means of low latency propagation of configuration changes and support for dynamic property changes in most aspects of Hystrix, which allows you to make real-time operational modifications with low latency feedback loops: 支持动态属性更改，这使您可以通过低延迟反馈回路进行实时操作修改
- Protecting against failures in the entire dependency client execution, not just in the network traffic: 防止整个依赖客户端执行失败，而不仅仅是网络流请求的客户端
2.演示环境
- JDK 1.8.0_201
- Spring Boot 2.2.0.RELEASE、Spring Cloud Hoxton.RELEASE
- 构建工具（apache maven 3.6.3）
- 开发工具（IntelliJ IDEA ）
3.演示代码
总体结构说明：
- hystrix-demo： hystrix 的简单使用例子
3.1 hystrix-demo
3.1.1 代码说明
hystrix 使用的例子
3.1.2 maven 依赖
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
</dependency>
3.1.3 配置文件
无
3.1.4 java代码
DemoController.java
@RestController
public class DemoController {
/**
* 设置线程的超时时间是100ms，默认值为50ms
* 如果超时触发降级之后，执行 fallbackMethod 中的 timeoutMethod 方法
*/
@GetMapping(value = "/demo")
@HystrixCommand(fallbackMethod = "timeoutMethod", commandProperties = {
@HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "100")})
public String demo(@RequestParam(name = "time", defaultValue = "50") String value) throws InterruptedException {
long timeout = Long.parseLong(value);
Thread.sleep(timeout);
return "hystrix demo";
}
public String timeoutMethod(String value) {
return "the request cost " + value + " ms, timeout!";
}
/**
* 使用自定义的 HelloCommand 定义降级策略
*/
@GetMapping(value = "/hello")
public String hello() {
return new HelloCommand().execute();
}
}
HelloCommand.java
public class HelloCommand extends HystrixCommand<String> {
/**
* 继承自 HystrixCommand，定义要支持降级的方法
*/
public HelloCommand() {
super(HystrixCommandGroupKey.Factory.asKey("hello"), 100);
}
/**
* 正常情况下执行
*/
@Override
protected String run() throws Exception {
int time = new Random().nextInt(200);
Thread.sleep(time);
return "hello hystrix command!";
}
/**
* 被降级后的执行
*/
@Override
protected String getFallback() {
return "hystrix command request timeout";
}
}
NetflixHystrixDemoApplication.java
@EnableHystrix // 启用 hystrix
@SpringBootApplication
public class NetflixHystrixDemoApplication {
public static void main(String[] args) {
SpringApplication.run(NetflixHystrixDemoApplication.class, args);
}
}
3.2 git 地址
spring-cloud-nfx-03-hystrix/hystrix-demo: Spring Cloud 整合 Hystrix 实现的分布式服务降级方案
4.效果展示
启动 NetflixHystrixDemoApplication.main 方法，在 netflix-hystrix-demo.http 访问下列地址，观察输出信息是否符合预期。
### GET /demo?time={value}
GET http://localhost:8080/demo?time=120
### GET /hello
GET http://localhost:8080/hello
5.源码分析
5.1 Hystrix 如何触发熔断？
hystrix 熔断的注解 @HystrixCommand，是通过 HystrixCommandAspect 切面来处理的。
切入点定义如下
@Pointcut("@annotation(com.netflix.hystrix.contrib.javanica.annotation.HystrixCommand)")
public void hystrixCommandAnnotationPointcut() {
}
它在 methodsAnnotatedWithHystrixCommand 上触发调用
@Around("hystrixCommandAnnotationPointcut() || hystrixCollapserAnnotationPointcut()")
public Object methodsAnnotatedWithHystrixCommand(final ProceedingJoinPoint joinPoint) throws Throwable {
// 获取目标方法
Method method = getMethodFromTarget(joinPoint);
Validate.notNull(method, "failed to get method from joinPoint: %s", joinPoint);
// 判断方法上是否存在 @HystrixCommand 注解
if (method.isAnnotationPresent(HystrixCommand.class) && method.isAnnotationPresent(HystrixCollapser.class)) {
throw new IllegalStateException("method cannot be annotated with HystrixCommand and HystrixCollapser " + "annotations at the same time");
}
MetaHolderFactory metaHolderFactory = META_HOLDER_FACTORY_MAP.get(HystrixPointcutType.of(method));
MetaHolder metaHolder = metaHolderFactory.create(joinPoint);
// 如果是异步，则创建GenericObservableCommand， 否则创建GenericCommand
HystrixInvokable invokable = HystrixCommandFactory.getInstance().create(metaHolder);
ExecutionType executionType = metaHolder.isCollapserAnnotationPresent() ?
metaHolder.getCollapserExecutionType() : metaHolder.getExecutionType();
Object result;
try {
// 是否响应式，默认为非响应式
if (!metaHolder.isObservable()) {
result = CommandExecutor.execute(invokable, executionType, metaHolder);
} else {
result = executeObservable(invokable, executionType, metaHolder);
}
} catch (HystrixBadRequestException e) {
throw e.getCause();
} catch (HystrixRuntimeException e) {
throw hystrixRuntimeExceptionToThrowable(metaHolder, e);
}
return result;
}
CommandExecutor.execute 实现如下
public static Object execute(HystrixInvokable invokable, ExecutionType executionType, MetaHolder metaHolder) throws RuntimeException {
Validate.notNull(invokable);
Validate.notNull(metaHolder);
switch (executionType) {
case SYNCHRONOUS: {
// 同步
return castToExecutable(invokable, executionType).execute();
}
case ASYNCHRONOUS: {
// 异步
HystrixExecutable executable = castToExecutable(invokable, executionType);
if (metaHolder.hasFallbackMethodCommand()
&& ExecutionType.ASYNCHRONOUS == metaHolder.getFallbackExecutionType()) {
return new FutureDecorator(executable.queue());
}
return executable.queue();
}
case OBSERVABLE: {
// 响应式
HystrixObservable observable = castToObservable(invokable);
return ObservableExecutionMode.EAGER == metaHolder.getObservableExecutionMode() ? observable.observe() : observable.toObservable();
}
default:
throw new RuntimeException("unsupported execution type: " + executionType);
}
}
由于是同步调用，所以到 HystrixCommand.execute，这里通过 queue() 返回一个 future 对象
public R execute() {
try {
return queue().get();
} catch (Exception e) {
throw Exceptions.sneakyThrow(decomposeException(e));
}
}
queue() 这个方法，返回了一个Future对象，这个future的实际处理委派给 f 实现，f是匿名内部类，当调用queue().get()方法时，最终调用 delegate.get 方法
public Future<R> queue() {
/*
* The Future returned by Observable.toBlocking().toFuture() does not implement the
* interruption of the execution thread when the "mayInterrupt" flag of Future.cancel(boolean) is set to true;
* thus, to comply with the contract of Future, we must wrap around it.
*/
// 创建一个委派对象
final Future<R> delegate = toObservable().toBlocking().toFuture();
final Future<R> f = new Future<R>() {
@Override
public boolean cancel(boolean mayInterruptIfRunning) {
if (delegate.isCancelled()) {
return false;
}
if (HystrixCommand.this.getProperties().executionIsolationThreadInterruptOnFutureCancel().get()) {
/*
* The only valid transition here is false -> true. If there are two futures, say f1 and f2, created by this command
* (which is super-weird, but has never been prohibited), and calls to f1.cancel(true) and to f2.cancel(false) are
* issued by different threads, it's unclear about what value would be used by the time mayInterruptOnCancel is checked.
* The most consistent way to deal with this scenario is to say that if *any* cancellation is invoked with interruption,
* than that interruption request cannot be taken back.
*/
interruptOnFutureCancel.compareAndSet(false, mayInterruptIfRunning);
}
final boolean res = delegate.cancel(interruptOnFutureCancel.get());
if (!isExecutionComplete() && interruptOnFutureCancel.get()) {
final Thread t = executionThread.get();
if (t != null && !t.equals(Thread.currentThread())) {
t.interrupt();
}
}
return res;
}
@Override
public boolean isCancelled() {
return delegate.isCancelled();
}
@Override
public boolean isDone() {
return delegate.isDone();
}
@Override
public R get() throws InterruptedException, ExecutionException {
return delegate.get();
}
@Override
public R get(long timeout, TimeUnit unit) throws InterruptedException, ExecutionException, TimeoutException {
return delegate.get(timeout, unit);
}
};
/* special handling of error states that throw immediately */
if (f.isDone()) {
try {
f.get();
return f;
} catch (Exception e) {
Throwable t = decomposeException(e);
if (t instanceof HystrixBadRequestException) {
return f;
} else if (t instanceof HystrixRuntimeException) {
HystrixRuntimeException hre = (HystrixRuntimeException) t;
switch (hre.getFailureType()) {
case COMMAND_EXCEPTION:
case TIMEOUT:
// we don't throw these types from queue() only from queue().get() as they are execution errors
return f;
default:
// these are errors we throw from queue() as they as rejection type errors
throw hre;
}
} else {
throw Exceptions.sneakyThrow(t);
}
}
}
return f;
}
delegate 对象由 toObservable() 创建，toObservable() 中调用了 applyHystrixSemantics() 方法
Observable<R> hystrixObservable =
Observable.defer(applyHystrixSemantics)
.map(wrapWithAllOnNextHooks);
在 applyHystrixSemantics 中先通过 circuitBreaker.allowRequest() 判断是否允许当前请求，如果允许执行后续逻辑；否则 调用 handleShortCircuitViaFallback 执行 fallback 方法。
handleShortCircuitViaFallback 的调用路劲为：handleShortCircuitViaFallback() -> getFallbackOrThrowException() -> getFallbackObservable() -> HystrixCommand.getFallbackObservable() -> getFallback() -> GenericCommand.getFallback()
private Observable<R> applyHystrixSemantics(final AbstractCommand<R> _cmd) {
// mark that we're starting execution on the ExecutionHook
// if this hook throws an exception, then a fast-fail occurs with no fallback. No state is left inconsistent
executionHook.onStart(_cmd);
/* determine if we're allowed to execute */
// 判断是否允许当前请求
if (circuitBreaker.allowRequest()) {
final TryableSemaphore executionSemaphore = getExecutionSemaphore();
final AtomicBoolean semaphoreHasBeenReleased = new AtomicBoolean(false);
final Action0 singleSemaphoreRelease = new Action0() {
@Override
public void call() {
if (semaphoreHasBeenReleased.compareAndSet(false, true)) {
executionSemaphore.release();
}
}
};
final Action1<Throwable> markExceptionThrown = new Action1<Throwable>() {
@Override
public void call(Throwable t) {
eventNotifier.markEvent(HystrixEventType.EXCEPTION_THROWN, commandKey);
}
};
// 获取信号量，获取到执行 executeCommandAndObserve 方法
if (executionSemaphore.tryAcquire()) {
try {
/* used to track userThreadExecutionTime */
executionResult = executionResult.setInvocationStartTime(System.currentTimeMillis());
return executeCommandAndObserve(_cmd)
.doOnError(markExceptionThrown)
.doOnTerminate(singleSemaphoreRelease)
.doOnUnsubscribe(singleSemaphoreRelease);
} catch (RuntimeException e) {
return Observable.error(e);
}
} else {
// 拒绝，执行fallback方法
return handleSemaphoreRejectionViaFallback();
}
} else {
// 不允许执行，直接调用fallback
return handleShortCircuitViaFallback();
}
}
通过以上多次调用后，最终到 GenericCommand.run()
@Override
protected Object run() throws Exception {
LOGGER.debug("execute command: {}", getCommandKey().name());
return process(new Action() {
@Override
Object execute() {
return getCommandAction().execute(getExecutionType());
}
});
}