1.异常的原因:
(1).DocumentDB重启导致一段时间服务不可以使用,并且DocumentDB无法实现主备的切换;
(2).statistic_record_service, thirdparty_control,queue_message这三个gen_server由于在访问DocumentDB的时候没有做异常的处理,导致监控xxxxxx_sup多次重启这些服务,一定频率之后({one_for_one, 10, 10})xxxxxx_sup会停掉所有监控的服务,最终导致整个iot云端的服务不可使用。
备注:xxxxxx_sup监控下的服务如果在最近的 MaxT 秒内发生的重启次数超过了 MaxR 次,那么督程会终止所有的子进程,然后结束自己。
参考文档:https://erldoc.com/doc/otp-design-principles/supervisor.html
(3)关键信息:reached_max_restart_intensity
2019-08-02 07:01:27.600 [error] <0.360.0> Supervisor xxxxxx_suphad child queue_message started with octopus_queue_message
:start_link([]) at <0.16789.4084> exit with reason reached_max_restart_intensity in context shutdown
2.服务器中错误log的输出:
(1).queue_message
2019-08-02 07:01:20.771 [error] <0.609.0>@queue_message:terminate:137 _Reason={{case_clause,{error,{connection_failure,{can
t_connect,econnrefused}}}},[{db_helper,fetch_list,1,[{file,"src/model/db_helper.erl"},{line,87}]},{queue_message,resend_off
line_message_internal,2,[{file,"src/octopus_log/octopus_queue_message.erl"},{line,197}]},{queue_message,handle_cast,2,[{fil
e,"src/log/queue_message.erl"},{line,110}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,616}]},{gen_s
erver,handle_msg,6,[{file,"gen_server.erl"},{line,686}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
(2).statistic_record_service
1).由于在函数terminate(_Reason, _State)中没有输出日志lager:error("_Reason=~p", [_Reason]),,所以没有最关键的信息;
2).建议在所有使用gen_server的terminate函数都要输出错误日志;
error.log.6:2019-08-02 07:01:20.750 [error] <0.667.0>@db_helper:fetch:132 gen_server statistic_record_service terminated with reaso
n: no case clause matching {error,{connection_failure,{cant_connect,econnrefused}}} in db_helper:fetch/1 line 132
(2).thirdparty_control
1).由于在函数terminate(_Reason, _State)中没有输出日志lager:error("_Reason=~p", [_Reason]),,所以没有最关键的信息
2).建议在所有使用gen_server的terminate函数都要输出错误日志;
error.log.6:2019-08-02 07:01:20.750 [error] <0.661.0>@db_helper:fetch:132 CRASH REPORT Process thirdparty_control with 0 neighbours
crashed with reason: no case clause matching {error,{connection_failure,{cant_connect,econnrefused}}} in db_helper:fetch/1 line 132