说下hangfire吧

最近因工作需要开发计划任务模块（严格来说应该是修改bug吧，其他同事负责的）接触到了Hangfire。早前听同事说hangfire有点坑，怀着好奇，趁这两天bug改的差不多了，在github上面down了hangfire源码，下面分享一下，自己读hangfire源码的一些理解，和工作中需要注意的地方。介绍大概分为以下几个部分吧。1.准备工作，2.简单使用，3.源码分析，4.避坑。需要说明一下接触hangfire源码的时间不长，也就几天时间理解不到位，或者说错了的，希望在评论指正。

准备工作：hangfire源代码的代码量不多，github地址： https://github.com/HangfireIO/Hangfire，有兴趣的朋友可以自己下载瞅瞅源码。功能上大概可以分为客户端模式和服务端模式。用到的技术大概有Multi Thread、Expression、Dapper、Cron等。可以这么说，它的定时任务完全就是基于多线程协作实现的。因为是多线程环境，所以个人觉得看起来有点费力。

简单使用：.Net&.Net Core环境都可以使用，下面就以.Net Core的使用为例。

1.客户端和服务端独立部署

client端

public IServiceProvider ConfigureServices(IServiceCollection services)
        {
            // 其他代码
             services.AddHangfire(config =>
            {
                 config.UseSqlServerStorage(...);
            });
        }
 
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            // 其他代码...
            // 启用Dashboard看板
            app.UseHangfireDashboard();
        }

server端

public void Configuration(IAppBuilder app)
        {
            GlobalConfiguration.Configuration
                 .UseSqlServerStorage("连接字符串", new SqlServerStorageOptions
                {
                    // options
                });
            app.UseHangfireServer(new BackgroundJobServerOptions
            {
            });
        }
 
 

或者

services.AddHangfireServer(options =>
                    {
                        // 基于IHostedService接口实现
                    });

PS：server端还有一种实现方式，实现IHostedService接口 其实跟上面的使用方法一样的，注入到服务就ok，在程序启动阶段会自动执行IHostedService接口的两个方法，可以简单看下IHostedService接口的定义。

  public interface IHostedService
  {
    Task StartAsync(CancellationToken cancellationToken);
    Task StopAsync(CancellationToken cancellationToken);
  }

接口就定义了两个方法，start是在程序启动的时候执行，当然stop就是在程序停止的时候执行。我们用一张图简单描绘一下它的执行时机，图是盗的。

以上就是hangfire的client端和server端分开部署的一个简单应用，下面我们看下第二种，client&server部署在同一台机器上。

2.客户端和服务端统一部署

public void Configuration(IAppBuilder app)
{
    GlobalConfiguration.Configuration.UseSqlServerStorage(); // 配置数据库连接
    
    app.UseHangfireServer(); // 启用server
    app.UseHangfireDashboard(); // 启用看板
}

简单的几行代码，当然我也只会简单的用法。以上服务注入并执行，接下来就是往hangfire里面添加任务。

BackgroundJob.Enqueue(() => Console.WriteLine("Simple!")); // 立即执行
BackgroundJob.Schedule(() => Console.WriteLine("Reliable!"), TimeSpan.FromDays(7)); // 延后执行
RecurringJob.AddOrUpdate(() => Console.WriteLine("Transparent!"), Cron.Daily); // 循环执行，支持cron表达式

简单使用就到这吧，我们继续大纲的第三部分，源码分析。

 

源码分析

客户端模式就不用说了，说白了就是往hangfire数据库里面写任务，我们主要是看看服务端的执行原理。我们先找到入口，也可以看做是NetCore里面的一个中间件吧。看代码

app.UseHangfireServer(); // 启用server

UseHangfireServer实现

public static IAppBuilder UseHangfireServer(
            [NotNull] this IAppBuilder builder,
            [NotNull] JobStorage storage,
            [NotNull] BackgroundJobServerOptions options,
            [NotNull] params IBackgroundProcess[] additionalProcesses)
        {
            // 其他代码...
            var server = new BackgroundJobServer(options, storage,  additionalProcesses); 
            
            return builder;
        }

UseHangfireServer扩展方法实现里面，比较重要的一行代码就是创建BackgroundJobServer，BackgroundJobServer实现了IBackgroundProcessingServer接口，server的启动也就是间接在它的构造器里面完成的。我们不妨先瞅瞅IBackgroundProcessingServer接口和BackgroundJobServer类的定义。

// IBackgroundProcessingServer
public interface IBackgroundProcessingServer : IDisposable
    {
        void SendStop();
        bool WaitForShutdown(TimeSpan timeout);
        Task WaitForShutdownAsync(CancellationToken cancellationToken);
    }
 
// BackgroundJobServer
public class BackgroundJobServer : IBackgroundProcessingServer
    {
        // 其他成员...
        public BackgroundJobServer(
            [NotNull] BackgroundJobServerOptions options,
            [NotNull] JobStorage storage,
            [NotNull] IEnumerable<IBackgroundProcess> additionalProcesses,
            [CanBeNull] IJobFilterProvider filterProvider,
            [CanBeNull] JobActivator activator,
            [CanBeNull] IBackgroundJobFactory factory,
            [CanBeNull] IBackgroundJobPerformer performer,
            [CanBeNull] IBackgroundJobStateChanger stateChanger)
        {
            // 其他代码
            var processes = new List<IBackgroundProcessDispatcherBuilder>();
            processes.AddRange(GetRequiredProcesses(filterProvider, activator,  factory, performer, stateChanger));
            processes.AddRange(additionalProcesses.Select(x =>  x.UseBackgroundPool(1)));
            var properties = new Dictionary<string, object>
            {
                { "Queues", options.Queues },
                { "WorkerCount", options.WorkerCount }
            };
            
            _processingServer = new BackgroundProcessingServer(
                storage,
                processes,
                properties,
                GetProcessingServerOptions());
        }
        public void SendStop()
        {
        }
        public void Dispose()
        {
        }
        [Obsolete("This method is a stub. There is no need to call the `Start`  method. Will be removed in version 2.0.0.")]
        public void Start()
        {
        }
        [Obsolete("Please call the `Shutdown` method instead. Will be removed in  version 2.0.0.")]
        public void Stop()
        {
        }
        [Obsolete("Please call the `Shutdown` method instead. Will be removed in  version 2.0.0.")]
        public void Stop(bool force)
        {
        }
        public bool WaitForShutdown(TimeSpan timeout)
        {
        }
        public Task WaitForShutdownAsync(CancellationToken cancellationToken)
        {
        }

IBackgroundProcessingServer接口里面的这几个方法都是跟停用server，取消任务清理资源相关的。BackgroundJobServer类里面真正完成接口的实现是由BackgroundProcessingServer类型的同名函数实现，这个对象是在构造函数里面初始化的，在初始化BackgroundProcessingServer类型的同时，创建了若干IBackgroundProcessDispatcherBuilder实现类BackgroundProcessDispatcherBuilder的实例，hangfire默认实现了7种dispatcher，我们任务、日志、心跳等等独立线程都是由它的Create方法完成，这个地方不算server启动主线，会在后面细说。我们继续看看BackgroundProcessingServer这个类型。这里需要注意的是里面有几个方法好像是被停用了，start、stop等方法，官方也注释了，被删除了。start方法被停用了，难道我们的server启动是在BackgroundProcessingServer类型里面？继续看BackgroundProcessingServer的定义。

public sealed class BackgroundProcessingServer : IBackgroundProcessingServer
    {
        // 其他成员
        internal BackgroundProcessingServer(
            [NotNull] BackgroundServerProcess process,
            [NotNull] BackgroundProcessingServerOptions options)
        {
            _process = process ?? throw new ArgumentNullException(nameof(process));
            _options = options ?? throw new ArgumentNullException(nameof(options));
            _dispatcher = CreateDispatcher();
#if !NETSTANDARD1_3
            AppDomain.CurrentDomain.DomainUnload += OnCurrentDomainUnload;
            AppDomain.CurrentDomain.ProcessExit += OnCurrentDomainUnload;
#endif
        }
        public void SendStop()
        {
        }
        public bool WaitForShutdown(TimeSpan timeout)
        {
        }
        public async Task WaitForShutdownAsync(CancellationToken cancellationToken)
        {
        }
        public void Dispose()
        {
            
        }
        private void OnCurrentDomainUnload(object sender, EventArgs args)
        {
            
        }
        private IBackgroundDispatcher CreateDispatcher()
        {
            var execution = new BackgroundExecution(
                _stoppingCts.Token,
                new BackgroundExecutionOptions
                {
                    Name = nameof(BackgroundServerProcess),
                    ErrorThreshold = TimeSpan.Zero,
                    StillErrorThreshold = TimeSpan.Zero,
                    RetryDelay = retry => _options.RestartDelay
                });
            return new BackgroundDispatcher(
                execution,
                RunServer,
                execution,
                ThreadFactory);
        }
        private void RunServer(Guid executionId, object state)
        {
            _process.Execute(executionId, (BackgroundExecution)state,  _stoppingCts.Token, _stoppedCts.Token, _shutdownCts.Token);
        }
        private static IEnumerable<Thread> ThreadFactory(ThreadStart threadStart)
        {
            yield return new Thread(threadStart)
            {
                IsBackground = true,
                Name = $"{nameof(BackgroundServerProcess)}  #{Interlocked.Increment(ref _lastThreadId)}",
            };
        }
    }

果不其然，server的启动快要揭开神秘的面纱了，RunServer？翻译过来应该是启动服务吧，我们暂且不去管他，先记一下这个有个runserver，我们继续跟踪。在构造函数里面调用了一个CreateDispatcher()的方法，我们看下它的实现

private IBackgroundDispatcher CreateDispatcher()
        {
            var execution = new BackgroundExecution(
                _stoppingCts.Token,
                new BackgroundExecutionOptions
                {
                    Name = nameof(BackgroundServerProcess),
                    ErrorThreshold = TimeSpan.Zero,
                    StillErrorThreshold = TimeSpan.Zero,
                    RetryDelay = retry => _options.RestartDelay
                });
            return new BackgroundDispatcher(
                execution,
                RunServer,
                execution,
                ThreadFactory);
        }

在CreateDispatcher方法里面返回了一个BackgroundDispatcher，字面意思好像是后台分发器，并且指定了回调runserver，BackgroundDispatcher实现了IBackgroundDispatcher接口，我们先看下它们的定义。

// IBackgroundDispatcher
public interface IBackgroundDispatcher : IDisposable
    {
        bool Wait(TimeSpan timeout);
        Task WaitAsync(TimeSpan timeout, CancellationToken cancellationToken);
    }
 
// BackgroundDispatcher
internal sealed class BackgroundDispatcher : IBackgroundDispatcher
    {
        // 其他成员
        public BackgroundDispatcher(
            [NotNull] IBackgroundExecution execution,
            [NotNull] Action<Guid, object> action,
            [CanBeNull] object state,
            [NotNull] Func<ThreadStart, IEnumerable<Thread>> threadFactory)
        {
            if (threadFactory == null) throw new  ArgumentNullException(nameof(threadFactory));
            _execution = execution ?? throw new  ArgumentNullException(nameof(execution));
            _action = action ?? throw new ArgumentNullException(nameof(action));
            _state = state;
#if !NETSTANDARD1_3
            AppDomainUnloadMonitor.EnsureInitialized();
#endif
            var threads = threadFactory(DispatchLoop)?.ToArray();
            if (threads == null || threads.Length == 0)
            {
                throw new ArgumentException("At least one unstarted thread should be  created.", nameof(threadFactory));
            }
            if (threads.Any(thread => thread == null || (thread.ThreadState &  ThreadState.Unstarted) == 0))
            {
                throw new ArgumentException("All the threads should be non-null and  in the ThreadState.Unstarted state.", nameof(threadFactory));
            }
            _stopped = new CountdownEvent(threads.Length);
            foreach (var thread in threads)
            {
                thread.Start();
            }
        }
        public bool Wait(TimeSpan timeout)
        {
            return _stopped.WaitHandle.WaitOne(timeout);
        }
        public async Task WaitAsync(TimeSpan timeout, CancellationToken  cancellationToken)
        {
            await _stopped.WaitHandle.WaitOneAsync(timeout,  cancellationToken).ConfigureAwait(false);
        }
        public void Dispose()
        {
        }
        public override string ToString()
        {
        }
        private void DispatchLoop()
        {
            try
            {
                _execution.Run(_action, _state);
            }
            catch (Exception ex)
            {
 
            }
            finally
            {
                try
                {
                    _stopped.Signal();
                }
                catch (ObjectDisposedException)
                {
 
                }
            }
        }
    }

从IBackgroundDispatcher接口的定义来看，分发器应该是负责协调资源处理，我们具体看看BackgroundDispatcher的实现。以上代码就是server的启动执行核心代码并且我以加粗，其实就是启动线程Loop执行。在DispatchLoop方法里面间接调用了我上面说的runserver方法。在runserver方法里面实现了整个server端的初始化工作。我们接着看DispatchLoop方法的实现 ，在这个方法里面调用了IBackgroundExecution接口的run方法，继续IBackgroundExecution接口的定义。

public interface IBackgroundExecution : IDisposable
    {
        void Run([NotNull] Action<Guid, object> callback, [CanBeNull] object  state);
        Task RunAsync([NotNull] Func<Guid, object, Task> callback, [CanBeNull]  object state);
    }

就两方法，run包含同步和异步，看看它的唯一实现类BackgroundExecution。

  internal sealed class BackgroundExecution : IBackgroundExecution
    {
                // 其他成员
        public void Run(Action<Guid, object> callback, object state)
        {
            if (callback == null) throw new ArgumentNullException(nameof(callback));
            var executionId = Guid.NewGuid();
           
            {
#if !NETSTANDARD1_3
                try
#endif
                {
                    HandleStarted(executionId, out var nextDelay);
                    while (true)
                    {
                        // Don't place anything here.
                        try
                        {
                           
                            if (StopRequested) break;
                            if (nextDelay > TimeSpan.Zero)
                            {
                                HandleDelay(executionId, nextDelay);
                            }
                            callback(executionId, state);
                            HandleSuccess(out nextDelay);
                        }
#if !NETSTANDARD1_3
                        catch (ThreadAbortException) when  (AppDomainUnloadMonitor.IsUnloading)
                        {
                            // Our thread is aborted due to AppDomain unload. It's  better to give up to
                            // not to cause the host to be more aggressive.
                            throw;
                        }
#endif
                        catch (OperationCanceledException) when (StopRequested)
                        {
                            break;
                        }
                        catch (Exception ex)
                        {
                            HandleException(executionId, ex, out nextDelay);
                        }
                    }
                    HandleStop(executionId);
                }
#if !NETSTANDARD1_3
                catch (ThreadAbortException ex)
                {
                    HandleThreadAbort(executionId, ex);
                }
#endif
            }
        }
}

hangfire里面所有的独立线程都是通过run方法执行，然后回调到自己的实现类Execute方法，自此每个独立的功能线程就循环干着自己独立的工作(这个后面会单独分析RecurringJobScheduler)。继续我们的主线，server启动，我们以run的同步方法为例，第一个线程（我们就叫它主线程吧）启动了一个while循环，在循环里面并且callback调用了我们的runserver方法。

    

private void RunServer(Guid executionId, object state)
        {
            _process.Execute(executionId, (BackgroundExecution)state,  _stoppingCts.Token, _stoppedCts.Token, _shutdownCts.Token);
        }

在runserver方法里面的实现很简单，直接调用了_process的execute方法，我们简单看下_process类型IBackgroundServerProcess的定义。

internal interface IBackgroundServerProcess
    {
        void Execute(
            Guid executionId,
            BackgroundExecution execution,
            CancellationToken stoppingToken,
            CancellationToken stoppedToken,
            CancellationToken shutdownToken);
    }

IBackgroundServerProcess的定义就一个execute方法，这个接口的工作其实就是初始化server服务端，我们看看它的唯一实现类BackgroundServerProcess。

internal sealed class BackgroundServerProcess : IBackgroundServerProcess
    {
        
        // 其他成员
        public BackgroundServerProcess(
            [NotNull] JobStorage storage,
            [NotNull] IEnumerable<IBackgroundProcessDispatcherBuilder> dispatcherBuilders,
            [NotNull] BackgroundProcessingServerOptions options,
            [NotNull] IDictionary<string, object> properties)
        {
            if (dispatcherBuilders == null) throw new ArgumentNullException(nameof(dispatcherBuilders));
 
 
            _storage = storage ?? throw new ArgumentNullException(nameof(storage));
            _options = options ?? throw new ArgumentNullException(nameof(options));
            _properties = properties ?? throw new ArgumentNullException(nameof(properties));
 
 
            var builders = new List<IBackgroundProcessDispatcherBuilder>();
            builders.AddRange(GetRequiredProcesses()); // 添加默认的工作dispatcher也就是独立线程
            builders.AddRange(GetStorageComponents());
            builders.AddRange(dispatcherBuilders);
 
 
            _dispatcherBuilders = builders.ToArray();
        }
 
 
        public void Execute(Guid executionId, BackgroundExecution execution, CancellationToken stoppingToken,
            CancellationToken stoppedToken, CancellationToken shutdownToken)  // server初始化
        {
            var serverId = GetServerId();
            Stopwatch stoppedAt = null;
 
 
            void HandleRestartSignal()
            {
                if (!stoppingToken.IsCancellationRequested)
                {
                    _logger.Info($"{GetServerTemplate(serverId)} caught restart signal...");
                }
            }
            using (var restartCts = new CancellationTokenSource())
            using (var restartStoppingCts = CancellationTokenSource.CreateLinkedTokenSource(stoppingToken, restartCts.Token))
            using (var restartStoppedCts = CancellationTokenSource.CreateLinkedTokenSource(stoppedToken, restartCts.Token))
            using (var restartShutdownCts = CancellationTokenSource.CreateLinkedTokenSource(shutdownToken, restartCts.Token))
            using (restartStoppingCts.Token.Register(HandleStopRestartSignal))
            using (stoppingToken.Register(HandleStoppingSignal))
            using (stoppedToken.Register(HandleStoppedSignal))
            using (shutdownToken.Register(HandleShutdownSignal))
            using (restartCts.Token.Register(HandleRestartSignal))
            {
                var context = new BackgroundServerContext(
                    serverId,
                    _storage,
                    _properties,
                    restartStoppingCts.Token,
                    restartStoppedCts.Token,
                    restartShutdownCts.Token);
                var dispatchers = new List<IBackgroundDispatcher>();
                CreateServer(context);
                try
                {
                    // ReSharper disable once AccessToDisposedClosure
                    using (var heartbeat = CreateHeartbeatProcess(context, () => restartCts.Cancel())) // 创建守护线程
                    {
                        StartDispatchers(context, dispatchers); // 启动hangfire默认初始化的所有独立任务线程
                        execution.NotifySucceeded();
                        WaitForDispatchers(context, dispatchers);
 
 
                        restartCts.Cancel();
 
                        heartbeat.WaitAsync(Timeout.InfiniteTimeSpan, shutdownToken).GetAwaiter().GetResult();
                    }
                }
                finally
                {
                    DisposeDispatchers(dispatchers);
                    ServerDelete(context, stoppedAt);
                }
            }
        }
 
 
        private IBackgroundDispatcher CreateHeartbeatProcess(BackgroundServerContext context, Action requestRestart) // 创建守护线程
        {
            return new ServerHeartbeatProcess(_options.HeartbeatInterval, _options.ServerTimeout, requestRestart)
                .UseBackgroundPool(threadCount: 1)
                .Create(context, _options);
        }
 
 
        private IEnumerable<IBackgroundProcessDispatcherBuilder> GetRequiredProcesses() // 初始化日志和任务监控线程
        {
            yield return new ServerWatchdog(_options.ServerCheckInterval, _options.ServerTimeout).UseBackgroundPool(threadCount: 1);
            yield return new ServerJobCancellationWatcher(_options.CancellationCheckInterval).UseBackgroundPool(threadCount: 1);
        }
        private string GetServerId() // 获取serverid
        {
            var serverName = _options.ServerName
                 ?? Environment.GetEnvironmentVariable("COMPUTERNAME")
                 ?? Environment.GetEnvironmentVariable("HOSTNAME");
            var guid = Guid.NewGuid().ToString();
 
            return !String.IsNullOrWhiteSpace(serverName) ? $"{serverName.ToLowerInvariant()}:{guid}" : guid;
        }
 
        
        private void CreateServer(BackgroundServerContext context) // 创建server，写入Server数据表
        {
            var stopwatch = Stopwatch.StartNew();
            using (var connection = _storage.GetConnection())
            {
                connection.AnnounceServer(context.ServerId, GetServerContext(_properties));
            }
            stopwatch.Stop();
 
 
            ServerJobCancellationToken.AddServer(context.ServerId);
            _logger.Info($"{GetServerTemplate(context.ServerId)} successfully announced in {stopwatch.Elapsed.TotalMilliseconds} ms");
        }
 
 
        private void StartDispatchers(BackgroundServerContext context, ICollection<IBackgroundDispatcher> dispatchers) // 启动所有独立的任务线程，包括我们的队列计划、循环计划、日志、守护等等线程
        {
 
            foreach (var dispatcherBuilder in _dispatcherBuilders)
            {
                dispatchers.Add(dispatcherBuilder.Create(context, _options));
            }
        }
 
    }

以上代码我有做精简处理，不要纠结里面的实现，代码注释也比较详细。下面我做一个简单的总结吧，第一个线程（暂时叫主线程吧）从startup里面调用usehangfireserver扩展方法-》启动一个新的worker线程用于初始化&启动server-》主程返回-》启动hangfire所有任务线程-》创建的第一个worker线程挂起（用于处理所有任务线程的资源释放）。server的初始化工作大概就是这些，下面详细看看hangfire的任务线程的执行原理，这里我们以RecurringJobScheduler循环任务为例。

 

RecurringJobScheduler实现机制

还记得上面提到的7个dispatcher任务线程的创建吗？这7个默认的任务线程初始化就发生在上面加粗的代码里面StartDispatchers方法，我们看代码。

private void StartDispatchers(BackgroundServerContext context,  ICollection<IBackgroundDispatcher> dispatchers)
        {
               // 其他代码...
            foreach (var dispatcherBuilder in _dispatcherBuilders)
            {
                dispatchers.Add(dispatcherBuilder.Create(context, _options)); // 初始化独立任务线程
            }
        }

遍历_dispatcherBuilders数组，7种任务类型，分别调用它们的Create方法。继续看create方法。

   

 public IBackgroundDispatcher Create(BackgroundServerContext context,  BackgroundProcessingServerOptions options) // 第一步
        {
            // 其他代码
            var execution = new BackgroundExecution(
                context.StoppingToken,
                new BackgroundExecutionOptions
                {
                    Name = _process.GetType().Name,
                    RetryDelay = options.RetryDelay
                }); // 定义自己的execution
            return new BackgroundDispatcher( // 创建BackgroundDispatcher 
                execution,
                ExecuteProcess, // 指定回调
                Tuple.Create(_process, context, execution), // 创建三元组上下文，注意一下1元组这个对象
                _threadFactory);
        }
 
public BackgroundDispatcher(  // 第二步
            [NotNull] IBackgroundExecution execution,
            [NotNull] Action<Guid, object> action,
            [CanBeNull] object state,
            [NotNull] Func<ThreadStart, IEnumerable<Thread>> threadFactory)
        {
   
            _state = state;
 
            var threads = threadFactory(DispatchLoop)?.ToArray();
           
            foreach (var thread in threads)
            {
                thread.Start(); // 执行线程
            }
        }
 
private void DispatchLoop() // 第三步
        {
            try
            {
                _execution.Run(_action, _state);  // 在run里面回调_action
            }
            catch (Exception ex)
            {
            }
            finally
            {
                try
                {
                    _stopped.Signal();
                }
                catch (ObjectDisposedException)
                {
                }
            }
        }
 
private static void ExecuteProcess(Guid executionId, object state) // 第四步 回调方法，对应上面的指定回调
        {
            var tuple = (Tuple<IBackgroundProcess, BackgroundServerContext,  BackgroundExecution>)state;
            var serverContext = tuple.Item2;
            var context = new BackgroundProcessContext( // 创建公共上下文
                serverContext.ServerId,
                serverContext.Storage,
                serverContext.Properties.ToDictionary(x => x.Key, x => x.Value),
                executionId,
                serverContext.StoppingToken,
                serverContext.StoppedToken,
                serverContext.ShutdownToken);
            while (!context.IsStopping)
            {
                tuple.Item1.Execute(context); // 执行自己元组对应的实例
                tuple.Item3.NotifySucceeded();
            }
        }

上面有点乱啊，我大概简单串起来说一下。第一步在create方法里面创建了BackgroundDispatcher并指定了元组参数-》第二步绑定线程的执行函数Loop并且执行-》第三步执行Loop并且回调_action委托-》第四步_action参数对应的函数地址就是ExecuteProcess，最后在ExecuteProcess里面通过元组参数调用对应的任务类型，自此7种任务类型启动并开始工作。以上代码还有个细节需要说明一下，Tuple.Create(_process, context, execution)。元组的第一个参数，其类型为IBackgroundProcess，看下定义。

public interface IBackgroundProcess : IServerProcess
    {
        void Execute([NotNull] BackgroundProcessContext context);
    }

接口就定义了一个方法，没什么特别的，但是它的几个实现类就是我们单独的任务类，我们下面要说的RecurringJobScheduler循环任务类也实现了这个接口。到此我们的RecurringJobScheduler循环定时任务线程就算开始执行了。

RecurringJobScheduler循环定时任务机制

照旧看下这个类型的定义

public class RecurringJobScheduler : IBackgroundProcess
    {
        // 其他代码
        public RecurringJobScheduler(
            [NotNull] IBackgroundJobFactory factory,
            TimeSpan pollingDelay,
            [NotNull] ITimeZoneResolver timeZoneResolver,
            [NotNull] Func<DateTime> nowFactory)
        {
            if (factory == null) throw new ArgumentNullException(nameof(factory));
            if (nowFactory == null) throw new ArgumentNullException(nameof(nowFactory));
            if (timeZoneResolver == null) throw new ArgumentNullException(nameof(timeZoneResolver));
 
 
            _factory = factory;
            _nowFactory = nowFactory;
            _timeZoneResolver = timeZoneResolver;
            _pollingDelay = pollingDelay;
            _profiler = new SlowLogProfiler(_logger);
        }
 
 
        /// <inheritdoc />
        public void Execute(BackgroundProcessContext context) // 实现方法
        {
            if (context == null) throw new ArgumentNullException(nameof(context));
 
 
            var jobsEnqueued = 0;
 
 
            while (EnqueueNextRecurringJobs(context)) // 从数据库获取定时任务
            {
                jobsEnqueued++;
 
 
                if (context.IsStopping)
                {
                    break;
                }
            }
 
 
            if (jobsEnqueued != 0)
            {
                _logger.Debug($"{jobsEnqueued} recurring job(s) enqueued.");
            }
 
 
            if (_pollingDelay > TimeSpan.Zero)
            {
                context.Wait(_pollingDelay);
            }
            else
            {
                var now = _nowFactory();
                context.Wait(now.AddMilliseconds(-now.Millisecond).AddSeconds(-now.Second).AddMinutes(1) - now);
            }
        }
    }

承上，调用元组的第一个参数的execute方法，RecurringJobScheduler的execute方法得以执行，该方法就干一件事，每隔15秒从数据库获取待执行的计划，每次1000条数据。通过EnqueueNextRecurringJobs方法获取任务。

private bool EnqueueNextRecurringJobs(BackgroundProcessContext context)
        {
            return UseConnectionDistributedLock(context.Storage, connection => 
            {
                var result = false;
                if (IsBatchingAvailable(connection))
                {
                    var now = _nowFactory();
                    var timestamp = JobHelper.ToTimestamp(now);
                    var recurringJobIds =  ((JobStorageConnection)connection).GetFirstByLowestScoreFromSet("recurring-jobs", 0,  timestamp, BatchSize); // 从数据库里面查询
                    if (recurringJobIds == null || recurringJobIds.Count == 0) return  false;
                    foreach (var recurringJobId in recurringJobIds)
                    {
                        if (context.IsStopping) return false;
                        if (TryEnqueueBackgroundJob(context, connection, recurringJobId,  now))// 排队执行
                        {
                            result = true;
                        }
                    }
                }
                else
                {
                    for (var i = 0; i < BatchSize; i++)
                    {
                        if (context.IsStopping) return false;
                        var now = _nowFactory();
                        var timestamp = JobHelper.ToTimestamp(now);
                        var recurringJobId =  connection.GetFirstByLowestScoreFromSet("recurring-jobs", 0, timestamp);
                        if (recurringJobId == null) return false;
                        if (!TryEnqueueBackgroundJob(context, connection, recurringJobId,  now))
                        {
                            return false;
                        }
                    }
                }
                return result;
            });
        }

GetFirstByLowestScoreFromSet方法从数据库Set表里面查询top1000数据，条件是key为recurring-jobs字符串（表示定时任务）并且 时间范围是0到当前时间。随后遍历这些jobids，排队执行，往下看TryEnqueueBackgroundJob方法的实现。

private bool EnqueueBackgroundJob(
            BackgroundProcessContext context,
            IStorageConnection connection,
            string recurringJobId,
            DateTime now)
        {
            // 其他代码
            using (connection.AcquireDistributedRecurringJobLock(recurringJobId,  LockTimeout))
            {
                try
                {
                    var recurringJob = connection.GetRecurringJob(recurringJobId,  _timeZoneResolver, now);
                    if (recurringJob == null)
                    {
                        using (var transaction = connection.CreateWriteTransaction())
                        {
                            transaction.RemoveFromSet("recurring-jobs", recurringJobId);
                            transaction.Commit();
                        }
                        return false;
                    }
          
                    BackgroundJob backgroundJob = null;
                    IReadOnlyDictionary<string, string> changedFields;
                    if (recurringJob.TrySchedule(out var nextExecution, out var error))
                    {
                        if (nextExecution.HasValue && nextExecution <= now)
                        {
                            backgroundJob = _factory.TriggerRecurringJob(context.Storage,  connection, _profiler, recurringJob, now);
                            if (String.IsNullOrEmpty(backgroundJob?.Id))
                            {
                                _logger.Debug($"Recurring job '{recurringJobId}' execution  at '{nextExecution}' has been canceled.");
                            }
                        }
                        recurringJob.IsChanged(out changedFields, out nextExecution);
                    }
                    else if (recurringJob.RetryAttempt < MaxRetryAttemptCount)
                    {
                        var delay = _pollingDelay > TimeSpan.Zero ? _pollingDelay :  TimeSpan.FromMinutes(1);
                        
                        _logger.WarnException($"Recurring job '{recurringJobId}' can't be  scheduled due to an error and will be retried in {delay}.", error);
                        recurringJob.ScheduleRetry(delay, out changedFields, out  nextExecution);
                    }
                    else
                    {
                        _logger.ErrorException($"Recurring job '{recurringJobId}' can't be  scheduled due to an error and will be disabled.", error);
                        recurringJob.Disable(error, out changedFields, out nextExecution);
                    }
              
                    using (var transaction = connection.CreateWriteTransaction())
                    {
                        if (backgroundJob != null)
                        {
                            _factory.StateMachine.EnqueueBackgroundJob(
                                context.Storage,
                                connection,
                                transaction,
                                recurringJob,
                                backgroundJob,
                                "Triggered by recurring job scheduler",
                                _profiler);
                        }
                        transaction.UpdateRecurringJob(recurringJob, changedFields,  nextExecution, _logger);
                        transaction.Commit();
                        return true;
                    }
                }
                catch (TimeZoneNotFoundException ex)
                {
                catch (Exception ex)
                {
   
                }
                return false;
            }
        }

需要注意的地方我都有加粗，该方法大概流程是：1.GetRecurringJob根据jobid从Hash表里面查询一条完整的定时任务，2.TrySchedule获取该任务的下次执行时间，如果下次执行时间小于当前，执行这条任务（并非真正执行定时任务，只是往job表里面写数据，真正执行任务由worker完成），3.获取下次执行时间&所有任务字段，4.状态机修改任务状态。定时任务就这样周而复始的重复执行以上流程。这里简单说下worker的执行机制，其实际就是轮询检索job表里面的数据执行任务表达式树，worker在hangfire里面默认开启了20个线程。第三部分就到这吧。

 

避坑

简单说下个人在改bug期间遇到的一些问题啊。

1.时区问题，在添加定时任务时如果不指定时区信息，默认使用的是utc时间，我们中国是东8区，也就是说解析出来的执行时间会晚8个小时执行。解决办法有几种可以通过全局指定options的ITimeZoneResolver属性指定，也可以通过AddorUpdate方法指定，如果是指定时区信息，需要注意看板上面的异常信息，如果有异常会导致任务不执行，时区信息它是从系统里面检索出来的，没有就抛异常。就这样吧。