如何防止 Hangfire 重复作业在连续执行 30 分钟后重新启动 [英] How to prevent a Hangfire recurring job from restarting after 30 minutes of continuous execution

问题描述

我正在开发一个 asp.net mvc-5 Web 应用程序，但在使用 Hangfire 工具运行长时间运行的后台作业时遇到了问题.问题是，如果作业执行时间超过 30 分钟，那么hangfire 会自动启动另一个作业，所以我最终会同时运行两个类似的作业.

I am working on an asp.net mvc-5 web application, and I am facing a problem in using Hangfire tool to run long running background jobs. the problem is that if the job execution exceed 30 minutes, then hangfire will automatically initiate another job, so I will end up having two similar jobs running at the same time.

现在我有以下内容:-

1. Asp.net mvc-5
2. IIS-8
3. Hangfire 1.4.6
4. Windows 服务器 2012

现在我已经定义了一个在每天 17:00 运行的 hangfire 循环作业.后台作业主要扫描我们的网络寻找服务器和虚拟机并更新数据库，循环作业在完成执行后会发送一封电子邮件.当执行时间少于 30 分钟时，重复性作业过去运行良好.但是今天随着我们系统的发展，重复性工作在 40 分钟后完成，而不是过去的 22-25 分钟.我收到了 2 封电子邮件而不是一封电子邮件(电子邮件之间的时间约为 30 分钟).现在我手动重新运行作业，我注意到问题如下:-

Now I have defined a hangfire recurring job to run at 17:00 each day. The background job mainly scan our network for servers and vms and update the DB, and the recurring job will send an email after completing the execution. The recurring job used to work well when its execution was less than 30 minutes. But today as our system grows, the recurring job completed after 40 minutes instead of 22-25 minutes as it used to be. and I received 2 emails instead of one email (and the time between the emails was around 30 minutes). Now I re-run the job manually and I have noted that that the problem is as follow:-

"当循环作业达到连续执行 30 分钟时，一个循环作业的新实例将开始，所以我将有两个多个实例而不是同时运行一个，所以我收到了 2 封电子邮件."

"when the recurring job reaches 30 minutes of continuous execution, a new instance of the recurring job will start, so I will have two instances instead of one running at the same time, so that why I received 2 emails."

现在，如果重复作业的时间少于 30 分钟(例如 29 分钟)，我将不会遇到任何问题，但如果重复作业的执行时间超过 30 分钟，那么出于某种原因或其他挂起，将启动新作业.尽管在作业执行期间访问 hangfire 仪表板时，我可以发现只有一个活动作业，但是当我监视我们的数据库时，我可以从 sql 探查器中看到有两个作业正在访问数据库.这发生在循环作业开始 30 分钟后(在我们的例子中是 17:30)，这就是为什么我收到了 2 封电子邮件，这意味着 2 个循环作业在后台运行，而不是一个.

Now if the recurring job takes less than 30 minutes (for example 29 minute) I will not face any problem, but if the recurring job execution exceeds 30 minutes then for a reason or another hangfire will initiate a new job. although when I access the hangfire dashboard during the execution of the job, I can find that there is only one active job, when I monitor our DB I can see from the sql profiler that there are two jobs accessing the DB. this happens after 30 minutes from the beginning of the recurring job (at 17:30 in our case), and that why I received 2 emails which mean 2 recurring jobs were running in the background instead of one.

那么任何人都可以就此提出建议，如果当前的重复作业执行时间超过 30 分钟，我如何避免 hangfire 自动启动新的重复作业?谢谢

So can anyone advice on this please, how I can avoid hangfire from automatically initiating a new recurring job if the current recurring job execution exceeds 30 minutes? Thanks

推荐答案

您是否查看了 Hangfire 文档?

Did you look at InvisibilityTimeout setting from the Hangfire docs?

默认的 SQL Server 作业存储实现使用常规表作为作业队列.确保在以下情况下不会丢失工作意外的进程终止，仅从队列中删除成功完成后.

Default SQL Server job storage implementation uses a regular table as a job queue. To be sure that a job will not be lost in case of unexpected process termination, it is deleted only from a queue only upon a successful completion.

为了使其对其他工作人员不可见，UPDATE 语句带有OUTPUT 子句用于获取排队的作业并更新 FetchedAt值(向其他工作人员发出信号，表明它已被提取)在一个原子方式.其他工作人员看到获取的时间戳并忽略作业.但是为了处理进程终止，他们只会忽略一个工作在指定的时间内(默认为 30 分钟).

To make it invisible from other workers, the UPDATE statement with OUTPUT clause is used to fetch a queued job and update the FetchedAt value (that signals for other workers that it was fetched) in an atomic way. Other workers see the fetched timestamp and ignore a job. But to handle the process termination, they will ignore a job only during a specified amount of time (defaults to 30 minutes).

虽然这个机制保证了每一个作业都会被处理，有时它可能会导致重试延迟过长或导致多个作业执行.考虑以下场景:

Although this mechanism ensures that every job will be processed, sometimes it may cause either long retry latency or lead to multiple job execution. Consider the following scenario:

1. 工人 A 获取了一个作业(运行了一个小时)并在 12:00 开始它.
2. 工人 B 在 12:30 获取了相同的作业，因为默认的隐身超时已过期.
3. 工人 C(未提取)在 13:00 执行相同的作业，因为(它表演成功后将被删除.)

如果您使用取消令牌，它将为工人 A 设置在12:30 和工人 B 的 13:00.这可能会导致您的永远不会执行长时间运行的作业.如果您不使用取消令牌，它将由 WorkerA 和工人 B(从 12:30 开始)，但工人 C 不会去取它，因为它表演成功后将被删除.

If you are using cancellation tokens, it will be set for Worker A at 12:30, and at 13:00 for Worker B. This may lead to the fact that your long-running job will never be executed. If you aren’t using cancellation tokens, it will be concurrently executed by WorkerA and Worker B (since 12:30), but Worker C will not fetch it, because it will be deleted after successful performance.

因此，如果您有长时间运行的作业，最好配置隐身超时间隔:

var options = new SqlServerStorageOptions
{
    InvisibilityTimeout = TimeSpan.FromMinutes(30) // default value
};

GlobalConfiguration.Configuration.UseSqlServerStorage("<name or connection string>", options);

截至 Hangfire 1.5 这个选项现在 Obsolete.其他工人看不到正在从事的工作.

As of Hangfire 1.5 this option is now Obsolete. Jobs that are being worked on are invisible to other workers.

告别将隐身超时与意外混淆使用 SQL 时后台作业在 30 分钟后重试(默认情况下)服务器.新的 Hangfire.SqlServer 实现使用普通的旧获取后台作业并将它们隐藏起来的事务工人.

Say goodbye to confusing invisibility timeout with unexpected background job retries after 30 minutes (by default) when using SQL Server. New Hangfire.SqlServer implementation uses plain old transactions to fetch background jobs and hide them from other workers.

即使在非正常关机后，该作业仍可用于其他工人立即，没有任何延误.

Even after ungraceful shutdown, the job will be available for other workers instantly, without any delays.

这篇关于如何防止 Hangfire 重复作业在连续执行 30 分钟后重新启动的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！