Quartz.NET 与 AdoNetJobStore 性能 [英] Quartz.NET with AdoNetJobStore performance

问题描述

我有大约 500 万个作业，并且每个作业在 Quartz.Net 中恰好安排了一个触发器，作为计划同时运行的最多大约 30 万个作业，我有一个限制，要在 3 小时内处理所有 30 万个作业(所以大约 10 万个作业)Jobs/Hour)，但现在当 Quartz.Net 配置为使用 AdoNetJobStore 时，我的测试应用每小时只能处理 10K.

I have ~5M Jobs and each Job has exactly one Trigger scheduled in Quartz.Net, as a maximum ~300K Jobs scheduled to run at same time, I have a constraint to proceed all 300K Jobs within 3 hours (so ~100K Jobs/Hour), but now my test app is able to proceed only 10K per hour when Quartz.Net is configured to use AdoNetJobStore.

我正在使用下一个 Quartz 配置:

I'm using next Quartz config:

<quartz>
    <add key="quartz.scheduler.instanceName" value="XxxDefaultQuartzScheduler" />
    <add key="quartz.scheduler.instanceId" value="instance_one" />
    <add key="quartz.threadPool.type" value="Quartz.Simpl.SimpleThreadPool, Quartz" />
    <add key="quartz.threadPool.threadCount" value="10" />
    <add key="quartz.threadPool.threadPriority" value="1" />
    <add key="quartz.jobStore.type" value="Quartz.Impl.AdoJobStore.JobStoreTX, Quartz" />
    <add key="quartz.jobStore.misfireThreshold" value="60000" />
    <add key="quartz.jobStore.dataSource" value="default" />
    <add key="quartz.jobStore.driverDelegateType" value="Quartz.Impl.AdoJobStore.SqlServerDelegate, Quartz" />
    <add key="quartz.jobStore.lockHandler.type" value="Quartz.Impl.AdoJobStore.UpdateLockRowSemaphore, Quartz" />
    <add key="quartz.jobStore.tablePrefix" value="QRTZ_" />
    <add key="quartz.jobStore.useProperties" value="false" />
    <add key="quartz.dataSource.default.connectionStringName" value="QuartzDbContext" />
    <add key="quartz.dataSource.default.provider" value="SqlServer-20" />
</quartz>

是否可以通过 SQL Job Store 配置 Quartz.Net 以提供这样的性能?

Is it possible to configure Quartz.Net with SQL Job Store to provide such performance?

推荐答案

对于这种大量使用，默认线程数 10 是相当低的.我会将它增加到诸如计算机中的 25 个内核之类的东西.您也可以自然地测试更高的值.现代硬件应该能够处理 100-200 个线程，但线程争用可能会成为一个问题.

The default thread count of 10 is quite low for this kind of massive usage. I'd increment it to something like, say, 25 x cores in computer. You can naturally test higher values too. Modern hardware should be able to handle 100-200 threads, but thread contention may come as an issue.

您没有提到您正在使用的版本，但是对于较新的版本，您可以删除 quartz.jobStore.lockHandler.type ，它会在检测到使用 SQL Server 时使用优化的行锁定.始终使用最新版本(如果可能)，最新版本往往是最优化且无错误的.

You didn't mention the version you are using, but with newer versions you can drop the quartz.jobStore.lockHandler.type , it's use optimized row locking when it detects that SQL Server is used. Always use the latest version (when possible), latest tends to be most optimized and bug-free.

在调整 Quartz 配置之后，您自然必须分析自己的工作代码.假设您的代码能够在 5 秒内完成，那么我们大约有:

After Quartz configuration is tweaked, you naturally must profile your own job code. Say your code would be able to complete in 5 seconds so what we would approximately have is:

5 秒 * 300 000 次触发/100 个线程 => 15 000 秒 ~ 250 分钟 ~ 4 小时

将这 5 秒的执行时间降低到 3 秒，您将达到 2 小时 30 米大关.或者只是增加线程数量并为一些资源之战做好准备 - 取决于您的工作正在做什么以及如何做.

Lower that five seconds of execution time to three and you'll be hitting the 2h 30m mark. Or just raise the amount of threads and be prepared for some battle of resources - depends on what your jobs are doing and how.

另一个考虑因素可能是集群，如果数据库不是瓶颈，则针对 Quartz 数据库运行多个工作程序.

Another consideration could be clustering, run multiple workers against Quartz database if the database isn't the bottleneck.

这篇关于Quartz.NET 与 AdoNetJobStore 性能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！