如何在多个节点上的云（AWS，heroku等）中建立时钟同步？ [英] How do I establish clock synchronization in the cloud (AWS, heroku, etc) across many nodes?
问题描述
我想在云中运行大型节点群集（AWS，Heroku或可能是自我管理的VMS），其时钟必须与预定义的容差同步。我正在寻找一个容差大概200毫秒。这意味着如果我有250个节点，任何250个节点之间的最大时钟差别不应该超过200毫秒。我真的不在乎关于世界的实际日期/时间。解决方案必须是容错的，并且不需要依赖任何一个系统的时钟的精度 - 事实上，很可能没有一个时钟将是非常准确的。
要求是足够强大的，如果由于任何原因，时钟同步被确定为不可靠的任何特定的节点，我喜欢从一个节点由于时钟失去同步 - 所以任何怀疑失败，我想能够对该节点执行某种类型的受控关闭。
我想使用类似NTP的东西，但根据NTP 已知问题twiki ：
NTP不是设计为在虚拟机内部运行。
它需要一个高分辨率的系统时钟，响应时间为时钟中断
，以高精度服务。没有已知的虚拟机是
能够满足这些要求。
虽然同样的twiki然后去描述各种方式解决这种情况（如在主机操作系统上运行ntp），我不相信我将有足够的能力使用AWS或在horoku上修改环境遵守解决方法。
即使我没有在VM中运行，一个信任的运营经理，有多年的运行ntp的经验告诉我，ntp可以和将丢弃同步（或明显地得到的时间错误）由于本地时钟漂移每偶尔。它不会经常发生，但它确实发生，当你增加机器，你增加这种情况发生的机会。 AFAIK，检测你离你多远需要停止ntpd，运行查询模式命令，并重新启动它，可能需要很长时间才能得到回答。
总结一下，我需要一个时钟同步，其主要目标如下：
- 在VM中运行良好，其中操作控制
- 所有参与者之间的集群中的时间容差在200ms左右
- 能够检测不良节点并做出反应
- 可扩展性（当您添加更多内容时，该功能不会崩溃）节点 - 绝对避免n ^ 2）
- 可以支持数百个节点
- 所有节点都不应被视为具有超越任何其他节点
- 整个群集漂移（在原因范围内） - 只要它一致漂移即可。
根据说明， Berkeley算法似乎是正确的选择，但是它已经实现？
很适合：
- 最小配置（节点自动注册参与） - 对于启动新节点很重要
- 报告参与时钟同步的节点的HTML信息中心或（REST？）API以及相对时间偏移量
- 漂亮的图形？
大多数机器都有一个RTC（实时时钟），在PC上它如何存储时间，以便你有一个'粗略的猜测，如果ntp不可用的时间，一旦系统加载了一个'tick'时钟是更高的分辨率 - 这是NTP设置。
该刻度时钟受虚拟机的漂移影响，因为刻度可能或可能不会以正确的间隔发生 - 您尝试使用的任何时间机制将是
如果机器A和B的增量为200毫秒，而机器B和C的虚拟机的虚拟机的虚拟机有一个200毫秒的增量，C可能400毫米远离A.你不能控制。
您最好使用像zeromq这样的集中式消息系统来保持每个人都与作业队列同步，这将是更多的开销，但依赖于系统滴答时间是一个诡异的事情。有许多集群解决方案使用各种可靠的机制来解决集群参与问题，以确保每个人都保持同步，查看corosync或spread - 他们已经为两阶段提交等问题解决了这一问题。
顺便说一句，ntp'放弃'当漂移太高可以通过指示它砰的时间到新的值，而不是转换规避。默认情况下，ntp将递增地更新系统时间，以说明其从实时的漂移。我忘记了如何在ntpd中配置这个，但如果你使用ntpdate的标志是-B
-B强制时间总是即使测量的
偏移量大于+ -128毫秒，也可以使用adjtime（2）系统调用来回转。如果偏移
大于+ -128 ms，则默认值为使用settimeofday（2）来计算时间。请注意，如果在这种情况下偏移远大于+ -128 ms，则
可能需要很长时间（小时）才能将时钟转换为正确的值。在此期间，主机
不应用于同步客户端。
I would like to run a large cluster of nodes in the cloud (AWS, Heroku, or maybe self-manged VMS), whose clocks must be synchronized with a predefined tolerance in mind. I'm looking for a tolerance of maybe 200 ms. That means if I have 250 nodes, the largest clock difference between any of the 250 nodes should never exceed 200 ms. I don't really care about the actual date / time with respect to the world. The solution has to be fault tolerant, and should not need to rely on the accuracy of the clock of any one system -- in fact, it's likely that none of the clocks will be terribly accurate.
The requirement is strong enough where if for any reason the clock synchronization is determined to be unreliable for any particular node, that I'd prefer to drop a node from the cluster due to clock desynchronization -- so on any suspected failure, I'd like to be able to perform some type of controlled shutdown of that node.
I'd love to use something like NTP, but according to the NTP known issues twiki:
NTP was not designed to run inside of a virtual machine. It requires a high resolution system clock, with response times to clock interrupts that are serviced with a high level of accuracy. No known virtual machine is capable of meeting these requirements.
And although the same twiki then goes to describe various ways of addressing the situation (such as running ntp on the host OS), I don't believe I'll have the ability to modify the environment enough using AWS or on horoku to comply with the workarounds.
Even if I was not running in VM's, a trusted operations manager who has years of experience running ntp tells me that ntp can and will drop synchronization (or plain get the time wrong) due to bad local clock drift every once in a while. It doesn't happen often, but it does happen, and as you increase machines, you increase your chances of this happening. AFAIK, detecting how far off you are requires stopping ntpd, running a query mode command, and starting it back up again, and it can take a long time to get an answer back.
To sum up -- I need a clock synchronization whose primary goal is as follows:
- Runs well in VM's where operational control is limited (ie: "cloud service providers")
- Time tolerances in the cluster at around 200ms between all participants
- Ability to detect bad node and react to that in an active way
- Fault tolerant (no single point of failure)
- Scalable (the thing can't fall over when you add more nodes -- definitely avoid n^2)
- Could support hundreds of nodes
- None of the nodes should be considered having superior notion of time over any other node
- It's OK for the entire cluster to drift (within reason) -- as long as it drifts in unison
From the description, it seems like the Berkeley Algorithm might be the right choice here, but is it already implemented?
Nice to haves:
- Minimal configuration (nodes auto register to participate) -- important for spinning up new nodes
- HTML dashboard or (REST?) API that reports the nodes that are participating in the clock synchronization and what the relative time offsets are
- Pretty graphs?
Since the FAQ for NTP specifically states why NTP time sync doesn't work 'right' under virtual machines, it's probably an insurmountable problem.
Most machines have a RTC (real-time clock) in them, on PCs its how you store the time so that you have a 'rough' guess as to what the time is if ntp is unavailable, once the system is loaded there's a 'tick' clock that is higher resolution - thats what NTP sets.
That tick clock is subject to the drift of the virtual machine since ticks may or may not happen at the correct intervals - any time mechanism you attempt to use is going to be subject to that drift.
It's probably suboptimal design to try to enforce ntp synchronization on virtual machines, if machine A and B have a delta of 200ms, and machine B and C have a delta of 200ms, C could 400ms away from A. You can't control that.
You're better off using a centralized messaging system like zeromq to keep everybody in sync with the job queue, it's going to be more overhead, but relying on system tick time is a dodgy affair at best. There are many clustering solutions that account for cluster participation using all sorts of reliable mechanisms to ensure that everyone is in sync, take a look at corosync or spread - they've solved this already for things like two-phase-commits.
Incidentally, ntp 'giving up' when drift is too high can be circumvented by instructing it to 'slam' the time to the new value rather than 'slew'. By default ntp will incrementally update the system time to account for its drift from 'real time'. I forget how to configure this in ntpd, but if you use ntpdate the flag is -B
-B Force the time to always be slewed using the adjtime(2) system call, even if the measured
offset is greater than +-128 ms. The default is to step the time using settimeofday(2) if the offset
is greater than +-128 ms. Note that, if the offset is much greater than +-128 ms in this case, it
can take a long time (hours) to slew the clock to the correct value. During this time, the host
should not be used to synchronize clients.
这篇关于如何在多个节点上的云（AWS，heroku等）中建立时钟同步？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
