模拟来自 Azure Service Fabric 群集的 10,000 个 Azure IoT 中心设备连接 [英] Simulate 10,000 Azure IoT Hub Device connections from Azure Service Fabric cluster

问题描述

我们正在开发应托管在 Azure Service Fabric 中的 .Net Core 服务.此 SF 服务需要通过其 AMQP 1.0 SSL TLS 端点与在 Azure IoT 中心注册的 10,000 个设备进行交互.每个 IoT 中心设备都有自己的安全令牌和 IoT 中心服务提供的连接字符串.

We are developing a .Net Core service that shall be hosted in Azure Service Fabric. This SF Service needs to interact with 10,000 devices registered in Azure IoT Hub via it's AMQP 1.0 SSL TLS endpoints. Each IoT Hub devices has it's own security tokens and connection string provided by the IoT Hub service.

对于我们的场景，我们需要侦听来自 10,000 个 IoT 中心设备实例的所有云到设备消息，并将这些消息路由"到中心服务总线主题，现场的实际网关"会侦听该主题.所以基本上我们希望将消息从 10,000 个服务总线队列转发到一个中央队列.

For our scenario we need to listen to all cloud-to-devices messages coming from the 10,000 IoT Hub device instances and "route" these to a central Service Bus topic to which the actual "gateways" in the field listen to. So basically we want to forward messages from 10,000 Service Bus Queues into one central Queue.

处理来自 SF 服务的这 10,000 个 AMQP 侦听器的最佳方法是什么?有没有办法重用 AMQP 连接、会话或链接，以便缓存/共享资源?以及如何将连接维护的负载动态分散到SF集群的5个节点上?

What is the best approach to handle these 10,000 AMQP listners from a SF Service? Is there a way we can reuse AMQP connections, sessions or links so we cache/share resources? And how can we dynamically spread the load of connection maintenance over the 5 nodes in the SF cluster?

我们正在评估这些 Nuget 包的实施情况:Microsoft.Azure.ServiceBusAMQPNetLiteMicrosoft.Azure.Devices.Client

We are evaluating these Nuget packages for the implementation: Microsoft.Azure.ServiceBus AMQPNetLite Microsoft.Azure.Devices.Client

我们正在使用 Microsoft.Azure.Devices.Client 库进行一些测试，请参阅下面的简化代码示例:

We are doing some tests using the Microsoft.Azure.Devices.Client lib, see a simplified code sample below:

using System;
using System.Fabric;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Azure.Devices.Client;
using Microsoft.ServiceFabric.Services.Runtime;

namespace ID.Monitoring.MonServer.ServiceFabric.ServiceBus
{
    /// <summary>
    /// An instance of this class is created for each service instance by the Service Fabric runtime.
    /// </summary>
    internal sealed class ServiceBus : StatelessService
    {
        private readonly DeviceClient _deviceClient;
        private ConnectionStatus _status;

        public ServiceBus(StatelessServiceContext context)
            : base(context)
        {
            _deviceClient = DeviceClient.CreateFromConnectionString("HostName=id-monitoring-dev.azure-devices.net;DeviceId=100;SharedAccessSignature=SharedAccessSignature sr=id-monitoring-dev.azure-devices.net%2Fdevices%2F100&sig={token}&se=1553265888", TransportType.Amqp_Tcp_Only);
        }

        /// <summary>
        /// This is the main entry point for your service instance.
        /// </summary>
        /// <param name="cancellationToken">Canceled when Service Fabric needs to shut down this service instance.</param>
        protected override async Task RunAsync(CancellationToken cancellationToken)
        {
            _deviceClient.SetConnectionStatusChangesHandler(ConnectionStatusChangeHandler);

            while (true)
            {
                if (_status != ConnectionStatus.Connected)
                {
                    await _deviceClient.OpenAsync();
                }
                var receivedMessage = await _deviceClient.ReceiveAsync(TimeSpan.FromSeconds(10)).ConfigureAwait(false);

                if (receivedMessage != null)
                {
                    var messageData = Encoding.ASCII.GetString(receivedMessage.GetBytes());
                    //TODO: handle incoming message and publish to common 
                    await _deviceClient.CompleteAsync(receivedMessage).ConfigureAwait(false);
                }
            }
        }

        private void ConnectionStatusChangeHandler(ConnectionStatus status, ConnectionStatusChangeReason reason)
        {
            _status = status;
        }
    }
}

问题:这是否可以很好地扩展到 10,000 个 Service Fabric 服务实例?或者是否有更有效的方法从 Service Fabric 服务环境中维护这么多 AMQP 服务总线侦听器?有没有办法可以应用 AMQP 连接多路复用?

Question: Does this scale well to 10,000 Service Fabric service instances? Or are there more efficient ways to have this many AMQP Service Bus Listners maintained from a Service Fabric Service environment? Is there a way we can apply AMQP connection multiplexing maybe?

推荐答案

您选择的设备监控方法不能很好地扩展并且难以维护.

The approach you choose to monitor your devices won't scale well and will be hard to maintain.

目前，服务结构对实例数量有限制您可以放置​​在单个节点中.例如:如果您使用 ServiceBus 服务创建一个应用程序并跨越 10000 个实例，您将遇到此限制，即节点数.即:如果您有一个 5 节点集群，您将只能使用默认扩展方法运行服务的 5 个实例.

Currently, service fabric has a limitation of how many instances you can place in a single node. For example: if you create an application with your ServiceBus service and span 10000 instances, you will hit this limitation, that is the number of nodes. i.e: if you have a 5 node cluster, you will be able to run only 5 instances of your service by using the default scaling approach.

要绕过此问题，您有一些选择:

To bypass this issue you have some options:

分区:

让单个无状态服务运行更多分区比节点数多，你必须对你的服务进行分区.假设您有一个 5 节点集群并需要 10000 个实例，您将需要在每个节点上运行 2000 个分区.如果您使用共享进程并且有足够的内存，这种方法可能对您有所帮助，请查看 此主题 和此主题在遵循这种方法之前

To have a single stateless service running more partitions than the node count, you have to partition your service. Assuming you have a 5 node cluster and need 10000 instances, you will need 2000 partitions running on each node. If you use shared process and have enough ram to this, this approach might help you, please take a look at this thread and this thread before following this approach

多命名服务:

命名服务是一种服务类型的运行服务定义，在这种情况下，您将为每个设备创建一个.喜欢:

Named service is the running service definition for one service type, in this case you would create one per device. like:

• 服务总线类型
  • ServiceBus-Device1
  • ServiceBus-Device2
  • ServiceBus-Device3

  这种方法会消耗您机器上的过多资源，因为您将为每个设备运行一个实例，但易于管理，因为您可以为每个新设备跨新实例而不影响其他正在运行的服务.

  This approach will consume too much resources in your machine, as you will be running one instance for each device, but easy to manage, as you can span new instances for each new device without affecting other running services.

  每个实例的并行处理:

  其中每个实例将负责同时处理多条消息，在这种情况下，您将为每个实例创建 2000 个连接(如果在每个集群 5 个实例/节点中运行).这将比其他资源消耗方法更轻，但维护起来有点困难，因为您必须自己处理平衡，并且可能需要额外的服务来监视和委派任务给所有服务并确保消息正在处理均匀.

  Where each instance, would be responsible for processing multiple messages concurrently, in this case you would create 2000 connections for each instance(if running in a 5 instance/node per cluster). This will be lighter than the other approaches on resources consumption, but is a bit harder to maintain, as you will have to handle the balance yourself and might need an extra service to monitor and delegate tasks to all the services and ensure the messages are being processing evenly.

  总结:

  一个实例处理一个连接一条消息需要10000个你的服务实例，分区类似但是你可以使用共享进程来减少内存消耗，但两种情况下内存消耗仍然很高.

  One instance handling one connection at one message a time will required 10000 instances of your service, the partitioning will be similar but you can use a shared process to reduce memory consumption, but the memory consumption will still be high in both cases.

  如果服务数量不是太多，可以选择多个命名服务，您也将无法共享连接.因此，我不会为您的场景推荐这种方法.

  Multiple named services could be an option if the number of services were not too high, You also wouldn't be able to share the connection. So I won't recommend this approach for your scenario.

  第三个选项对资源更友好，但您必须找到一种方法在整个集群节点中均匀地划分连接.

  The third option, is the more resource friendly but you will have to find a way to partition the connections evenly throughout the cluster nodes.

  您还可以使用混合方法，例如，您可以让服务并行处理多条消息，并使用分区服务来定义设备的关键范围.

  You can also use a mixed approach, for example, you can have service handling multiple messages in parallel and a partitioned service to define the key range of devices.

  请查看我提到的链接.

  这篇关于模拟来自 Azure Service Fabric 群集的 10,000 个 Azure IoT 中心设备连接的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！