面向服务的架构 - AMQP 或 HTTP [英] Service Oriented Architecture - AMQP or HTTP

问题描述

一点背景.

非常大的单体 Django 应用程序.所有组件都使用相同的数据库.我们需要将服务分开，以便我们可以独立升级系统的某些部分而不影响其余部分.

我们使用 RabbitMQ 作为 Celery 的代理.

现在我们有两个选择:

1. 使用 REST 接口的 HTTP 服务.
2. 通过 AMQP 的 JSONRPC 到事件循环服务

我的团队倾向于使用 HTTP，因为这是他们所熟悉的，但我认为使用 RPC 而非 AMQP 的优势远远超过它.

AMQP 为我们提供了轻松添加负载平衡和高可用性以及保证消息传递的功能.

对于 HTTP，我们必须创建客户端 HTTP 包装器以与 REST 接口配合使用，而我们必须放入负载均衡器并设置该基础架构以实现 HA 等.

使用 AMQP，我可以生成另一个服务实例，它将连接到与其他实例以及 bam、HA 和负载平衡相同的队列.

我对 AMQP 的想法是否遗漏了什么?

解决方案

起初，

• REST、RPC - 架构模式，AMQP - 线级和 HTTP - 运行在 TCP/IP 之上的应用协议
• AMQP 是 HTTP 时的特定协议 - 通用协议，因此，与 AMQP 相比，HTTP 的开销非常高
• AMQP 本质是异步的，而 HTTP 本质是同步的
• REST 和 RPC 都使用数据序列化，哪种格式取决于您并取决于基础架构.如果您在任何地方都使用 python，我认为您可以使用 python 本机序列化 - pickle 这应该比 JSON 或任何其他格式更快.<​​/li>
• HTTP+REST 和 AMQP+RPC 都可以在异构和/或分布式环境中运行

因此，如果您选择使用哪种方式:HTTP+REST 或 AMQP+RPC，答案实际上取决于基础架构的复杂性和资源使用情况.没有任何特定要求，这两种解决方案都可以正常工作，但我宁愿进行一些抽象，以便能够在它们之间透明地切换.

您告诉您的团队熟悉 HTTP 但不熟悉 AMQP.如果开发时间很重要，您会得到答案.

如果你想以最小的复杂性构建 HA 基础设施，我猜 AMQP 协议就是你想要的.

我对它们都有过经验，RESTful 服务的优点是:

• 他们很好地映射了网络界面
• 人们熟悉他们
• 易于调试(由于 HTTP 的通用目的)
• 轻松向第三方服务提供 API.

基于 AMQP 的解决方案的优势:

• 真他妈的快
• 灵活
• 具有成本效益(就资源使用而言)

请注意，您可以在基于 AMQP 的 API 之上向第三方服务提供 RESTful API，而 REST 不是协议而是范例，但您应该考虑构建 AQMP RPC api.我这样做是为了向外部第三方服务提供 API，并在那些在旧代码库上运行或无法添加 AMQP 支持的基础设施部分提供对 API 的访问.

如果我是对的，您的问题是关于如何更好地组织软件不同部分之间的通信，而不是如何向最终用户提供 API.

如果您有一个高负载项目，RabbitMQ 是一款非常好的软件，您可以轻松添加在不同机器上运行的任意数量的工作线程.它还具有开箱即用的镜像和集群功能.还有一点，RabbitMQ 是建立在 Erlang OTP 之上的，这是一个高可靠、稳定的平台......(bla-bla-bla)，它不仅对营销有好处，对工程师也有好处.当 nginx 日志占用了运行 RabbitMQ 的同一分区上的所有磁盘空间时，我只遇到了一次 RabbitMQ 问题.

UPD(2018 年 5 月):Saurabh Bhoomkar 发布了一个链接到 MQ vs. HTTP 文章由 Arnold Shoon 于 2012 年 6 月 7 日撰写，这是它的副本:

<块引用>

我正在浏览我的旧文件并偶然发现我关于 MQ 的笔记，并想分享一些使用 MQ 与 HTTP 的原因:

• 如果您的使用者以固定速率进行处理(即无法处理到 HTTP 服务器的洪水 [bursts])，那么使用 MQ 可以让服务灵活地缓冲其他请求，而不是让它陷入困境.
• 独立于时间的处理和消息交换模式 - 如果线程执行即发即忘，则 MQ 比 HTTP 更适合该模式.
• 长寿命进程更适合 MQ，因为您可以发送请求并有一个单独的线程侦听响应(注意 WS-Addressing 允许 HTTP 以这种方式处理，但需要两个端点都支持该功能).
莉>
• 松耦合，其中一个进程可以继续工作，即使另一个进程不可用，而 HTTP 必须重试.
• 请求优先级，更重要的消息可以跳到队列的前面.
• XA 事务——MQ 完全符合 XA——HTTP 不是.
• 容错 - MQ 消息可以在服务器或网络故障中存活 - HTTP 不能.
• MQ 提供一次且仅一次的可靠"消息传递，而 http 不提供.
• MQ 提供了对大消息进行消息分段和消息分组的能力——HTTP 没有这种能力，因为它单独处理每个事务.
• MQ 提供了一个发布/订阅接口，其中 HTTP 是点对点的.

UPD(2018 年 12 月):正如 @Kevin 在下面的评论中所注意到的，RabbitMQ 比 RESTful 服务更好地扩展是值得怀疑的.我最初的答案是基于简单地添加更多工作人员，这只是扩展的一部分，只要不超过单个 AMQP 代理容量，这是真的，但之后它需要更高级的技术，例如 高可用(镜像)队列，这使得基于 HTTP 和 AMQP 的服务在基础设施级别扩展具有一些非平凡的复杂性.>

经过深思熟虑，我还删除了维护 AMQP 代理 (RabbitMQ) 比任何 HTTP 服务器都更简单的内容:原始答案写于 2013 年 6 月，自那时以来发生了很多变化，但主要的变化是我对两种方法，所以我现在可以说你的里程可能会有所不同".

另请注意，在某种程度上比较 HTTP 和 AMQP 就像是苹果和橘子，所以请不要将此答案解释为您做出决定的最终指导，而是将其作为来源之一或作为参考您的进一步研究，以找出与您的特定情况相匹配的确切解决方案.

A little background.

Very big monolithic Django application. All components use the same database. We need to separate services so we can independently upgrade some parts of the system without affecting the rest.

We use RabbitMQ as a broker to Celery.

Right now we have two options:

1. HTTP Services using a REST interface.
2. JSONRPC over AMQP to a event loop service

My team is leaning towards HTTP because that's what they are familiar with but I think the advantages of using RPC over AMQP far outweigh it.

AMQP provides us with the capabilities to easily add in load balancing, and high availability, with guaranteed message deliveries.

Whereas with HTTP we have to create client HTTP wrappers to work with the REST interfaces, we have to put in a load balancer and set up that infrastructure in order to have HA etc.

With AMQP I can just spawn another instance of the service, it will connect to the same queue as the other instances and bam, HA and load balancing.

Am I missing something with my thoughts on AMQP?

解决方案

At first,

• REST, RPC - architecture patterns, AMQP - wire-level and HTTP - application protocol which run on top of TCP/IP
• AMQP is a specific protocol when HTTP - general-purpose protocol, thus, HTTP has damn high overhead comparing to AMQP
• AMQP nature is asynchronous where HTTP nature is synchronous
• both REST and RPC use data serialization, which format is up to you and it depends of infrastructure. If you are using python everywhere I think you can use python native serialization - pickle which should be faster than JSON or any other formats.
• both HTTP+REST and AMQP+RPC can run in heterogeneous and/or distributed environment

So if you are choosing what to use: HTTP+REST or AMQP+RPC, the answer is really subject of infrastructure complexity and resource usage. Without any specific requirements both solution will work fine, but i would rather make some abstraction to be able switch between them transparently.

You told that your team familiar with HTTP but not with AMQP. If development time is an important time you got an answer.

If you want to build HA infrastructure with minimal complexity I guess AMQP protocol is what you want.

I had an experience with both of them and advantages of RESTful services are:

• they well-mapped on web interface
• people are familiar with them
• easy to debug (due to general purpose of HTTP)
• easy provide API to third-party services.

Advantages of AMQP-based solution:

• damn fast
• flexible
• cost-effective (in resources usage meaning)

Note, that you can provide RESTful API to third-party services on top of your AMQP-based API while REST is not a protocol but rather paradigm, but you should think about it building your AQMP RPC api. I have done it in this way to provide API to external third-party services and provide access to API on those part of infrastructure which run on old codebase or where it is not possible to add AMQP support.

If I am right your question is about how to better organize communication between different parts of your software, not how to provide an API to end-users.

If you have a high-load project RabbitMQ is damn good piece of software and you can easily add any number of workers which run on different machines. Also it has mirroring and clustering out of the box. And one more thing, RabbitMQ is build on top of Erlang OTP, which is high-reliable,stable platform ... (bla-bla-bla), it is good not only for marketing but for engineers too. I had an issue with RabbitMQ only once when nginx logs took all disc space on the same partition where RabbitMQ run.

UPD (May 2018): Saurabh Bhoomkar posted a link to the MQ vs. HTTP article written by Arnold Shoon on June 7th, 2012, here's a copy of it:

I was going through my old files and came across my notes on MQ and thought I’d share some reasons to use MQ vs. HTTP:

• If your consumer processes at a fixed rate (i.e. can’t handle floods to the HTTP server [bursts]) then using MQ provides the flexibility for the service to buffer the other requests vs. bogging it down.
• Time independent processing and messaging exchange patterns — if the thread is performing a fire-and-forget, then MQ is better suited for that pattern vs. HTTP.
• Long-lived processes are better suited for MQ as you can send a request and have a seperate thread listening for responses (note WS-Addressing allows HTTP to process in this manner but requires both endpoints to support that capability).
• Loose coupling where one process can continue to do work even if the other process is not available vs. HTTP having to retry.
• Request prioritization where more important messages can jump to the front of the queue.
• XA transactions – MQ is fully XA compliant – HTTP is not.
• Fault tolerance – MQ messages survive server or network failures – HTTP does not.
• MQ provides for ‘assured’ delivery of messages once and only once, http does not.
• MQ provides the ability to do message segmentation and message grouping for large messages – HTTP does not have that ability as it treats each transaction seperately.
• MQ provides a pub/sub interface where-as HTTP is point-to-point.

UPD (Dec 2018): As noticed by @Kevin in comments below, it's questionable that RabbitMQ scales better then RESTful servies. My original answer was based on simply adding more workers, which is just a part of scaling and as long as single AMQP broker capacity not exceeded, it is true, though after that it requires more advanced techniques like Highly Available (Mirrored) Queues which makes both HTTP and AMQP-based services have some non-trivial complexity to scale at infrastructure level.

After careful thinking I also removed that maintaining AMQP broker (RabbitMQ) is simpler than any HTTP server: original answer was written in Jun 2013 and a lot of changed since that time, but the main change was that I get more insight in both of approaches, so the best I can say now that "your mileage may vary".

Also note, that comparing both HTTP and AMQP is apple to oranges to some extent, so please, do not interpret this answer as the ultimate guidance to base your decision on but rather take it as one of sources or as a reference for your further researches to find out what exact solution will match your particular case.

这篇关于面向服务的架构 - AMQP 或 HTTP的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！