测试大容量SMTP邮件发送代码的最佳方式？ [英] Best way to test high-volume SMTP email sending code?

问题描述

我在Windows服务（C＃）中编写了一个组件，它负责发送大量的电子邮件。这些电子邮件将转到许多域的收件人–真的，任何域名。 （是的，收件人想要电子邮件，不，我不是垃圾邮件，是的，我在抱怨CAN-SPAM。是的，我知道发送电子邮件从代码吮吸）

我不想依赖外部SMTP服务器。 （除了其他考虑之外，想要检查邮箱的反弹信息并尝试解析它们会给我带来不好的感觉。）

我的设计很简单。事务和批量消息都将生成并插入到数据库表中。此表包含电子邮件信封和内容，以及尝试次数和重试日期。

该服务运行一些工作线程，每次抓取20行，循环通过每一个。使用 Simple DNS Plus 库，我抓住MX记录收件人的域，然后使用 System.Net.Mail.SmtpClient 同步发送电子邮件。如果调用 Send（）成功，我可以将该邮件列出。如果暂时失败，我可以增加尝试次数并设置适当的重试日期。如果它永久失败，我可以出队并处理失败。

显然，将数千个测试电子邮件发送到数百个不同的实际域名是一个非常糟糕的想法。但是，我肯定需要强调测试我的多线程发送代码。我也不太清楚最好的方法是模拟SMTP的各种故障模式。另外，我想确保我可以通过各种垃圾邮件控制方法（灰色列表来命名与网络层最相关的东西）。

即使是我的小规模我最近发现我的ISP阻止连接到端口25的任何服务器，而不是我的ISP的SMTP服务器，加剧了测试困难。 （在生产中，这个东西当然是在一个正确的服务器上，端口25没有被阻止，这不能帮助我从我的开发机器测试。）

所以，我最好奇的两件事：

1. 我应该怎么去测试我的代码？


2. SmtpClient.Send（）可能会失败的各种方式是什么？列出六个例外; SmtpException 和 SmtpFailedRecipientsException 似乎是最相关的。

---

更新： Marc B的答案指出，我基本上是创建自己的SMTP服务器。他提出了我正在重塑车轮的有效要点，所以这里是不使用实际（Postfix等）的理由：

1. 电子邮件具有不同的发送优先级（尽管这与信封的 X-Priority 无关）。批量电子邮件是低优先级;交易高。 （并且任何电子邮件或电子邮件组可以进一步配置为具有任意的优先级。）我需要能够暂停发送较低优先级的电子邮件，这样可以首先发送更高优先级的电子邮件。 （为了完成这个工作，工作线程只要每次获得另外20个队列，就从队列中挑选最高优先级的项目。）

   
   
   

   如果我已经提交了数千个批量项目到外部SMTP服务器，我无法把它们搁置，而我想提交的项目现在被发送。粗略的 Google搜索显示Postfix并不真正支持优先级; Sendmail 对信封中的信息进行优先级排序不满足我的需要。




2. 我需要能够向我的用户显示爆炸（一组批量电子邮件）发送过程的进度。如果我把所有的电子邮件都交给外部服务器，我不知道实际交货的时间有多远。




3. 因为每个MTA的反弹信息都不一样，因此犹豫解析反弹消息。 Sendmail的不同于Exchange的不同于 [...] 。另外，我在哪个频率检查我的反弹收件箱？如果一个反弹消息本身没有交付，该怎么办？




4. 我并不太在意一个爆炸失败的中途。

   
   
   

   如果我们在讨论灾难性故障（应用程序终止未处理的异常，电源故障，无论如何）：由于工作线程成功交付后，数据库中的每个电子邮件都会出现，我可以知道谁已经收到了爆炸声，谁也没有。此外，当服务在失败后重置时，它只会在队列中取消它的位置。

   
   
   

   如果我们在说本地失败（一个 SmtpException ，DNS失败等）：我只是记录失败，增加电子邮件的尝试计数器，稍后再试。 （这基本上是SMTP规范要求的。）尝试之后，我可以永久失败消息（出队），并在以后记录故障进行检查。这样，我可以发现我的代码不处理&ndash的奇怪的边缘情况;即使我的代码不是100％完美的第一次。 （而且老实说，这不会是。）




5. 我希望我自己的路线最终能让我收到电子邮件比我不得不依赖外部SMTP服务器更快。如果服务器不在我的控制之下，我必须担心速率限制;即使是这样，它仍然是瓶颈。我已经走过的多线程架构意味着我并行连接到多个远程服务器，减少了传递 n 消息所需的总时间。

解决方案

假设你有两台服务器可用。一个将是发送者，一个将是接收者。您可以在一系列假域名上设置DNS（甚至只是托管文件）。就这两个服务器而言，这些域是完全有效的，因为本地DNS服务器对它们是权威的，但是对于网络的其余部分而言是完全无效的。只需确保解析器在DNS之前检查主机文件。

完成之后，您可以让发送服务器将接收服务器垃圾邮件发送到您的内心，如接收器做各种各样的事情来测试代码的反应。灰名单，TCP延迟，硬反弹，ICMP无法访问，超过ICMP跳数等...

当然，鉴于您必须测试所有这些条件，您基本上创建自己的SMTP服务器，那么为什么不使用实际的呢？我猜想做一些基本的反弹消息的分析所需的努力将远远小于必须提出代码块来处理postfix / sendmail / exim / etc的所有故障模式，它们已经在他们的自己。

当您认为您的发送代码必须从get-go完美时，这一点尤其如此。如果电子邮件爆炸事件单向通过，只有一半的收件人列表得到消息，那么您的数据空间远大于几百或几千个消息反弹。或者更糟糕的是，以多种不同的方式失败（某些服务器无法访问，某些灰名单用于流量过多等）。而弹跳将乐意坐在入站队列中，直到你手动处理它们，或者修改你的反弹解析器来处理它们。

I've written a component in a Windows service (C#) which is responsible for sending sometimes large volumes of emails. These emails will go to recipients on many domains – really, any domain. (Yes, the recipients want the email. No, I'm not spamming. Yes, I'm in complaince with CAN-SPAM. Yes, I'm aware sending email from code sucks.) Many of the emails are transactional (generated in response to user actions); some are bulk (mail-merges basically).

I do not want to rely on an external SMTP server. (Among other considerations, the thought of having to check a mailbox for bounce messages and trying to parse them gives me bad feelings.)

My design is fairly simple. Both the transactional and bulk messages are generated and inserted into a DB table. This table contains the email envelope and content, plus an attempt count and retry-after date.

The service runs a few worker threads which grab 20 rows at a time and loop through each. Using the Simple DNS Plus library, I grab the MX record(s) of the recipient's domain and then use System.Net.Mail.SmtpClient to synchronously send the email. If the call to Send() succeeds, I can dequeue the email. If it temporarily fails, I can increment the attempt count and set an appropriate retry-after date. If it permanently fails, I can dequeue and handle the failure.

Obviously, sending thousands of test emails to hundreds of different actual domains is a Very Bad Idea. However, I definitely need to stress-test my multi-threaded send code. I'm also not quite sure what the best way is to simulate the various failure modes of SMTP. Plus, I want to make sure I get past the various spam control methods (graylisting to name the most relevant to the network layer of things).

Even my small-scale testing difficulties are exacerbated by my recent discovery of my ISP blocking connections to port 25 on any server other than my ISP's SMTP server. (In production, this thing will of course be on a proper server where port 25 isn't blocked. That does not help me test from my dev machine.)

So, the two things I'm most curious about:

1. How should I go about testing my code?
2. What are the various ways that SmtpClient.Send() can fail? Six exceptions are listed; SmtpException and SmtpFailedRecipientsException seem to be the most relevant.

---

Update: Marc B's answer points out that I'm basically creating my own SMTP server. He makes the valid point that I'm reinventing the wheel, so here's my rationale for not using an 'actual' one (Postfix, etc) instead:

1. Emails have different send priorities (though this is unrelated to the envelope's X-Priority). Bulk email is low priority; transactional is high. (And any email or group of emails can be further configured to have an arbitrary priority.) I need to be able to suspend the sending of lower-priority emails so higher-priority emails can be delivered first. (To accomplish this, the worker threads simply pick up the highest priority items from the queue each time they get another 20.)

   If I've already submitted several thousand bulk items to an external SMTP server, I have no way of putting those on hold while the items I wish to submit now get sent. A cursory Google search shows Postfix doesn't really support priorities; Sendmail prioritizes on information in the envelope, which does not meet my needs.

2. I need to be able to display the progress of the send process of a blast (group of bulk emails) to my users. If I've simply handed all of my emails off to an external server, I have no idea how far along in actual delivery it is.

3. I'm hesitant to parse bounce messages because each MTA's bounce message is different. Sendmail's is different from Exchange's is different from [...]. Also, at what frequency do I check my bounce inbox? What if a bounce message itself isn't delivered?

4. I'm not too terribly concerned with a blast failing half-way through.

   If we're talking catastrophic failure (app-terminating unhandled exception, power failure, whatever): Since the worker threads dequeue each email from the database when it is successfully delivered, I can know who has received the blast and who hasn't. Further, when the service resets after a failure, it simply picks up where it left off in the queue.

   If we're talking local failure (a SmtpException, DNS failure, or so forth): I just log the failure, increment the email's attempt counter, and try again later. (Which is basically what the SMTP spec calls for.) After n attempts, I can permanently fail the message (dequeue it) and log the failure for examination later. This way, I can find weird edge cases that my code isn't handling – even if my code isn't 100% perfect the first time. (And let's be honest, it won't be.)

5. I'm hoping the roll-my-own route will ultimately allow me to get emails out faster than if I had to rely on an external SMTP server. I'd have to worry about rate-limiting if the server weren't under my control; even if it were, it's still a bottleneck. The multithreaded architecture I've gone with means I'm connecting to multiple remote servers in parallel, decreasing the total amount of time it takes to deliver n messages.

解决方案

Assume you've got two servers available. One will be the sender, one will be the receiver. You can set up DNS (or even just hosts files) on both with a long series of fake domains. As far as the two servers are concerned, those domains are perfectly valid as the local DNS servers are authoritative for them, but are completely invalid as far as the rest of the net is concerned. Just make sure the resolver checks the hosts file before DNS.

Once that's done, you can have the sending server spam the receiving server to your heart's content, as have the receiver do various things to test your code's reactions. Greylisting, TCP delays, hard bounces, ICMP unreachables, ICMP hops exceeded, etc...

Of course, given you have to test all these conditions, you're basically creating your own SMTP server, so why not use an actual one to begin with? I'd guess the effort required to do some basic parseing of bounce messages will be far less than having to come up with code chunks to handle all the failure modes that postfix/sendmail/exim/etc... already handle perfectly well on their own.

And this is especially true when you consider your sending code has to be perfect from the get-go. If an email blast fails part-way through and only half the recipient list gets the message, you're in a far bigger hole than if a few hundred or a few thousand messages bounce. Or worse yet, fails in multiple different ways (some servers unreachable, some greylisting you for excessive traffic, etc...). Whereas bounces will happily sit in the incoming queue until you process them manually, or patch up your bounce parser to handle them.

这篇关于测试大容量SMTP邮件发送代码的最佳方式？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！