保存fprintf调用的变量参数列表 [英] save variable argument list for fprintf calls

问题描述

我正在编写沉重的多线程[> 170线程] c ++ 11程序。每个线程都将信息记录到所有线程使用的一个文件中。出于性能原因，我想创建一个 log 线程，该线程通过 fprintf（）将信息写入全局文件。我不知道如何组织 worker 线程正在写入的信息的结构，然后 log 线程可以读取这些信息。

为什么不在每个 worker 线程中调用 sprintf（），然后只将输出缓冲区提供给 log 线程？对于日志文件的格式化输出，我在 fprintf（）函数中使用了 locale 线程的其余部分。因此，为了区别 locale 的输出，我将不得不永久地切换和锁定/保护 xprintf（）调用。
在 log 线程中，我有一个 locale 设置用于整个输出，而 worker 线程具有他们的语言环境版本。

log 线程的另一个原因是我必须将输出分组，否则来自每个 worker 线程的信息将不会放在一个块中：

错误：

 信息A线程＃1 
信息A线程＃2 
信息B线程＃1 
信息B线程＃2 
  

正确：

 信息A线程＃1 
信息B线程＃1 
信息A线程＃2 
信息B线程＃2 
  

为了实现此分组，我必须保护每个 worker 线程中的输出，这会减慢线程执行时间。

如何将 va_list 保存到结构中，以使其可以被 log 线程读取和pa退回到 fprintf（）？

解决方案

我看不到如何使用带有 va_list 的旧版C vprintf 轻松完成此操作当您想在线程之间传递信息时，迟早需要以某种方式使用堆。

下面是使用 Boost.Format 用于设置格式，并 Boost.Variant 进行参数传递。如果按顺序串联以下代码块，则该示例将完整且有效。如果使用GCC进行编译，则需要传递 -pthread 链接器标志。当然，您还需要两个Booster库，它们都是仅标头的。这是我们将使用的标头。

  #include< condition_variable> 
 #include< iostream> 
 #include< list> 
 #include< locale> 
 #include< mutex> 
 #include< random> 
 #include< string> 
 #include< thread> 
 #include< utility> 
 #include< vector> 
 
 #include< boost / format.hpp> 
 #include< boost / variant.hpp> 
  

首先，我们需要某种机制来异步执行某些任务，在这种情况下，请打印日志消息。由于概念很笼统，因此我为此使用了抽象基类 Spooler 。其代码基于Herb Cutter在CppCon 2014上的演讲无锁编程（或杂耍剃须刀刀片） （第1部分，第2部分）。我不会详细介绍此代码，因为它主要是脚手架，与您的问题没有直接关系，并且我想您已经具备了这一功能。我的后台打印程序使用受 std :: mutex std :: list c $ c>作为任务队列。也许值得考虑使用无锁数据结构。

  class Spooler 
 {
私人：
 
个布尔值已完成{}; 
 std :: list< std :: function< void（void）>> queue_ {}; 
 std :: mutex Mutex_ {}; 
 std :: condition_variable condvar_ {}; 
 std :: thread worker_ {}; 
 
 public：
 
 Spooler（）：worker_ {[this（）{work（）; }} 
 {
} 
 
〜Spooler（）
 {
自动中毒= [this]（）{done_ = true; }; 
 this-> submit（std :: move（poison））; 
 if（this-> worker_.joinable（））
 this-> worker_.join（）; 
} 
 
受保护：
 
 void 
 Submit（std :: function< void（void）> task）
 {
 //这基本上是一个push_back，但是避免了在关键部分中可能阻止
 //调用。 
 decltype（this-> queue_）tmp {std :: move（task）}; 
 {
 std :: unique_lock< std :: mutex> lck {this-> mutex_}; 
 this-> queue_.splice（this-> queue_.cend（），tmp）; 
} 
 this-> condvar_.notify_all（）; 
} 
 
私人：
 
无效
 work（）
 {
 do 
 {
 std :: unique_lock< std :: mutex> lck {this-> mutex_}; 
 while（this-> queue_.empty（））
 this-> condvar_.wait（lck）; 
 const auto task = std :: move（this-> queue_.front（））; 
 this-> queue_.pop_front（）; 
 lck.unlock（）; 
 task（）; 
} 
 while（！this-&d; done_）; 
} 
}; 
  

从后台处理程序，我们现在得出 Logger （私下）从 Spooler 继承了其异步功能，并添加了特定于日志记录的功能。它只有一个称为 log 的函数成员，该函数成员将格式字符串和零个或多个参数作为参数格式化为 std :: vector 的 boost :: variant s。

不幸的是，这将我们限制为一个固定数我们可以支持的类型，但这应该不是一个大问题，因为C printf 也不支持任意类型。就本例而言，我仅使用 int 和 double ，但您可以使用<$扩展列表c $ c> std :: string s， void ** $code>指针或您有什么东西。

log 函数构造一个lambda表达式，该表达式创建 boost :: format 对象，并将其全部输入参数，然后将其写入 std :: log 或希望格式化的消息发送到的任何地方。

构造函数 boost :: format 的重载可以接受格式字符串和语言环境。您可能对此感兴趣，因为您已经提到在注释中设置自定义语言环境。通常的构造函数只接受一个参数，即格式字符串。

请注意，所有格式化和输出都是在假脱机程序线程上完成的。

  class Logger：后台处理程序
 {
 public：
 
 void 
 log（const std :: string& ; fmt，
 const std :: vector< boost :: variant< int，double>& args）
 {
自动任务= [fmt，args]（）{
 boost :: format msg {fmt，std :: locale { C}}; //您的语言环境
用于（const auto& arg：args）
 msg％arg; //输入下一个参数
 std :: clog<< msg<< std :: endl; //打印格式化的消息
}; 
 this-> submit（std :: move（task））; 
} 
}; 
  

这就是全部。现在，我们可以像下面的示例一样使用 Logger 。重要的是，在销毁 Logger 之前，必须先对所有工作线程进行 join（）编辑，否则将无法处理所有工作线程。

  int 
 main（）
 {
 Logger logger {}; 
 std :: vector< std :: thread>线程{}; 
 std :: random_device rnddev {}; 
 for（int i = 0; i< 4; ++ i）
 {
 const auto seed = rnddev（）; 
自动任务= [& logger，i，种子]（）{
 std :: default_random_engine rndeng {seed}; 
 std :: uniform_real_distribution< double> rnddist {0.0，0.5}; 
 for（double p = 0.0; p< 1.0; p + = rnddist（rndeng））
 logger.log（线程＃％d已完成％6.2f %%，{i，100.0 * p}）; 
 logger.log（线程＃％d已完成其工作，{i}）； 
}; 
 thread.emplace_back（std :: move（task））; 
} 
表示（自动和线程：线程）
 thread.join（）; 
} 
  

可能的输出：

< 线程＃1是0.00％完成
线程＃0是0.00％完成
线程＃0是26.84％完成
线程＃0已完成76.15％
线程＃3已完成0.00％
线程＃0已完成其工作
线程＃3已完成34.70％已完成
线程＃3已完成78.92已完成的百分比
线程＃3是91.89％已完成的
线程＃3已完成工作
线程＃1的是26.98％已完成的
线程＃1的是73.84％已完成的
线程＃1已完成工作
线程＃2是0.00％完成
线程＃2是10.17％完成
线程＃2是29.85％完成
线程＃2是79.03％已完成
线程＃2已完成工作


I am writing a heavy multi threaded [>170 threads] c++11 program. Each thread is logging information into one file used by all threads. For performance reasons I want to create a log thread which is writing the information via fprintf() into the global file. I have no idea how to organize the structure into which the worker threads are writing the information which can be then read by the log thread.

Why do I not call sprintf() in each worker thread and then just provide the output buffer to the log thread? For the formatted output into the log file I am using a locale in the fprintf() functions which is different than in the rest of the thread. Therefore I would have to switch and lock/guard permanently the xprintf() calls in order to differ the locale output. In the log thread I have one locale setting used for the whole output while the worker threads have their locale version.

Another reason for the log thread is that I have to "group" the output otherwise the information from each worker thread would not be in a block:

Wrong:

Information A Thread #1
Information A Thread #2
Information B Thread #1
Information B Thread #2

Correct:

Information A Thread #1
Information B Thread #1
Information A Thread #2
Information B Thread #2

In order to achieve this grouping I have to guard the output in each worker thread which is slowing the thread execution time.

How can I save the va_list into a structure that way it can be read by the log thread and passed back to fprintf()?

解决方案

I don't see how this would be done easily using the legacy C vprintf with va_lists. As you want to pass things around between threads, sooner or later you will need to use the heap in some way.

Below is a solution that uses Boost.Format for the formatting and Boost.Variant for parameter passing. The example is complete and working if you concatenate the following code blocks in order. If you compile with GCC, you need to pass the -pthread linker flag. And of course, you'll also need the two Boost libraries which are header-only, however. Here are the headers we will use.

#include <condition_variable>
#include <iostream>
#include <list>
#include <locale>
#include <mutex>
#include <random>
#include <string>
#include <thread>
#include <utility>
#include <vector>

#include <boost/format.hpp>
#include <boost/variant.hpp>

At first, we need some mechanism to asynchronously execute some tasks, in this case, print our logging messages. Since the concept is general, I use an "abstract" base class Spooler for this. Its code is based on Herb Sutter's talk "Lock-Free Programming (or, Juggling Razor Blades)" on CppCon 2014 (part 1, part 2). I'm not going into detail about this code because it is mostly scaffolding not directly related to your question and I assume you already have this piece of functionality in place. My Spooler uses a std::list protected by a std::mutex as a task queue. It might be worthwhile to consider using a lock-free data structure instead.

class Spooler
{
private:

  bool done_ {};
  std::list<std::function<void(void)>> queue_ {};
  std::mutex mutex_ {};
  std::condition_variable condvar_ {};
  std::thread worker_ {};

public:

  Spooler() : worker_ {[this](){ work(); }}
  {
  }

  ~Spooler()
  {
    auto poison = [this](){ done_ = true; };
    this->submit(std::move(poison));
    if (this->worker_.joinable())
      this->worker_.join();
  }

protected:

  void
  submit(std::function<void(void)> task)
  {
    // This is basically a push_back but avoids potentially blocking
    // calls while in the critical section.
    decltype(this->queue_) tmp {std::move(task)};
    {
      std::unique_lock<std::mutex> lck {this->mutex_};
      this->queue_.splice(this->queue_.cend(), tmp);
    }
    this->condvar_.notify_all();
  }

private:

  void
  work()
  {
    do
      {
        std::unique_lock<std::mutex> lck {this->mutex_};
        while (this->queue_.empty())
          this->condvar_.wait(lck);
        const auto task = std::move(this->queue_.front());
        this->queue_.pop_front();
        lck.unlock();
        task();
      }
    while (!this->done_);
  }
};

From the Spooler, we now derive a Logger that (privately) inherits its asynchronous capabilities from the Spooler and adds the logging specific functionality. It has only one function member called log that takes as parameters a format string and zero or more arguments to format into it as a std::vector of boost::variants.

Unfortunately, this limits us to a fixed number of types we can support but that shouldn't be a large problem since the C printf doesn't support arbitrary types either. For the sake of this example, I'm only using int and double but you can extend the list with std::strings, void * pointers or what have you.

The log function constructs a lambda expression that creates a boost::format object, feeds it all the arguments and then writes it to std::log or wherever you want the formatted message to go.

The constructor of boost::format has an overload that accepts the format string and a locale. You might be interested in this one since you have mentioned setting a custom locale in the comments. The usual constructor only takes a single argument, the format string.

Note how all formatting and outputting is done on the spooler's thread.

class Logger : Spooler
{
 public:

  void
  log(const std::string& fmt,
      const std::vector<boost::variant<int, double>>& args)
  {
    auto task = [fmt, args](){
      boost::format msg {fmt, std::locale {"C"}};  // your locale here
      for (const auto& arg : args)
        msg % arg;  // feed the next argument
      std::clog << msg << std::endl;  // print the formatted message
    };
    this->submit(std::move(task));
  }
};

This is all it takes. We can now use the Logger like in this example. It is important that all worker threads are join() ed before the Logger is destructed or it won't process all messages.

int
main()
{
  Logger logger {};
  std::vector<std::thread> threads {};
  std::random_device rnddev {};
  for (int i = 0; i < 4; ++i)
    {
      const auto seed = rnddev();
      auto task = [&logger, i, seed](){
        std::default_random_engine rndeng {seed};
        std::uniform_real_distribution<double> rnddist {0.0, 0.5};
        for (double p = 0.0; p < 1.0; p += rnddist(rndeng))
          logger.log("thread #%d is %6.2f %% done", {i, 100.0 * p});
        logger.log("thread #%d has completed its work", {i});
      };
      threads.emplace_back(std::move(task));
    }
  for (auto& thread : threads)
    thread.join();
}

Possible output:

thread #1 is   0.00 % done
thread #0 is   0.00 % done
thread #0 is  26.84 % done
thread #0 is  76.15 % done
thread #3 is   0.00 % done
thread #0 has completed its work
thread #3 is  34.70 % done
thread #3 is  78.92 % done
thread #3 is  91.89 % done
thread #3 has completed its work
thread #1 is  26.98 % done
thread #1 is  73.84 % done
thread #1 has completed its work
thread #2 is   0.00 % done
thread #2 is  10.17 % done
thread #2 is  29.85 % done
thread #2 is  79.03 % done
thread #2 has completed its work

这篇关于保存fprintf调用的变量参数列表的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！