关于仅使用头的c ++库的使用的可量化度量(基准) [英] Quantifiable metrics (benchmarks) on the usage of header-only c++ libraries

查看:222
本文介绍了关于仅使用头的c ++库的使用的可量化度量(基准)的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我试图找到一个答案,使用SO。有许多问题列出了在c ++中构建一个只有头文件的库的各种优缺点,但是我还没有找到一个可量化的方法。



因此,在可量化的术语中,使用传统上分离的c ++头文件和实现文件与头文件有什么区别?

,我假设模板没有使用(因为它们只需要头)。



要详细说明,我列出了我从文章中看到的是利弊。显然,一些不容易量化(例如易于使用),因此对于可量化的比较是无用的。



只有标题的优点


  1. 它更容易包含,因为你不需要在构建系统中指定链接器选项。


  2. 这可能会更快一些,因为库代码与其他代码具有相同的编译器(选项)。 (可以定量)




    1. 它使代码膨胀。 (可量化)(如何影响执行时间和内存占用)

    2. 更长的编译时间。 (可量化)

    3. 接口和实现的分离丢失。

    4. 有时会导致难以解决的循环依赖。 li>
    5. 防止共享库/ DLL的二进制兼容性。

    6. 可能会加重喜欢使用C ++的传统方式的同事。

    您可以从更大的开源项目中使用的任何示例(比较类似大小的代码库)将非常感谢。或者,如果你知道一个项目可以在只有头文件和分离版本之间切换(使用包含这两个文件的第三个文件),这将是理想的。轶事数字也是有用的,因为他们给了我一个可以得到一些洞察力的球场。



    利弊来源:




    • Botan 被选择,因为它们潜在地计算昂贵,包含良好数量的单位,并且实际上具有很少或没有错误编译作为单个单元。许多其他项目被尝试,但消耗太多的时间修复作为一个单位的编译。内存占用是通过定期轮询内存占用并使用最大值来测量的,因此可能不会完全准确。



      此外,此基准测试不执行自动标头依赖性生成(以检测头部变化)。在使用不同构建系统的项目中,这可能为所有基准增加时间。



      基准中有3个编译器,每个具有5个配置。



      编译器:




      • gcc

      • icc

      • clang



      编译器配置:




      • 默认 - 默认编译器选项

      • 优化原生 - -O3 -march = native

      • 大小优化 - -Os

      • LTO / IPO native - -O3 -flto -march = native with clang and gcc, -O3 -ipo -march = native with icpc / icc

      • 零优化 - -Os



      轴承在单单元和多单元结构之间的比较。我包括LTO / IPO,所以我们可能会看到如何正确的方式来实现单一单元有效性比较。



      csv字段的解释:




      • 测试名称 - 基准名称。示例: Botan,Box2D

      • 测试配置 - 命名此测试的特定配置(特殊cxx标志等)。通常与测试名称相同。

      • 编译器 - 编译器名称用过的。示例: gcc,icc,clang

      • 编译器配置 - 一个使用的编译器选项的配置。示例: gcc opt native

      • 编译器版本字符串 - 第一行输出的编译器版本从编译器本身。示例: g ++ --version 在我的系统上生成 g ++(GCC)4.6.1 $ b
      • 仅头文件 - 值 True 如果此测试用例构建为单个单元,

      • 单位 - 如果是多单位项目,

      • 编译时间,链接时间,构建时间,运行时间
      • 重新编译时间AVG,重新编译时间MAX,重新链接时间AVG,重新链接时间MAX,Re -build时间AVG,重建时间MAX - 触摸单个文件后重建项目的时间。每个单元都被触摸,并且每个单元都被重建。最大时间和平均时间记录在这些字段中。

      • 编译内存,链接内存,内存,运行内存,可执行大小



      重现基准:




      • 牛肉盒是 run.py

      • 需要 psutil (内存足迹测量)。

      • 需要GNUMake。

      • 因此,在路径中需要gcc,clang,icc / icpc。

      • 每个基准测试都应该有一个数据文件,列出该基准测试的单位。然后, run.py 会创建两个测试用例,每个单元分别编译一个,每个单元一个编译在一起。示例: box2d.data 。文件格式定义为一个json字符串,包含具有以下键的字典

        • units - 列表 c / c / c / c / c 档案

        • code> - 要编译的可执行文件的名称。

        • link_libs

        • include_directores - 要包含在项目中的目录列表。

        • command - 可选。特殊命令执行以运行基准测试。例如,command:botan_test --benchmark


      • 不所有C ++项目都可以轻松完成;在单个单元中必须没有冲突/歧义。

      • 要向测试用例添加项目,请修改 test_base_cases run.py ,其中包含项目的信息,包括数据文件名。

      • 如果一切正常,输出文件 data.csv 应包含基准测试结果。



      要生成条形图:




      • 您应该从基准测试生成的data.csv文件开始。 li>
      • 获取 chart.py 。需要 matplotlib

      • 调整字段列表

      • 执行 python chart.py data.csv

      • c> 现在应包含结果。



      Box2D




      • Box2D用于 svn as is ,revision 251。

      • 基准取自此处,修改了此处,可能不能代表良好的Box2D基准,

      • 通过查找所有.cpp单位,手动写入box2d.data文件。



      Botan




      • 使用 Botan-1.10.3

      • 数据文件: botan_bench.data

      • 首先运行 ./ configure.py --disable-asm --with-openssl - enable-modules = asn1,benchmark,block,cms,engine,entropy,filters,hash,kdf,mac,bigint,ec_gfp,mp_generic,numbertheory,mutex,rng,ssl,stream,cvc

      • 我禁用了程序集,因为程序集可能会涉及到当函数边界不阻塞优化时可能发生的优化。

      • 然后执行 grep -o\。/ src。* cppMakefile grep -o\。/ checks。*Makefile 获取.cpp单位,并将它们放入 botan_bench.data 文件。

      • 修改 /checks/checks.cpp 不调用x509单元测试,并删除x509检查,因为Botan typedef和openssl之间存在冲突。

      • 使用Botan源代码中包含的基准。



      系统规格:




      • OpenSuse 11.4,32位

      • 4GB RAM

      • 英特尔酷睿i7 CPU Q 720 @ 1.60GHz


      I've tried to find an answer to this using SO. There are a number of questions that list the various pros and cons of building a header-only library in c++, but I haven't been able to find one that does so in quantifiable terms.

      So, in quantifiable terms, what's different between using traditionally separated c++ header and implementation files versus header only?

      For simplicity, I'm assuming that templates are not used (because they require header only).

      To elaborate, I've listed what I have seen from the articles to be the pros and cons. Obviously, some are not easily quantifiable (such as ease of use), and are therefore useless for quantifiable comparison. I'll mark those that I expect quantifiable metrics with a (quantifiable).

      Pros for header-only

      1. It's easier to include, since you don't need to specify linker options in your build system.
      2. You always compile all the library code with the same compiler (options) as the rest of your code, since the library's functions get inlined in your code.
      3. It may be a lot faster. (quantifiable)
      4. May give compiler/linker better opportunities for optimization (explanation/quantifiable, if possible)
      5. Is required if you use templates anyways.

      Cons for header-only

      1. It bloats the code. (quantifiable) (how does that affect both execution time and the memory footprint)
      2. Longer compile times. (quantifiable)
      3. Loss of separation of interface and implementation.
      4. Sometimes leads to hard-to-resolve circular dependencies.
      5. Prevents binary compatibility of shared libraries/DLLs.
      6. It may aggravate co-workers who prefer the traditional ways of using C++.

      Any examples that you can use from larger, open source projects (comparing similarly-sized codebases) would be very much appreciated. Or, if you know of a project that can switch between header-only and separated versions (using a third file that includes both), that would be ideal. Anecdotal numbers are useful too because they give me a ballpark with which I can gain some insight.

      sources for pros and cons:

      Thanks in advance...

      UPDATE:

      For anyone that may be reading this later and is interested in getting a bit of background information on linking and compiling, I found these resources useful:

      UPDATE: (in response to the comments below)

      Just because answers may vary, doesn't mean that measurement is useless. You have to start measuring as some point. And the more measurements you have, the clearer the picture is. What I'm asking for in this question is not the whole story, but a glimpse of the picture. Sure, anyone can use numbers to skew an argument if they wanted to unethically promote their bias. However, if someone is curious about the differences between two options and publishes those results, I think that information is useful.

      Has no one been curious about this topic, enough to measure it?

      I love the shootout project. We could start by removing most of those variables. Only use one version of gcc on one version of linux. Only use the same hardware for all benchmarks. Do not compile with multiple threads.

      Then, we can measure:

      • executable size
      • runtime
      • memory footprint
      • compile time (for both entire project and by changing one file)
      • link time

      解决方案

      Summary (notable points):

      • Two packages benchmarked (one with 78 compilation units, one with 301 compilation units)
      • Traditional Compiling (Multi Unit Compilation) resulted in a 7% faster application (in the 78 unit package); no change in application runtime in the 301 unit package.
      • Both Traditional Compiling and Header-only benchmarks used the same amount of memory when running (in both packages).
      • Header-only Compiling (Single Unit Compilation) resulted in an executable size that was 10% smaller in the 301 unit package (only 1% smaller in the 78 unit package).
      • Traditional Compiling used about a third of the memory to build over both packages.
      • Traditional Compiling took three times as long to compile (on the first compilation) and took only 4% of the time on recompile (as header-only has to recompile the all sources).
      • Traditional Compiling took longer to link on both the first compilation and subsequent compilations.

      Box2D benchmark, data:

      box2d_data_gcc.csv

      Botan benchmark, data:

      botan_data_gcc.csv

      Box2D SUMMARY (78 Units)

      Botan SUMMARY (301 Units)

      NICE CHARTS:

      Box2D executable size:

      Box2D compile/link/build/run time:

      Box2D compile/link/build/run max memory usage:

      Botan executable size:

      Botan compile/link/build/run time:

      Botan compile/link/build/run max memory usage:


      Benchmark Details

      TL;DR


      The projects tested, Box2D and Botan were chosen because they are potentially computationally expensive, contain a good number of units, and actually had few or no errors compiling as a single unit. Many other projects were attempted but were consuming too much time to "fix" into compiling as one unit. The memory footprint is measured by polling the memory footprint at regular intervals and using the maximum, and thus might not be fully accurate.

      Also, this benchmark does not do automatic header dependency generation (to detect header changes). In a project using a different build system, this may add time to all benchmarks.

      There are 3 compilers in the benchmark, each with 5 configurations.

      Compilers:

      • gcc
      • icc
      • clang

      Compiler configurations:

      • Default - default compiler options
      • Optimized native - -O3 -march=native
      • Size optimized - -Os
      • LTO/IPO native - -O3 -flto -march=native with clang and gcc, -O3 -ipo -march=native with icpc/icc
      • Zero optimization - -Os

      I think these each can have different bearings on the comparisons between single-unit and multi-unit builds. I included LTO/IPO so we might see how the "proper" way to achieve single-unit-effectiveness compares.

      Explanation of csv fields:

      • Test Name - name of the benchmark. Examples: Botan, Box2D.
      • Test Configuration - name a particular configuration of this test (special cxx flags etc.). Usually the same as Test Name.
      • Compiler - name of the compiler used. Examples: gcc,icc,clang.
      • Compiler Configuration - name of a configuration of compiler options used. Example: gcc opt native
      • Compiler Version String - first line of output of compiler version from the compiler itself. Example: g++ --version produces g++ (GCC) 4.6.1 on my system.
      • Header only - a value of True if this test case was built as a single unit, False if it was built as a multi-unit project.
      • Units - number of units in the test case, even if it is built as a single unit.
      • Compile Time,Link Time,Build Time,Run Time - as it sounds.
      • Re-compile Time AVG,Re-compile Time MAX,Re-link Time AVG,Re-link Time MAX,Re-build Time AVG,Re-build Time MAX - the times across rebuilding the project after touching a single file. Each unit is touched, and for each, the project is rebuilt. The maximum times, and average times are recorded in these fields.
      • Compile Memory,Link Memory,Build Memory,Run Memory,Executable Size - as they sound.

      To reproduce the benchmarks:

      • The bullwork is run.py.
      • Requires psutil (for memory footprint measurements).
      • Requires GNUMake.
      • As it is, requires gcc, clang, icc/icpc in the path. Can be modified to remove any of these of course.
      • Each benchmark should have a data-file that lists the units of that benchmarks. run.py will then create two test cases, one with each unit compiled separately, and one with each unit compiled together. Example: box2d.data. The file format is defined as a json string, containing a dictionary with the following keys
        • "units" - a list of c/cpp/cc files that make up the units of this project
        • "executable" - A name of the executable to be compiled.
        • "link_libs" - A space separated list of installed libraries to link to.
        • "include_directores" - A list of directories to include in the project.
        • "command" - optional. special command to execute to run the benchmark. For example, "command": "botan_test --benchmark"
      • Not all C++ projects can this be easily done with; there must be no conflicts/ambiguities in the single unit.
      • To add a project to the test cases, modify the list test_base_cases in run.py with the information for the project, including the data file name.
      • If everything runs well, the output file data.csv should contain the benchmark results.

      To produce the bar charts:

      • You should start with a data.csv file produced by the benchmark.
      • Get chart.py. Requires matplotlib.
      • Adjust the fields list to decide which graphs to produce.
      • Run python chart.py data.csv.
      • A file, test.png should now contain the result.

      Box2D

      • Box2D was used from svn as is, revision 251.
      • The benchmark was taken from here, modified here and might not be representative of a good Box2D benchmark, and it might not use enough of Box2D to do this compiler benchmark justice.
      • The box2d.data file was manually written, by finding all the .cpp units.

      Botan

      • Using Botan-1.10.3.
      • Data file: botan_bench.data.
      • First ran ./configure.py --disable-asm --with-openssl --enable-modules=asn1,benchmark,block,cms,engine,entropy,filters,hash,kdf,mac,bigint,ec_gfp,mp_generic,numbertheory,mutex,rng,ssl,stream,cvc, this generates the header files and Makefile.
      • I disabled assembly, because assembly might intefere with optimizations that can occure when the function boundaries do not block optimization. However, this is conjecture and might be totally wrong.
      • Then ran commands like grep -o "\./src.*cpp" Makefile and grep -o "\./checks.*" Makefile to obtain the .cpp units and put them into botan_bench.data file.
      • Modified /checks/checks.cpp to not call the x509 unit tests, and removed x509 check, because of conflict between Botan typedef and openssl.
      • The benchmark included in the Botan source was used.

      System specs:

      • OpenSuse 11.4, 32-bit
      • 4GB RAM
      • Intel(R) Core(TM) i7 CPU Q 720 @ 1.60GHz

      这篇关于关于仅使用头的c ++库的使用的可量化度量(基准)的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆