我可以加速 YAML 吗? [英] Can I speedup YAML?

问题描述

我做了一个小测试用例来比较 YAML 和 JSON 的速度:

I made a little test case to compare YAML and JSON speed :

import json
import yaml
from datetime import datetime
from random import randint

NB_ROW=1024

print 'Does yaml is using libyaml ? ',yaml.__with_libyaml__ and 'yes' or 'no'

dummy_data = [ { 'dummy_key_A_%s' % i: i, 'dummy_key_B_%s' % i: i } for i in xrange(NB_ROW) ]


with open('perf_json_yaml.yaml','w') as fh:
    t1 = datetime.now()
    yaml.safe_dump(dummy_data, fh, encoding='utf-8', default_flow_style=False)
    t2 = datetime.now()
    dty = (t2 - t1).total_seconds()
    print 'Dumping %s row into a yaml file : %s' % (NB_ROW,dty)

with open('perf_json_yaml.json','w') as fh:
    t1 = datetime.now()
    json.dump(dummy_data,fh)
    t2 = datetime.now()
    dtj = (t2 - t1).total_seconds()
    print 'Dumping %s row into a json file : %s' % (NB_ROW,dtj)

print "json is %dx faster for dumping" % (dty/dtj)

with open('perf_json_yaml.yaml') as fh:
    t1 = datetime.now()
    data = yaml.safe_load(fh)
    t2 = datetime.now()
    dty = (t2 - t1).total_seconds()
    print 'Loading %s row from a yaml file : %s' % (NB_ROW,dty)

with open('perf_json_yaml.json') as fh:
    t1 = datetime.now()
    data = json.load(fh)
    t2 = datetime.now()
    dtj = (t2 - t1).total_seconds()
    print 'Loading %s row into from json file : %s' % (NB_ROW,dtj)

print "json is %dx faster for loading" % (dty/dtj)

结果是:

Does yaml is using libyaml ?  yes
Dumping 1024 row into a yaml file : 0.251139
Dumping 1024 row into a json file : 0.007725
json is 32x faster for dumping
Loading 1024 row from a yaml file : 0.401224
Loading 1024 row into from json file : 0.001793
json is 223x faster for loading

我在 ubuntu 12.04 上使用 PyYAML 3.11 和 libyaml C 库.我知道 json 比 yaml 简单得多，但是 json 和 yaml 之间的比率是 223 倍，我想知道我的配置是否正确.

I am using PyYAML 3.11 with libyaml C library on ubuntu 12.04. I know that json is much more simple than yaml, but with a 223x ratio between json and yaml I am wondering whether my configuration is correct or not.

你们的速比一样吗?
如何加快 yaml.load() 的速度?

Do you have same speed ratio ?
How can I speed up yaml.load() ?

推荐答案

您可能已经注意到 Python 的数据结构语法非常类似于 JSON 的语法.

You've probably noticed that Python's syntax for data structures is very similar to JSON's syntax.

发生了什么是 Python 的 json 库对 Python 的内置数据类型进行编码 直接插入文本块，将 ' 替换为 " 并在此处和此处删除 , (过于简化有点).

What's happening is Python's json library encodes Python's builtin datatypes directly into text chunks, replacing ' into " and deleting , here and there (to oversimplify a bit).

另一方面，pyyaml 必须构造一个完整的表示图 在将其序列化为字符串之前.

On the other hand, pyyaml has to construct a whole representation graph before serialising it into a string.

加载时必须向后发生相同类型的事情.

The same kind of stuff has to happen backwards when loading.

加速 yaml.load() 的唯一方法是编写一个新的 Loader，但我怀疑这可能是性能上的巨大飞跃，除非你'愿意编写自己的单一用途的 YAML 解析器，采用 考虑以下评论:

The only way to speedup yaml.load() would be to write a new Loader, but I doubt it could be a huge leap in performance, except if you're willing to write your own single-purpose sort-of YAML parser, taking the following comment in consideration:

YAML 构建图是因为它是通用的序列化能够表示对相同的多个引用的格式目的.如果您知道没有对象重复并且只出现基本类型，您可以使用 json 序列化程序，它仍然是有效的 YAML.

YAML builds a graph because it is a general-purpose serialisation format that is able to represent multiple references to the same object. If you know no object is repeated and only basic types appear, you can use a json serialiser, it will still be valid YAML.

-- 更新

我之前所说的仍然正确，但是如果您正在运行 Linux，则有一种方法可以加快 Yaml 解析.默认情况下，Python 的 yaml 使用 Python 解析器.你必须告诉它你想使用 PyYaml C 解析器.

What I said before remains true, but if you're running Linux there's a way to speed up Yaml parsing. By default, Python's yaml uses the Python parser. You have to tell it that you want to use PyYaml C parser.

你可以这样做:

import yaml
from yaml import CLoader as Loader, CDumper as Dumper

dump = yaml.dump(dummy_data, fh, encoding='utf-8', default_flow_style=False, Dumper=Dumper)
data = yaml.load(fh, Loader=Loader)

为此，您需要安装 yaml-cpp-dev(包后来重命名为 libyaml-cpp-dev)，例如使用 apt-get:

In order to do so, you need yaml-cpp-dev (package later renamed to libyaml-cpp-dev) installed, for instance with apt-get:

$ apt-get install yaml-cpp-dev

PyYaml 和 LibYaml 也是如此.但根据您的输出，情况已经如此.

And PyYaml with LibYaml as well. But that's already the case based on your output.

我现在无法测试它，因为我正在运行 OS X 并且 brew 在安装 yaml-cpp-dev 时遇到了一些问题，但是如果您遵循 PyYaml 文档，他们很清楚性能会更好.

I can't test it right now because I'm running OS X and brew has some trouble installing yaml-cpp-dev but if you follow PyYaml documentation, they are pretty clear that performance will be much better.

这篇关于我可以加速 YAML 吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！