使用更大的空间模型加速测试运行的好方法是什么? [英] What is a good way to speed up test runs utilizing larger spacy models?

问题描述

我已经构建了一些依赖于 en_core_web_md 模型的测试.该模型需要大约 15 秒才能加载到我计算机的内存中，这使得测试运行起来很痛苦.

有什么聪明的方法可以加快速度吗?

解决方案

v2.2.[0-5] md 模型有一个小错误，使其加载速度特别慢(请参阅https://github.com/explosion/spaCy/pull/4990).>

您可以重新格式化模型包中的一个文件以缩短加载时间.在模型包的 vocab 目录中(例如，lib/python3.7/site-packages/en_core_web_md/en_core_web_md-2.2.5/vocab):

import srslyorig_data = srsly.read_msgpack("key2row")新数据 = {}对于 orig_data.items() 中的键、值:new_data[int(key)] = int(value)srsly.write_msgpack("key2row", new_data)

在我的测试中，这几乎将加载时间减半(18 秒到 10 秒).剩下的时间主要是为模型加载字符串和词素，此时很难进一步优化.因此，这稍微改善了一些情况，但对于短期测试而言，整体加载时间仍然相对繁重.

I have constructed some tests relying on the en_core_web_md model. The model takes ~15 sec to load into memory on my computer making the tests a pain to run.

Is there a smart way to speed it up?

解决方案

The v2.2.[0-5] md models have a minor bug that make them particularly slow to load (see https://github.com/explosion/spaCy/pull/4990).

You can reformat one file in the model package to improve the load time. In the vocab directory for the model package (e.g., lib/python3.7/site-packages/en_core_web_md/en_core_web_md-2.2.5/vocab):

import srsly
orig_data = srsly.read_msgpack("key2row")
new_data = {}
for key, value in orig_data.items():
    new_data[int(key)] = int(value)
srsly.write_msgpack("key2row", new_data)

In my tests, this nearly halves the loading time (18s to 10s). The remaining time is mostly loading strings and lexemes for the model, which is harder to optimize further at this point. So this improves things a bit but the overall load time is still relatively burdensome for short tests.

这篇关于使用更大的空间模型加速测试运行的好方法是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！