使用 cqlsh 复制非常大的 cassandra 表时出现 PicklingError [英] PicklingError when copying a very large cassandra table using cqlsh

问题描述

当我尝试使用以下命令将表复制到 cassandra 时:

When I try to copy a table to cassandra using the command:

copy images from 'images.csv'

我收到错误:

'PicklingError: Can't pickle <class 'cqlshlib.copyutil.ImmutableDict'>: attribute lookup cqlshlib.copyutil.ImmutableDict failed'

我已经成功导入了我所有的其他表，但是这个表不起作用.与此唯一的区别是它包含用于图像的大型二进制 blob.

I have successfully imported all of my other tables, but this one is not working. The only difference with this one is that it contains large binary blobs for images.

这是来自 csv 文件的示例行:

Here is a sample row from the csv file:

b267ba01-5420-4be5-b962-7e563dc245b0,,0x89504e...[large binary blob]...426082,0,7e700538-cce3-495f-bfd2-6a4fa968bdf6,pentium_e6600,01fa819e-3425-47ca-82aa-a3eec319a998,0,7e700538-cce3-495f-bfd2-6a4fa968bdf6,,,png,0

这是导致错误的文件:https://www.dropbox.com/s/5mrl6nuwelpf3lz/images.csv?dl=0

这是我的架构:

CREATE TABLE dealtech.images (
    id uuid PRIMARY KEY,
    attributes map<text, text>,
    data blob,
    height int,
    item_id uuid,
    name text,
    product_id uuid,
    scale double,
    seller_id uuid,
    text_bottom int,
    text_top int,
    type text,
    width int
)

表是使用 cassandra 2.x 导出的，我目前使用 cassandra 3.0.9 导入它们.

The tables were exported using cassandra 2.x and I am currently using cassandra 3.0.9 to import them.

推荐答案

我在使用 apache cassandra 3.9 时遇到了同样的问题，尽管我的数据集相当小(一个表中有 46 行，另一个表中有 262 行).

I ran into this same issue with apache cassandra 3.9, although my datasets were fairly small (46 rows in one table, 262 rows in another table).

PicklingError: Can't pickle : 属性查找 cqlshlib.copyutil.link 失败

PicklingError: Can't pickle : 属性查找 cqlshlib.copyutil.attribute 失败

其中 link 和 attribute 是我定义的类型.

Where link and attribute are types I defined.

COPY 命令是在 Docker 容器内运行的 .cql 脚本的一部分，作为其设置过程的一部分.

The COPY commands were apart of a .cql script that was being run inside a Docker container as apart of it's setup process.

我在几个地方看到人们在 Windows 上看到这个 PicklingError(似乎与 NTFS 相关)，但在这种情况下 Docker 容器使用的是 Alpine Linux.

I read in a few places where people were seeing this PicklingError on Windows (seemed to be related to NTFS), but the Docker container in this case was using Alpine Linux.

解决方法是将这些选项添加到我的 COPY 命令的末尾:

The fix was to add these options to the end of my COPY commands:

WITH MINBATCHSIZE=1 AND MAXBATCHSIZE=1 AND PAGESIZE=10;

http://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlshCopy.html

我没有看到 PicklingError 在本地运行这些包含 COPY 命令的 .cql 脚本，所以这似乎是一个只有在内存不足的情况下才会出现的问题.

I was not seeing the PicklingError running these .cql scripts containing COPY commands locally, so it seems to be an issue that only rears it's head in a low memory situation.

相关问题:

• 运行 COPY 命令时出现酸洗错误:Windows 上的 CQLShell
• Cassandra 多处理无法pickle _thread.lock 对象

这篇关于使用 cqlsh 复制非常大的 cassandra 表时出现 PicklingError的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！