使用来自 MonetdbLite C API 的 monetdb_append [英] Using monetdb_append from MonetdbLite C API

问题描述

我正在尝试在应用程序中使用 MonetDBLite C.根据 PDF (https://arxiv.org/pdf/1805.08520.pdf)，我将受益于使用 monetdb_append 函数加载大量数据的速度提升.来自 PDF:

I am trying to use MonetDBLite C in an application. According to the PDF (https://arxiv.org/pdf/1805.08520.pdf), I would benefit from a boost in speed in loading massive amount of data using monetdb_append function. From PDF:

除了发出 SQL 查询外，嵌入式进程还可以使用有效地将大量数据批量附加到数据库中monetdb_append 函数.此函数采用架构和要附加到的表的名称，以及对要附加到的数据的引用表的列.此功能允许高效批量插入，因为解析涉及大量开销单独的 INSERT INTO 语句，当用户想要插入大量数据.

In addition to issuing SQL queries, the embedded process can efficiently bulk append large amounts of data to the database using the monetdb_append function. This function takes the schema and the name of a table to append to, and a reference to the data to append to the columns of the table. This function allows for efficient bulk insertions, as there is significant overhead involved in parsing individual INSERT INTO statements, which becomes a bottleneck when the user wants to insert a large amount of data.

这是embedded.h中的声明

This is the declaration in embedded.h

char* monetdb_append(monetdb_connection conn, const char* schema, const char* table, append_data *data, int ncols);

有没有人举例说明如何使用这个功能?我假设 append_data 结构的 batid 是 BAT 结构的标识.但尚不清楚如何将其与现有 API 结合使用.

Has anybody an example how to use this function? I assume that batid of the append_data structure is the identification of a BAT structure. But it is not clear how that can be used with the existing API.

推荐答案

二进制追加确实需要构建与要追加的列一样多的 BAT 结构.需要包含一些额外的 MonetDBLite 头文件(monetdb_config.h 和 gdk.h).重要的部分是:

The binary append indeed requires construction of as many BAT structures as you have columns to append. Some additional MonetDBLite headers need to be included (monetdb_config.h and gdk.h). The important parts are:

1. 使用具有正确类型和计数的 COLnew 创建 BAT
2. 为它们添加一些值，例如通过指针访问(正确类型长度)到 bat->theap.base[i]
3. 为追加设置 BAT 属性(BATsetcount、BATsettrivprop 和 BBPkeepref)
4. 分配并填充append_data 数据结构.
5. 调用monetdb_append.

1. Create the BATs using COLnew with the correct type and count
2. Add some values to them, e.g. by pointer access (of the correct type length) to bat->theap.base[i]
3. Set BAT properties (BATsetcount, BATsettrivprop and BBPkeepref) for the append
4. Allocate and populate the append_data data structure.
5. Call monetdb_append.

下面是如何将 42 个值附加到包含整数的单列表的简短示例 (CREATE TABLE test (my_column INTEGER);)

Below is a short example how to append 42 values to a one-column table containing integers (CREATE TABLE test (my_column INTEGER);)

// startup, connect etc. before

size_t n = 42;
BAT* b = COLnew(0, TYPE_int, n, TRANSIENT);
for (size_t i = 0; i < n; i++) {
    ((int*)b->theap.base)[i] = i; // or whatever
}

BATsetcount(b, n);
BATsettrivprop(b);
BBPkeepref(b->batCacheid);

append_data *ad = NULL;
ad = malloc(1 * sizeof(append_data));
ad[0].colname = "my_column";
ad[0].batid = b->batCacheid;

if (monetdb_append(conn, "sys", "test", ad, 1) != NULL) { /* handle error */}

这篇关于使用来自 MonetdbLite C API 的 monetdb_append的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！