Graphql + Dgraph如何批量导入json数据? [英] Graphql + Dgraph how to batch import json data?

问题描述

我刚刚开始了一个简单的graphql模式:

I just started a trivial graphql schema:

type Product {
    productID: ID!
    name: String @search(by: [term])
    reviews: [Review] @hasInverse(field: about)
}

type Review {
    id: ID!
    about: Product! @hasInverse(field: reviews)
    by: Customer! @hasInverse(field: reviews)
    comment: String @search(by: [fulltext])
    rating: Int @search
}

type Customer {
    custID: ID!
    name: String @search(by: [hash, regexp])
    reviews: [Review] @hasInverse(field: by)
}

现在，我想用数百万个json条目填充数据库，而无需调用graphql突变(太慢).例如，我有一个文件夹，其中充满了以下形状的几个json文件(客户和产品).

Now I want to populate the DB with millions of json entries without calling the graphql mutation (too slow). For instance I have a folder full of several json files (customers and products) of the following shape.

json客户文件的示例:

Example of a json customer file:

{
id: "deadbeef",
name: "Bill Gates",
reviews: [
   {
      id:"1234",
      comment: "nice product"
      rating: 5,
      productId: "5678"
   }
]
}

json产品文件示例:

Example of a json product file:

{
id: "5678",
name: "Bluetooth headset",
}

据我了解，到节点之间定义的边缘，我首先必须用uid

To what I understood, to defined edges between nodes, I first have to overload each object with an uid

客户将成为:

{
id: "deadbeef",
uid: "_:deadbeef",
...
reviews: [
   {
      id:"1234",
      uid:"_:1234",
      productId: {uid: "_:5678"}
   }
]
}

和产品

{
id: "5678",
uid: "_:5678"
...
}

然后我们可以批量导入它们(这纯粹是猜测，我从未尝试过).尽管这应该导入条目，但我想知道数据库如何将这些条目与类型相关联，因为对于我们要插入的数据还没有任何线索.是否可以像__typename这样的属性添加到每个条目中以进行键入?

Then we could batch import them (this is pure speculation, I never tried this). While this should import the entries, I would like to know how the DB would associate those entries with a type, because there is no clue yet on the data we want to insert. Is there a property like __typename I could add to each of my entries to type them?

[edit]我发现2个可能的属性class和dgraph.type仍在怀疑应该使用哪个属性以及如何使用它们

[edit] I've found 2 possible properties class and dgraph.type still wondering which one and how I should use them

推荐答案

上面的graphql模式将生成以下谓词:

The graphql schema above will generate the following predicates:

Customer.name
Customer.reviews
Product.name
Product.reviews
Review.about
Review.by
Review.comment
Review.rating
Schema.date
Schema.schema

即Type.property要批量导入值，无需指定类型，只需使用正确的属性名称即可.

i.e. Type.property To batch import values, there is no need to specify the type, just use the right property name.

这是一个有效的示例:

    const product = {
        "dgraph.type":"Product",
        "uid": "_:5678",
        "Product.name": "Bluetooth headset"
    };

    const customer = {
        "uid": "_:deadbeef",
        "dgraph.type":"Customer",
        "Customer.name": "Bill Gates",
        "Customer.reviews": [
            {                    
                "uid": "_:1234",
                "dgraph.type":"Review",
                "Review.comment": "nice product",
                "Review.rating": 5,
                "Review.by": {"uid": "_:deadbeef"},
                "Review.about": {"uid": "_:5678"}
            }
        ]
    };

    // Run mutation.
    const mu = new Mutation();
    mu.setSetJson({set: [product, customer]});

如果要导入成千上万个条目的块，则需要找到一种方法来保留事务中的空白ID.为此，我建议使用一个负责将地图保留在块导入中的类.这是我的POC

If you want to import blocks of thousands of entries, you need find a way to keep the blank ids across the transactions. To achieve this, I suggest to use a class responsible to keep the maps among the blocks imports. Here is my POC

import {DgraphClient, DgraphClientStub, Mutation} from "dgraph-js";
import * as jspb from 'google-protobuf';

type uidMap = jspb.Map<string, string>;

class UidMapper {

    constructor(private uidMap: uidMap = UidMapper.emptyMap()) {
    }

    private static emptyMap(): uidMap {
        return new jspb.Map<string, string>([]);
    }

    public uid(uid: string): string {
        return this.uidMap.get(uid) || `_:${uid}`;
    }

    public addMap(anotherMap: uidMap): void {
        anotherMap.forEach((value, key) => {
            this.uidMap.set(key, value);
        });
    }
}

class Importer {
    public async importTest(): Promise<void> {
        try {
            const clientStub = new DgraphClientStub(
                "localhost:9080",
                grpc.credentials.createInsecure(),
            );
            const dgraphClient: DgraphClient = new DgraphClient(clientStub);

            await this.createData(dgraphClient);

            clientStub.close();
        } catch (error) {
            console.log(error);
        }
    }

    private async createData(dgraphClient: DgraphClient): Promise<void> {
        const mapper = new UidMapper();

        const product = {
        "dgraph.type":"Product",
        "uid": mapper.uid("5678"),
        "Product.name": "Bluetooth headset"
        };

        const customer = ...;
        const addMoreInfo = ...;

        await this.setJsonData(dgraphClient, mapper, [product, customer]);
        await this.setJsonData(dgraphClient, mapper, [addMoreInfo]);
    }

    private async setJsonData(dgraphClient: DgraphClient, mapper: UidMapper, data: any[]) {
        // Create a new transaction.
        const txn = dgraphClient.newTxn();
        try {
            // Run mutation.
            const mu = new Mutation();

            mu.setSetJson({set: data});
            let response = await txn.mutate(mu);
            // Commit transaction.
            mapper.addMap(response.getUidsMap());
            await txn.commit();

        } finally {
            // Clean up. Calling this after txn.commit() is a no-op and hence safe.
            await txn.discard();
        }
    }
}

这篇关于Graphql + Dgraph如何批量导入json数据?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！