从Node.js批量加载titan db中的数据 [英] Bulk load data in titan db from nodejs

问题描述

我当前的情况就像

1. 我有一个兔子mq，可以给我下订单的详细信息.
2. 另一方面，我有我的titan db(cassandra存储，es索引后端和gremlin服务器).
3. 还有另一个我有nodejs应用程序，它可以使用 https://www通过http api与gremlin服务器进行交互.npmjs.com/package/gremlin .我可以从这里点击我的图形数据库.

1. I have a rabbit mq which gives me the details of the order placed.
2. On the other side I have my titan db (cassandra storage, es index backends and gremlin server).
3. Yet another I have nodejs application which can interact with gremlin server through http api using https://www.npmjs.com/package/gremlin . I am able to make hits to my graph database from here.

现在我要做的是将兔子mq的数据加载到titan db中.

Now what I am trying to do is load data from rabbit mq into titan db.

到目前为止，我只能使用gremlin节点模块从nodejs文件中加载数据

What I have been able to do till now is load the data from nodejs file using gremlin node module

    var createClient = require('gremlin').createClient;
//import { createClient } from 'gremlin';
 
const client = createClient();

client.execute('tx=graph.newTransaction();tx.addVertex(T.label,"product","id",991);tx.commit()', {}, function(err, results){
  if (err) {
    return console.error(err)
  }
    console.log(results)
});

接下来应该如何移动，以便可以利用现有的兔子mq订单并将其推入titan db.

How should I move next so that I can harness existing rabbit mq of orders and push them into titan db.

由于某些限制，我不能使用Java.

Due to some constraints I can not use java.

推荐答案

您最有可能在寻找类似 node-amqp ，它是RabbitMQ的Node.js客户端.您想做的是:

You're most likely looking for something like node-amqp, which is a Node.js client for RabbitMQ. What you want to do is:

1. 建立与Gremlin Server的连接
2. 建立与RabbitMQ的连接
3. 收听RabbitMQ队列中的消息
4. 将这些消息发送给Gremlin，创建图形元素

您必须注意的事情否则可能会损害您的表现:

Things you must watch for that will otherwise likely kill your performance:

1. 发送带有绑定参数的Gremlin查询
2. 批处理消息:创建多个顶点并在同一事务中提交它们(=相同的Gremlin查询，除非在您自己 .commit()的会话模式下).数以千计的数字应该起作用.
3. 当心背压，并确保您不会向Titan实例注入过多的消息而无法处理.

1. Send Gremlin queries with bound parameters
2. Batch messages: create multiple vertices and commit them in the same transaction (= same Gremlin query, unless in session mode where you .commit() yourself). Numbers in the couple thousands should work.
3. Watchout for back-pressure and make sure you don't flood your Titan instances with more messages than they can handle.

我对RabbitMQ不熟悉，但希望这可以帮助您入门.

I'm not familiar with RabbitMQ but hopefully this should get you started.

注意:Gremlin javascript驱动程序通过WebSocket连接与Gremlin Server交互，该连接是永久的和双向的.客户端尚不支持HTTP Channelizer(这不是您希望在当前情况下建立的连接).

Note: Gremlin javascript driver interacts with Gremlin Server via a WebSocket connection, which is permanent and bi-directional. The client doesn't support the HTTP Channelizer yet (which is not the kind of connection that you wish to establish in the current scenario).

这篇关于从Node.js批量加载titan db中的数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！