Talend作业以较低的传输速率运行 [英] Talend job running with slow transfer rate

问题描述

我是talend的新手，经验很有限， 我的任务需要执行每天从sql rds到daliy的redshift的增量更新， 但是我的工作传输速度很慢 详细信息如下

I am new to talend and have very limited experience in it , My task required to perform daily incremental update from sql rds to redshift on daliy basis, however my job runs with very slow transfer rate details are listed below

我的sql rds查询是

my sql rds query is

SELECT
    *  
FROM
    test.ankit2 
WHERE
    id > (SELECT COALESCE(max(id), 0) as id FROM test.stagetable)

ankit2是神话中的表格 和 舞台是红移中的表 并使用tmap组件将组件从rds输入链接到redshift输出组件 请看图片

ankit2 is the table in myrds and stagetable is table in redshift and used tmap component to link the component from rds input to redshift output component Please have a look at the image

请看看并提供您的建议 任何帮助将不胜感激 问候 Ankit Talend Etl作业

Please have a look and provide your suggestion Any help will be appreciated regards AnkitTalend Etl job

推荐答案

为获得最佳Redshift性能，请使用tRedshiftOutputBulkExec组件而不是tRedshiftOutput.它将使用选定的S3存储桶将数据存储为csv，并利用超快COPY命令(通过这种方法，我设法达到20000行/秒的写入速度).

For the best Redshift performance, use tRedshiftOutputBulkExec component instead of tRedshiftOutput. It will use selected S3 bucket for storing data as csv, and utilize superfast COPY command (with this approach I managed to hae 20000 rows/s write speed).

如果可能，最好从RDS直接卸载到S3.

It will be better to do direct unload from RDS to S3 if possible.

这篇关于Talend作业以较低的传输速率运行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！