Apache Sqoop和Flume可以互换使用吗？ [英] Can Apache Sqoop and Flume be used interchangeably?

问题描述

我是大数据新手。从 Flume和Sqoop的区别是什么？ ，Flume和Sqoop都可以从源数据中提取数据并推送到Hadoop。任何人都可以指定使用水槽的地方以及sqoop在哪里？这两个都可以用于相同的任务？

解决方案

Flume 和 Sqoop 都可以使用不同类型的数据源。

Sqoop适用于任何支持JDBC连接的RDBMS系统。另一方面，Flume可以很好地处理流式数据源，例如在您的环境中不断生成的日志数据。

特别是，

• Sqoop可用于从RDBMS系统（如Oracle，MS SQL Server，MySQL，PostgreSQL，Netezza，Teradata和其他支持JDBC连接的系统）导入/导出数据。 li>
  
• Flume可用于从下面的来源获取高吞吐量数据并插入到下面的目标（汇点）中。
  
  
  
  • 常用的flume资源：
    
    
    
    • 后台打印目录 - 创建大量文件的目录，主要用于收集并汇总日志数据
    
    
    • JMS - 从基于JMS的系统收集指标
    
    
    • 和更多
    
    
  
  
  • 常用的水槽接收器：
    
    
    
    • HDFS
    
    
    • HBase
    
    
    • Solr
    
    
    • ElasticSearch
    
    
    • 和更多
    
    
  
  

不，这两种工具都不能用于实现相同的任务，例如flume不能与数据库一起使用，sqoop不能用于流数据源或平面文件。

如果您有兴趣，flume还有一个替代方案，它可以做同样的事情，称为 chukwa 。

I am new to Big data. From some of the answers to What's the difference between Flume and Sqoop?, both Flume and Sqoop can pull data from source and push to Hadoop. Can anyone please specify exaclty where flume is used and where sqoop is? Can both be used for the same tasks?

解决方案

Flume and Sqoop are both designed to work with different kind of data sources.

Sqoop works with any kind of RDBMS system that supports JDBC connectivity. Flume on the other hand works well with streaming data sources like log data which is being generated continuously in your environment.

Specifically,

• Sqoop could be used to import/export data to/from RDBMS systems like Oracle, MS SQL Server, MySQL, PostgreSQL, Netezza, Teradata and some others which supports JDBC connectivity.
• Flume could be used to ingest high throughput data from sources like below and insert into destinations (sinks) below.
  • Commonly used flume sources:
    • Spooling directory - directory in which lot of files are being created, used mostly for collecting and aggregating log data
    • JMS - collect metrics from JMS based systems
    • And lots more
  • Commonly used flume sinks:
    • HDFS
    • HBase
    • Solr
    • ElasticSearch
    • And lots more

No, both tools cannot be used to achieve the same task like for example flume cannot be used with databases and sqoop cannot be used with streaming data sources or flat files.

If you are interested flume also has an alternate which does the same thing called as chukwa.

这篇关于Apache Sqoop和Flume可以互换使用吗？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！