ADF/数据流-将多个CSV转换为实木复合地板 [英] ADF / Dataflow - Convert Multiple CSV to Parquet

问题描述

在ADLS Gen2中，TextFiles文件夹包含3个CSV文件.每个文件中的列名都不同.

In ADLS Gen2, TextFiles folder has 3 CSV files. Column names are different in each file.

我们需要将所有3个CSV文件转换为3个实木复合地板文件，并将其放入ParquetFiles文件夹中

We need to convert all 3 CSV files to 3 parquet files and put it in ParquetFiles folder

我尝试使用复制活动"，但失败，因为列名称中包含空白，并且镶木地板文件不允许使用

I tried to use Copy Activity and it fails because the column names have empty space in it and parquet files doesn't allow it

要删除空格，我使用了数据流:源->".选择(在列名中用下划线替换空格)并接收.这适用于单个文件.当我尝试对所有3个文件执行此操作时，它会尝试合并3个文件并生成包含错误数据的单个文件.

To remove spaces, I used Data flow: Source -> Select (replace space by underscore in col name) and sink. This worked for a single file. When I tried to do it for all 3 files, it tries to merge 3 files and generates single file with incorrect data.

如何解决此问题，主要是在所有文件的列名中删除空格.这里还有其他选择吗?

How to solve this, mainly removing spaces from column names in all files. What would be the other options here?

推荐答案

管道:ForEach活动(循环遍历文件夹中的CSV文件，并将当前迭代项作为参数发送到数据流中)->源指向该文件夹的数据流活动(参数化源路径中的文件名)

Pipeline: ForEach activity (loop over CSV files in folder and send in current iteration item to data flow as param) -> Data Flow activity with source that points to that folder (parameterize the file name in the source path)

这篇关于ADF/数据流-将多个CSV转换为实木复合地板的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！