将多个文件从Google Cloud Storage加载到单个Pandas Dataframe中 [英] Loading multiple files from Google Cloud Storage into a single Pandas Dataframe

问题描述

我一直在尝试编写一个函数，将Google Cloud Storage存储桶中的多个文件加载到单个Pandas Dataframe中，但是我似乎无法使其正常工作.

I have been trying to write a function that loads multiple files from a Google Cloud Storage bucket into a single Pandas Dataframe, however I cannot seem to make it work.

import pandas as pd
from google.datalab import storage
from io import BytesIO


def gcs_loader(bucket_name, prefix):
  bucket = storage.Bucket(bucket_name)
  df = pd.DataFrame()
  for shard in bucket.objects(prefix=prefix):
    fp = shard.uri
    %gcs read -o $fp -v tmp
    df.append(read_csv(BytesIO(tmp))
  return df

当我尝试运行时说:

在命令行中引用的未定义变量:$ fp

undefined variable referenced in command line: $fp

推荐答案

当然，这是一个示例: https://colab.research.google.com/notebook#fileId=0B7I8C_4vGdF6Ynl1X25iTHE4MGc

Sure, here's an example: https://colab.research.google.com/notebook#fileId=0B7I8C_4vGdF6Ynl1X25iTHE4MGc

此笔记本显示以下内容:

This notebook shows the following:

1. 创建两个随机CSV
2. 将两个CSV文件都上传到GCS存储桶
3. 使用GCS Python API遍历存储桶中的文件.而且，
4. 将每个文件合并到单个Pandas DataFrame中.

这篇关于将多个文件从Google Cloud Storage加载到单个Pandas Dataframe中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！