从一个大的 CSV 文件中读取一个小的随机样本到一个 Python 数据框中 [英] Read a small random sample from a big CSV file into a Python data frame

问题描述

我想读取的 CSV 文件不适合主内存.如何读取其中的几行(~10K)随机行并对选定的数据框进行一些简单的统计?

The CSV file that I want to read does not fit into main memory. How can I read a few (~10K) random lines of it and do some simple statistics on the selected data frame?

推荐答案

假设 CSV 文件中没有标题:

Assuming no header in the CSV file:

import pandas
import random

n = 1000000 #number of records in file
s = 10000 #desired sample size
filename = "data.txt"
skip = sorted(random.sample(range(n),n-s))
df = pandas.read_csv(filename, skiprows=skip)

如果 read_csv 有一个 keeprows 会更好，或者如果 skiprows 采用回调函数而不是列表.

would be better if read_csv had a keeprows, or if skiprows took a callback func instead of a list.

带有标题和未知文件长度:

With header and unknown file length:

import pandas
import random

filename = "data.txt"
n = sum(1 for line in open(filename)) - 1 #number of records in file (excludes header)
s = 10000 #desired sample size
skip = sorted(random.sample(range(1,n+1),n-s)) #the 0-indexed header will not be included in the skip list
df = pandas.read_csv(filename, skiprows=skip)

这篇关于从一个大的 CSV 文件中读取一个小的随机样本到一个 Python 数据框中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！