将数据帧转换为无副本的data.table [英] Convert a data frame to a data.table without copy
问题描述
我有一个大数据框(大约几GB的数量级),我想转换为 data.table
。使用 as.data.table
创建数据帧的副本,这意味着我需要的可用内存至少是数据大小的两倍。有没有办法转换没有副本?
I have a large data frame (in the order of several GB) that I'd like to convert to a data.table
. Using as.data.table
creates a copy of the data frame, which means I need available memory to be at least twice the size of the data. Is there a way to do the conversion without a copy?
这是一个简单的例子来证明:
Here's a simple example to demonstrate:
library(data.table)
N <- 1e6
K <- 1e2
data <- as.data.frame(rep(data.frame(rnorm(N)), K))
gc(reset=TRUE)
tracemem(data)
data <- as.data.table(data)
gc()
输出:
library(data.table)
# data.table 1.8.10 For help type: help("data.table")
N <- 1e6
K <- 1e2
data <- as.data.frame(rep(data.frame(rnorm(N)), K))
gc(reset=TRUE)
# used (Mb) gc trigger (Mb) max used (Mb)
# Ncells 303759 16.3 597831 32.0 303759 16.3
# Vcells 100442572 766.4 402928632 3074.2 100442572 766.4
tracemem(data)
# [1] "<0x363fda0>"
data <- as.data.table(data)
# tracemem[0x363fda0 -> 0x31e4260]: copy as.data.table.data.frame as.data.table
gc()
# used (Mb) gc trigger (Mb) max used (Mb)
# Ncells 304519 16.3 597831 32.0 306162 16.4
# Vcells 100444242 766.4 322342905 2459.3 200933219 1533.0
推荐答案
o正在关注这个SO post ,现在实现一个函数
setDT
,它需要一个列表
(命名和/或未命名)data.frame
(或data.table
)作为输入并返回与data.table
按参考(无任何副本)。请参阅?setDT
更多示例。
o Following this S.O. post, a function
setDT
is now implemented that takes alist
(named and/or unnamed),data.frame
(ordata.table
) as input and returns the same object as adata.table
by reference (without any copy). See?setDT
examples for more.
c $ c> data.table 命名约定 - 所有 set *
函数通过引用修改。 :=
是唯一也通过引用修改的其他。
This is in accordance with data.table
naming convention - all set*
functions modifies by reference. :=
is the only other that also modifies by reference.
require(data.table) # v1.9.0+
setDT(data) # converts data which is a data.frame to data.table *by reference*
查看较早(现已过时)答案的历史记录。
See history for older (now outdated) answers.
这篇关于将数据帧转换为无副本的data.table的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!