下载大文件时,httr GET 函数空间不足 [英] httr GET function running out of space when downloading a large file
问题描述
我正在尝试使用 httr
下载 1.1 GB 的文件,但遇到以下错误:
i'm trying to download a file that's 1.1 gigabytes with httr
but i'm hitting the following error:
x <- GET( extract.path )
Error in curlPerform(curl = handle$handle, .opts = curl_opts$values) :
cannot allocate more space: 1728053248 bytes
我的 C 盘有 400GB 可用空间..
my C drive has 400GB free..
在 RCurl
包中,我在使用 getCurlOptionsConstants()
时看到了 maxfilesize
和 maxfilesize.large
选项但我不明白这些是否/如何通过 config
或 set_config
传递给 httr
.. 或者如果我需要切换到RCurl
为此..即使我确实需要切换,增加最大文件大小会起作用吗?
in the RCurl
package, i see the maxfilesize
and maxfilesize.large
options when using getCurlOptionsConstants()
but i don't understand if/how these might be passed to httr
through config
or set_config
.. or if i need to switch over to RCurl
for this.. and even if i do need to switch, will increasing the maximum filesize work?
这是我的 sessionInfo..
here's my sessionInfo..
> sessionInfo()
R version 3.0.0 (2013-04-03)
Platform: i386-w64-mingw32/i386 (32-bit)
locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252 LC_MONETARY=English_United States.1252 LC_NUMERIC=C LC_TIME=English_United States.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] XML_3.96-1.1 httr_0.2
loaded via a namespace (and not attached):
[1] digest_0.6.0 RCurl_1.95-4.1 stringr_0.6.2 tools_3.0.0
..and(不推荐这样做,只是因为它会花费你一段时间)如果你想重现我的错误,你可以去https://usa.ipums.org/usa-action/samples,注册一个新账号,选择2011 5年acs提取,添加大约一百个变量,然后等待提取准备就绪.然后编辑前三行并运行下面的代码.(再次,不推荐)
..and (this is not recommended, just because it will take you a while) if you want to reproduce my error, you can go to https://usa.ipums.org/usa-action/samples, register for a new account, choose the 2011 5-year acs extract, add about a hundred variables, and then wait for the extract to be ready. then edit the first three lines and run the code below. (again, not recommended)
your.email <- "email@address.com"
your.password <- "password"
extract.path <- "https://usa.ipums.org/usa-action/downloads/extract_files/some_file.csv.gz"
require(httr)
values <-
list(
"login[email]" = your.email ,
"login[password]" = your.password ,
"login[is_for_login]" = 1
)
POST( "https://usa.ipums.org/usa-action/users/validate_login" , body = values )
GET( "https://usa.ipums.org/usa-action/extract_requests/download" , query = values )
# this line breaks
x <- GET( extract.path )
推荐答案
仅供参考 - 这已添加到 httr
的 write_disk()
控件中:https://github.com/hadley/httr/blob/master/man/write_disk.Rd
FYI - this has been added in the write_disk()
control in httr
:
https://github.com/hadley/httr/blob/master/man/write_disk.Rd
这篇关于下载大文件时,httr GET 函数空间不足的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!