在R中上传超过2.15 GB的文件 [英] Upload a file over 2.15 GB in R

问题描述

我有一个手动过程，我通过curl将5-6 GB的文件上传到Web服务器：

I've got a manual process where I'm uploading 5-6 GB file to a web server via curl:

curl -X POST --data-binary @myfile.csv http://myserver::port/path/to/api

此过程运行良好，但我很想使用R将其自动化。问题是，我要么不知道自己在做什么，要么curl的R库不知道如何处理大于2GB的文件：

This process works fine, but I'd love to automate it using R. The problem is, I either don't know what I'm doing, or the R libraries for curl don't know how to handle files bigger than ~2GB:

library(RCurl)
postForm(
     "http://myserver::port/path/to/api",
      file = fileUpload(
        filename = path.expand("myfile.csv"),
        contentType = "text/csv"
      ),.encoding="utf-8")

Yields 错误：内部服务器错误

httr也不起作用：

httr doesn't work either:

library(httr)
POST(
      url = "http://myserver:port/path/to/api",
      body = upload_file(
        path =  path.expand("myfile.csv"),
        type = 'text/csv'),
      verbose()
    )

哪个收益：

Response [http://myserver:port/path/to/api]
  Date: 2015-06-30 11:11
  Status: 400
  Content-Type: <unknown>
<EMPTY BODY>

httr使用 verbose（）选项，告诉我：

httr is a little more informative with the verbose() option, telling me:

-> POST http://myserver:port/path/to/api
-> User-Agent: libcurl/7.35.0 r-curl/0.9 httr/1.0.0
-> Host: http://myserver::port
-> Accept-Encoding: gzip, deflate
-> Accept: application/json, text/xml, application/xml, */*
-> Content-Type: text/csv
-> Content-Length: -2147483648
-> Expect: 100-continue
-> 
<- HTTP/1.1 400 Bad Request
<- Server: Apache-Coyote/1.1
<- Transfer-Encoding: chunked
<- Date: Tue, 30 Jun 2015 11:11:11 GMT
<- Connection: close
<- 

Content-Length：-2147483648 看起来像一个32位整数溢出，因此我认为这是httr中的错误。我怀疑RCurl遇到了类似的故障。

The Content-Length: -2147483648 looks suspiciously like a 32 bit integer overflow, so I think this is a bug in httr. I suspect RCurl is experiencing a similar failure.

我真的很喜欢在 curl -X POST --data-binary 周围的最小包装。 code>，但除​​非如此，从R上传较大文件的方法是什么？

I'd really love a minimal wrapper around curl -X POST --data-binary, but barring that, what are my options for uploading fairly large files from R?

推荐答案

已在httr / curl的开发版本中修复：

This bug is fixed in the dev version of httr/curl:

devtools::install_github("jeroenooms/curl")
devtools::install_github("hadley/httr")

这是 httr 和 curl 包（用于R）。该错误已在在GitHub上修复。 2015年2月2日，此更改将很快推广到CRAN。

This is a bug in the httr and curl packages for R. The bug has been fixed on GitHub as of July 2, 2015, and the change will roll out to CRAN soon.

在上面的命令中我也有可能错误地调用了RCurl，但我永远无法找出正确的调用。

It is also possible I was calling RCurl incorrectly in the above command, but I could never figure out the correct invocation.

这篇关于在R中上传超过2.15 GB的文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！