通过 Web 服务下载文件并通过 Node/Express 将其推送到 Azure Blob 存储 [英] Download file via Webservice and Push it to Azure Blob Storage via Node/Express

问题描述

我需要从供应商的 Web 服务中检索文件并将这些文件推送到一个唯一的 blob 容器中，以便用户拥有一个唯一的工作区".本质上，我会从供应商那里获取文件，用户可以通过他们自己的 blob 容器中的文件来编辑这些文件，这样他们就不会相互交叉工作文件.我有独特的 blob 容器在工作，但需要从我的供应商 API 中下载"/获取文件并将它们推送到 blob 容器中.我能够成功检索文件，这将是单独调用以获取 PDF、文本文件和图像......但是如果我尝试将它们上传到 Azure Blob 存储，我会在 Node.js 中收到以下错误:

I need to retrieve files from a vendor's Web service and push these into a unique blob container so users have a unique "workspace". Essentially, I would get files down from the vendor and users could edit these via the files in their own blob container so they don't cross each others working files. I have the unique blob containers working, but need to "download"/GET the files from my vendors API and push them into a blob container. I am able to successfully retrieve the files, which will be separate calls to get a PDF, text files, and images... but if I attempt to upload them to Azure Blob Storage, I get the following error in Node.js:

TypeError: 无法读取 null 的属性 'length'

TypeError: Cannot read property 'length' of null

我想我需要在客户端将文件编码为 base64 以正确获取长度，并且已经看到一些使用带有 toDataURL 的 Canvas 的示例，但我不确定这是否是本质上下载和直接推送到 Azure Blob 存储，特别是因为我有 PDF 之类的文件(不确定 PDF 是否可以进行 base64 编码).

I am thinking that I need to encode the files as base64 on the client side to properly get the length, and have seen some examples of using a Canvas with toDataURL, but am unsure if this is the best method for essentially downloading and pushing directly to Azure Blob Storage, especially since I have files such as PDFs (not sure if PDFs can be base64 encoded).

这是我调用服务的 AngularJS 控制器(请注意，实际端点可能会根据他们调用的文件而变化，因此我使用客户端 GET 文件来控制用户可能在表单中输入的变量):

Here is my AngularJS controller that calls the service (note that the actual endpoint may change depending on which files they call, so I am using a client side GET of files to control variables that a user may enter in a form):

$scope.getFiles = function () {

$.ajax({
url: 'http://vendorwebservice.net/ws/file1',
type: "GET",
success: function (result) {
console.log(result);
var filename = 'Texture_0.png';

$http.post('/postFile', { filename: filename, file: result }).success(function (data) {
console.log(data);
}, function (err) {
console.log(err);
});

alert("Files Retrieved!");
},
error: function (error) {
console.log("Failed to download image!");
}
})
}

这是我的后端/节点/Express 代码:

Here is my backend/Node/Express code:

app.post('/postFile', function (req, res, next) {
    var filename = req.body.filename;
    var file = req.body.file;
    var base64Data;
    fileBuffer = decodeBase64Image(file);
    blobSvc.createBlockBlobFromText('blob5', filename, fileBuffer.data, { 'contentType': fileBuffer.type }, function (error, result, response) {
        if (!error) {
            console.log("Uploaded" + result);
        }
        else {
            console.log(error);
        }
    });
})

// Decode file for upload
function decodeBase64Image(dataString) {
    var matches = dataString.match(/^data:([A-Za-z-+\/]+);base64,(.+)$/),
        response = {};

    if (matches.length !== 3) {
        return new Error('Invalid input string');
    }

    response.type = matches[1];
    response.data = new Buffer(matches[2], 'base64');

    return response;
}

更新 1:根据 Gary 的建议，我尝试了以下操作，但是由于我的供应商 API 没有文件 URI 而是在 GET 上生成文件的端点，所以我尝试了以下代码有道理的例子).例如，我的供应商端点 'http://vendorapi.net/ws/texture_0' 返回一个文件命名为Texture_0.png".

Update 1: Per Gary's suggestion I have tried the following, but messed up the code a bit since my vendors API does not have file URIs but rather endpoints that produce a file on a GET (aka, I am lost on how to pass the endpoint in versus Gary's example that makes sense). For example, my vendors endpoint of 'http://vendorapi.net/ws/texture_0' returns a file named "Texture_0.png".

前端角度代码:

 $scope.getFromVendor = function () {
            var filename = 'Texture_0.png';//jpg,txt...
            $http.post('/uploadapifiles', { filename: filename, url: 'http://vendorapi.net/ws/texture_0' }).success(function (data) {
                console.log(data);
            }, function (err) {
                console.log(err);
            });
        }

服务器端下载处理(我相信这是最混乱的一个:

Server Side Download Processing (I believe this is the one that is the most messed up:

app.get(http://vendorapi.net/ws/texture_0', function (req, res, next) {
    res.download('http://vendorapi.net/ws/texture_0' + req.params.filename);
})

服务器端上传处理:

app.post('/uploadapifiles', function (req, res, next) {

    var filename = req.body.filename;
    var r = request(req.body.url).pipe(fs.createWriteStream('http://vendorapi.net/ws/texture_0' + filename))
    r.on('close', function () {
        blobsrv.createBlockBlobFromLocalFile('blob5', filename, 'http://vendorapi.net/ws/texture_0' + filename, function (error, result, response) {
            if (!error) {
                console.log("Uploaded" + result);
            }
            else {
                console.log(error);
            }
        });
    })
});

推荐答案

在您最初的想法中，首先您在客户端获取文件内容数据，然后将数据发布到 Express Web 服务器.

In your original idea, at first you get file content data in client side and then post the data to the Express web server.

如果你得到的文件很大，因为文件数据会通过HTTP传输两次，这会导致你的网站变慢，并且可能会出现其他问题.

If the file you get is in a large size, it will slow down your site because of the file data will be transferred twice via HTTP, and it may occur other problem.

此外，在我的测试项目中，很难直接处理文件内容数据.

Furthermore, in my test project, it is hardly to handle with file content data directly.

所以我尝试了另一个想法作为解决方法.

So I tried another idea as a workaround.

我只是将获取特定文件的API发布到服务器，将文件另存为服务器目录中的文件，然后将文件上传到服务器端的存储.这是我的代码片段:

I just post the API of getting specific file to the server, pull the file save as a file on server directory and upload file to Storage on server side. Here is my code snippet:

角度前端:

$scope.upload =function(){
    var filename = (new Date()).getTime()+'.pdf';//jpg,txt...
    $http.post('http://localhost:1337/uploadfile', { filename: filename, url: 'http://localhost:1337/output/123.pdf'}).success(function (data) {
        console.log(data);
    },function(err){
        console.log(err);
    });
  }

后端:

我怀疑您获得文件表单的 API 会像后面一样.

I suspect the API which you get the file form would be like behind.

router.get('/output/:filename', function (req, res, next) {
    res.download('upload/'+req.params.filename);
})

post 请求处理程序利用包 request，无需弄清楚文件类型或编码类型，createBlockBlobFromLocalFile 将在您提供的 blob 存储位置上传文件，API 参考:

The post request handler leverages package request, and there is no necessary to figure out file type or encoding type, createBlockBlobFromLocalFile will upload the file at the location you provide on blob storage, API reference:

router.post('/uploadfile', function (req, res, next) {
    var request = require('request');
    var filename = req.body.filename;
    var tmpFolder = 'upload/', //this folder is to save files download from vendor URL, and should be created in the root directory previously. 
        tmpFileSavedLocation = tmpFolder + filename; //this is the location of download files, e.g. 'upload/Texture_0.png'
    var r = request(req.body.url).pipe(fs.createWriteStream(tmpFileSavedLocation))//this syntax will download file from the URL and save in the location asyns
   r.on('close', function (){
        blobsrv.createBlockBlobFromLocalFile('vsdeploy', filename, tmpFileSavedLocation, function (error, result, response) {
            if (!error) {
                console.log("Uploaded" + result);
           }
            else {
                console.log(error);
            }
        });
    })

})

这篇关于通过 Web 服务下载文件并通过 Node/Express 将其推送到 Azure Blob 存储的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！