Azure批处理NodeFiles到Blob Stotage [英] Azure batch NodeFiles to Blob Stotage

问题描述

我正在Microsoft Azure Batch Services上运行任务，其中每个任务都会在节点上创建一组文件。我必须将这些文件复制到Blob存储中。

I'm running Tasks on Microsoft Azure Batch Services, where each task create a set of files on the node. I have to copy these files to a Blob Storage.

任务是通过不属于批处理池的vm创建和管理的。

The task are created and managed from a vm which is not part of the batch pool

我能够访问节点文件，并且可以将内容写入blob存储，但是这意味着我在驱动vm上以字符串形式获取文件并将其上传到blobstorage。

I'm able to acces the node files and i can write the content to a blob storage however this means I get the file as a string on my driving vm and upload it to the blobstorage.

        var container = BlobClient.GetContainerReference(containerName);
        container.CreateIfNotExists();
        var content = nodeFile.ReadAsString();
        var blob = container.GetBlockBlobReference(nodeFile.Name);
        blob.UploadText(content);

为防止流量增加，有人知道我可以将文件直接上传到BlobStorage吗？

To prevent extra trafic, does anybody know a way I can upload the files directly to the BlobStorage?

我无法控制任务中的exe，因此无法直接从任务中上传它

I have no control over the exe in the task so uploading it from the task directly is not an option

推荐答案

更新后的答案2017-10-27：

您现在可以使用以下命令直接上传任务中的工件任务输出文件，且API版本大于或等于 2017-05-01 （带有虚拟机配置池）。

You can now directly upload artifacts from your task with task output files with API versions greater than or equal to 2017-05-01 with Virtual Machine Configuration pools.

原始答案：

如果能够将可执行文件打包到bat / cmd文件或Shell脚本中，则可以直接从计算节点上载到存储。如果您的VM是Windows，则可以使用 AzCopy 或 blobxfer （如果您的VM是Linux（或Windows），则可以在程序退出后传输文件）。您需要将程序作为计算节点启动任务的一部分进行安装，作为作业准备任务的一部分进行安装，或者将其包括在资源文件中（如果是AzCopy），以便可用于您的任务。

You can upload to storage directly from the compute nodes if you are able to wrap up your executable in a bat/cmd file or shell script. You can use AzCopy if your VM is Windows or blobxfer if your VM is Linux (or Windows) to transfer your files after your program has exited. You will need to install the programs as part of your compute node start task, install as part of job prep task, or include it as part of your resource files (if AzCopy) so it is available to your task.

例如在Windows节点上：

For example on Windows nodes:

   @echo off
   myprogram.exe arg1 arg2
   set /a rc=%ERRORLEVEL%
   REM assuming return code of 0 is success
   IF %rc% EQU 0 (
       AzCopy.exe <azcopy args>
   )
   exit /b %rc%

例如在Linux节点上： / p>

For example on Linux nodes:

    #!/usr/bin/env bash
    set -e
    # your program below
    myprogram arg1 arg2
    # invoke blobxfer to transfer output data to storage, see docs for more info
    blobxfer <blobxfer args>

这篇关于Azure批处理NodeFiles到Blob Stotage的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！