Azure数据湖分析IOutputter E_RUNTIME_USER_ROWTOOBIG [英] Azure Data Lake Analytics IOutputter E_RUNTIME_USER_ROWTOOBIG

问题描述

我正在尝试将自定义IOutputter的结果写入本地磁盘上的中间文件.

I'm trying to write the results of my custom IOutputter to an intermediate file on the local disk.

之后，我要将数据库文件(〜20MB)复制到adl输出存储中.

After that I want to copy the database file (~20MB) to the adl output store.

可悲的是，脚本以以下字符终止:

Sadly the script terminates with:

类型的未处理异常 'Microsoft.Cosmos.ScopeStudio.BusinessObjects.Debugger.ScopeDebugException' 发生在Microsoft.Cosmos.ScopeStudio.BusinessObjects.Debugger.dll

An unhandled exception of type 'Microsoft.Cosmos.ScopeStudio.BusinessObjects.Debugger.ScopeDebugException' occurred in Microsoft.Cosmos.ScopeStudio.BusinessObjects.Debugger.dll

其他信息: {"diagnosticCode":195887112，"severity":"Error"，"component":"RUNTIME"，"source":"User"，"errorId":"E_RUNTIME_USER_ROWTOOBIG"，"message":"The 行已超过最大允许大小 4MB，" description:"，" resolution:"，" helpLink:"，" details:"该行 已超出允许的最大大小4MB，" internalDiagnostics:" 7ffe97231797 \ tScopeEngine！?ToStringInternal @ KeySampleCollection @ SSLibV3 @ ScopeEngine @@ AEAA?AV?$ basic_string @ DU?$ char_traits @ D @ std @@ V?$ allocator @ D @ 2 @@ std @@ XZ + 11b7 \ t \ n 7ffe971d7261 \ tScopeEngine！?? 0ExceptionWithStack @ ScopeEngine @@ QEAA @ W4ErrorNumber @ 1 @ AEBV?$ initializer_list @ VScopeErrorArg @ ScopeCommon @@@ std @@ _ N @ Z + 121 \ t \ n 7ffe971d7f6a \ tScopeEngine！?? 0RuntimeException @ ScopeEngine @@ QEAA @ W4ErrorNumber @ 1 @ PEBD @ Z + aa \ t \ n 7ffe6de06aca \ t(无模块)！(无名称)\ t \ n

Additional information: {"diagnosticCode":195887112,"severity":"Error","component":"RUNTIME","source":"User","errorId":"E_RUNTIME_USER_ROWTOOBIG","message":"The row has exceeded the maximum allowed size of 4MB","description":"","resolution":"","helpLink":"","details":"The row has exceeded the maximum allowed size of 4MB","internalDiagnostics":" 7ffe97231797\tScopeEngine!?ToStringInternal@KeySampleCollection@SSLibV3@ScopeEngine@@AEAA?AV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@XZ + 11b7\t\n 7ffe971d7261\tScopeEngine!??0ExceptionWithStack@ScopeEngine@@QEAA@W4ErrorNumber@1@AEBV?$initializer_list@VScopeErrorArg@ScopeCommon@@@std@@_N@Z + 121\t\n 7ffe971d7f6a\tScopeEngine!??0RuntimeException@ScopeEngine@@QEAA@W4ErrorNumber@1@PEBD@Z + aa\t\n 7ffe6de06aca\t(no module)!(no name)\t\n

    public class CustomOutputter : IOutputter
    {
        private Stream stream;

        public override void Close()
        {
            base.Close();

            using (var fs = File.Open("mydb.data", FileMode.Open))
            {
                fs.CopyTo(stream);
            }
        }

        public override void Output(IRow input, IUnstructuredWriter output)
        {
            if(stream == null)
                stream = output.BaseStream;

            myDb.Insert("somestuff");
        }
    }

关于这个问题有什么想法吗?

Any ideas on this problem?

推荐答案

如错误消息所示，当前对USQL读取或写入的行的长度有限制，即4MB.如果您使用CSV等面向记录的文件，则会达到此限制.

As the error message indicates there is currently a limit to the length of rows read or written by USQL and that is 4MB. If you use record-oriented files like CSVs you will hit this limit.

在

There is an example of a byte-oriented file read/write UDO that can help you handle files as binaries at https://github.com/Azure/usql/tree/master/Examples/FileCopyUDOs/FileCopyUDOs. You can effectively chunk data using this.

这篇关于Azure数据湖分析IOutputter E_RUNTIME_USER_ROWTOOBIG的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！