如何处理“模板参数无效"?使用Cloud Dataflow REST API/Python启动自定义模板时 [英] How to deal with "The template parameters are invalid" when launching a custom template using Cloud Dataflow REST API/Python

问题描述

我一直在使用Dataprep构建数据流模板.

I have been using Dataprep to build a Dataflow template.

从 https://console.cloud.google.com/dataflow/createjob -没问题.它会提示您输入参数(区域端点，输入位置，输出位置，临时文件的自定义位置)，而元数据文件基本上会为我提供答案.

Running it from https://console.cloud.google.com/dataflow/createjob - no problems. It prompts for parameters (regional endpoint, input locations, output locations, custom location for temp files) and the metadata file basically hands me the answers.

当我开始使用REST API从Python运行自定义模板时，我将包含以下参数(大量的引号转义)

When I come to run the custom template from Python using the REST API, I am including the parameters like the below (lots of quote escaping)

BODY = {
    "jobName": "{jobname}".format(jobname=JOBNAME),
    "parameters": {
        "customGcsTempLocation": "gs://{bucket}/dts/temp".format(bucket=BUCKET),
        "inputLocations":"{{\"location1\": \"projectname:DATASET.table\",\"location2\": \"projectname:DATASET.table\",\"location3\": \"projectname:DATASET.table\",\"location4\": \"gs://bucket/filename.csv\",\"location5\":\"projectname:DATASET.table\",\"location6\":\"projectname:DATASET.table\",\"location7\":\"projectname:DATASET.table\",\"location8\":\"projectname:DATASET.table\",\"location9\": [\"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename.tsv\", \"gs://bucket/folder/filename\"]}}",
        "outputLocations":"{{\"projectname:DATASET.table\"}}"
    },
     "environment": {
        "tempLocation": "gs://{bucket}/dts/temp".format(bucket=BUCKET)
     }
}

我得到的错误是模板参数无效"，但没有更多详细信息.我不确定是否遗漏了某些东西，包括我不应该错过的东西，或者是错误地转义了某些东西(其他问题，例如

The error I get back is "The template parameters are invalid", but there is no more detail; I am not sure if I am missing something, including something I shouldn't, or escaping something incorrectly (other questions like this seem to indicate the latter isn't uncommon). Any advice appreciated, thanks

推荐答案

克里斯蒂安(Cristian)是正确的，但我犯了另一个错误，以防在类似情况下对其他人有所帮助.

Cristian was correct but there was one other error I had made, posting in case it helps others in similar situations.

我应该以不同的方式对location9中的文件名进行重载，因为它们显然位于列表中(而不只是一个值).下面显示了如何完成此操作:

I should have excaped the filenames in location9 differently because they are in a list (rather than just being a value) apparently. The below shows how this was done:

\"location9\":\"[\\\"gs://bucket/folder/folder.tsv\\\",\\\"gs://bucket/folder/filename1122.tsv\\\",\\\"gs://bucket/folder/filename1123.tsv\\\",\\\"gs://bucket/folder/filename1124.tsv\\\",\\\"gs://bucket/folder/filename1125.tsv\\\",\\\"gs://bucket/folder/filename1126.tsv\\\",\\\"gs://bucket/folder/filename1127.tsv\\\",\\\"gs://bucket/folder/filename1128.tsv\\\",\\\"gs://bucket/folder/filename1129.tsv\\\",\\\"gs://bucket/folder/filename1201.tsv\\\",\\\"gs://bucket/folder/filename1202.tsv\\\",\\\"gs://bucket/folder/filename1203.tsv\\\",\\\"gs://bucket/folder/filename1204.tsv\\\",\\\"gs://bucket/folder/filename1205.tsv\\\",\\\"gs://bucket/folder/filename1206.tsv\\\",\\\"gs://bucket/folder/filename1207.tsv\\\",\\\"gs://bucket/folder/filename1208.tsv\\\",\\\"gs://bucket/folder/filename1209.tsv\\\",\\\"gs://bucket/folder/filename1210.tsv\\\",\\\"gs://bucket/folder/filename1211.tsv\\\",\\\"gs://bucket/folder/filename1212.tsv\\\"]\"

这篇关于如何处理“模板参数无效"?使用Cloud Dataflow REST API/Python启动自定义模板时的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！