胶水虚拟视图(已创建地形)未出现在雅典娜中 [英] Glue Virtual View (terraform created) not appearing in Athena

问题描述

我正在尝试使用Terraform aws_glue_catalog_table 创建Virtual_View，据我了解，该视图应在Athena中作为视图出现.

I am trying to use Terraform aws_glue_catalog_table to create a Virtual_View , which I understand, should appear in Athena as a View.

到目前为止，我的代码似乎在Glue中创建了一个目录表，但是在Athena Views库存中什么都没有出现.

So far my code seems to create a catalog table in Glue, but nothing appears in the Athena Views inventory.

很难确切地知道问题的哪一部分.我尝试将生成的胶水表与手动创建但具有相同规格的现有胶水表进行比较，但是显示的信息中没有差异-但是原始文本"部分很难比较编码.

It is hard to know exactly which part is the issue. I have tried comparing the generated glue table to an existing one, manually created but of the same specification, but no differences appear in the info shown - but the 'originaltext' part is hard to compare being encoded.

曾尝试删除ser_de_info部分，但似乎没有任何作用.

Have tried removing the ser_de_info section, but doesn't seem to make any difference.

感谢这里的任何提示！

完全不确定为什么TF不仅仅允许我们提交一个简单的SQL DDL语句来创建这些语句，因为这种胶合方法太复杂了以至于没有实际意义-用2种不同的格式两次声明列，编码脚本-都不好

resource "aws_glue_catalog_table" "aws_gluetable_getresources_vw" {
    name = "getresources_vw" 
    database_name = "mydatabase" 
    table_type = "VIRTUAL _VIEW"

    view_original_text = "/* Presto View: ${base64encode(file("${path.module}/originaltexts/getresources.txt"))} */"
    view_expanded_text = "/* Presto View */" 

    parameters = {
        presto_view = "true"
        comment = "Presto View"
    }

    storage_descriptor { 
        ser_de_info {
            name = " " 
            serialization_library = " " 
        } 

        columns {
            name = "key" 
            type = "string"
        }
        columns {
            name = "value" 
            type = "string"
        }
        columns {
            name = "resourcearn" 
            type = "string"
        }
        columns {
            name = "tags" 
            type = "array<struct<key:string,value:string>>"
        }
         .... more
        }
    }
}

getresources.txt

{
    "catalog":"awsdatacatalog",
    "schema":"mydatabase",
    "columns":[
        {"name":"key","type":"varchar"},
        {"name":"value","type":"varchar"},
        {"name":"resourcearn","type":"varchar"},
        {"name":"tags","type":"array(row(key varchar,value varchar))"},
        {"name":"arn1","type":"varchar"},
        {"name":"arn2","type":"varchar"},
        {"name":"arn3","type":"varchar"},
        {"name":"arn4","type":"varchar"}
    ],
    "originalSql":"SELECT  g.tag.key, g.tag.value, t.resource.resourcearn, t.resource.tags, split_part(t.resource.resourcearn, ':', 1) arn1, split_part(t.resource.resourcearn, ':', 2) arn2, split_part(t.resource.resourcearn, ':', 3) arn3, split_part(t.resource.resourcearn, ':', 6) arn4 FROM  ((ap_ath_meta_use_sbx.getresources h CROSS JOIN UNNEST(h.resourcetagmappinglist) t (resource)) CROSS JOIN UNNEST(t.resource.tags) g (tag))"
}

推荐答案

使用Glue API创建与Athena兼容的视图非常困难.我不知道Terraform提供程序是如何做到的，但是我认为它缺少正确使用所必需的许多细节之一.我在这个答案中一起写了一个非正式的文档: https://stackoverflow.com/a/56347331/1109

Creating an Athena-compatible view using the Glue APIs is difficult. I don't know how the Terraform provider does it, but I assume it's missing one of the many details that are necessary to get right. I wrote together an unofficial documentation in this answer: https://stackoverflow.com/a/56347331/1109

这篇关于胶水虚拟视图(已创建地形)未出现在雅典娜中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！