如何列出 AWS Glue Catalog 中的所有数据库和表? [英] How to list all databases and tables in AWS Glue Catalog?
问题描述
我在 AWS Glue 控制台中创建了一个开发终端节点,现在我可以在gluepyspark 控制台中访问 SparkContext 和 SQLContext.
I created a Development Endpoint in the AWS Glue console and now I have access to SparkContext and SQLContext in gluepyspark console.
如何访问目录并列出所有数据库和表?通常的 sqlContext.sql("show tables").show()
不起作用.
How can I access the catalog and list all databases and tables? The usual sqlContext.sql("show tables").show()
does not work.
CatalogConnection Class 但我不知道它在哪个包中.我尝试从 awsglue.context 导入但没有成功.
What might help is the CatalogConnection Class but I have no idea in which package it is. I tried importing from awsglue.context and no success.
推荐答案
我花了几个小时试图找到有关 CatalogConnection 类的一些信息,但没有找到任何信息.(即使在 aws-glue-lib 存储库中 https://github.com/awslabs/aws-glue-libs)
I spend several hours trying to find some info about CatalogConnection class but haven't found anything. (Even in the aws-glue-lib repository https://github.com/awslabs/aws-glue-libs)
就我而言,我需要在 Glue 作业脚本控制台中输入表名
In my case I needed table names in Glue Job Script console
最后我使用了 boto 库并通过 Glue 客户端检索数据库和表名:
Finally I used boto library and retrieved database and table names with Glue client:
import boto3
client = boto3.client('glue',region_name='us-east-1')
responseGetDatabases = client.get_databases()
databaseList = responseGetDatabases['DatabaseList']
for databaseDict in databaseList:
databaseName = databaseDict['Name']
print '\ndatabaseName: ' + databaseName
responseGetTables = client.get_tables( DatabaseName = databaseName )
tableList = responseGetTables['TableList']
for tableDict in tableList:
tableName = tableDict['Name']
print '\n-- tableName: '+tableName
重要的是正确设置区域
参考:get_databases - http://boto3.readthedocs.io/en/latest/reference/services/glue.html#Glue.Client.get_databases
Reference: get_databases - http://boto3.readthedocs.io/en/latest/reference/services/glue.html#Glue.Client.get_databases
get_tables - http://boto3.readthedocs.io/en/latest/reference/services/glue.html#Glue.Client.get_tables
get_tables - http://boto3.readthedocs.io/en/latest/reference/services/glue.html#Glue.Client.get_tables
这篇关于如何列出 AWS Glue Catalog 中的所有数据库和表?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!