如何使用python unittest模拟AWS库GlueContext [英] How to mock an aws library GlueContext using python unittest

问题描述

无法从AWS GLue模拟GlueContext

我有这些代码行

sc = SparkContext()
gluecontext = GlueContext(sc)
spark = gluecontext.spark_session

如何使用python单元测试模拟GlueconText？

How to mock in a python unittest a library not installed locally?

推荐答案

一种方法是将粘合库和Spark下载到项目的根目录(或按您喜欢的方式进行配置)

wget https://github.com/awslabs/aws-glue-libs/archive/glue-1.0.zip
wget https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-1.0/spark-2.4.3-bin-hadoop2.8.tgz
unzip glue-1.0.zip -d $PROJECT_ROOT
tar -xf spark-2.4.3-bin-hadoop2.8.tgz -C $PROJECT_ROOT
export SPARK_HOME=$PROJECT_ROOT/spark-2.4.3-bin-spark-2.4.3-bin-hadoop2.8

然后只需模拟Gluecontext

from mock import patch
class Test(unittest.TestCase):
    @patch('awsglue.context.GlueContext')
    @patch('awsglue.utils.getResolvedOptions', side_effect=mock_get_resolved_options)
    def test_method(self, mock_resolve_options, mock_glue_context):
        <your code>

在本地提交测试

$PROJECT_ROOT/aws-glue-libs-glue-1.0/bin/gluepytest $PROJECT_ROOT/tests/

这篇关于如何使用python unittest模拟AWS库GlueContext的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！