如何在Dataflow中进行这种类型的测试(在Twitter上称为功能测试)? [英] How to do this type of testing in Dataflow(called feature testing at twitter)?
问题描述
我们像这样进行称为功能测试的事情-> https://blog.twitter.com/engineering/en_us/topics/insights/2017/the-testing-renaissance.html
We do something called feature testing like so -> https://blog.twitter.com/engineering/en_us/topics/insights/2017/the-testing-renaissance.html
该文章的TLDR，我们将请求发送到微服务(带有主体的REST POST)，模拟GCP存储，模拟下游api调用，以便可以重构整个微服务.另外，我们可以交换平台/库，而无需更改测试，这使我们非常灵活.
TLDR of that article, we send request to microservice(REST POST with body), mock GCP Storage, mock downstream api call so the entire microservice can be refactored. Also, we can swap out our platforms/libs with no changes in our testing which makes us extremely agile.
我的第一个问题是DataFlow(Apache Beam)可以接收REST请求来触发作业吗?我看到很多api都围绕创建作业"，但是我没有看到文档中的执行作业"，但我确实看到get status返回作业执行的状态.我只是看不到触发工作的方法
My first questions is can DataFlow (apache beam) receive a REST request to trigger the job? I see much of the api is around 'create job' but I don't see 'execute job' in the docs while I do see get status returns the status of job execution. I just don't see a way to trigger a job to
- 从我的存储api(可模拟，位于GCP前面)中读取
- 希望跨多个节点处理文件
- 调用下游的api(这也是可模拟的)
然后，我只是想在我的测试中模拟http调用，然后在读取文件时，返回真实的客户文件，然后在完成后，我的测试将验证所有正确的请求是否已发送到下游的api.
Then, I simply want to in my test simulate the http call, then when file is read, return a real customer file and then after done, my test will verify all the correct requests were sent to the apis downstream.
我们在功能测试中使用的是apache Beam，但不确定是否与google的dataflow版本相同:(那样会是最理想的!!!!!!-> hmmm，是否有报告的google dataflow的apache beam版本我们可以得到吗?
We are using apache beam in our feature tests though not sure if it's the same version as google's dataflow :( as that would be the most ideal!!! -> hmmm, is there a reported apache beam version of google's dataflow we can get?
谢谢，院长
谢谢，院长
推荐答案
Apache Beam的DirectRunner应该非常接近Dataflow的环境，这是我们建议用于这种单进程管道测试的类型.
Apache Beam's DirectRunner should be very close to Dataflow's environment, and it's what we recommend for this type of single-process pipeline test.
我的建议是相同的:使用DirectRunner进行功能测试.
My advise would be the same: Use the DirectRunner for your feature tests.
您也可以使用Dataflow运行器，但这听起来像是一次完整的集成测试.根据数据源/数据接收器的不同，您也许可以将其传递给模拟实用程序.
You can also use the Dataflow runner, but that sounds like it would be a full integration test. Depending on the data source / data sink, you may be able to pass it mocking utilities.
BigQueryIO is a good example. It has a withTestServices method that you can use to pass objects that mock the behavior of external services
这篇关于如何在Dataflow中进行这种类型的测试(在Twitter上称为功能测试)?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
