测试数百万条DB2记录的最佳方法 [英] Best Method to test millions of DB2 records

问题描述

嗨朋友们，

我的要求是验证db2行。我将简要说明我的项目是什么。我需要根据一些公式验证前一天生成的账单。只有对DB2的读访问权限。作为测试的一部分，我获取所有以前的日期记录（被视为计算的输入）并存储在本地SQL DB中。然后我开始用公式计算值，最后我将使用db2上的其他一些表进行验证。

我编写了一个测试方法来验证db2行。但我觉得它不是一种有效的方法来验证数百万条记录。所以任何人都应该提供一些可以提高执行速度的灯。



给出以下是我要遵循的步骤。

步骤1：从DB2获取所有输入值并保存在本地SQL中。为此，我使用datatable.I有一个写入存储过程，通过传递datatable作为参数来插入值。

步骤2：从SQL中获取所有值进行计算。我将所有值存储在数据集中。

步骤3：遍历数据行并进行计算。将计算值存储在变量中。

步骤4：进行db2调用以获取检索到的值。

步骤5：与计算值和检索值进行比较。如果值匹配其传递，则其他方式失败。

步骤6：在SQL中保存计算和重新传递的值以用于报告

步骤7：生成报告



我的问题是

我可以使用线程吗？如果是，请给出一些想法

我可以分批执行测试，即每个1000条记录。



请给我一些想法进一步

Hi Friends,
My requirement is to validate db2 rows.I will give a brief description on what my project is.I need to validate the bill generated on previous day based on some formulas.I have only Read access to DB2.As part of testing i fetch all the previous date records(which is considered as input for calculations) and store in local SQL DB. Then i start calculating the values with formula and finally i will validate with some other tables on the db2.
I have written a test method to validate the db2 rows. But i feel its not an efficient method to validate millions records.So any one please shed up some lights on which i can improve the execution speed.

Given Below are the steps i am following.
Step1: Fetch all input values from DB2 and save in Local SQL. For this i am using datatable.I have a written a stored procedure to insert values by passing datatable as parameter.
Step2: Fetch all values from SQL for the calculation. I will store all the values in dataset.
Step3: iterate through the datarows and do calculations. store calculated values in variables.
Step4: Make a db2 Call to get the retrieved values.
Step5: Compare with calculated value and retrieved value. If values matches its pass other wise fail.
Step6: Save calculated&Retrived values in SQL for Reporting
step7: Generate Report

My questions are
Can i use threading? If yes please give some idea
Can i execute the test in batches ie a 1000 records each.

Please give me some ideas to step further

推荐答案

我可以使用线程吗？

我们无法告诉你：问题是我们不知道你的计算是怎么做的。如果它们是独立的，那么可能你可以在各个步骤中使用线程，但在我看来，第3步取决于第2步完成，第4步依赖于第3步完成，等等 - 所以你不能使用线程来运行你的步骤同时。



我可以批量执行测试，即每次1000条记录。

不知道。我们不知道你对你的记录做了什么 - 见上面的线程。



说真的，你将不得不看整个过程，看看它是如何工作的 - 细节在这样的决定中至关重要。但要注意：线程并不是出于性能问题的免于监禁卡：你只能在拥有可用内核的情况下同时执行多个线程，并且每个线程在线程设置和管理中都会增加它自己的开销，所以它是相当的通常发现线程版本需要更长才能完成无线版本，特别是如果任务很小但很多。

"Can i use threading?"
We can't tell you: the problem is that we have no idea what your calculations are trying to do. If they are independent, then possibly you can use threading within individual steps, but to my mind Step 3 depends on Step 2 completing, Step 4 depends on Step 3 completing, and so on - so you can't use threading to run your steps at the same time.

"Can i execute the test in batches ie a 1000 records each."
No idea. We don't know what you are doing with your records - see threading above.

Seriously, you are going to have to look at the process as a whole and see how it works - the details are critical in decisions like this. But beware: threading is not a "get out of jail free" card for performance problems: you can only execute as many thread simultaneously as you have free cores, and each thread adds it's own overhead in thread setup and management, so it is quite common to find that the "threaded" version takes longer to finish than the "unthreaded" version, particularly if the tasks are small but numerous.

这篇关于测试数百万条DB2记录的最佳方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！