用于附加 xml 节点的 Groovy 脚本代码需要 15 多个小时才能处理 6K 员工记录 [英] Groovy script code to append xml node taking 15+ hours for 6K employee records
问题描述
下面的代码可以工作,但需要 15 多个小时来执行 6000 条员工记录,是否有任何改进的可能?
Below code is working but taking 15+ hours to execute 6000 employee records, any improvements possible?
对于 6000 名员工,我有两个员工记录结构(员工数据和员工福利),我已使用人员编号将它们合并为单个 xml(要检查 xml 结构,请查看我之前的问题 - https://stackoverflow.com/questions/65174244/multiple-different-xml-structures-to-one-using-xml-using-xsl).
I have two employee record structures (employee data and employee benefits) for each of 6000 employees I have merged them into single xml using personnel number (to check the xml structure please check my previous question - https://stackoverflow.com/questions/65174244/multiple-different-xml-structures-to-one-using-xml-using-xsl).
现在我必须在 xml 员工记录中附加一个节点/子节点,当 ID(multimap:Message1 中的 personIdExternal 在 multimap:Message2 中找到相同的 ID/PERNR.
Now I have to append a node/subnode in xml employee record when ID (personIdExternal in multimap:Message1 finds same ID / PERNR in multimap:Message2.
xml.'**'.findAll{it.name() == 'EmpEmployment'}.each{ p->
def perID = xml.'**'.find{it.personIdExternal.text() == p.personIdExternal.text()}
def pernr = xml.'**'.find{it.PERNR.text() == '000'+perID.personIdExternal.text()}
if(pernr != null)
{
perID.appendNode {
erpBenEligibility(pernr.PARDT.text()) }
}
}
message.setBody(groovy.xml.XmlUtil.serialize(xml))
示例 XML:
<?xml version='1.0' encoding='UTF-8'?>
<multimap:Messages xmlns:multimap="http://sap.com/xi/XI/SplitAndMerge">
<multimap:Message1>
<person>
<person>
<street>test_stree1</street>
<city>test_city1</city>
<state>test_state1</state>
<EmpEmployment>
<personIdExternal> 001 </personIdExternal>
</EmpEmployment>
</person>
<person>
<street>test_stree2</street>
<city>test_city2</city>
<state>test_state2</state>
<EmpEmployment>
<personIdExternal> 002 </personIdExternal>
</EmpEmployment>
</person>
<person>
<street>test_stree3</street>
<city>test_city3</city>
<state>test_state3</state>
<EmpEmployment>
<personIdExternal> 003</personIdExternal>
</EmpEmployment>
</person>
</person>
</multimap:Message1>
<multimap:Message2>
<rfc:ZHR_GET_EMP_BENEFIT_DETAILS.Response xmlns:rfc="urn:sap-
com:document:sap:rfc:functions">
<phone>
<home>
<phone>number1</phone>
</home>
<PERNR> 001 </PERNR>
<PARDT>#### 1 ####</PARDT>
<home>
<phone>number2</phone>
</home>
<PERNR> 002 </PERNR>
<PARDT>#### 2 ####</PARDT>
<home>
<phone>number3</phone>
</home>
<PERNR> 003 </PERNR>
<PARDT>#### 3 ####</PARDT>
</phone>
</rfc:ZHR_GET_EMP_BENEFIT_DETAILS.Response xmlns:rfc="urn:sap-com:document:sap:rfc:functions">
</multimap:Message2>
</multimap:Messages>
推荐答案
代码中的一些主要问题:
some major issues in your code:
- 使用
.**
访问器.如果 message1 中有 10000 个persons
,则xml.**
将返回一个数组,其中count(person)+count(EmpEmployment)+count(personIdExternal) =10000*3
个元素.并在这个数组上调用 findAll 应该扫描所有这些元素 - 在主循环
xml.'**'.findAll{it.name() == 'EmpEmployment'}.each{
中,您无缘无故地构建嵌套的大数组.例如在这个表达式之后def perID = xml.'**'.find{it.personIdExternal.text() == p.personIdExternal.text()}
你有perID
等于p
- using
.**
accessors. if you have 10000persons
in message1, thenxml.**
will return an array withcount(person)+count(EmpEmployment)+count(personIdExternal) = 10000*3
elements. and calling findAll on this array should scan all those elements - inside the main loop
xml.'**'.findAll{it.name() == 'EmpEmployment'}.each{
you are building nested large arrays for no reason. for example after this expressiondef perID = xml.'**'.find{it.personIdExternal.text() == p.personIdExternal.text()}
you haveperID
equals top
您的代码仍然与 xml 示例不对应.
your code still does not correspond to the xml sample.
所以,我将做一些假设来展示如何在没有 .**.
的情况下构建 gpath:
so, i'm going to make some assumptions to show how you could build gpath without .**.
:
让我们有这样的xml:
let we have xml like this:
<?xml version='1.0' encoding='UTF-8'?>
<multimap:Messages xmlns:multimap="http://sap.com/xi/XI/SplitAndMerge">
<multimap:Message1>
<person>
<person>
<EmpEmployment>
<personIdExternal>001</personIdExternal>
</EmpEmployment>
</person>
</person>
</multimap:Message1>
<multimap:Message2>
<phone>
<xyz>
<PERNR>000001</PERNR>
<PARDT>#### 1 ####</PARDT>
</xyz>
</phone>
</multimap:Message2>
</multimap:Messages>
这是构建大型xml消息的代码部分:
this is a code part to build large xml message:
def count = 60000 //just for test let's create xml with 60K elements
def msg = '''<?xml version='1.0' encoding='UTF-8'?>
<multimap:Messages xmlns:multimap="http://sap.com/xi/XI/SplitAndMerge">
<multimap:Message1>
<person>
'''+
(1..count).collect{"""\
<person>
<EmpEmployment>
<personIdExternal>${String.format('%03d',it)}</personIdExternal>
</EmpEmployment>
</person>
"""}.join()+
''' </person>
</multimap:Message1>
<multimap:Message2>
<phone>
'''+
(1..count).collect{"""\
<xyz>
<PERNR>${String.format('%06d',it)}</PERNR>
<PARDT>#### ${it} ####</PARDT>
</xyz>
"""}.join()+
''' </phone>
</multimap:Message2>
</multimap:Messages>
'''
现在是修改后的变换算法:
and now the modified transforming algorithm:
def xml = new XmlParser().parseText(msg)
def t = System.currentTimeMillis()
def ns = new groovy.xml.Namespace('http://sap.com/xi/XI/SplitAndMerge')
//for fast search let map PERNR value to a node that contains it
def pernrMap=xml[ns.Message2][0].phone[0].children().collectEntries{ [it.PERNR.text(), it] }
//itearte msg1 -> find entry in pernrMap -> add node
xml[ns.Message1][0].person[0].person.each{p->
def emp = p.EmpEmployment[0]
def pernr = pernrMap['000'+emp.personIdExternal.text()]
if(pernr) emp.appendNode('erpBenEligibility', null, pernr.PARDT.text() )
}
groovy.xml.XmlUtil.serialize(xml)
println "t = ${(System.currentTimeMillis()-t)/1000} sec"
即使对于 msg1 & 中的 60k 个元素msg2 它在不到 1 秒的时间内完成转换.
even for 60k elements in msg1 & msg2 it does transformation in less then 1 sec.
这篇关于用于附加 xml 节点的 Groovy 脚本代码需要 15 多个小时才能处理 6K 员工记录的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!